KiRoSe: Using Kinematic Data for Efficient Segmentation of Robot Demonstrations

Shuangyu Xie1, Kaiyuan Chen1, Muhammad Zubair Irshad2, Richard Cheng2, Sergey Zakharov2, Ken Goldberg1
1University of California, Berkeley   2Toyota Research Institute
Paper arXiv Models Videos

Sparse Motion Tokens

The same motion makes the same tokens. Recurring kinematic events, such as a bimanual lift, a handover, a pick or a place, map to recurring codes across tasks and objects.

How a Sparse Motion Token is made

From motion to tokens. ① End-effector kinematics (14 channels at 30 Hz). ② A dilated temporal convolution encoder: each layer doubles its dilation, so one output feature sees seconds of motion around it (highlighted). ③ During training only, a head on these features predicts where annotated subtasks change (focal loss on soft targets around each boundary); afterwards the head is dropped and the encoder is frozen. ④ Features are pooled to 2 per second and snapped to the nearest of 512 k-means centroids. ⑤ The resulting token stream, plus the task instruction, is what the Qwen3.5-9B reader sees. Illustration: 3 layers with kernel 3; the real encoder has 7 blocks with kernel 5 and dilations 1–64 (17 s receptive field, 2.76M parameters).

Figure 1. Recurring motion, recurring tokens. Each panel shows one kinematic event in two YAM episodes from different tasks (A, B). Sparse Motion Tokens appear in sync with the robot (2 per second, from a 512-code vocabulary) on a shared time axis. As soon as a panel’s two clips end, lines join the tokens the two rows share in the same order (their longest common subsequence), and the shared sequence is spelled out: a single-arm pick-up is 438 → 456 → 445 → 431 → 296 in both tasks, and a single-arm place is 126 → 308 → 443 → 48 → 208. Pairs are chosen examples; across all held-out segments, clips of the same motion category share about twice as many codes as clips of different categories. A GIF version is at static/media/smt_motion.gif.

Architecture

KiRoSe has two stages. A kinematic encoder is trained with subtask-transition supervision, and its features are pooled and quantized into Sparse Motion Tokens. The tokenizer is then frozen while a pretrained language model learns to read the tokens and write segments.

KiRoSe pipeline: kinematic stream to transition-supervised encoder, temporal pooling and codebook quantization into motion tokens, then a language model that outputs timed action descriptions
Figure 3. KiRoSe pipeline. Given a kinematic stream and the instruction “fold the paper box,” KiRoSe encodes motion with a transition-supervised encoder, then pools and quantizes its features into Sparse Motion Tokens. The language model reads the tokens, time markers and instruction, and outputs action descriptions with start and end times such as “pick up the object,” “place in the box” and “adjust.”

Shared end-effector input

Every embodiment is mapped to the same 14 channels at 30 Hz: per arm or hand, linear velocity, angular velocity and gripper opening rate. For robots these come from end-effector motion and gripper aperture; for human hands, from wrist motion, palm orientation and thumb–index separation.

xt = ⊕h∈{L,R} [ vh,t , ωh,t , ġh,t ] ∈ ℝ14

Transition-supervised temporal encoder

Residual temporal convolutions map the stream to contextual features. During training, an auxiliary head predicts a transition probability at every step against soft targets centred on annotated subtask boundaries, with a focal binary cross-entropy loss. The head is discarded afterwards; the 2.76M-parameter encoder is kept.

Sparse Motion Tokens

Encoder features are pooled on a uniform grid and assigned to the nearest of K = 512 k-means centroids fitted on training trajectories only. Token positions do not depend on annotated or predicted boundaries; supervision shapes what the features encode, not where tokens fall.

Language-model reader

Qwen3.5-9B receives 512 new vocabulary entries, <k0> to <k511>, plus textual time markers every second and the task instruction. LoRA adaptation trains it to emit the reference segmentation as JSON with start time, end time and a description per segment. The tokenizer stays frozen, so the model learns to read the motion vocabulary rather than reshape it.

Training and Evaluation Data

All trainable components see only the YAM bimanual robot in ABC-130K. AgiBot G1 and Ego-Exo4D human hands are evaluated without any target-embodiment training, with fewer boundaries per minute and less verb overlap than YAM.

DatasetEmbodimentPurposeEpisodesTasksDuration (s)Segments / ep.Boundaries / minVerbs (% in train)
ABC-130KYAMTraining5,478140173.3 ± 66.526.6 ± 15.09.283 (100%)
ABC-130KYAMHeld-out test20010154.6 ± 57.624.3 ± 13.49.424 (99%)
AgiBot WorldAgiBot G1Cross-embodiment, test1,1991658.0 ± 23.37.4 ± 3.37.617 (73%)
AgiBot WorldAgiBot G1Cross-embodiment, held-out tasks2,653394.4 ± 31.110.8 ± 3.16.912 (61%)
Ego-Exo4DHuman handsTransfer to human1931281.4 ± 53.510.1 ± 5.97.469 (35%)
Example tasks and annotated timelines for YAM, AgiBot and human hands, with 40-second traces of linear speed, angular speed and opening rate per arm
Figure 4. Evaluation data. Top: example tasks and annotated action timelines, colored by action category. Bottom: representative 40-second traces of linear speed, angular speed and opening rate for the left (black) and right (orange) arms or hands. The datasets differ in task semantics, temporal structure and motion.

Segmentation Experiments

Every kinematic tokenizer is paired with the same LoRA-adapted Qwen3.5-9B reader. Boundary F1 matches predicted and reference boundaries within a tolerance; Grounded F1 also requires the action description to agree.

200 YAM episodes, 10 held-out tasks
MethodBoundary F1Grounded F1
±0.5 s±1 s±2 sLex.Emb.
Qwen3.5-9B zero-shot (video)0.090.160.220.050.07
FAST0.220.420.500.120.17
OAT0.220.410.510.140.20
MotionGPT0.230.410.500.130.16
KiRoSe: Latent + MLP0.440.520.56––
KiRoSe: SMT codes + MLP0.380.450.49––
KiRoSe0.480.630.710.430.49

FAST, OAT and MotionGPT use their native 32-D joint-state input and a slightly larger training set, so the comparison is between complete tokenizer interfaces.

The released checkpoint (sy-xie/KiRoSe-Reader) is a retrain on the same split with the location boilerplate removed from the training descriptions: the ABC-130K phrase "the center of the board" tags 38% of segments regardless of arm position. Its Boundary F1 is 0.48 / 0.65 / 0.72 on the held-out YAM tasks and 0.40 / 0.62 / 0.76 on all 3,852 AgiBot episodes, and it names a location only when the task implies one. The video examples below come from this checkpoint; the tables on this page report the paper model.

Boundary F1 at plus or minus 1 second as a function of motion-token rate for each tokenizer
Boundary F1 ±1 s as a function of motion-token rate.

Cross-Embodiment Transfer

The encoder, codebook and reader are trained on YAM only. On AgiBot G1 and human hands they run zero-shot through the shared end-effector representation.

Bar chart of Boundary F1 at plus or minus 0.5 seconds for YAM, AgiBot and human hands, comparing Qwen zero-shot video, Latent plus MLP, KiRoSe, and GPT-6 Astra on a matched subset
Figure 5. Boundary F1 ±0.5 s per embodiment. Hatched: GPT-6 Astra and KiRoSe on the same 50 / 20 / 30-episode subsets.
  • Timing transfers. KiRoSe reaches 0.40 Boundary F1 ±0.5 s on all 3,852 AgiBot episodes without AgiBot data, against 0.48 on held-out YAM. The video Qwen3.5-9B baseline reaches 0.19 on the 357 AgiBot episodes with video.
  • Ahead of a frontier video model. On 20 matched AgiBot episodes, GPT-6 Astra reaches 0.30 and KiRoSe 0.61.
  • The motion signal is used. Removing the motion tokens, or swapping in tokens from another trajectory, drops Boundary F1 ±1 s from 0.61–0.63 to 0.31–0.35 on both robots.
  • Actions transfer, object names need vision. On the matched AgiBot episodes, action-level Grounded F1 is 0.593 for KiRoSe and 0.349 for Astra; on full descriptions with object names Astra leads (0.35 vs 0.23 embedding).

Video Examples

Pick an episode and press play. The lanes follow the video: the human annotation, KiRoSe reading kinematics only, and Qwen3.5-9B reading the video and instruction zero-shot. Click the timeline to jump.

0.0 / 0.0 s
Annotation
KiRoSe
Video VLM
F1 = boundary F1 at ±1 s against the annotation

Robot episodes are the paper's appendix examples followed by more chosen with the same rule: per task, the episode whose KiRoSe score is closest to that task's median. Videos play at the recorded 5 fps.

Progress Estimation

A useful progress signal should rise while an action is executed and fall when it is undone. We play one AgiBot subtask backwards in place (video reversed, kinematic velocities negated) and check whether predicted progress follows.

Two AgiBot episodes with one subtask played backwards; ReWiND progress keeps rising during the reversed segment while KiRoSe progress falls
Figure 6. Reversal probe on AgiBot (zero-shot). Red frames are played backwards; stamps give original recording time. Curves show the signed progress rate (green rising, red falling); dashed lines are the same model on the original video. ReWiND keeps counting up while the task is undone, and KiRoSe's progress falls. These curves use an earlier absolute-progress probe; the table reports the final signed-rate probe.
2,586 subtasks from 357 AgiBot episodesforward and reversed
ModelFwd. ρ ↑Rev. ρ ↓Rev. ρ < 0 ↑
ReWiND (MetaWorld)+0.88+0.881%
ReWiND (Open-X)+0.65+0.634%
Robometer-4B+0.71+0.0646%
KiRoSe+0.51−0.6886%
  • The 2.76M-parameter SMT encoder, trained only for transition detection on YAM, stays frozen. A 68k-parameter probe reads its features and the displacement accumulated since the subtask start.
  • The probe is trained on YAM subtasks played forward and in reverse. Neither the encoder nor the probe sees AgiBot.
  • A model that tracks elapsed time stays positive in both directions. KiRoSe's progress falls on 86% of reversed subtasks.

Summary

KiRoSe shows that a robot's own end-effector kinematics are enough to segment its demonstrations. Compressed into two Sparse Motion Tokens per second, kinematics locate subtask boundaries better than a zero-shot video model, and transition supervision raises Boundary F1 ±1 s from about 0.42 to 0.63. Because the tokens live in a shared end-effector space, a model trained only on YAM keeps most of its boundary accuracy on AgiBot G1 and runs on human hands, while its open-vocabulary reader writes task words it never saw in training. Motion supplies the action and vision the object: KiRoSe leads GPT-6 Astra on action-level Grounded F1 and trails it where descriptions need object names. The same frozen encoder also tells forward execution from an action being undone, which video reward models miss, all at a small fraction of the cost of video-based segmentation.

BibTeX

@misc{kirose2026,
    title  = {KiRoSe: Using Kinematic Data for Efficient Segmentation of Robot Demonstrations},
    author = {Xie, Shuangyu and Chen, Kaiyuan and Irshad, Muhammad Zubair and Cheng, Richard and Zakharov, Sergey and Goldberg, Ken},
    year   = {2026}
}