Flip Anything

TL;DR

Two YAM arms must turn an industrial hand tool face down on a mat and let go, using only static RGB-D cameras and the joint encoders (four cameras in simulation). A coding agent builds the policy from scratch as ordinary, inspectable code and improves it in a self-learning loop: run 360 simulated episodes, group the failures by mechanism, change the source, and regression-test. Over 36 versions the agent added 20 motion primitives in 5 families, combined into 4 flip strategies, and took the tune success rate from 24.7 % to 95.0 %. Nothing is learned inside the policy; what is learned lives in the code.

95.0%best tune run, v036 (342 / 360)

90.8%held-out, frozen v016 (109 / 120)

88.3%v016 with depth noise and calibration error

36policy versions, 26 full tune evaluations

Real-world experiment

real rig · 8× speed

Spring clamp. One hand picks up the clamp, the other takes it in the air, and it is set down turned over. 3 min 44 s of real time, shown at 8×.

real rig · 8× speed

Tape roll. The left hand picks up the roll, the right hand takes it in the air, and it is set down turned over. 2 min 29 s of real time, shown at 8×.

Same arms, different eyes. The real bench uses the two YAM arms from simulation, but sees the scene through one overhead ZED Mini and a D405 wrist camera instead of four static cameras.
Same policy code. The code learned in simulation runs on the rig through a bridge that adds safety checks.
Not solved yet. Real runs fail in ways the simulator rarely produced. The last section shows a failure and what the gap is made of.

Task Setup

FlipAnything: 24 hand tools, each spawned face up at a random pose on a gridded mat in front of a bimanual YAM rig. One policy has to flip all of them, without being told which tool it is looking at.

Figure 1. The 24 objects. Each tile turns one tool from face up to face down. The set spans 5 mm surgical scissors, lock nuts, tape rolls, boards, pliers and a 0.6 kg power drill.

Robot. Two 6-DoF YAM arms with parallel grippers. Action: 14 joint and gripper targets at 50 Hz.
Sensing in simulation. RGB and depth from four fixed cameras (top, bench, close, profile) plus joint and finger encoders. No wrist camera. The real rig differs; see Perception.
Never read. Object pose, meshes, tool names, segmentation masks or the score.
Success. Face down (cos ≤ −0.9), on the mat, at rest, released, both hands clear, held for 1.5 s, all within 80 s.
Splits. Tune: 24 tools × episodes 0–14 (360). Held-out: episodes 15–19 (120), one registered run per frozen version.

The Self-Learning Loop

Each iteration is a frozen snapshot evaluated on all 360 tune episodes. The agent reads the policy's own event log, groups the failures by the first mechanism recorded (no plan, lost grip, squeeze-out, receiving-hand pinch, false "done", lost track), writes a probe for the largest group, changes the code and evaluates again. Every version is kept, including the regressions. Run-to-run spread on identical code is about ±1.4 points.

Tune success, all 24 tools (360 episodes) Regression, later fixed or reverted Held-out (one run per frozen version) Hover a point for what that version changed.

Six phases of learning

v001 → v005 · 24.7 → 80.6 %Build the mechanisms. 235 of 271 early failures were "no plan". The agent added in-hand ICP registration, then stand and topple for compact parts, then a strategy cascade.
v006 → v011 · 73.9 → 81.4 %Make the pinch hold. Pads move to the widest band, gentle release, verified B pinch. Three regressions, each a grasp rule that removed one tool's only grasp.
v012 → v016 · 86.9 → 89.4 %Reach 90 %. Push look-ahead, bridged sections, fingertip touch-off and a guarded-descent fix. v016 frozen: 90.8 % held-out, 88.3 % under sensor noise.
v017 → v024 · 89–91 %Plateau. Test lift, reposition, slow roll, partial-turn finish, face-normal placement, B deepen. Each moves one or two tools; the total stays inside the noise.
v025 → v030 · 91.9 %Graph-as-Policy audit. The loop was re-expressed as a GaP workflow. Same score, but per-node failure attribution pointed at the next two levers.
v033 → v036 · 94.7 → 95.0 %Toward 98 %. A guarded-contact pinch takes the 5 mm scissors from 0 to 13 of 15; a face check after drops removes most false "face down" beliefs.

Learned Motion Primitives

The policy flips a tool with one of four strategies, chosen per observation from geometry and reachability, with both hand assignments searched. The strategies are built from 20 motion primitives. The chip on each primitive is the version where the self-learning loop added it or made it work; unmarked ones were in the first version.

claw hammer · 27 s

1 · In-air handover v001

Hand A pinches top-down, lifts and rolls 90°. Hand B pinches the far face, A releases, and B rolls back and sets the tool down face down. Used for 19 of 24 tools.

pipe wrench · 39 s

2 · Reposition, then handover v017

When no handover is reachable, A carries the tool to the middle of the mat, rotating it if needed. The target pose is chosen by planning the handover on the predicted scene.

nylon lock nut · 21 s

3 · Stand, then topple v003

For compact parts. A pivots the part about its bottom edge onto its side, past the tipping angle atan(W/H) read from the height map. B pushes it the rest of the way over.

multimeter · 16 s

4 · Push-stand, then topple v005

For parts too wide to pinch. Two arc pushes with closed fingertips and no grasp. Since v012 the first direction is chosen so that the second push is also reachable.

20 primitives in 5 families

Each row is a family. The table-assisted and in-hand families grew most during self-learning, because those were where episodes failed.

Family	Primitives
Free motion & safety 4	joint-space approachmove homeguarded descent, stop on stallfixed v015fingertip touch-offv014
Grasping 4	pinch closewidest band v0061 cm test liftv017open-to-width gentle releasev008back off along the fingersv008
In-hand & bimanual 6	lift and roll 90°slow v019A in-hand pose correction (ICP)v002B approach and insertB deepen and re-closev024A put-back, reversing its own pathv009B roll by the face normalv022
Table-assisted 5	pinch pivot about the bottom edgev003arc push with press-in biasv003brace the far edge while pushingpress a propped part flatset down to contact
Transport 1	pick, carry, rotate, placev017

Checks learned from failures

Most of the gain after v010 came from runtime checks wrapped around the primitives. Each turns a drop into a retry.

Verify before letting go

B's pinch is checked from the finger encoder before A releases (v010); A's grip after closing (v011). A width narrower than the planned section sends B 8 mm deeper (v024).

Catch slips early

A 1 cm test lift catches a pinch squeezing out before the full lift (v017). The part is set down and re-planned instead of dropped mid-air.

Know the outcome

A tool stopped part-way through a topple is finished with another push (v021). After a drop, the visible top is aligned against the first-seen top to tell up from down (v036).

Perception Pipeline

Four fixed RGB-D cameras, no wrist camera and no learned models. Every step is explicit geometry: filtering, RANSAC, connected components, PCA and ICP. The pipeline runs every time the policy re-observes: at the start, after each stage, and when checking a grasp or a placement.

4static RGB-D cameras

0wrist cameras

0neural nets or segmentation channel

4 mmheight-map grid

Bimanual YAM setup: two 6-DoF arms with parallel-jaw grippers, a ZED Mini overhead camera and a D405 wrist camera, with example overhead and right-wrist RGB frames — **Figure 2. The real bimanual YAM rig.** Two 6-DoF arms with parallel-jaw grippers in the RoboSimStudio reconstruction of the real bench, with its ZED Mini overhead camera and D405 wrist camera. Figure from the open Gemini harness post.

Camera setting	Self-learning (simulation)	Real rig (Figure 2)
Static cameras	4 rendered RGB-D cameras: top, bench, close, profile	1 ZED Mini stereo camera overhead
Wrist camera	None	D405 on the right wrist
Depth	Rendered planar depth; optional noise ∝ z², dropout and extrinsic error	Stereo depth from real sensors
Viewpoints of the object	Four views from fixed, separated poses	One fixed overhead view, plus the moving wrist view

The two camera settings differ. Every self-learning result on this page (the learning curve, the primitives, the per-tool scores and the videos) was measured with the four static simulation cameras below. The real rig has a single overhead camera and a wrist camera, so the policy sees fewer and different viewpoints there. This is one of the sim-to-real gaps.

Simulation pipeline, frame to object model

Capture RGB-D

RGB + planar depth, 4 cameras

Segmentation is never requested. Optional robustness noise: depth noise ∝ z², dropout, per-camera extrinsic error.

sensors.py · RGBD.capture()

Clean depth

5×5 median + flying-pixel rejection

Drops pixels more than 2 cm from their neighbourhood median.

perception.py · clean_depth()

Back-project and fuse

pinhole → world frame

Every 2nd pixel, camera-to-world transform, all 4 clouds merged with colour and source camera. ROI x 0.30–0.92 m, y ±0.40 m.

perception.py · backproject()

Mask out the robot

forward kinematics + collision spheres

Both arms posed from the joint encoders; points within 18 mm of any collision primitive removed.

perception.py · robot_mask()

Fit the support plane

sequential RANSAC + least squares

Up to 3 near-horizontal planes; the highest large one is the mat. One plane per camera absorbs that camera's calibration offset.
If the cameras disagree by more than 2 mm, a fingertip touch-off measures the true height.

perception.py · fit_table()

Segment the object

height threshold + 3-D connected components

Points from max(3 mm, 3σ) to 30 cm above the plane, 8 mm voxels, 26-connected. Largest cluster is the object; occlusion splits are merged back.

perception.py · segment_object()

Describe the object

2.5-D height map + PCA footprint

Max height and top colour on a 4 mm grid; the unseen underside is assumed to reach the table. PCA gives axes, extents, centre and height.

perception.py · HeightMap, describe()

Grasp candidates

Pinch lines every 8 mm and 15° of yaw, kept when the section fits the 94 mm gripper. Pads at the widest band; wedges rejected; ranked by gravity stability.

cost = 0.6·|d_b| + 1.2·|d_c|·(3 cm / w)

grasp.py · stability_cost()

Which face is up

A yaw sweep plus 2-D ICP aligns the first-seen top surface to the current one. Footprint overlap, height RMS and colour difference decide up, side or down.

faces.py

In-hand pose at handover

Trimmed point-to-point ICP registers the pick-time model onto the points around the held tool (40 iterations, 12 mm trim). A corrects its pose or B re-plans.

perception.py · icp()

Statics, not physics

No mass or friction is estimated. Tipping angle and slide-vs-tip come from the perceived shape.

θ_tip = atan2(com→edge, H/2)

topple.py

Selected grasps, all 24 tools

Offline run of the policy on the saved episode-0 camera frames. Top-down view of the perceived height map, darker cells higher, world +x up the page. Everything shown is computed from the cameras; the tool mesh is never read.

chosen pinch, pads at the ends next-ranked candidates rejected (sample) estimated centre of mass

Results

Tune column: v036, the best full run (342 / 360). Held-out column: v016, the frozen policy's single registered run on episodes 15–19, from before the scissors contact pinch existed. Strategy column: what the policy chose in the example episode for that tool.

Tool	Tune v036 (/ 15)	Held-out v016 (/ 5)	Strategy in example episode

Robustness

Version	Tune (360)	Held-out (120)	Tune 0–4 with depth noise + calibration error
v001 handover only	24.7 %	–	–
v012 push look-ahead	86.9 %	85.8 %	58.5 %
v016 touch-off (frozen)	89.4 %	90.8 %	88.3 %
v025 Graph-as-Policy	91.9 %	–	–
v036 face check	95.0 %	–	–

The fingertip touch-off is what made the policy robust to calibration error: under noise it moved the score from 65.8 % (v015) to 88.3 % (v016) while leaving the clean tune score unchanged.

Video Examples

One recorded episode per tool from the frozen v021 policy. Left half: overview camera. Right half: side view. Pick a tool to play it.

claw hammer

Outcome
Time
Strategy
Tune v036

Success is judged by the simulator. The policy never sees the score; it decides it is done from its own observation.

Failure Modes and the Sim-to-Real Gap

In simulation the policy reaches 95 %. On the real bench it fails far more often, and in different ways. This section lists what still fails in simulation, measures how the real rig differs, and reports what happened when those differences were put back into the simulator for another round of self-learning.

real rig · 8× speed

A real failure: the folding toolkit. The tool is picked up and passed between the hands, but the run ends without it face down. 2 min 43 s of real time, shown at 8×.

Still failing in simulation (v036, 18 of 360)

Resting just short of face down. ESP32 board at cos −0.88 (3) and box cutter propped at −0.83 (2), against the −0.9 threshold.
Drops. Measuring tape after drops (3), scissors dropped in the roll (2).
Receiving pinch. Power drill (2): hand B cannot close on the far face.
Singles. Pipe wrench (2), hammer, pliers, screwdriver and tape measure (1 each).

The same policy signals, real vs simulated

Measured from the policy's own event logs on 27 real sessions (YAM bench, 21–22 Oct 2026) and on the simulated runs.

Signal	Real rig	Simulation	Note
Missed grasps	21 %	3.7 %	Simulation pools v001–v036; v036 alone is 1.8 %. Without 11 unusable real sessions (a gripper-direction bug), real misses are 14 %.
Slip in the air after a passed lift test	29 %	0 %	Zero by construction in simulation, which ran without the slip guard.
Hand A's in-hand correction diverged	15 %	0 %
Close width vs plan	−15 mm	−1 mm	Real grippers close well past the planned width.
Fingertip height at close	15.6 mm	8.4 mm	Real closes end higher above the mat.

What the gap is made of

Each difference was modelled as a physical mechanism in the simulator and checked against a first-principles prediction. No rate was fitted to the real failures.

Soft arm

The real joints run a PD controller at kp 80/80/80/40/10/10, against 300 on every joint in the old twin. The hand sags 12 mm under a 0.47 kg load (real: 10 mm).

Force-limited gripper

A blocked real grip holds about 50 N whatever gap is commanded. The old twin held 18 N.

Pinch ride-up

With a soft arm (about 2.3 mm/N at the fingertips), a 50 N squeeze on a rounded section above its equator lifts the hand over the part. This explains the higher closes and the misses on round or thin parts.

Soft foam mat

The EVA mat lets a pressing fingertip sink 2–5 mm before it stiffens. The old twin's mat gave 0.4 mm.

Real objects

Round rubber handles modelled as curved sleeves, not one convex hull. The spring clamp as two halves on a sprung pivot, not one rigid body.

Sensors and cameras

Stereo depth artefacts, rig calibration error, encoder offsets, gripper reading error and one step of latency. The real rig also has different cameras (see Perception).

Self-learning again, in the gap simulator

About 250 episodes over 10 hours on the rig's own code path, with the clamp, tape roll and folding tool at their real positions. A new primitive counted only if removing it lost episodes.

0 of 4 new primitives were effective. Hold-z close, scoop push, push-through flip and a B drop guard each came out within one episode of their ablation on the same seeds.
Ride-up is real but not the limit. Cancelling it did not change clamp success.
The folding tool is bounded by kinematics. No handover is reachable, and a pinch-stand needs 79° of wrist pitch where the arm reaches 25–65°.
The clamp is a perception problem. 7 of 10 clamp failures end face up while the overhead face check reads "unknown". A close look with the wrist camera did not separate the faces either.