Benchmark details

1The six task classes

Each class fixes an operational objective and an application setting, then names the axes along which instances vary. Tasks 1–4 run on the single-arm Franka; tasks 5 and 6 use the bimanual YAM.

WorkshopFrankaSemantic perception

Task 01Tool Sorting

The robot must place industrial tools into corresponding bins. Instance variation is introduced by changing the tools, their poses, and tool density.

L1: Singulated tools with no distractors.
L2: Cluttered tools that overlap and occlude one another.

Reference graph (Newton): L1 47.1% · 421.6 s; L2 33.7% · 913.8 s. Clip: Level 2 · 9× speed.

WorkshopFrankaConstrained placement

Task 02Tool Hanging

The robot must pick up one or more industrial tools (wrenches, screwdrivers, hammers, scissors) and hang or insert them into designated hooks, clips or slots on a pegboard. Variation comes from tool type, the number of tools, initial tool pose and holder pose.

L1: One tool; known fixture pose.
L2: Multiple tools and fixture types; randomized poses.

Reference graph (Newton): L1 86.7% · 86.1 s; L2 81.7% · 158.4 s. Clip: Side view · 4× speed.

AssemblyFrankaTight tolerance

Task 03Gear Meshing

The task input is gears and axles. The robot must pick up multiple gears, assemble them onto axles so that they properly mesh, and press a button to activate the mechanism. Variation covers the number of gears, gear size and initial gear poses. To ensure physically valid instances, scenes can be initialised from a valid assembled configuration before selected gears are removed and repositioned.

L1: Assemble two gears.
L2: Assemble three gears.

Reference graph (Newton): L1 100.0% · 165.9 s; L2 90.0% · 235.5 s. Clip: Front view · 4× speed.

HealthcareFrankaConstrained insertion

Task 04Syringe Packing

The robot must place syringes into a disposal container while avoiding distractor objects. Instance variation includes the number of objects and the disposal box type.

L1: Singulated targets.
L2: Cluttered scenes with diverse syringe types.

Reference graph (Newton): L1 87.0% · 228.8 s; L2 63.0% · 472.3 s. Clip: Agent view · 9× speed.

Data centreYAM, bimanualPrecision alignment

Task 05Cable Insertion

The robot must pick up a connector, align it with a USB-C port and fully insert it. Instance variation includes the position and orientation of the connector and the port.

L1: Fixed, visible port; nearby connector with small misalignment.
L2: Randomized port pose; moderate connector misalignment.

Reference graph (Newton): L1 86.7% · 78.8 s; L2 30.0% · 74.2 s. Clip: Wrist + support · 3.5× speed.

Data centreYAM, bimanualDeformable objects

Task 06Cable Routing

The robot must route a flexible cable around a specified ordered sequence of cylindrical fixtures mounted on a planar routing board. The task varies the number, position and spacing of pegs; the ordered target sequence; cable length; initial cable configuration; cable slack and curvature; endpoint position; and the 3D pose of the routing board.

L1: 2–3 widely spaced pegs; short route; low turning angle.
L2: 4–6 moderately spaced pegs; multiple turns; moderate initial cable variation.

Reference graph (Newton): L1 46.7% · 116.0 s; L2 46.7% · 189.8 s. Clip: Corner view · 5× speed.

2What the benchmark ships

Component	Choice	Purpose
Asset library	55+ normalised assets	Industrial tools, mechanical components, medical supplies, fixtures, connectors and deformable objects, normalised in format, scale, appearance and collision geometry.
Robot embodiment	Single-arm Franka, bimanual YAM	Franka for reproducible tabletop manipulation; YAM for coordinated bimanual interaction, particularly with deformable objects.
Task specification	Extended BDDL	Tasks declare objects, initial configurations, variations and success conditions, for rigid and deformable objects alike.
Physics	NVIDIA Newton	Unified rigid and deformable dynamics, spanning rigid manipulation, contact-rich assembly and insertion, and cable routing.

Table 1. The four components of the suite.

The asset catalogue on one bench: a pegboard with hand tools, a sharps container, a tray of syringes, gear rails, cable routing boards, and rows of tools, gears and fasteners. — **Figure 1.** The asset catalogue on one bench, rebuilt from the sources the tasks themselves use: the syringes carry the suite's own collision hulls, the gears and routing boards its own fixtures. Normalisation lets a variation axis mean the same thing in every task that uses it.

3Task execution records

Task-level success says whether an execution achieved its objective, not where or why it failed. VABAR records the evidence needed to relate outcomes to both the program's decisions and the resulting physical interaction.

Physics states_t→Observation & estimationo_t→ŝ_t→Decision & actiona_t→Physics outcomes_t+1, C_t→Verificatione_t+1

Mapping	Failures it localises
o_t → ŝ_t	Perception or state-estimation failures
ŝ_t → a_t	Program-generation, planning, skill-selection or decision failures
a_t → (s_t+1, C_t)	Grasping, motion-execution, collision, contact or other physical-interaction failures
(s_t+1, C_t) → e_t+1	Task verification, failure detection and recovery triggering

Table 2. The four mappings that localise a failure.

Object poses and velocities describe the resulting scene configuration; contact records describe how that configuration arose. A failed placement may leave an object in an incorrect pose, while its contact history reveals whether it collided with a fixture, was supported by another object, or became jammed. VABAR records both, so failure attribution is grounded in the physical execution rather than in task-level success or software error messages alone.