← robovista home

RoboVista · eval board

Multiple-choice spatial-reasoning questions over robot scenes — 460 scored (474 minus a 14-question blacklist). 27 models, each run with a standard and a chain-of-thought prompt. updated 2026-07-06

RoboVista: Evaluating Vision-Language Models for Diverse Robot ApplicationsRSS 2026

Best overall

68.9%
gemini-3-flash-preview · API

Best open-weight · new

55.2%
qwen3.5-397b-a17b · passes every GPT row

Models evaluated

27 total
23 open-weight · 4 API
Contents
  1. Leaderboard — standard vs chain-of-thought
  2. Accuracy by application domain
  3. Scene understanding vs planning
  4. All runs

Leaderboard — standard vs chain-of-thought

Sorted by standard-prompt accuracy. CoT helps the Gemma 4 family (+3 to +4), the thinking-tuned variants and the Qwen3.6 dense model; it costs accuracy on most other open models.

standard prompt chain-of-thought Δ = CoT − standard

Accuracy by application domain

Standard-prompt runs. Domains follow the benchmark's taxonomy; Driving and Surgery are the smallest columns (n=20, n=32), so read those with wider error bars in mind.

Scene understanding vs planning

Standard-prompt runs, split by ability type (SC n=306, PL n=148 — planning here bundles decision-making, motion awareness and recovery). Gemini holds its accuracy on planning questions; nearly every other model drops.

scene understanding planning Δ = planning − scene

All runs

Exact figures for every completed pass.

modelweightsstandardcotΔSCPLreleased