wtf is robovista?

RoboVista is a benchmark dataset for testing how well vision-language AI models understand and reason about robots. It was built by researchers at UC Berkeley, Princeton, and Google DeepMind, and it has been accepted to the RSS 2026 conference. The benchmark contains 474 multiple-choice questions, each paired with one or more photos from real robots. Every question comes with five answer choices, the correct answer, and a detailed explanation written by an expert, most of whom hold doctoral degrees in robotics. The images come from 18 published research papers and existing robot datasets, and they cover six different areas where robots are used: open datasets, industrial automation, agriculture, home tasks, surgery, and self driving cars, adding up to 39 distinct task types. The questions are designed to test four different skills: recognizing what is happening in a scene, planning next steps, judging whether a motion is physically possible, and detecting or recovering from failures. When tested against this benchmark, even the strongest AI models available today only answer correctly around 56 percent of the time, and the results line up closely with how well those same models actually perform when controlling physical robots. For people who want to use RoboVista themselves, the repository includes a Python tool to load the dataset directly from Hugging Face, a local web viewer for browsing questions and images in a browser, and a script for benchmarking any vision-language model that supports an OpenAI-style API, whether that model runs through OpenAI's own service or a locally hosted server. The repository also includes a separate experiment comparing two different training methods for teaching a model to answer these kinds of questions, with a full write-up and reproduction instructions included. This project is aimed at robotics and AI researchers rather than general users, and using it requires some comfort with Python and machine learning tools.