wtf is wandr?

WANDR is a benchmark, meaning a standardized test, built by Perplexity AI to measure how well an AI system performs at wide and deep research tasks. These are the kinds of jobs that require searching broadly across many sources, gathering a lot of detail from each one, pulling out specific facts, correctly telling similar entities apart, such as two people with the same name, and then producing an answer that is backed by evidence rather than guesswork. The project is organized into four separate pieces that work together. First, there is the raw task data itself, made up of editable configuration files, prompts, and supporting materials. Second, an adapter converts that raw data into a format used by a testing system called Harbor. Third, the generated Harbor-ready task packages are what actually get run and published. Fourth, a component called Relay acts as a general purpose bridge that lets different AI providers, such as OpenAI, Anthropic, or Google, plug into the Harbor testing system in a consistent way. Running WANDR requires Python 3.12, a tool called uv for managing the project, and Docker for running tasks in a controlled environment, plus API keys for whichever AI providers you want to test. A cheap smoke test is available to check that everything is wired up correctly, though it still makes small paid calls to OpenAI and Perplexity. Larger runs, such as validating two representative tasks or running the entire benchmark across six different AI providers, involve significantly more paid API usage, and the README warns clearly that costs are not capped by the system itself. When a task runs, Harbor sets up an isolated environment, Relay sends the task to the AI provider being tested and collects its output files, and then WANDR's own verifier checks the submitted work, cross-references facts, removes duplicate entities, and scores the result. Results are saved as a set of files per run, including a machine readable score, a detailed trajectory of what the AI did, and a human readable HTML report. Anyone wanting to edit or add new tasks works from the original source files, then regenerates the Harbor packages from them rather than editing the generated files directly.