wtf is harvey-labs?

Harvey LAB is an open-source benchmark for testing how well AI agents can perform real legal work. Instead of measuring general knowledge, it gives an AI agent realistic legal tasks, such as reviewing documents or working through a business deal, and checks how well the agent completes them compared to what a human lawyer would expect. The project has two main parts. The first is a growing collection of tasks, over 1,600 of them, spanning more than two dozen legal practice areas plus contract work. Each task includes instructions for the agent, the relevant documents it needs to read, and a scoring rubric that defines what a correct or complete answer looks like. The second part is an execution harness, the software that actually runs an AI agent against these tasks and scores its output using the rubrics, sometimes with the help of another AI model acting as a judge. The maintainers describe LAB as an ongoing project, meaning new tasks and improvements to the scoring system are expected to be added over time rather than the benchmark staying fixed. The documentation includes a full walkthrough that follows one realistic example, a mergers and acquisitions data review task, from setup through running an agent, scoring its work, and comparing results across different runs. This project is licensed under the MIT license, a permissive license that allows free use, modification, and redistribution, including for commercial purposes. Harvey LAB is intended for AI researchers and engineers who are building or evaluating AI agents meant for legal work, as well as legal technology teams who want a standardized way to measure how capable a given agent actually is at real legal tasks rather than relying on general purpose benchmarks.