wtf is legalita?

LegalITA is a benchmark for testing how well large language models handle Italian case law reasoning. It gives a model a legal question along with a set of PASS or FAIL criteria written by the project's authors, and checks whether the model's answer satisfies those criteria. A second part of the benchmark deliberately gives the model questions that reference documents which were never actually supplied, to see whether the model notices the documents are missing instead of inventing plausible sounding legal advice about content it never read. The README is upfront that cloning this repository alone does not give a person the actual benchmark questions. The 107 task definitions, split into 67 case law reasoning tasks and 40 missing document detection tasks, are distributed separately as a password free ZIP file that has to be requested by email, then extracted into specific folders inside the project before anything can run. The project also describes a more advanced citation grounding pipeline and an internal gold standard builder, but explicitly states that neither is included here because they depend on private search indexes and cloud storage that only the authors control. The public repository is only meant to be run with citation grounding turned off. Once the task bundle is in place, running the benchmark requires API keys for whichever language model is being tested, plus one or more separate judge models that grade each answer. By default, evaluation uses an adaptive process where two judge models look at the same answer independently, and only calls in a third judge model to break a tie if the first two disagree. Setup involves installing the Python package and creating a local file holding API keys for providers like Anthropic and OpenAI. The README does not state a license for this project.