wtf is read?

READ is a research toolkit from IBM for testing whether an AI system used as a judge, one that decides if content violates a written policy, is actually reading and reasoning about that policy or just guessing from what it learned during training. It is the companion code for an academic paper under review at a 2026 AI ethics and society conference. The tool works by asking the same question to a judge model multiple times with small variations: once with the full policy, once with no policy at all, once with the specific rule that was supposedly broken removed, and once with other unrelated rules removed one at a time. By comparing how the judge's verdict changes across these variations, the framework can tell whether the judge is genuinely anchored to the correct rule, ignoring the policy entirely, or getting confused by unrelated details. Results get sorted into labeled categories such as Faithful, Confused, Prior Based, or Policy Degraded. It ships as a Python package with a command line interface, and works with judge models from OpenAI or Anthropic, requiring you to supply your own API key for whichever provider you use. It downloads its test dataset automatically from Hugging Face on first run, drawing on a published benchmark covering domains like HR policy, education, and social media moderation. The output is a spreadsheet style CSV of verdicts that can then be summarized into statistics with confidence intervals. This project fits AI safety researchers and practitioners who build or evaluate LLM based judges and want a rigorous way to check whether those judges are trustworthy, rather than developers building general purpose applications.