wtf is whatbroke?

whatbroke is a command-line tool that compares what an AI agent did across two runs. When you change a model, prompt, or framework version, the tool shows exactly which tool calls disappeared, which arguments shifted, where costs and latency moved, and which outputs changed. The core problem it addresses is that text-based diffs miss behavioral changes. An agent might say "your subscription is cancelled" while silently skipping the cancellation call entirely. The words look correct, but the behavior broke. The workflow is straightforward. Record a trace of your agent doing its job, which is a plain JSONL file with one event per line. Change something about your setup. Record a new trace. Run a diff command to compare them. whatbroke aligns the runs and reports findings at three severity levels: breaking (like dropped tool calls or failures), changed (like argument drift or cost increases), and info (like model swaps or token swings). The exit code is 1 when something breaking appears, so you can plug it directly into continuous integration pipelines. AI agents are naturally inconsistent, so a single before-and-after comparison can blame your change for noise the agent was already producing. whatbroke handles this by letting you record the same scenario multiple times with suffixes like "refund-flow#1, refund-flow#2". It compares every baseline sample against every new sample and attaches a rate to each finding. Issues that also appear between baseline samples get demoted to flaky, since the agent behaved that way before your change. To get traces, you can use a proxy that requires zero code changes. Point your agent at a local address and every call gets recorded. There are also SDK wrappers for OpenAI and Anthropic in Node, plus manual methods for other languages. If you already record traces through OpenTelemetry, Langfuse, or Langsmith, an import command converts those into whatbroke's format. The full README is longer than what was shown.