wtf is agentbench?

AgentBench is a testing framework designed for AI agents. It lets developers replay agent interactions, evaluate the results, assert expected behaviors, and catch regressions before they reach users. Think of it as a test runner like Jest, but built for non-deterministic AI systems instead of predictable code. The motivation is straightforward. AI agents are unpredictable, and small changes to a prompt, model, or tool can silently break behavior. Most teams only notice when users complain. AgentBench addresses this by providing automated test suites with assertions, cross-model replay to catch drift, full execution traces, and CI integration so regressions are caught before merging. The framework centers on a layered testing strategy. Deterministic assertions, such as checking which tools were called, token counts, and latency, run on every pull request. LLM-assisted quality scores across eight dimensions like correctness, safety, and faithfulness gate pre-release evaluation. The README references an "Agent Testing Pyramid" for the full methodology and an anti-patterns guide for common mistakes to avoid. Writing a test involves creating a suite, defining a test case, calling your agent, and asserting expectations. The assertion DSL supports chaining checks together. You can verify that a specific tool was called with certain arguments, that the output contains expected text, that token usage stays below a threshold, that latency meets a bound, and that quality scores exceed a target. The framework includes 14 rule evaluators, an LLM-as-judge mode with configurable voting strategies, snapshot management for saving and restoring agent state, a replay engine for deterministic or cross-model replay, a diff engine for comparisons, and automatic regression detection. The project is built in TypeScript and offers a CLI installed via npm, a web dashboard, a Python SDK, and a VS Code extension. It supports 12 or more AI providers. The README is longer than what was shown.