wtf is basically-agent-evals?

This repository is a hands-on TypeScript course that teaches how to properly test AI language model systems, and how easy it is to build tests that quietly give you false confidence. It was built to accompany a 60 minute talk, and each folder is a self-contained, runnable lesson that builds on the one before it, starting from a simple assertion and ending with a live diagnostic tool pointed at a real retrieval-based AI system. The core idea is that testing an AI system is like a normal software test, except the input can be phrased endless ways, the notion of a correct answer is fuzzy rather than exact, and the output can change between runs. So instead of checking for an exact match, you need a set of varied example inputs, some definition of what counts as good, and a checker that tolerates variation. The whole point is repeatability: change your prompt, rerun the test, and see whether the score went up or down. A large part of the course focuses on a common but flawed technique: using one AI model to judge the output of another. Using a return policy scenario as a running example, it shows how a judge model can be fooled into preferring a longer, more confident but factually wrong answer over a short, correct one. It walks through several biases that cause this, such as favoring whichever answer appears first, favoring longer answers, and favoring an agreeable tone over the truth. The fixes taught include giving the judge model the correct answer to compare against, checking how often the judge agrees with real human graders before trusting it, replacing vague scoring with specific pass or fail checks where possible, and saving every past failure as a permanent test case so a codebase does not regress over time. Later sections apply these same techniques to a real, working retrieval-based AI pipeline, showing how to tell whether a wrong answer came from failing to find the right information or from ignoring information it did find. The course can run fully offline using canned responses, or against a live AI model using an API key. It is written for developers who are building or maintaining systems that use large language models and want a reliable way to know if a change actually made things better or worse.