wtf is rigorloop?

RigorLoop is a Python framework that uses AI agents to build small solutions to data-transformation tasks, like extracting fields from messy text or reformatting one structure into another, and evaluates the result the way a careful data scientist would rather than trusting a single test run. You give it a plain-language task description, a set of example inputs and expected outputs, and a list of checks, and it repeatedly runs a strategy agent that directs several executor agents to write and refine candidate solutions. What makes RigorLoop different from a single coding-agent session is how it measures success. Your examples get split once into three separate groups: a development set the building agents see while working, a validation set used to decide which candidate is actually improving and when to stop, and a held-out test set that no agent ever sees until the very end. This split is meant to catch a solution that only looks good because it overfit to the examples it was built and checked against, and the final report shows pass rates with confidence intervals on all three splits so the score can actually be trusted. The output can take one of three forms: a plain executable Python script, a Claude-style agent skill document, or a guidance file meant to steer a coding agent, and any of these can be copied out and used on their own without RigorLoop installed. Checks that decide whether an example passes include exact string matching, matching after normalization, JSON equality, regular expression matching, numeric tolerance, a custom Python script, or having a model judge the output against a rubric. The project requires Python 3.12 or newer on Linux or macOS, along with the Claude CLI installed and authenticated, since it invokes agents through that tool. It warns that small example sets cannot prove much statistically and recommends at least a hundred examples, and it estimates how many model calls a run will cost before you commit to running it.