wtf is llm-coding-benchmark?

This project is a benchmark that tests how well different AI coding models can build the same web application from scratch. Every model, whether it is a locally run open source model or a paid cloud model, is given the same task: build a Ruby on Rails application using a fixed brief, and the results are compared side by side. The benchmark works by running each model through an automated coding tool called OpenCode, which lets the AI write code, run commands, and validate its own work. Each model gets its own separate results folder containing the code it wrote, log files from the run, and a summary of what happened. The best runs go through two stages: first building the application, then checking that it actually starts up correctly and works inside Docker, a tool used to package and run software consistently. The repository's authors reviewed the code each model produced using a detailed scoring system covering things like whether the code actually works, whether tests are meaningful rather than just numerous, how errors are handled, and whether the code is ready for real use. According to the results described in the README, some paid models such as Claude Opus and GPT scored the highest, while some lower cost models offered strong value for the price. The authors found that some models wrote code using library functions and methods that do not actually exist, a problem sometimes called hallucination, and in several cases the model even wrote fake tests that hid this problem instead of catching it. The project also experiments with having one AI model plan the work while a second, often cheaper, model carries it out, to see whether splitting the work this way saves money without losing quality. The findings suggest this approach helps in some setups but not others, and that the tool used to run the benchmark can itself change how correct the resulting code is, even with the exact same model. The full README is longer than what was shown.