wtf is imario-benchmark?

This repository is an open benchmark for checking how accurately a product called iMario can predict how real people would answer survey questions, using simulated respondents instead of actual human panels. Rather than just publishing a single accuracy number, it publishes the underlying data and a small script so that anyone can rerun the check themselves and confirm the published results independently. Running the included score.py file, which needs nothing beyond a standard Python install, recomputes the accuracy scores from the raw simulated answers and compares them against the real survey distributions. This is meant to prove the numbers are not just claimed but reproducible from evidence anyone can inspect, including thousands of individual simulated answers for each question, not just the final averages. The benchmark measures how close the distribution of simulated answers is to the distribution of real human answers on the same question, across eleven countries and one open ended question set. It groups questions into three honesty tiers: brand new questions with no prior data to learn from, questions repeated from an earlier year, and questions with a very recent prior answer to draw on, so that easier and harder cases are not blended together misleadingly. Its source survey question banks come from well known public research sources such as Pew Research and the Stack Overflow Developer Survey, and the underlying licensed survey microdata itself is not included, only extraction scripts to rebuild the question bank from official downloads. This repository is for verifying published accuracy claims only. It does not let you run the actual simulated survey engine on your own questions, that requires a separate paid product. It is aimed at researchers, data scientists, and buyers evaluating synthetic survey tools who want to audit accuracy claims rather than take a vendor's word for it. The code is released under the Apache 2.0 license.