wtf is ml-over-searching?

OverSearchQA is a dataset that tests whether AI language models know when they should not look something up. When AI assistants are connected to search tools, they tend to search for answers to every question they receive. But sometimes searching actually makes things worse, like when nobody could possibly know the answer yet, or when the question itself is flawed. This benchmark helps researchers measure whether a model can recognize those situations and simply say "I shouldn't search for this" instead of going down a rabbit hole. The dataset covers three tricky scenarios. The first is questions about genuinely unknowable things, like which stock will perform best over the next 15 years. The second is questions with false premises built in, assumptions that just aren't true, which no search can fix. The third is questions so vague that there isn't enough context to give a definitive answer. Each category has a balanced set of questions where the model should abstain from searching and questions where searching is perfectly fine, so you can measure whether the model correctly tells the difference. The dataset contains 1,188 questions total, split evenly between cases where a model should hold back and cases where it should go ahead and search. Each entry includes the question, the expected answer or explanation, and some metadata about where the question came from. It's distributed as three JSON files (one per scenario) with a straightforward format that makes it easy to load and evaluate against. Researchers and engineers building search-augmented AI systems would use this to evaluate two things: whether their model correctly identifies when to skip the search, and whether it avoids wasting unnecessary search calls. If you're building a customer support bot or a research assistant, for instance, you want it to handle unanswerable questions gracefully rather than confidently returning irrelevant search results. The project comes from a team at Apple and is associated with a paper forthcoming at a major natural language processing conference.