wtf is som?

Set-of-Mark (SoM) is a technique that helps AI vision models like GPT-4V understand and reference specific parts of an image. It works by overlaying numbered labels directly onto objects in a picture, so the AI can point to and talk about exact regions instead of guessing. At a high level, the project uses segmentation models to identify objects and regions in an image, then draws numbered marks on each one. You can run a demo that generates these marked images automatically or interactively, choosing how detailed the labels should be. Once the image is marked, you send it to GPT-4V along with a text prompt, and the model can refer to specific objects by their numbers, answer questions about them, or follow instructions tied to particular regions. This would be useful for anyone building applications where an AI needs to reason about specific things in a photo. For example, it can help solve CAPTCHA puzzles by identifying which numbered squares match a description, guide users through smartphone interfaces by labeling each button, teach someone what every control on a game controller does, or plan moves in a 2D game by referencing marked elements on screen. It also handles food identification, knowledge sharing about dish ingredients, and simulated navigation tasks. What makes this approach notable is that it bridges a gap in vision models: GPT-4V alone can describe images but struggles to ground its answers to specific regions. By adding numbered visual marks, the model gains precise spatial references, and the authors found it outperforms specialist models on most vision tasks they tested. The project itself is built on top of several existing segmentation models, and the README provides install commands for those dependencies along with a demo script to try it locally.