wtf is rondine?

Rondine is a tool for running large language models on your own computer instead of through a cloud service. It figures out how much memory and graphics power your machine has, recommends models that will actually fit, downloads them, and starts a server that other software can talk to. It works on Mac computers with Apple Silicon chips, computers with NVIDIA graphics cards, and NVIDIA DGX Spark systems. The tool does not build its own engine for running models. Instead, it acts as a layer on top of three existing engines called llama.cpp, MLX-LM, and vLLM. Rondine's job is to pick the right engine for your hardware, configure it with tuned settings like batch sizes and memory limits, and handle the workflow of getting a model up and running. You start it with a single command, and it guides you through checking your system, picking a model, downloading it, and starting the server. Rondine offers profiles for different use cases. A "coding" profile picks models suited for programming tasks, sets a larger context window of 32,000 tokens, and tunes the engine for a single user. A "chat" profile uses a smaller context and settings geared toward conversation. The tool can also search Hugging Face, a popular model repository, to discover models that fit your hardware. Results are labeled to show whether they come from Rondine's curated catalog, a live search, or a hardware-targeted recommendation. For models too large to fit entirely in graphics memory, Rondine supports a hybrid mode where some data stays in system RAM while the rest goes to the graphics card. There is also an experimental mode that lets very large models read directly from a fast SSD, though this can be extremely slow. Rondine will never choose this SSD mode automatically because performance may be unusable. The project includes documentation on installation, command-line options, engine tuning, and benchmarks. On a 32GB M2 Pro Mac, one test model ran at about 66.6 tokens per second. The code is open source under the Apache 2.0 license.