wtf is cachy-router?

Cachy Router is a traffic router for people running their own local AI models with llama.cpp, a popular tool for running large language models on your own hardware. If you have several machines each running a copy of a model, this router sits in front of them and presents one single address that behaves like the standard OpenAI API, so any tool built to talk to OpenAI can talk to your own setup instead. Beyond simple routing, the project's main feature is cache awareness. Language models can reuse previously processed text, called a prefix cache, to avoid redoing work and to respond faster on follow up requests. Cachy Router adds a way to explicitly build, use, and refresh this cache across a cluster of worker machines, storing it in a durable cache store the router itself manages, on top of whatever caching each individual worker already keeps locally. The project is explicit and detailed about what is finished versus still being tested. Using its own tracking system, it reports 168 total capability checks, with the large majority marked as done through offline automated tests, a handful still needing live testing on real hardware, and a few blocked on external dependencies outside the project's control. Features already working through automated checks include normal request routing and fallback if a worker goes down, per tenant access rules so different users' cached data cannot leak into each other's requests, and optional request queuing when workers are busy. Features still requiring further real world validation include certain advanced caching modes and long running stability tests over many hours. Installation for basic checks needs only Python's standard library with no extra packages. A full working deployment additionally needs Python 3 on every machine involved and a compatible llama.cpp server setup on the worker machines. The documentation is very clear that the default setup is meant only for a trusted local network and should never be exposed directly to the public internet, since it does not include the security hardening that would require. This is aimed at people already comfortable running their own AI infrastructure, not casual users, and the author describes it as an early public release of a working setup rather than finished production software.