wtf is llama.cpp?

llama.cpp lets you run large language models on your own computer using plain C and C++ code, with no external dependencies required. Its goal is to make LLM inference work with minimal setup and strong performance across a wide range of hardware, whether that hardware is a laptop, a desktop with a GPU, or a server in the cloud. The project treats Apple silicon as a first class target, using ARM NEON, Accelerate, and Metal to speed things up on Mac hardware, and it also supports AVX, AVX2, AVX512, and AMX on x86 chips, plus several extensions for RISC-V processors. It supports custom CUDA code for NVIDIA GPUs, with AMD GPUs handled through HIP and Moore Threads GPUs through MUSA, along with Vulkan and SYCL backends. Models can be shrunk using quantization ranging from 1.5 bit up to 8 bit integers, which trades some accuracy for faster inference and lower memory use, and the project can split a model across CPU and GPU together when it is too large to fit entirely in available video memory. Getting started involves installing the software through a package manager such as brew, nix, winget, or conda-forge, running it with Docker, downloading a prebuilt binary from the releases page, or building it from source. Once installed, you point the llama-cli tool at a local model file, or download one directly from Hugging Face with a single flag. There is also a llama-server command that launches an OpenAI compatible API server, along with a web interface for chatting with a loaded model. The project supports a very large number of model families, including several generations of LLaMA, Mistral, Mixtral, Falcon, Gemma, Qwen, Phi, and many others, and finetuned versions of these models generally work as well. It also serves as the main testing ground for new features in the underlying ggml library that llama.cpp is built on. The full README is longer than what was shown.