wtf is lmcache-tests?

lmcache-tests is a benchmarking and testing toolkit for LMCache, a system designed to speed up large language model serving. If you're running an LLM-powered application and want to understand whether caching intermediate computation results actually improves response times and throughput, this project gives you a structured way to measure that. At a high level, the project works by simulating realistic traffic to an LLM serving engine. You configure a workload, specifying things like request rate, context length, and session duration, and the system generates a stream of requests. It then spins up one or more serving engines (using vLLM, with or without LMCache enabled), sends the identical traffic to each, and records performance metrics. The output is a CSV file capturing per-request stats like time-to-first-token, throughput, latency, and GPU memory usage, along with a script to generate a PDF report from those results. The project includes a variety of pre-built test scenarios covering different storage backends (local CPU, GPU, disk, remote Redis) and usage patterns like multi-turn conversations, variable-length workloads, and chunked prefill. You can run all tests or filter by name to target specific functionality. The test runner also supports specifying which model to use, such as Meta's Llama 3.1 8B. The primary audience is developers contributing to LMCache itself, the README explicitly notes it's currently an onboarding doc for new developers. But it's also useful for anyone evaluating LLM serving infrastructure who wants hard numbers on whether caching helps their specific workload. A team building a customer support chatbot, for instance, could use it to quantify how much faster repeat queries return when cached versus recomputed from scratch. One practical limitation: the workload generator currently only produces dummy text, not realistic conversational data, so the performance numbers reflect infrastructure behavior rather than real-world language patterns. The project is designed to be extensible, with clear separation between workload generation, engine bootstrapping, and experiment execution, making it straightforward to add new test scenarios.