wtf is kube-state-metrics?

Kube-state-metrics is a monitoring tool that watches your Kubernetes cluster and reports what's happening to it. Instead of measuring CPU usage or response times, it focuses on tracking the state of things like deployments, pods, nodes, and other Kubernetes objects, whether they exist, how many replicas are running, what labels they have, and so on. Think of it as a detailed inventory system for everything in your cluster. The tool works by continuously connecting to your Kubernetes API server and reading the current state of all your cluster objects. It then converts this information into metrics, a standardized format that monitoring systems like Prometheus can understand and store. These metrics are exposed on a simple web endpoint that Prometheus (or any compatible monitoring tool) can scrape periodically, usually every 15-30 seconds. The key insight is that kube-state-metrics reports raw, unfiltered data directly from Kubernetes, without trying to interpret or summarize it the way the kubectl command-line tool does. Teams use this when they want to build alerts and dashboards around cluster health and resource management. For example, a DevOps engineer might use these metrics to alert when a deployment doesn't have the expected number of replicas running, or to track how many nodes are in a "not ready" state. It's particularly useful in large clusters where manual inspection is impractical, and it integrates naturally with existing Prometheus-based monitoring stacks. One practical consideration: as your cluster grows, kube-state-metrics consumes more memory and CPU to track all those objects. The README suggests starting with about 200MB of memory and 0.1 CPU cores for smaller clusters, then scaling up as needed. For very large clusters, you can run multiple copies of kube-state-metrics and shard the work among them, so each instance only tracks a subset of your objects.