CPU usage at pod level in k8s

`Published: Dec 2019`

How is pod-level CPU usage % calculated in Kubernetes?

If you google for a PromQL query to get pod CPU utilization against its limit, you’ll usually come across something that looks like this,

sum(rate(container_cpu_usage_seconds_total{name!~".*prometheus.*", image!="", container_name!="POD"}[5m])) by (pod_name, namespace) /
sum(container_spec_cpu_quota{name!~".*prometheus.*", image!="", container_name!="POD"}/container_spec_cpu_period{name!~".*prometheus.*", image!="", container_name!="POD"}) by (pod_name, namespace)

Looks messy at first glance, so let’s break down the three cAdvisor metrics inside it:

container_cpu_usage_seconds_total: Counter of total CPU seconds consumed by the container since it started.
container_spec_cpu_period: The CFS (Completely Fair Scheduler) time window in microseconds. Almost always 100000 (100ms).
container_spec_cpu_quota: How many microseconds of CPU time the container is allowed to use per container_spec_cpu_period window, based on the resources.limits.cpu you set on the pod.

Why do we divide quota by period in the denominator?

Because that gives you the container’s CPU limit in cores!

For example, suppose you set limits.cpu: 500m (0.5 cores) on a container:

container_spec_cpu_period = 100,000 (100ms)
container_spec_cpu_quota = 50,000 (50ms)
quota / period = 50000 / 100000 = 0.5 cores.

And for the numerator: since container_cpu_usage_seconds_total is a cumulative counter of CPU seconds used, taking rate(container_cpu_usage_seconds_total[5m]) gives you how many CPU seconds the container is burning per second over the last 5 minutes - which is literally the number of cores it is currently using!

So the math is just:

CPU utilization ratio = (cores currently used) / (cores allowed by limit)
                      = rate(container_cpu_usage_seconds_total[5m]) / (container_spec_cpu_quota / container_spec_cpu_period)

And wrapping both sides in sum(...) by (pod) (while filtering out the pause container container_name!="POD" and empty image!="" cgroup parent slices) aggregates all containers in the pod:

sum(rate(container_cpu_usage_seconds_total{name!~".*prometheus.*", image!="", container_name!="POD"}[5m])) by (pod)
/
sum(container_spec_cpu_quota{name!~".*prometheus.*", image!="", container_name!="POD"}/container_spec_cpu_period{name!~".*prometheus.*", image!="", container_name!="POD"}) by (pod)

(Note: on newer kubelet/cAdvisor versions, the label is pod and container instead of pod_name and container_name, and kube-state-metrics exposes kube_pod_container_resource_limits{resource="cpu"} which is much cleaner to use in the denominator than quota / period.)