CPU usage at pod level in k8s

`Published: Dec 2019`

How is pod level CPU usage metric calculated in k8s?

The common PromQL query found on the internet may resemble this,

sum(rate(container_cpu_usage_seconds_total{name!~".*prometheus.*", image!="", container_name!="POD"}[5m])) by (pod_name, namespace) /
sum(container_spec_cpu_quota{name!~".*prometheus.*", image!="", container_name!="POD"}/container_spec_cpu_period{name!~".*prometheus.*", image!="", container_name!="POD"}) by (pod_name, namespace)

Let’s break down how this works. There are three metrics in the above equation, let’s understand them first:

**container_cpu_usage_seconds_total**: This is the cumulative CPU time consumed by a container, expressed as a counter.
**container_spec_cpu_quota**: This represents the CPU limit set for the container as a fraction of one core, multiplied by **container_spec_cpu_period**.
**container_spec_cpu_period**: This denotes the time window used by the Completely Fair Scheduler (CFS) when limiting a container’s CPU usage.

To calculate CPU utilization, we need to divide the cumulative CPU time consumed by the total number of CPU cores available to the container:

CPU Utilization = Number of CPU seconds utilized / Number of system cores available

We can derive the number of CPU cores specified as the container limit using this formula:

Number of cores = container_spec_cpu_quota / container_spec_cpu_period

As an example, if the CPU limit is set to 500m (500 millicores) for a container and **container_spec_cpu_period** is set to 100,000, **container_spec_cpu_quota** will be 50,000.

Substituting these metrics into the CPU Utilization formula, we get:

CPU Utilization = container_cpu_usage_seconds_total / (container_spec_cpu_quota / container_spec_cpu_period)

However, because **container_cpu_usage_seconds_total** is a counter (not a gauge), we need to apply the **rate()** function, which calculates the per-second average rate of increase for the given time series:

CPU Utilization = rate(container_cpu_usage_seconds_total) / (container_spec_cpu_quota / container_spec_cpu_period)

This raw equation isn’t suitable as a PromQL query due to the many potential label values. After filtering, if you want to compute the CPU usage percentage per pod, the PromQL query might look something like this:

sum(rate(container_cpu_usage_seconds_total{name!~".*prometheus.*", image!="", container_name!="POD"}[5m])) by (pod)
/
sum(container_spec_cpu_quota{name!~".*prometheus.*", image!="", container_name!="POD"}/container_spec_cpu_period{name!~".*prometheus.*", image!="", container_name!="POD"}) by (pod)

Please note, these metrics and labels may vary depending on your Kubernetes setup, so make sure to validate your queries with the actual metrics and labels from your specific Kubernetes cluster. Additionally, it’s important to note that Kubernetes itself does not provide a direct metric for CPU percentage usage. The metrics mentioned in the query are from cAdvisor, which is part of the kubelet’s functionality. Kubernetes exposes metrics like **cpu/usage_rate** and **cpu/usage_time**, which are used to derive the CPU utilization.