Ran into a weird bug while setting up OpenTelemetry auto-instrumentation for a Python FastAPI service on k8s. Everything looked configured properly, pod started fine, zero errors in logs - and zero traces showing up in Jaeger.
Dumping the debugging steps and the root cause here coz it took me hours to figure out.
The local repro setup
Spun up a local minikube cluster with Jaeger and the OpenTelemetry Operator to isolate the issue:
minikube start --memory=8192 --cpus=4
kubectl create namespace observability
# Install Jaeger
kubectl apply -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.52.0/jaeger-operator.yaml
kubectl apply -f - <<EOF
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: jaeger
namespace: observability
spec:
strategy: AllInOne
EOF
# Install OpenTelemetry Operator
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
Configured the OpenTelemetryCollector and the Python Instrumentation CR:
# collector.yaml
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: otel-gateway
namespace: observability
spec:
mode: deployment
config: |
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
exporters:
jaeger:
endpoint: jaeger-collector.observability:14250
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
exporters: [jaeger]
---
# instrumentation.yaml
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: python-instrumentation
namespace: default
spec:
exporter:
endpoint: http://otel-gateway-collector.observability:4318
propagators:
- tracecontext
- baggage
python:
env:
- name: OTEL_PYTHON_LOG_CORRELATION
value: "true"
And added the instrumentation.opentelemetry.io/inject-python: "true" annotation on the FastAPI deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: fastapi-app
spec:
template:
metadata:
annotations:
instrumentation.opentelemetry.io/inject-python: "true"
spec:
containers:
- name: app
image: myapp:latest
ports:
- containerPort: 8000
Port-forwarded Jaeger (kubectl port-forward -n observability svc/jaeger-query 16686:16686), hit the FastAPI endpoints a bunch of times -> nothing in Jaeger.
Checking from inside the pod
Exec’d into the pod to see if the OTel operator webhook actually injected the init container and sitecustomize.py:
kubectl exec -it deployment/fastapi-app -- bash
Opened a python shell inside the container:
$ python
>>> import sys
>>> 'sitecustomize' in sys.modules
True
>>> import sitecustomize
>>> print(sitecustomize.__file__)
/otel-auto-instrumentation-python/sitecustomize.py
Okay, so sitecustomize.py is getting loaded when I run python in the shell.
Next check: is the collector reachable over the cluster network?
>>> import requests
>>> endpoint = "http://otel-gateway-collector.observability:4318"
>>> r = requests.post(f"{endpoint}/v1/traces",
... data=b"junk",
... headers={"content-type": "application/x-protobuf"})
>>> r.status_code, r.text
(400, 'proto: illegal wireType 6')
400 from the collector protobuf parser, so network connectivity is fine.
What if I emit a manual span from the Python REPL?
>>> from opentelemetry import trace
>>> tracer = trace.get_tracer("manual-test")
>>> with tracer.start_as_current_span("test-span"):
... print("Hello from manual span")
...
>>> trace.get_tracer_provider()._active_span_processor.force_flush()
Checked Jaeger UI - manual-test showed up immediately!
So why does the interactive python shell send traces, while the FastAPI server running in the exact same container sends nothing?
The actual culprit: PEX -sE flags
Checked the environment variables and command line of PID 1 (the actual FastAPI server process):
$ tr '\0' '\n' < /proc/1/environ | grep -E '^(PYTHONPATH|OTEL_)'
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-gateway-collector.observability:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_SERVICE_NAME=fastapi-app
_PEX_PYTHONPATH=/otel-auto-instrumentation-python
Notice PYTHONPATH got renamed to _PEX_PYTHONPATH. And then look at ps aux:
$ ps aux | head -2
USER PID COMMAND
root 1 /usr/bin/python3.11 -sE /app/.bootstrap/pex/pex.py --python /usr/bin/python3.11 /app/service.pex
Look at the flags PEX passes to Python: -sE.
-s= don’t add user site directory tosys.path-E= ignore allPYTHON*environment variables (includingPYTHONPATH)!
How OpenTelemetry Python auto-instrumentation works under the hood is super simple: the k8s operator mounts /otel-auto-instrumentation-python into your pod and sets PYTHONPATH=/otel-auto-instrumentation-python. When Python starts up normally, it reads PYTHONPATH, finds sitecustomize.py in that folder, and sitecustomize.py monkey-patches FastAPI/requests/etc.
Because our service was packaged as a PEX (Python EXecutable), PEX runs Python in hermetic mode with -sE by default. It strips PYTHONPATH, stashes it in _PEX_PYTHONPATH, and sitecustomize.py never gets loaded! Meanwhile, when I typed python in bash earlier, that launched plain /usr/bin/python3.11 without -sE, which respected PYTHONPATH and loaded OTel fine.
Verified this by dumping sys.path from inside /app/service.pex:
$ cat > check_path.py << EOF
import sys, json
with open('/tmp/syspath.json', 'w') as f:
json.dump(sys.path, f)
EOF
$ python /app/service.pex check_path.py
$ cat /tmp/syspath.json | jq
# '/otel-auto-instrumentation-python' is completely missing from sys.path
How to fix it
Two ways to fix this depending on whether you can change the PEX build:
Option 1: Build the PEX with --non-hermetic-venv-scripts (proper fix)
If you pass --venv and --non-hermetic-venv-scripts when building the PEX file, PEX omits the -sE flags so PYTHONPATH works normally:
pex -r requirements.txt \
-c gunicorn \
--venv service \
--non-hermetic-venv-scripts \
-o service.pex .
(Note: PEX_EXTRA_SYS_PATH and PEX_INHERIT_PATH do NOT fix this because PEX appends those to sys.path after the Python interpreter has already finished its sitecustomize.py startup hook.)
Option 2: Run the unpacked PEX or invoke opentelemetry-instrument directly
If you’re stuck with an existing image or can’t change the build pipeline, override the container entrypoint to export PYTHONPATH and invoke python without -E, or wrap the command with opentelemetry-instrument:
apiVersion: apps/v1
kind: Deployment
spec:
template:
spec:
containers:
- name: app
command: ["/bin/sh", "-c"]
args:
- |
export PYTHONPATH="/otel-auto-instrumentation-python"
exec python /app/service.pex
Quick checklist if OTel Python auto-instrumentation isn’t working
Whenever the OTel operator injects your pod and zero spans show up, run these 3 commands first:
# 1. Does plain python load sitecustomize.py?
kubectl exec <pod> -- python -c "import sitecustomize; print('Loaded from:', sitecustomize.__file__)"
# 2. Did something strip PYTHONPATH on PID 1?
kubectl exec <pod> -- sh -c 'tr "\0" "\n" < /proc/1/environ | grep -E "^(PYTHONPATH|_PEX_)"'
# 3. Is PID 1 running with python -E or -sE?
kubectl exec <pod> -- ps aux | grep python