loc bengaluru, ist | local --:-- srijanshukla18@gmail.com →
[post]/tech/when-opentelemetry-auto-instrumentation-meets-python-pex-a-debugging-journey

why OpenTelemetry Python auto-instrumentation silently fails with PEX

/ 3 min read· infra

OpenTelemetry auto-instrumentation worked in a Python REPL but not in the actual FastAPI server. The reason was PEX running Python with -sE flags, which ignore PYTHONPATH. How I found it, and two ways to fix it.

Ran into a weird bug while setting up OpenTelemetry auto-instrumentation for a Python FastAPI service on k8s. Everything looked configured properly, pod started fine, zero errors in logs - and zero traces showing up in Jaeger.

Dumping the debugging steps and the root cause here coz it took me hours to figure out.

The local repro setup

Spun up a local minikube cluster with Jaeger and the OpenTelemetry Operator to isolate the issue:

minikube start --memory=8192 --cpus=4
kubectl create namespace observability

# Install Jaeger
kubectl apply -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.52.0/jaeger-operator.yaml
kubectl apply -f - <<EOF
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger
  namespace: observability
spec:
  strategy: AllInOne
EOF

# Install OpenTelemetry Operator
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

Configured the OpenTelemetryCollector and the Python Instrumentation CR:

# collector.yaml
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel-gateway
  namespace: observability
spec:
  mode: deployment
  config: |
    receivers:
      otlp:
        protocols:
          http:
            endpoint: 0.0.0.0:4318
    exporters:
      jaeger:
        endpoint: jaeger-collector.observability:14250
        tls:
          insecure: true
    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [jaeger]
---
# instrumentation.yaml
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: python-instrumentation
  namespace: default
spec:
  exporter:
    endpoint: http://otel-gateway-collector.observability:4318
  propagators:
    - tracecontext
    - baggage
  python:
    env:
      - name: OTEL_PYTHON_LOG_CORRELATION
        value: "true"

And added the instrumentation.opentelemetry.io/inject-python: "true" annotation on the FastAPI deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fastapi-app
spec:
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-python: "true"
    spec:
      containers:
      - name: app
        image: myapp:latest
        ports:
        - containerPort: 8000

Port-forwarded Jaeger (kubectl port-forward -n observability svc/jaeger-query 16686:16686), hit the FastAPI endpoints a bunch of times -> nothing in Jaeger.

Checking from inside the pod

Exec’d into the pod to see if the OTel operator webhook actually injected the init container and sitecustomize.py:

kubectl exec -it deployment/fastapi-app -- bash

Opened a python shell inside the container:

$ python
>>> import sys
>>> 'sitecustomize' in sys.modules
True
>>> import sitecustomize
>>> print(sitecustomize.__file__)
/otel-auto-instrumentation-python/sitecustomize.py

Okay, so sitecustomize.py is getting loaded when I run python in the shell.

Next check: is the collector reachable over the cluster network?

>>> import requests
>>> endpoint = "http://otel-gateway-collector.observability:4318"
>>> r = requests.post(f"{endpoint}/v1/traces", 
...                   data=b"junk", 
...                   headers={"content-type": "application/x-protobuf"})
>>> r.status_code, r.text
(400, 'proto: illegal wireType 6')

400 from the collector protobuf parser, so network connectivity is fine.

What if I emit a manual span from the Python REPL?

>>> from opentelemetry import trace
>>> tracer = trace.get_tracer("manual-test")
>>> with tracer.start_as_current_span("test-span"):
...     print("Hello from manual span")
... 
>>> trace.get_tracer_provider()._active_span_processor.force_flush()

Checked Jaeger UI - manual-test showed up immediately!

So why does the interactive python shell send traces, while the FastAPI server running in the exact same container sends nothing?

The actual culprit: PEX -sE flags

Checked the environment variables and command line of PID 1 (the actual FastAPI server process):

$ tr '\0' '\n' < /proc/1/environ | grep -E '^(PYTHONPATH|OTEL_)'
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-gateway-collector.observability:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_SERVICE_NAME=fastapi-app
_PEX_PYTHONPATH=/otel-auto-instrumentation-python

Notice PYTHONPATH got renamed to _PEX_PYTHONPATH. And then look at ps aux:

$ ps aux | head -2
USER  PID  COMMAND
root    1  /usr/bin/python3.11 -sE /app/.bootstrap/pex/pex.py --python /usr/bin/python3.11 /app/service.pex

Look at the flags PEX passes to Python: -sE.

  • -s = don’t add user site directory to sys.path
  • -E = ignore all PYTHON* environment variables (including PYTHONPATH)!

How OpenTelemetry Python auto-instrumentation works under the hood is super simple: the k8s operator mounts /otel-auto-instrumentation-python into your pod and sets PYTHONPATH=/otel-auto-instrumentation-python. When Python starts up normally, it reads PYTHONPATH, finds sitecustomize.py in that folder, and sitecustomize.py monkey-patches FastAPI/requests/etc.

Because our service was packaged as a PEX (Python EXecutable), PEX runs Python in hermetic mode with -sE by default. It strips PYTHONPATH, stashes it in _PEX_PYTHONPATH, and sitecustomize.py never gets loaded! Meanwhile, when I typed python in bash earlier, that launched plain /usr/bin/python3.11 without -sE, which respected PYTHONPATH and loaded OTel fine.

Verified this by dumping sys.path from inside /app/service.pex:

$ cat > check_path.py << EOF
import sys, json
with open('/tmp/syspath.json', 'w') as f:
    json.dump(sys.path, f)
EOF

$ python /app/service.pex check_path.py
$ cat /tmp/syspath.json | jq
# '/otel-auto-instrumentation-python' is completely missing from sys.path

How to fix it

Two ways to fix this depending on whether you can change the PEX build:

Option 1: Build the PEX with --non-hermetic-venv-scripts (proper fix)

If you pass --venv and --non-hermetic-venv-scripts when building the PEX file, PEX omits the -sE flags so PYTHONPATH works normally:

pex -r requirements.txt \
    -c gunicorn \
    --venv service \
    --non-hermetic-venv-scripts \
    -o service.pex .

(Note: PEX_EXTRA_SYS_PATH and PEX_INHERIT_PATH do NOT fix this because PEX appends those to sys.path after the Python interpreter has already finished its sitecustomize.py startup hook.)

Option 2: Run the unpacked PEX or invoke opentelemetry-instrument directly

If you’re stuck with an existing image or can’t change the build pipeline, override the container entrypoint to export PYTHONPATH and invoke python without -E, or wrap the command with opentelemetry-instrument:

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: app
        command: ["/bin/sh", "-c"]
        args:
          - |
            export PYTHONPATH="/otel-auto-instrumentation-python"
            exec python /app/service.pex

Quick checklist if OTel Python auto-instrumentation isn’t working

Whenever the OTel operator injects your pod and zero spans show up, run these 3 commands first:

# 1. Does plain python load sitecustomize.py?
kubectl exec <pod> -- python -c "import sitecustomize; print('Loaded from:', sitecustomize.__file__)"

# 2. Did something strip PYTHONPATH on PID 1?
kubectl exec <pod> -- sh -c 'tr "\0" "\n" < /proc/1/environ | grep -E "^(PYTHONPATH|_PEX_)"'

# 3. Is PID 1 running with python -E or -sE?
kubectl exec <pod> -- ps aux | grep python