When OpenTelemetry Auto-Instrumentation Meets Python PEX, A Debugging Journey

OpenTelemetry auto-instrumentation was supposed to be the easy part of setting up distributed tracing. Six hours later, Jaeger was still empty. The FastAPI service had not sent a single span.

The Setup

I started with a local minikube cluster to test OpenTelemetry auto-instrumentation. The plan:

• Deploy Jaeger for trace storage/UI
• Deploy OpenTelemetry Collector as a gateway
• Use the OpenTelemetry Operator to auto-instrument Python apps
• Watch traces flow without touching application code

# Start fresh
minikube start --memory=8192 --cpus=4
kubectl create namespace observability

# Install Jaeger
kubectl apply -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.52.0/jaeger-operator.yaml
kubectl apply -f - <<EOF
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger
  namespace: observability
spec:
  strategy: AllInOne
EOF

# Install OpenTelemetry Operator
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

Zero traces

Created the collector and instrumentation:

# collector.yaml
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel-gateway
  namespace: observability
spec:
  mode: deployment
  config: |
    receivers:
      otlp:
        protocols:
          http:
            endpoint: 0.0.0.0:4318
    exporters:
      jaeger:
        endpoint: jaeger-collector.observability:14250
        tls:
          insecure: true
    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [jaeger]
---
# instrumentation.yaml
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: python-instrumentation
  namespace: default
spec:
  exporter:
    endpoint: http://otel-gateway-collector.observability:4318
  propagators:
    - tracecontext
    - baggage
  python:
    env:
      - name: OTEL_PYTHON_LOG_CORRELATION
        value: "true"

Deployed a test FastAPI app with the magic annotation:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fastapi-app
spec:
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-python: "true"
    spec:
      containers:
      - name: app
        image: myapp:latest
        ports:
        - containerPort: 8000

Port-forwarded to Jaeger UI:

kubectl port-forward -n observability svc/jaeger-query 16686:16686

Result? Nothing. Zero traces.

REPL detective work

Time to get hands dirty. Exec’d into the pod:

kubectl exec -it deployment/fastapi-app -- bash

First: is auto-instrumentation even loaded?

$ python
>>> import sys
>>> 'sitecustomize' in sys.modules
True
>>> import sitecustomize
>>> print(sitecustomize.__file__)
/otel-auto-instrumentation-python/sitecustomize.py

Good! The operator injected its magic.

Next: can we reach the collector?

>>> import requests
>>> endpoint = "http://otel-gateway-collector.observability:4318"
>>> r = requests.post(f"{endpoint}/v1/traces", 
...                   data=b"junk", 
...                   headers={"content-type": "application/x-protobuf"})
>>> r.status_code, r.text
(400, 'proto: illegal wireType 6')

Perfect! The collector is reachable and trying to parse our junk.

Last: can we manually send a span?

>>> from opentelemetry import trace
>>> tracer = trace.get_tracer("manual-test")
>>> with tracer.start_as_current_span("test-span"):
...     print("Hello from manual span")
... 
>>> # Force flush
>>> trace.get_tracer_provider()._active_span_processor.force_flush()

Checked Jaeger. The manual span appeared.

So the REPL can send traces, but the FastAPI server can’t.

The smoking gun

Let’s check what the actual server process sees:

# Check PID 1 environment
$ tr '\0' '\n' < /proc/1/environ | grep -E '^(PYTHONPATH|OTEL_)'
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-gateway-collector.observability:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_SERVICE_NAME=fastapi-app
_PEX_PYTHONPATH=/otel-auto-instrumentation-python

Hold on… _PEX_PYTHONPATH? Not PYTHONPATH?

# Check the actual process
$ ps aux | head -2
USER  PID  COMMAND
root    1  /usr/bin/python3.11 -sE /app/.bootstrap/pex/pex.py --python /usr/bin/python3.11 /app/service.pex

There it is. The app is packaged as a PEX, running with -sE flags:

• -E: Ignores all PYTHON* environment variables
• -s: Ignores user site directory

Why PEX breaks it

PEX (Python EXecutable) creates hermetic Python environments. Here’s what’s happening:

1. OpenTelemetry Operator sets PYTHONPATH=/otel-auto-instrumentation-python
2. PEX launcher starts Python with -E flag, ignoring PYTHONPATH
3. Python never loads /otel-auto-instrumentation-python/sitecustomize.py
4. No auto-instrumentation happens

But why does the REPL work? Because python command bypasses the PEX launcher!

Let’s verify this theory:

# In REPL (works)
>>> import sys
>>> '/otel-auto-instrumentation-python' in sys.path
True

# Check server's sys.path
$ cat > check_path.py << EOF
import sys
import json
with open('/tmp/syspath.json', 'w') as f:
    json.dump(sys.path, f)
EOF

$ python /app/service.pex check_path.py
$ cat /tmp/syspath.json | jq
# Result: No /otel-auto-instrumentation-python!

The fixes

Fix A: rebuild the PEX with non-hermetic scripts (the clean one)

# Original PEX build
pex -r requirements.txt -c gunicorn -o service.pex .

# Fixed PEX build
pex -r requirements.txt \
    -c gunicorn \
    --venv service \
    --non-hermetic-venv-scripts \
    -o service.pex .

The --non-hermetic-venv-scripts flag creates venv scripts that respect environment variables.

Fix B: runtime wrapper (quick and dirty)

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: app
        command: ["/bin/sh"]
        args:
          - -c
          - |
            export PYTHONPATH="/otel-auto-instrumentation-python"
            exec python /app/service.pex

What doesn’t work

• PEX_EXTRA_SYS_PATH: Appends to sys.path after startup (too late for sitecustomize.py)
• PEX_INHERIT_PATH: Still blocked by -E flag
• Manual instrumentation: Defeats the whole “zero-code” purpose

For next time

When OpenTelemetry auto-instrumentation seems broken, the quick diagnostic:

# 1. Check if instrumentation is loaded in REPL
kubectl exec <pod> -- python -c "import sitecustomize; print('Loaded from:', sitecustomize.__file__)"

# 2. Check PID 1 environment
kubectl exec <pod> -- sh -c 'tr "\0" "\n" < /proc/1/environ | grep -E "^(PYTHONPATH|_PEX_)"'

# 3. Check the actual process command
kubectl exec <pod> -- ps aux | grep python

# 4. Test manual spans
kubectl exec -it <pod> -- python
>>> from opentelemetry import trace
>>> with trace.get_tracer("test").start_as_current_span("test"): pass
>>> trace.get_tracer_provider()._active_span_processor.force_flush()

How your packaging affects it:

The whole auto-instrumentation mechanism is one trick: set PYTHONPATH to include the instrumentation, Python loads sitecustomize.py at startup, and that hooks and instruments your code. Any packaging that breaks Python’s normal startup breaks it. In this case, PEX was the package in the way.