Open Telemetry The Easy Way
This article simplifies the process of setting up jaegar with open telemetry in your code base. Instead of unending function wrappers and confusion about aggregating telemetry data, this article will explain a simple way to add function calls your trace data.
Why Telemetry?
Telemetry is a means for collecting metrics and traces that explain and measure latency. This is especially useful for pinpointing performance issues and identifying the full request dependency tree among other things.
There are two main types within telemetry data that you will want to know. These are:
- Traces
- Spans
Traces represent the lifecycle of a single request as it flows through a service.
Spans represent a singular timed operated within a given trace such as a function call.
Now that you are familiar with the absolute basics, let's spin up a simple FastAPI server and Jaegar frontend to demonstrate the capabilities.
Code Walkthrough
Environment Setup
Start by creating the following files
OTEL_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
SERVICE_NAME=my-fastapi
APP_ENV=development
fastapi>=0.110.0
uvicorn[standard]>=0.27.0
python-dotenv>=1.0.1
opentelemetry-api>=1.23.0
opentelemetry-sdk>=1.23.0
opentelemetry-exporter-otlp>=1.23.0
opentelemetry-instrumentation>=0.44b0
opentelemetry-instrumentation-fastapi>=0.44b0
opentelemetry-instrumentation-asgi>=0.44b0
opentelemetry-instrumentation-requests>=0.44b0
Now, create a virtual environment and install the dependencies
python -m venv venv
source venv/bin/active
pip install -r requirements.txtFastAPI Server
Copy the following FastAPI server code to a file called main.py
import os
from dotenv import load_dotenv
from fastapi import FastAPI
from telemetry import setup_telemetry, addToTrace
load_dotenv()
app = FastAPI(title="My API")
@app.on_event("startup")
async def startup():
os.environ.setdefault("OTEL_ENABLED", "true")
os.environ.setdefault("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317")
os.environ.setdefault("SERVICE_NAME", "my-fastapi")
os.environ.setdefault("APP_ENV", "development")
setup_telemetry(
app,
service_name=os.environ["SERVICE_NAME"]
)
print("[Successfully setup jaegar connection]")
@app.get("/")
@addToTrace
def health():
return {"status": "ok"}
@addToTrace
async def expensive_logic(x: int) -> int:
return x * 2
@app.get("/work")
@addToTrace
async def do_work(x: int = 2):
y = await expensive_logic(x)
return {"x": x, "y": y}
Open Telemetry Config
Copy the following code to a file called telemetry.py
import os
import functools
import inspect
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.propagate import set_global_textmap
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
def setup_telemetry(app, service_name: str):
"""
Initializes OpenTelemetry for FastAPI + requests.
Controlled by env var OTEL_ENABLED=true.
Exports spans via OTLP gRPC to OTEL_EXPORTER_OTLP_ENDPOINT.
"""
if os.getenv("OTEL_ENABLED", "false").lower() != "true":
return
set_global_textmap(TraceContextTextMapPropagator())
resource = Resource.create({
SERVICE_NAME: service_name,
"deployment.environment": os.getenv("APP_ENV", "development"),
})
provider = TracerProvider(resource=resource)
otlp_exporter = OTLPSpanExporter(
endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317")
)
provider.add_span_processor(BatchSpanProcessor(otlp_exporter))
trace.set_tracer_provider(provider)
RequestsInstrumentor().instrument()
FastAPIInstrumentor.instrument_app(
app,
tracer_provider=provider,
client_response_hook=response_hook,
)
def response_hook(span: trace.Span, scope: dict, message: dict):
"""Captures HTTP response status codes."""
if message.get("type") == "http.response.start":
status = message.get("status")
if status is not None:
span.set_attribute("http.response.status_code", status)
if status >= 500:
span.set_status(Status(StatusCode.ERROR))
def addToTrace(func):
"""Decorator for internal business-logic child spans (async + sync)."""
tracer = trace.get_tracer(__name__)
@functools.wraps(func)
async def async_wrapper(*args, **kwargs):
with tracer.start_as_current_span(func.__name__) as span:
try:
result = await func(*args, **kwargs)
span.set_status(Status(StatusCode.OK))
return result
except Exception as e:
span.record_exception(e)
span.set_status(Status(StatusCode.ERROR, str(e)))
raise
@functools.wraps(func)
def sync_wrapper(*args, **kwargs):
with tracer.start_as_current_span(func.__name__) as span:
try:
result = func(*args, **kwargs)
span.set_status(Status(StatusCode.OK))
return result
except Exception as e:
span.record_exception(e)
span.set_status(Status(StatusCode.ERROR, str(e)))
raise
return async_wrapper if inspect.iscoroutinefunction(func) else sync_wrapper
Jaegar Frontend
First, make sure that docker is running. After that, we can run the following command to spin up the Jaegar frontend server.
docker run -d --name jaeger \
-e COLLECTOR_OTLP_ENABLED=true \
-p 16686:16686 \
-p 4317:4317 \
-p 4318:4318 \
jaegertracing/all-in-one:1.56
Putting it all Together
Now that your dependencies are installed, your files are present, and jaegar is running, all that is left to do is start your server and you should see the following logs
$ uvicorn main:app --host 0.0.0.0 --port 8000 --reload
INFO: Will watch for changes in these directories: ['/']
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO: Started reloader process [4021] using WatchFiles
INFO: Started server process [4024]
INFO: Waiting for application startup.
[Successfully setup jaegar connection]
INFO: Application startup complete.Next, make a simple call to each endpoint
curl http://localhost:8000/
curl http://localhost:8000/workNow we can look at the frontend by going to http://localhost:16686
↳ Jaegar frontend view
Additionally, we can click into the /work trace to see the full latency profile
That's it! All you need to do now to add additionally route or functions is simply include the @addToTrace decorator and the rest is done automatically.