Skip to content

Add OpenTelemetry distributed tracing support#1234

Open
lovasoa wants to merge 39 commits intomainfrom
otel
Open

Add OpenTelemetry distributed tracing support#1234
lovasoa wants to merge 39 commits intomainfrom
otel

Conversation

@lovasoa
Copy link
Collaborator

@lovasoa lovasoa commented Mar 8, 2026 •
edited
Loading

Summary

  • Add OpenTelemetry tracing support, activated when OTEL_EXPORTER_OTLP_ENDPOINT is set
  • Replace actix Logger middleware with TracingLogger for W3C traceparent extraction and HTTP request spans
  • Add tracing spans at key instrumentation points: sqlpage.exec (SQL file execution), db.pool.acquire (connection pool), db.query (individual SQL statements)
  • Bridge existing log:: calls to tracing via tracing-log. Logs are now in logfmt format instead of the old nginx-style. They have structured data from the requests attached.
image

Spans produced per request

Span Key attributes
HTTP request (from tracing-actix-web) http.method, http.route, http.status_code
sqlpage.exec sqlpage.file (path to .sql file)
db.pool.acquire db.pool.size
db.query db.statement, db.system

Configuration

Standard OTel environment variables — no custom SQLPage config needed:

OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318
OTEL_SERVICE_NAME=sqlpage

Example

Includes examples/opentelemetry/ with a complete docker-compose stack:
nginx (OTel module) → SQLPage → PostgreSQL → OTel Collector → Tempo → Grafana

Test plan

  • cargo check compiles
  • cargo test — all 57 existing tests pass
  • Run without OTEL_EXPORTER_OTLP_ENDPOINT — verify env_logger behavior unchanged

lovasoa and others added 30 commits March 8, 2026 18:51
@lovasoa @claude
add OpenTelemetry distributed tracing support
482e11e 
When OTEL_EXPORTER_OTLP_ENDPOINT is set, enables full tracing pipeline
with OTLP export, W3C traceparent propagation, and spans for HTTP
requests, SQL file execution, DB pool acquire, and query execution.
Falls back to env_logger when unset.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovasoa @claude
fix OTel example: tracing init, Dockerfile, Tempo config
f3a9ee2 
- Fix tracing-log bridge initialization order (set subscriber first,
  then LogTracer) to avoid double-set panic
- Add dedicated Dockerfile for example using release profile (avoids
  OOM with superoptimized LTO in Docker)
- Use debian:trixie-slim runtime for glibc compatibility
- Fix nginx image to nginx:otel (official image with OTel module)
- Fix nginx.conf: move otel_trace directives into location block
- Pin Tempo to 2.6.1 (latest has partition ring issues in single-node)
- Fix otel-collector exporter alias (otlp → otlp_grpc)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovasoa @claude
propagate trace context to PostgreSQL via application_name
c742abe 
After acquiring a DB connection, set the W3C traceparent as the
PostgreSQL application_name (or MySQL session variable). This makes
trace IDs visible in pg_stat_activity and PostgreSQL logs, enabling
direct correlation between Grafana Tempo traces and database-side
monitoring.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovasoa @claude
rewrite OTel example README with setup guides for all major providers
2bb370a 
Comprehensive documentation covering:
- Step-by-step quick start for the Docker Compose example
- How OpenTelemetry works (spans, collectors, backends)
- Setup guides for Grafana Tempo, Jaeger, Grafana Cloud, Datadog,
  Honeycomb, New Relic, and Axiom with exact env vars and doc links
- PostgreSQL trace correlation via application_name
- Environment variable reference
- Troubleshooting section

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovasoa @claude
fix todo example: use :title (POST variable) instead of $title
9a879e5 
The form submits via POST, so the title field must be referenced with
the : prefix (POST parameter) rather than $ (GET parameter).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovasoa @claude
set code.filepath and code.lineno span attributes to user SQL files
ec9f3f9 
OTel span attributes now reference the user's .sql file path and line
number instead of the SQLPage Rust source code. Also improves span
naming, adds JSON log formatting, custom root span builder, and
Grafana dashboard provisioning for the OTel example.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovasoa
npm run fix
0aebd0c
@lovasoa @claude
use stable OTel semantic convention attribute names
712fb14 
- code.filepath → code.file.path, code.lineno → code.line.number
- db.statement → db.query.text, db.system → db.system.name
- Disable auto code location (.with_location(false)) so spans
  reference user SQL files, not SQLPage Rust source
- Remove redundant sqlpage.file attribute (code.file.path suffices)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovasoa @claude
use OTel semantic convention values for db.system.name
85dd9b8 
Use well-known values from the OpenTelemetry registry instead of raw
DBMS name strings. Cast line numbers to i64 for correct span recording.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovasoa @claude
remove json-subscriber dependency
dafbac2 
The custom logfmt layer in telemetry.rs replaces it with zero extra
dependencies and precise control over field selection and ordering.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovasoa @claude
add Loki + Promtail log aggregation to OTel example
2a8a89a 
Adds two new services (Loki, Promtail) to scrape SQLPage container logs
and display them in Grafana alongside traces. The home dashboard now
shows a logs panel with trace_id derived fields linking to Tempo.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovasoa @claude
add OTel spans for request parsing, rendering, sqlpage functions, and…
d492ccc 
… OIDC

Add targeted spans to account for previously untraced time:
- http.parse_request: request/form parsing before SQL execution
- render: template rendering and response streaming
- subprocess: sqlpage.exec() with process.command attribute
- http.client: sqlpage.fetch()/fetch_with_meta() with OTel HTTP client
  semantic conventions (http.request.method, url.full, http.response.status_code)
- sqlpage.file: sqlpage.run_sql() nested file execution
- oidc.callback + http.client: OIDC token exchange

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovasoa @claude
add oidc.jwt.verify span for OIDC token verification
cb7bdf6 
This span covers JWT signature verification and claims validation,
which runs on every authenticated request via get_token_claims().

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovasoa @claude
add enduser.id attribute to oidc.jwt.verify span
0eeb5c3 
Records the OIDC subject claim (sub) as enduser.id after successful
JWT verification, following OTel semantic conventions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovasoa @claude
add OTel user.* attributes to oidc.jwt.verify span
fab9350 
Record user.id (sub), user.name (preferred_username), user.full_name
(name), and user.email from OIDC claims after JWT verification.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovasoa @claude
add sqlpage.file.load span and attributes to http.parse_request
ff8800e 
The gap before http.parse_request was the SQL file cache lookup -
now covered by the sqlpage.file.load span with code.file.path.

http.parse_request now records http.request.method and content_type,
which helps identify slow multipart/form-data parsing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovasoa
Fix clippy pedantic warnings
f8fdc24
@lovasoa @claude
unify log format: logfmt with colors, no OTel noise
3b0c37c 
Use the custom logfmt layer for both OTel and non-OTel modes instead
of falling back to env_logger. This eliminates the tracing→log bridge
dumping all span fields (user agents, otel.kind, request_id, etc.)
and only shows: ts, level, target, msg, method, path, status,
file, client_ip, and trace_id (when valid).

Adds terminal color support (bold red for errors, green for info,
dim for timestamps/targets). Emits one log line per completed
successful request. Errors are logged once by the error handler.
Suppresses trace_id=000...0 when no real trace context exists.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovasoa @claude
use .instrument() instead of .entered() for async spans
1309725 
Span guards from .entered() do not propagate correctly across await
points. Switch to tracing::Instrument to ensure spans are properly
associated with their async tasks throughout their lifetime.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovasoa @claude
preserve multi-line error formatting in terminal log output
f884eab 
When stderr is a terminal and the log message contains newlines
(e.g. SQL syntax errors with source highlighting and arrows),
print the metadata on the first line and the message below with
its original formatting. Machine output (non-terminal) remains
single-line logfmt.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovasoa @claude
use root Dockerfile for OTel example, add CARGO_PROFILE build arg
5bfaa08 
Remove the example's custom Dockerfile and use the main one with a
CARGO_PROFILE=release build arg to avoid OOM from fat LTO in
memory-constrained Docker environments. The build scripts now
read CARGO_PROFILE from the environment, defaulting to
superoptimized for backward compatibility.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovasoa @claude
add db.query.parameter and db.response.returned_rows span attributes
3aaf82e 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovasoa
Add official blog post about tracing
4c88b1e
@lovasoa @claude
add http.request.body.size and url.query span attributes
bf7a863 
Add http.request.body.size to HTTP client spans (fetch, fetch_with_meta)
and to the server-side http.parse_request span (from Content-Length header).
Add url.query to the http.parse_request span.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@lovasoa
Replace Promtail with the OpenTelemetry Collector
10dec87
@lovasoa
Refactor telemetry logging helpers
7e057e2
@lovasoa
Clamp traced fetch body size
7e156c5
@lovasoa
Clamp traced fetch_with_meta body size
36aa465
@lovasoa
Silence noisy PostgreSQL collector logs
e02ed67
@lovasoa
make startup logs parseable
d285c57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@lovasoa