Metrics and tracing
Prometheus metrics on the admin listener, and OpenTelemetry traces that continue the caller's W3C trace context.
Prometheus metrics
Served at GET /metrics on the admin listener (:9090).
| Metric | Labels | Meaning |
|---|---|---|
jwksbridge_snapshot_age_seconds | keyset | Seconds since the source last confirmed the keyset. |
jwksbridge_refresh_total | keyset, result | Refreshes: updated, unchanged, error, invalid. |
jwksbridge_keys | keyset, status | Keys in the served snapshot per status. |
jwksbridge_key_info | keyset, kid, status, signing | One series per key, value 1. signing="true" marks the key Kong signs with, which differs from the active key while the staging guard holds. |
jwksbridge_requests_total | endpoint, code | Requests to jwks and keyset. |
jwksbridge_request_duration_seconds | endpoint | Request latency histogram. |
jwksbridge_source_errors_total | source | Errors returned by Vault or the file source. |
jwksbridge_build_info | version, commit | Always 1. |
Go runtime and process metrics are included.
Suggested alerts
- alert: JwksBridgeKeysetStale
expr: max by (keyset) (jwksbridge_snapshot_age_seconds) > 1800
- alert: JwksBridgeRefreshFailing
expr: sum by (keyset) (rate(jwksbridge_refresh_total{result=~"error|invalid"}[5m])) > 0
for: 5m
- alert: JwksBridgeNoActiveKey
expr: max by (keyset) (jwksbridge_keys{status="active"}) == 0
- alert: JwksBridgeKeysetErrors
expr: sum(rate(jwksbridge_requests_total{endpoint="keyset",code!~"2..|304"}[5m])) > 0W3C trace context
Every request on the public and private listeners extracts the traceparent and baggage headers, so a span started by Kong's opentelemetry plugin continues in the bridge. Calls to Vault carry the trace context on, and each background refresh is a keyset.refresh span with the Vault request as a child.
Spans are exported only when an exporter is configured with the standard OpenTelemetry variables:
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.observability:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf # or grpc
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1
OTEL_SERVICE_NAME=jwks-bridgeWithout them, propagation still works and nothing is exported. With JWKSB_LOG_LEVEL=debug, access log lines include trace_id and span_id.
Kong's jwt-signer plugin fetches the keyset from a background timer and sends no traceparent, so GET /keyset spans start a new trace. Requests that Kong proxies to the bridge, such as the public JWKS route, join the client's trace.
Grafana dashboard
The repository ships a dashboard in deploy/grafana/jwks-bridge.json. It picks the Prometheus, Loki and Jaeger datasources through variables, so it imports into any Grafana. Panels:
- Signing kid, keys per status, snapshot age and source errors.
- Key lifecycle: one bar per kid, colored staged, active (held back by the staging guard), signing or retired.
- Kong data planes: requests per pod, failed requests and latency, from the Kong
prometheusplugin. - Bridge requests, latency and refresh results.
- Links to the Jaeger UI searches (set the
jaeger_uivariable to its address), and the bridge logs from Loki with akeyset loadedannotation on every rotation.
Import it through the Grafana API, or copy it into a dashboard provider directory:
jq '{dashboard: (. + {id: null}), overwrite: true}' deploy/grafana/jwks-bridge.json \
| curl -u admin:"$GRAFANA_PASSWORD" -H 'Content-Type: application/json' \
-X POST "$GRAFANA_URL/api/dashboards/db" -d @-The cluster variable reads the cluster label from jwksbridge_build_info. Without that label it stays empty and every panel still matches.