System: Observability & Telemetry Stack
Category: DevOps · Prometheus & Telemetry
Reading Time: 6 min read
Why deploying without metrics is flying blind: setting up Prometheus and PromQL
Capturing real runtime signals, p95 latencies, and USE method metrics.
Covered: Prometheus · Node Exporter · /metrics Scrape · PromQL p95 Latency · GitHub Actions
01. WHY I BUILT IT
The Trigger & Underlying Problem
The Trigger: When an application slowed down in testing, I had no way of knowing whether the bottleneck was CPU saturation, database connection pools, or an external API timeout.
The Core Problem: Looking at logs alone is insufficient for understanding performance degradation. Systems need actionable numeric telemetry to detect saturation before outages occur.
Why Standard Solutions Were Insufficient: Hands-on practice configuring Prometheus scrapers and writing PromQL queries.
02. PRACTICAL IMPACT
What It Solves in Practice
PRACTICAL PROBLEM
Scrapes host metrics (CPU, RAM, disk I/O) via Node Exporter and application metrics (request duration, status code counters) via custom `/metrics` endpoints.
BENEFICIARIES
Provides visibility into system health and alert signals when error rates spike.
BEFORE THIS SYSTEM
Running `top` and inspecting logs manually after failures.
03. IMPLEMENTATION
Architecture & Key Engineering Decisions
A containerized Prometheus instance configured to scrape Node Exporter on the host and application endpoints on a 15-second interval.
Key Decisions:
- Instrumented FastAPI with Prometheus client counters (`http_requests_total`) and histograms (`http_request_duration_seconds_bucket`).
- Wrote PromQL queries calculating 95th-percentile latencies using `histogram_quantile(0.95, sum(rate(...)) by (le))`.
- Applied the USE method (Utilization, Saturation, Errors) for monitoring host resources.
04. CHALLENGES & SOLUTIONS
What Failed in Practice & How It Was Addressed
WHAT WENT WRONG
- High cardinality explosion: initially included user IDs as labels on request metrics, causing Prometheus memory usage to balloon rapidly.
- Prometheus containers lost historical metric data whenever the container restarted because storage was not bound to a persistent volume.
HOW I SOLVED IT
- Removed high-cardinality labels, restricting labels strictly to low-cardinality metadata (endpoint route, HTTP method, and status code class).
- Configured a dedicated Docker volume mounted at `/prometheus` with explicit `--storage.tsdb.retention.time=15d` retention limits.
05. REFLECTION & LESSONS
Takeaways & Future Iterations
What I Learned:
- Metrics show you that something is wrong; logs tell you why it is wrong; traces show you where it is wrong.
- PromQL query semantics: understanding the difference between `rate()`, `irate()`, and `increase()`.
WHAT I WOULD CHANGE NOW:
I would add OpenTelemetry tracing to capture distributed request traces across service boundaries alongside Prometheus metrics.