Monitoring & Logging¶

Build observability pipelines that detect issues early, enable fast triage, and provide clear insights for postmortems.

🎯 Goals¶

Know when something breaks (fast, reliable alerts)
See why it broke (logs, metrics, traces)
Measure reliability (SLOs and error budgets)
Improve based on data, not guesswork

📊 Metrics¶

Four Golden Signals (Google SRE)¶

Latency — time to service a request
Traffic — demand on the system
Errors — failed requests
Saturation — resource usage

Common Metric Types¶

Counters — ever-increasing (e.g., http_requests_total)
Gauges — up/down (e.g., cpu_usage_percent)
Histograms — distribution (e.g., request latency buckets)
Summaries — percentiles (e.g., p95 response time)

📈 Prometheus Basics¶

PromQL Examples¶

# Requests per second (RPS) over 5 minutes
sum(rate(http_requests_total[5m]))

# Error rate %
(sum(rate(http_requests_total{status!~"2.."}[5m])) /
 sum(rate(http_requests_total[5m]))) * 100

# 99th percentile latency
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
)

# Node CPU usage
100 - (avg by(instance)(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

📢 Alerting¶

Alert Rules (Prometheus)¶

groups:
  - name: sre.rules
    rules:
      - alert: HighErrorRate
        expr: (sum(rate(http_requests_total{status=~"5.."}[5m])) /
               sum(rate(http_requests_total[5m]))) > 0.05
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "High HTTP 5xx error rate"
          description: "More than 5% of requests are failing over 5m"

Alert Best Practices¶

Tie to user experience (SLOs), not just raw resource metrics
Use multi-window, multi-burn rate alerts for error budgets
Group alerts to avoid noisy pages
Have a runbook link in every alert

📉 Dashboards¶

Grafana Essentials - Group panels by service or golden signals - Use templating variables for service/namespace/env - Add threshold lines for SLO targets - Include annotations for deploys and incidents

Golden Signals Example Layout 1. Latency (p50/p90/p99) 2. RPS (total & per endpoint) 3. Error rate 4. CPU/memory saturation

🗂 Logging¶

Logging Levels¶

DEBUG — noisy, dev-only
INFO — state changes, startup/shutdown
WARN — unusual, not yet broken
ERROR — failed operation, recoverable
FATAL — crash/stop

Best Practices¶

Use structured logging (JSON)
Always include timestamp, service, request ID, trace ID
No secrets in logs
Keep logs consistent across services

📜 Loki (Promtail → Loki → Grafana)¶

Promtail config snippet

server:
  http_listen_port: 9080
positions:
  filename: /tmp/positions.yaml
clients:
  - url: http://loki:3100/loki/api/v1/push
scrape_configs:
  - job_name: varlogs
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*.log

Grafana LogQL Examples

# Error logs in last 5m
{job="api"} |= "error" | logfmt

# Count 5xx errors
count_over_time({job="nginx"} |= " 5" [5m])

# Join logs with metrics using traceID

🔍 ELK / OpenSearch¶

Logstash pipeline example

input {
  beats { port => 5044 }
}
filter {
  grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
  date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] }
}
output {
  elasticsearch { hosts => ["http://es:9200"] index => "nginx-%{+YYYY.MM.dd}" }
}

Kibana Query

status:[500 TO 599] AND path:"/api/v1/orders"

🧵 Tracing¶

OpenTelemetry (OTel) + Jaeger¶

Instrument HTTP/gRPC clients & servers
Pass traceparent header between services
Export to Jaeger/Tempo/Cloud Trace

OTel Collector Example

receivers:
  otlp:
    protocols:
      grpc:
      http:
exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [jaeger]

📏 SLOs & Error Budgets¶

Example SLO¶

Availability: 99.9% monthly
Error budget: 0.1% downtime (~43.2 min/month)

Prometheus burn rate alert (multi-window)

# Fast-burn (1h window, budget 2%)
sum(rate(http_requests_total{status!~"2.."}[1h])) /
sum(rate(http_requests_total[1h])) > (0.001 * 14.4)

🛡️ Observability Checklist¶

[ ] Metrics, logs, and traces for all critical paths
[ ] Golden signals dashboards per service
[ ] Alerts mapped to SLOs
[ ] Log retention policy + cost controls
[ ] Trace sampling tuned for cost & usefulness
[ ] Runbooks linked from alerts

If you can’t see it, you can’t fix it.