Prometheus Concepts 🚀

Prometheus is the de-facto open-source monitoring standard for cloud-native environments. If you work with AWS ECS, Kubernetes, or any distributed system, you will encounter it. In this blog, we tear down every component of the Prometheus architecture — visually, with real-world ECS context — so you walk away truly understanding how the pieces fit together.

In this blog, you will learn the following:

• What is Prometheus Architecture?
• Prometheus Server
• Time-Series Database (TSDB)
• Prometheus Targets
• Prometheus Exporters
• Prometheus Service Discovery
• Prometheus Pushgateway
• Prometheus Client Libraries
• Prometheus Alert Manager
• PromQL

Prometheus Architecture — ECS Flow

The diagram below shows how all Prometheus components interact in a typical AWS ECS environment. Watch the data flows between components.

1. What is Prometheus?

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud, now a graduated CNCF project. It is written in Go and follows a pull-based metrics collection model — instead of agents pushing data to a central server, Prometheus actively scrapes (HTTP GET) the /metrics endpoint of targets at a regular interval.

Key idea: Prometheus stores everything as time-series data — a stream of timestamped values identified by a metric name and key-value labels. e.g. http_requests_total{method="GET", status="200"}

2. Prometheus Server

The Prometheus server is the heart of the system. It has three internal sub-components that work together:

3. Time-Series Database (TSDB)

Prometheus's built-in TSDB is purpose-built for time-series workloads. Unlike relational databases, it stores data in chunks of 2-hour blocks that are compacted over time to save disk space.

# /prometheus/data/
.
├── 01H8G3...  # 2-hour block (immutable)
│   ├── chunks/
│   │   └── 000001
│   ├── index
│   └── meta.json
├── 01H8G4...  # another block
├── wal/        # Write-Ahead Log (current)
│   ├── 00000001
│   └── checkpoint.000005/
└── lock

4. Prometheus Targets

A target is any endpoint that exposes metrics in the Prometheus text format at a /metrics HTTP path. In an ECS environment, targets can be:

• ECS Tasks — your microservices instrumented with a client library
• Node Exporter — running as a sidecar or daemon on the EC2 host
• cAdvisor — container-level CPU, memory, and network metrics
• Custom exporters — e.g., ECS task metadata endpoint wrapped as a Prometheus target
• AWS services — via yet-another-cloudwatch-exporter (YACE)

5. Prometheus Exporters

Exporters are adapter processes that translate metrics from systems that don't natively speak the Prometheus format into something Prometheus can scrape. They run alongside the target and expose a /metrics endpoint.

6. Prometheus Service Discovery

Hard-coding scrape targets is not scalable. Service Discovery (SD) lets Prometheus automatically find and track targets as they come and go — critical in dynamic ECS environments where tasks start and stop frequently.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:

  # File-based SD — ECS task IPs written by deployment script
  - job_name: 'ecs-tasks'
    file_sd_configs:
      - files: ['/etc/prometheus/targets/ecs-*.json']
        refresh_interval: 30s

  # DNS SD for ECS Service Connect / Cloud Map
  - job_name: 'ecs-service-connect'
    dns_sd_configs:
      - names: ['_metrics._tcp.my-svc.local']
        type: SRV

  # Static targets — node_exporter on EC2 hosts
  - job_name: 'node'
    static_configs:
      - targets: ['10.0.1.10:9100', '10.0.1.11:9100']

7. Prometheus Pushgateway

The Pushgateway solves a specific problem: short-lived jobs (ECS batch tasks, cron jobs, one-off tasks) that finish before Prometheus gets a chance to scrape them. These jobs push their metrics to the Pushgateway, which Prometheus then scrapes like any other target.

When to use: ECS Fargate batch jobs, nightly ETL pipelines, data export tasks. Do NOT use as a general-purpose proxy — it breaks stale data detection and the pull model.

#!/bin/bash
PUSHGATEWAY="http://pushgateway:9091"
JOB="nightly_etl"

cat <<EOF | curl --data-binary @- "${PUSHGATEWAY}/metrics/job/${JOB}"
# HELP etl_duration_seconds Time taken for the ETL job
# TYPE etl_duration_seconds gauge
etl_duration_seconds 142
# HELP etl_records_processed Total records processed
# TYPE etl_records_processed counter
etl_records_processed 48503
EOF

8. Prometheus Client Libraries

Client libraries let you instrument your application code directly — exposing custom business metrics alongside default runtime metrics.

from prometheus_client import Counter, Histogram, start_http_server
import time

REQUEST_COUNT = Counter(
    'http_requests_total', 'Total HTTP requests',
    ['method', 'endpoint', 'status']
)
REQUEST_LATENCY = Histogram(
    'http_request_duration_seconds', 'Request latency',
    ['endpoint'],
    buckets=[0.01, 0.05, 0.1, 0.5, 1.0]
)

def handle_request(method, endpoint):
    start = time.time()
    # ... process request ...
    REQUEST_COUNT.labels(method=method, endpoint=endpoint, status="200").inc()
    REQUEST_LATENCY.labels(endpoint=endpoint).observe(time.time() - start)

start_http_server(8000)  # exposes /metrics on :8000

9. Prometheus AlertManager

AlertManager handles alerts sent by the Prometheus server. It is responsible for deduplication, grouping, routing, silencing, and inhibition before dispatching notifications.

groups:
  - name: ecs-alerts
    rules:

      # Alert when ECS container CPU > 90% for 5 minutes
      - alert: ECSHighCPU
        expr: rate(container_cpu_usage_seconds_total[5m]) * 100 > 90
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "ECS task {{ $labels.container_name }} high CPU"
          description: "CPU at {{ $value }}%"

      # Alert when error rate > 5%
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          / sum(rate(http_requests_total[5m])) * 100 > 5
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "HTTP error rate above 5%: {{ $value | printf \"%.1f\" }}%"

10. PromQL — Prometheus Query Language

PromQL is a functional query language for selecting and aggregating time-series data. Every Grafana panel and alert expression uses PromQL under the hood.

The Four Metric Types

Essential PromQL Cheat Sheet

PromQL Operators Quick Reference

• rate(metric[5m]) — per-second rate of a counter over 5 minutes
• irate(metric[5m]) — instantaneous rate (last 2 samples) — more responsive but spiky
• increase(metric[1h]) — total increase of a counter over 1 hour
• sum by (label) — aggregate, grouped by a label
• avg without (instance) — average, dropping the instance label
• topk(5, metric) — top 5 time-series by value
• predict_linear(metric[1h], 3600) — predict value in 1 hour via linear regression

Architecture Summary — How it all flows

Prometheus + Grafana + AlertManager is the golden observability stack for ECS environments. Master these three tools and you have complete visibility into every layer of your infrastructure — from EC2 host metrics all the way up to business-level request rates and SLOs.