Inside prometheus/prometheus: Architecture & Production Teardown — How Does It Work in Production?

TL;DR: Prometheus is a Cloud Native Computing Foundation (CNCF) systems and service monitoring system built in Go, distinguished by its multi-dimensional time-series data model, autonomous single-server nodes, and an HTTP pull-based collection mechanism. For engineering teams, it provides a highly concurrent, low-blast-radius observability primitive that scales without the operational overhead of distributed storage dependencies.

What Is Inside prometheus/prometheus: Architecture & Production Teardown & Why Is It Blowing Up?

In the landscape of modern infrastructure observability, few projects have achieved the ubiquity of Prometheus. Currently tracking over 67,600 stars on its primary repository, Prometheus has become the de facto standard for metrics collection in cloud-native environments. Originally developed at SoundCloud and now a graduated project under the Cloud Native Computing Foundation (CNCF), Prometheus fundamentally shifted the industry away from hierarchical, push-based metrics (like StatsD) toward a multi-dimensional, pull-based time-series architecture.

When we evaluate the Prometheus GitHub Readme, the core value proposition is explicitly defined: it is a systems and service monitoring system that collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and triggers alerts. However, the reason it is "blowing up" and maintaining its dominance lies in its strict architectural boundaries. Prometheus explicitly avoids dependency on distributed storage. Single server nodes are entirely autonomous. This design choice means that when your cluster is experiencing a catastrophic network partition or a cascading failure, your monitoring system—the very tool you need to diagnose the outage—remains operational because it does not rely on a complex, distributed consensus backend to write its data.

Furthermore, Prometheus introduced PromQL, a powerful and flexible query language designed specifically to leverage its multi-dimensional data model. Instead of embedding metadata into long, dot-separated metric names, Prometheus defines time series by a metric name and a set of key/value dimensions (labels). This allows engineers to slice, dice, and aggregate telemetry data with mathematical precision, calculating error rates, Apdex scores, and P99 latencies dynamically at query time.

Real-World Field Use Cases: Where This Moves the Needle in the Field

To understand how Prometheus operates in production, we must examine its application across different engineering topologies. Here are three concrete field use cases demonstrating its implementation and tangible impact.

1. Developer Platform Integration: Embedding into CI/CD and Microservice Pipelines

The Everyday Problem: As organizations scale their internal developer platforms, they deploy hundreds of microservices, background workers, and increasingly, AI agent runtimes. Monitoring these disparate workloads using legacy push-based systems often leads to dropped UDP packets, overwhelmed metrics aggregators, and a lack of standardized telemetry. How It Works in Practice: Platform engineering teams embed Prometheus client libraries directly into their application templates. For instance, if a team is deploying autonomous AI agents using frameworks like the Agent Development Kit (ADK), they can expose a /metrics HTTP endpoint on the agent's API server. Prometheus is then configured via Service Discovery (e.g., Kubernetes SD) to automatically detect these new agent pods as they spin up. The Prometheus server periodically scrapes these endpoints, pulling metrics like workflow execution times, LLM token usage, and graph routing errors. The Tangible Impact: This creates a zero-touch observability pipeline. Developers do not need to configure metrics routing or worry about overwhelming a central metrics server; the pull model inherently provides backpressure. If an ADK agent is under heavy load and cannot respond to the scrape request, Prometheus records a scrape failure, which itself is a valuable metric, rather than the agent crashing while trying to push telemetry over a saturated network.

2. Concurrency & Memory Footprint: Evaluating P99 Latency Under Load

The Everyday Problem: High-throughput systems generate massive amounts of telemetry. Monitoring systems must ingest millions of active time series without consuming so much CPU and RAM that they starve the actual production workloads they are monitoring. How It Works in Practice: Prometheus is engineered in Go, leveraging goroutines to handle highly concurrent scraping operations. In production, a single Prometheus instance can scrape thousands of targets and ingest hundreds of thousands of samples per second. The internal Time Series Database (TSDB) uses a memory-mapped architecture, appending incoming samples to an in-memory head block and periodically compacting them into immutable, highly compressed blocks on disk. Engineers tune the --storage.tsdb.retention.time and memory limits to ensure the footprint remains predictable. The Tangible Impact: The operational efficiency is staggering. A single, well-provisioned Prometheus node can handle the metrics ingestion for an entire mid-sized Kubernetes cluster. By keeping the memory footprint optimized and relying on Go's efficient concurrency model, teams achieve sub-millisecond query latencies for real-time dashboards, ensuring that P99 latency spikes in application code are detected and alerted upon instantly.

3. Build-vs-Buy Adoption Verdict: Operational Trade-offs vs. Managed Cloud

The Everyday Problem: CTOs and infrastructure leads constantly weigh the cost of self-hosting open-source observability stacks against purchasing managed SaaS solutions (like Datadog or New Relic), which often come with exorbitant, volume-based pricing models that penalize high-cardinality metrics. How It Works in Practice: Teams deploy Prometheus as the foundational metrics layer. Because Prometheus supports hierarchical and horizontal federation, organizations can deploy edge Prometheus servers in individual clusters that scrape high-resolution data, and then federate aggregated, lower-resolution data to a global Prometheus instance. For long-term, durable storage, they utilize the Prometheus Remote Write API—publishing data to open-source distributed backends like Thanos or Cortex, or to managed cloud endpoints. The Tangible Impact: This hybrid approach drastically reduces observability bills. Teams retain the high-cardinality, granular data locally for short-term debugging (e.g., 15 days) where it is cheapest to store, while only paying to transmit and store aggregated, business-critical metrics in a centralized or managed system. The build-vs-buy verdict heavily favors Prometheus as the collection agent, even if the long-term storage is outsourced.

Under the Hood: Architecture & Design Choices

Architecturally, Prometheus is a masterclass in bounded contexts and operational resilience. By inspecting the repository and its documentation, we can dissect the specific design choices that enable its performance.

The most defining characteristic of Prometheus is its HTTP pull model. Unlike push-based systems where applications blindly fire metrics at a centralized daemon, Prometheus actively reaches out to targets. This inversion of control is critical for several reasons. First, it allows Prometheus to know immediately if a target is down (the scrape fails). Second, it prevents the monitoring system from being DDoS'd by its own infrastructure; Prometheus controls the scrape interval and concurrency. If pushing time series is absolutely necessary—such as for ephemeral batch jobs that do not live long enough to be scraped—Prometheus supports this via an intermediary Pushgateway.

Another major design choice is the pluggable Service Discovery (SD) mechanism. In dynamic cloud environments, IP addresses change constantly. Prometheus does not rely on static configuration alone; it integrates natively with Kubernetes, Consul, EC2, Azure, and others to dynamically discover targets. The Go codebase allows operators to compile custom binaries that include or exclude specific service discovery plugins to reduce the binary size and attack surface.

Data storage is handled by a custom-built, local Time Series Database (TSDB). The TSDB is optimized for the specific workload of metrics: append-only writes, high concurrency, and massive compression of timestamp/float64 pairs. Because single server nodes are autonomous, there is no clustering logic within Prometheus itself. This intentional omission of distributed consensus (like Raft or Paxos) eliminates a massive class of distributed systems failures.

Below is a Mermaid flowchart illustrating the internal execution pipeline and architectural topology of a Prometheus deployment:

flowchart LR
    subgraph CloudEnvironment["Cloud Environment"]
        A[Microservice Pods]
        B[Batch Jobs]
        C[Pushgateway]
    end

    subgraph PrometheusServer["Prometheus Server"]
        D[Service Discovery]
        E[Scraper / Retrieval]
        F[(Local TSDB)]
        G[PromQL Engine]
        H[HTTP Server]
    end

    subgraph Ecosystem
        I[Alertmanager]
        J[Grafana Dashboards]
        K[Remote Storage]
    end

    B -- Pushes Metrics --> C
    D -. Discovers Targets .-> A
    D -. Discovers Targets .-> C
    
    E -- HTTP GET /metrics --> A
    E -- HTTP GET /metrics --> C
    
    E -- Appends Data --> F
    F -- Queried by --> G
    G -- Serves Results --> H
    
    H -- Pushes Alerts --> I
    H -- JSON API --> J
    F -- Remote Write Protocol --> K

This diagram highlights the separation of concerns. The Retrieval component relies on Service Discovery to find targets, scrapes them via HTTP, and writes to the local TSDB. The PromQL engine reads from the TSDB to serve the HTTP API, which in turn feeds Grafana and evaluates alerting rules to send to the Alertmanager.

Hands-On Quickstart & Code Walkthrough

Deploying and compiling Prometheus is straightforward, thanks to its Go architecture and comprehensive tooling. The project provides multiple avenues for installation, from precompiled binaries to Docker images and source builds.

Running via Docker

For immediate evaluation, the official Docker image is the fastest route. As documented in the repository, you can launch a Prometheus container bound to your local port 9090:

docker run --name prometheus -d -p 127.0.0.1:9090:9090 prom/prometheus

Once running, the Prometheus web UI and API are reachable at http://127.0.0.1:9090/.

Building from Source and Managing Build Tags

For platform engineers who need to customize the binary, building from source requires Go and NodeJS (for the React UI). Cloning the repository and using the make toolchain is the recommended path:

git clone https://github.com/prometheus/prometheus.git
cd prometheus
make build
./prometheus --config.file=your_config.yml

A fascinating architectural detail in the Prometheus build system is the management of Service Discovery plugins via Go build tags. Prometheus is bundled with many SD plugins, which can bloat the binary if you only operate in a specific environment (e.g., Kubernetes). You can customize the build by modifying the .promu.yml file or passing tags directly to the Go compiler.

To compile a lean binary that excludes all optional service discoveries except Kubernetes, you would execute:

go build -tags "remove_all_sd,enable_kubernetes_sd" ./cmd/prometheus

This command utilizes the remove_all_sd tag to strip out all optional SDs (keeping only file_sd, static_sd, and http_sd), and then selectively re-enables Kubernetes via enable_kubernetes_sd.

Using Prometheus as a Go Library (And Its Caveats)

It is crucial to note the project's stance on library usage. The prometheus/prometheus repository builds a stand-alone program and is not designed for use as a library. The maintainers explicitly state that no care has been taken to make it work well as an imported module, and you may encounter errors that only surface in library contexts. Reusable components are instead housed in prometheus/common and prometheus/client_golang.

However, for teams building custom metrics forwarders, Prometheus publishes its Remote Write protobuf independently. You can import this experimental library via buf.build:

go get buf.build/gen/go/prometheus/prometheus/protocolbuffers/go@latest

Additionally, to comply with Go module versioning rules without breaking semantic expectations, the Prometheus Releases documentation notes a specific versioning mapping. For Prometheus v3.y.z releases, they publish equivalent v0.3y.z tags. Therefore, to import Prometheus v3.0.0 as a library, you must use:

go get github.com/prometheus/prometheus@v0.300.0

My Honest Verdict: Where It Fits in Your Stack (Pros & Trade-offs)

In our architectural evaluation of the observability landscape, Prometheus stands out as an uncompromising, highly optimized tool that does exactly what it claims to do, with remarkable efficiency. However, understanding where it fits in your stack requires acknowledging both its immense strengths and its intentional limitations.

The Pros

1. Operational Simplicity and Resilience: The decision to make single server nodes autonomous is Prometheus's greatest strength. By eschewing distributed storage, Prometheus requires virtually no operational hand-holding. There are no cluster states to manage, no split-brain scenarios to resolve, and no complex consensus algorithms to debug during an outage.
2. PromQL: The query language is unparalleled in the open-source ecosystem. Its ability to perform vector matching, calculate derivatives over time, and aggregate across dynamic labels makes it an incredibly powerful analytical tool for site reliability engineers.
3. The Pull Model: HTTP-based scraping provides implicit backpressure, simplifies target configuration (targets just expose a /metrics endpoint), and makes debugging trivial (you can simply curl the endpoint yourself to verify what the application is exposing).
4. Ecosystem Ubiquity: Almost every modern infrastructure tool, database, and message queue natively exposes Prometheus metrics or has an open-source exporter available.

The Trade-offs

1. Not a Long-Term Distributed Datastore: Prometheus is not designed to store years of historical data across a highly available cluster. If you lose the disk attached to a Prometheus node, you lose that node's historical data. For global views, high availability, and long-term retention, you must pair Prometheus with a remote storage backend like Thanos, Cortex, or Mimir. Prometheus is the collector and short-term buffer; it is not your data lake.
2. High Cardinality Hazards: While the multi-dimensional data model is powerful, it is a double-edged sword. If developers attach unbounded labels to metrics (e.g., user IDs, session tokens, or raw HTTP paths), the cardinality of the time series explodes. This will rapidly consume the memory of the Prometheus server, leading to Out-Of-Memory (OOM) crashes. Strict governance over label cardinality is required in production.
3. Not Designed as a Library: As explicitly noted in the repository, attempting to import the core Prometheus engine into your own Go applications is fraught with peril. It is built as a standalone binary, and teams looking for embeddable TSDBs or metrics engines must look elsewhere or rely strictly on the remote write protobufs.

Final Thoughts

Prometheus is an indispensable primitive in the modern infrastructure stack. It should be deployed as the primary metrics collection and alerting engine in every cluster you operate. By respecting its architectural boundaries—using it for highly concurrent, short-to-medium term metrics collection and relying on external systems for long-term durable storage—engineering teams can achieve world-class observability with a remarkably low operational burden. It is not a silver bullet for all telemetry (it does not handle logs or distributed traces), but for time-series metrics, it remains the undisputed king of the cloud-native ecosystem.