Prometheus Monitoring Guide: Architecture, PromQL, Grafana, and Kubernetes

By /

Prometheus Monitoring Guide: Architecture, PromQL, Grafana, and Kubernetes

Prometheus is an open-source monitoring and alerting system that collects numerical metrics from applications, servers, containers, and Kubernetes clusters.

It stores these metrics as time-series data, meaning every value is saved with a timestamp and optional labels. Prometheus was originally created at SoundCloud and is now an independent cloud-native project. (Prometheus)

Why Is Prometheus Used?

Prometheus helps DevOps and SRE teams:

  • Monitor CPU, memory, disk, network, and application performance
  • Detect failures and abnormal behavior
  • Query historical and real-time metrics
  • Create alerts before users experience major problems
  • Visualize infrastructure health through Grafana
  • Monitor dynamic Kubernetes environments

Prometheus commonly runs on port 9090, while Alertmanager commonly uses port 9093.

How Prometheus Works

The basic monitoring flow is:

Applications, Servers, Kubernetes
              ↓
     Exporters or /metrics endpoints
              ↓
       Prometheus scrapes metrics
              ↓
       Time-Series Database
              ↓
     PromQL, Rules, and Grafana
              ↓
          Alertmanager

Prometheus normally uses a pull model. It periodically sends HTTP requests to configured targets and collects metrics from their /metrics endpoints.

For short-lived batch jobs that cannot be scraped reliably, the Prometheus Pushgateway can be used carefully. (Prometheus)

Core Prometheus Components

Prometheus Server

The main server:

  • Discovers monitoring targets
  • Scrapes metrics
  • Stores metrics in its local TSDB
  • Evaluates recording and alerting rules
  • Processes PromQL queries

Prometheus includes local disk storage and can also integrate with compatible remote storage systems. (Prometheus)

Exporters

Exporters convert metrics from systems that do not directly expose Prometheus-compatible metrics.

Common exporters include:

  • Node Exporter: Linux hardware and operating-system metrics
  • Windows Exporter: Windows server metrics
  • Blackbox Exporter: Website, API, TCP, DNS, and endpoint checks
  • MySQL Exporter: MySQL database metrics

Node Exporter exposes Linux hardware and kernel metrics for Prometheus to scrape. (Prometheus)

Service Discovery

Service discovery automatically finds changing scrape targets from platforms such as:

  • Kubernetes
  • AWS
  • Azure
  • Consul
  • Docker
  • File-based target lists

It discovers endpoints; it does not automatically monitor every service without the required scrape configuration and permissions. (Prometheus)

PromQL

PromQL is the Prometheus Query Language. It is used through the Prometheus web interface, HTTP API, Grafana, and other compatible tools.

PromQL helps engineers filter, aggregate, calculate rates, troubleshoot failures, and build alerting rules.

Alertmanager

Prometheus evaluates alerting rules and sends active alerts to Alertmanager.

Alertmanager handles:

  • Deduplication
  • Grouping
  • Routing
  • Silencing
  • Inhibition
  • Notifications to email, chat, and incident-management systems

Grafana also supports alerting, but it should not automatically be described as a direct replacement for every Alertmanager workflow. (Prometheus)

Grafana

Grafana connects to Prometheus as a data source and displays metrics through dashboards, charts, tables, and alerts.

Grafana includes built-in Prometheus support, so a separate Prometheus data-source plugin is not required. (Grafana Labs)

Four Prometheus Metric Types

1. Counter

A value that increases or resets.

Examples:

  • Total HTTP requests
  • Failed login attempts
  • Processed messages
http_requests_total

2. Gauge

A value that can increase or decrease.

Examples:

  • Memory usage
  • CPU temperature
  • Active connections
node_memory_MemAvailable_bytes

3. Histogram

Records observations inside configurable buckets.

Examples:

  • Request duration
  • Response size
  • Database query latency

4. Summary

Records observations and calculates quantiles over a configured time window.

Prometheus officially supports Counter, Gauge, Histogram, and Summary as its four primary metric types. (Prometheus)

Basic Prometheus Configuration

Prometheus scrape targets are configured inside prometheus.yml.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "linux-server"
    static_configs:
      - targets: ["10.0.1.20:9100"]

In Prometheus terminology, a scrape endpoint is an instance. A collection of similar instances is called a job. (Prometheus)

Useful PromQL Queries

Check whether a target is available

up
  • 1 means the target is reachable.
  • 0 means the scrape failed.

View CPU usage rate

rate(node_cpu_seconds_total[5m])

Calculate HTTP request rate

sum(rate(http_requests_total[5m]))

Find unavailable targets

up == 0

Calculate error percentage

100 *
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

The rate() function is mainly used with counters to calculate their per-second rate of increase. (Prometheus)

Prometheus and Grafana with Kubernetes

A common Kubernetes monitoring solution is kube-prometheus-stack.

It can deploy:

  • Prometheus
  • Prometheus Operator
  • Alertmanager
  • Grafana
  • Node Exporter
  • kube-state-metrics
  • Preconfigured dashboards and alerting rules
helm repo add prometheus-community \
https://prometheus-community.github.io/helm-charts

helm repo update

helm install monitoring \
prometheus-community/kube-prometheus-stack \
--namespace monitoring \
--create-namespace

The official chart provides an end-to-end Kubernetes monitoring stack using Prometheus Operator. (GitHub)

Accessing Prometheus Securely

For temporary local access:

kubectl port-forward \
service/monitoring-kube-prometheus-prometheus \
9090:9090 \
-n monitoring

Do not expose Prometheus directly to the public internet through an unrestricted LoadBalancer.

For production access, use:

  • Private LoadBalancer
  • Authenticated Ingress
  • VPN
  • Identity-aware proxy
  • TLS
  • Network policies and firewall rules

Prometheus endpoints may operate without built-in authentication unless security is deliberately configured. (Prometheus)

Common Daily Prometheus Tasks

A DevOps engineer may:

  • Check target health on the /targets page
  • Write and optimize PromQL queries
  • Add new exporters and scrape jobs
  • Manage Kubernetes ServiceMonitor resources
  • Create Grafana dashboards
  • Configure alerting and recording rules
  • Investigate missing metrics
  • Monitor Prometheus storage and memory
  • Review alert noise and false positives
  • Upgrade Helm charts carefully

These responsibilities align with the practical tasks in the original notes.

Common Errors and Solutions

Target Shows as Down

Check:

  • Target IP and port
  • /metrics endpoint
  • Security groups and firewalls
  • Exporter process
  • DNS resolution
  • Prometheus configuration syntax

Helm Chart Installation Fails

Run:

helm repo update
helm list -n monitoring
helm status monitoring -n monitoring

Verify the repository URL, chart values, Kubernetes context, namespace, and permissions.

Metrics Disappear After Restart

Prometheus may be using temporary storage. Configure persistent storage and define an appropriate retention policy.

Prometheus Uses Too Much Memory

Possible causes:

  • Too many targets
  • Very short scrape intervals
  • High-cardinality labels
  • Unnecessary metrics
  • Expensive PromQL queries

Avoid labels containing uncontrolled values such as request IDs, session IDs, timestamps, or complete URLs.

Too Many Alerts

  • Alert on user-impacting symptoms
  • Add an appropriate pending period
  • Group related alerts
  • Remove alerts without a clear action
  • Use dashboards to investigate root causes

Prometheus recommends keeping alerts actionable and focusing on symptoms rather than producing unnecessary pages. (Prometheus)

Prometheus Interview Questions

What is Prometheus?

Prometheus is an open-source monitoring and alerting system that collects and stores metrics as labeled time-series data.

How does Prometheus collect metrics?

It normally pulls metrics by periodically scraping configured HTTP endpoints.

What is PromQL?

PromQL is the query language used to filter, aggregate, analyze, visualize, and alert on Prometheus metrics.

Why are exporters needed?

Exporters expose metrics from systems that do not provide Prometheus-compatible metrics directly.

Prometheus vs. Grafana?

  • Prometheus: Collects, stores, queries, and evaluates metrics.
  • Grafana: Visualizes metrics and can manage dashboards and alerts.

What does Alertmanager do?

It groups, deduplicates, routes, silences, and sends alert notifications.

What is a time series?

A time series is a sequence of metric values recorded over time and identified by a metric name and labels.

What is high cardinality?

High cardinality occurs when labels create too many unique time-series combinations, increasing memory, storage, and query costs.

Conclusion

Prometheus is a core DevOps monitoring tool for servers, applications, containers, and Kubernetes.

For job and interview preparation, focus on:

  • Prometheus architecture
  • Pull-based scraping
  • Exporters and service discovery
  • Metric types
  • PromQL
  • Grafana integration
  • Alertmanager
  • Kubernetes monitoring
  • Cardinality and security

Related Posts

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top