How to Create Prometheus Alerting Rules with Alertmanager

Prometheus and Alertmanager have different responsibilities.

Prometheus evaluates alerting rules. When an expression stays active long enough, Prometheus marks an alert Pending or Firing and sends firing alerts to Alertmanager.

Alertmanager routes notifications. It groups, deduplicates, silences, inhibits, and forwards alerts to configured receivers.

Keeping that boundary clear makes troubleshooting much easier. If the Prometheus alert never reaches Firing, changing a Slack or webhook receiver will not fix it. If the alert is Firing in Prometheus but no notification is delivered, investigate the Prometheus-to-Alertmanager path and Alertmanager routing.

This tutorial creates one Linux host alert using Node Exporter metrics, validates the rule with promtool, validates Alertmanager configuration with amtool, connects Prometheus to Alertmanager, and uses a neutral webhook receiver concept for safe verification.

Last verified: September 11, 2026. The Alertmanager GitHub Releases page currently marks 0.32.1 as Latest, while the repository changelog already contains a 0.32.2 entry. Because those sources are not perfectly aligned, this guide does not hard-code an Alertmanager download version. Install the official current release offered for your maintained environment and record the actual version with alertmanager --version.

What You'll Accomplish

You will:

1. create a meaningful host alert rule;
2. validate the rule file with promtool check rules;
3. load the rule from prometheus.yml;
4. configure Prometheus to send alerts to Alertmanager;
5. create and validate an Alertmanager route/receiver;
6. prepare Alertmanager configuration for UTF-8 strict parsing;
7. observe Pending and Firing states; and
8. verify delivery with a safe test receiver.

Prerequisites

You need:

• a running Prometheus server;
• a Node Exporter target already UP;
• the promtool binary matching your Prometheus installation;
• Alertmanager and its amtool utility;
• permission to edit/reload Prometheus and Alertmanager configuration;
• a safe webhook or test notification destination.

Before creating an alert, confirm this returns data:

up{job="node"}

If the metric does not exist, fix scraping before building alert rules.

Quick Answer

Create a rule file such as linux-alerts.yml containing a Prometheus alert expression, for example up{job="node"} == 0 with a for: duration. Validate it using promtool check rules linux-alerts.yml. Add the file under rule_files in prometheus.yml, configure alerting.alertmanagers to point at Alertmanager on port 9093, and run promtool check config prometheus.yml. In Alertmanager, define a route and a safe receiver, then run both amtool check-config alertmanager.yml and amtool check-config alertmanager.yml --enable-feature="utf8-strict-mode". Reload the services, confirm the rule appears in Prometheus, trigger it in a test environment, and verify the alert progresses from Pending to Firing and appears in Alertmanager.

Step 1: Choose a First Alert That Tests the Plumbing

For the first rule, choose a simple signal whose behavior is easy to understand.

A scrape-health rule is useful:

up{job="node"} == 0

It means Prometheus knows the Node Exporter target but cannot scrape it successfully.

This is a better first integration test than a complex CPU saturation expression because:

• it is easy to reason about;
• it has a clear target label;
• it can be triggered safely on a test host;
• it tests the rule-to-Alertmanager path.

Do not confuse this with a complete production alert strategy. Prometheus best practices recommend alerting on user-visible symptoms where possible, keeping alerts actionable, and allowing slack for small blips.

Step 2: Create the Rule File

Create linux-alerts.yml:

groups:
  - name: linux-host
    rules:
      - alert: NodeExporterTargetDown
        expr: up{job="node"} == 0
        for: 2m
        labels:
          severity: warning
          team: platform
        annotations:
          summary: "Node Exporter target is down"
          description: "Prometheus cannot scrape {{ $labels.instance }} for more than 2 minutes."

Understand for:

Prometheus alerting rules support an optional for duration.

When the expression first becomes true, the alert is Pending. If it remains true continuously for the for duration, it becomes Firing.

This reduces notifications for brief scrape failures.

Two minutes is an example for the tutorial, not a universal recommendation. Tune it to:

• scrape interval;
• acceptable detection delay;
• known transient behavior;
• service criticality.

Step 3: Validate the Rule with promtool

Run:

promtool check rules linux-alerts.yml

Current Prometheus documentation defines promtool check rules specifically for rule-file validation.

A successful result confirms the rule file parses.

For production workflows, consider linting/CI around rule changes so malformed alert files never reach the running server.

Test rule logic separately

Syntax validation does not prove the expression is useful.

Run the expression directly in Prometheus:

up{job="node"} == 0

When all hosts are healthy, an empty result is expected.

Also inspect:

up{job="node"}

so you understand which instance labels would produce alert instances.

Step 4: Load the Rule File in Prometheus

Edit prometheus.yml:

rule_files:
  - "linux-alerts.yml"

If the rule file lives in another directory, use the path Prometheus can read.

A compact configuration might look like:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "linux-alerts.yml"

scrape_configs:
  - job_name: node
    static_configs:
      - targets:
          - "10.0.0.12:9100"

Validate the whole configuration:

promtool check config prometheus.yml

The current promtool check config command validates Prometheus configuration and referenced resources.

Reload Prometheus using your deployment's supported mechanism.

Step 5: Confirm the Rule Is Loaded

Open the Prometheus Rules page and find:

NodeExporterTargetDown

Verify:

• the expression is correct;
• labels/annotations are present;
• current state is Inactive while the target is healthy.

If the rule does not appear, inspect:

• rule_files path;
• filesystem permissions;
• Prometheus reload/startup logs;
• configuration validation.

Do not continue to Alertmanager troubleshooting until Prometheus has actually loaded the rule.

Step 6: Configure Prometheus to Send Alerts to Alertmanager

Add:

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - "127.0.0.1:9093"

Use the real Alertmanager address reachable from Prometheus.

Official Prometheus high-availability guidance says that when you run multiple Alertmanager instances, configure Prometheus with all Alertmanager instances rather than hiding them behind a load balancer.

For a single-host lab, 127.0.0.1:9093 is fine when both services run on that host.

Validate again:

promtool check config prometheus.yml

Step 7: Create a Minimal Alertmanager Configuration

Create alertmanager.yml:

route:
  receiver: "test-webhook"

receivers:
  - name: "test-webhook"
    webhook_configs:
      - url: "http://127.0.0.1:5001/alerts"
        send_resolved: true

This defines one default route and one webhook receiver.

The webhook URL is an example local test destination. It is not an instruction to expose an unauthenticated webhook receiver publicly.

In production, configure your real receiver—PagerDuty, email, Slack-compatible integration, webhook gateway, or another supported integration—using the current Alertmanager documentation and your secret-management practices.

Step 8: Validate Alertmanager Configuration

First run:

amtool check-config alertmanager.yml

Current Prometheus Alertmanager documentation also recommends checking UTF-8 strict compatibility.

Run:

amtool check-config \
  alertmanager.yml \
  --enable-feature="utf8-strict-mode"

For a new Alertmanager installation, the current documentation recommends moving to UTF-8 strict mode rather than building new configuration around classic matcher parsing.

When you later add matcher expressions, quote values carefully according to current UTF-8 matcher syntax.

Why this matters now

Alertmanager versions from 0.27 onward introduced a UTF-8 matcher parser and a transition period. Configuration that parses only in classic fallback mode can produce warnings today and break when strict behavior becomes the default.

A minimal configuration with no route matchers avoids most transition complexity, but validating strict mode now is still good practice.

Step 9: Start or Reload Alertmanager

Check the installed version:

alertmanager --version

For a new installation prepared for strict parsing, start according to the current Alertmanager documentation, for example:

alertmanager \
  --config.file=alertmanager.yml \
  --enable-feature="utf8-strict-mode"

If your production Alertmanager is an existing deployment still using fallback mode, do not turn on strict mode blindly. Run amtool validation first, resolve warnings, and migrate under change control.

Confirm the Alertmanager UI/API is reachable from Prometheus at the configured address.

Step 10: Use a Safe Webhook Receiver for Verification

You need a destination that proves Alertmanager actually sent the notification.

Use one of:

• a dedicated non-production webhook service;
• an internal test receiver;
• a local development HTTP endpoint designed to log POST bodies.

Do not route the first plumbing test to the production paging rotation.

A simple local receiver can be a tiny application that accepts POST requests at /alerts and prints only the test payload. If you build one, keep it on localhost/private networking and do not use it as a production notification system.

The purpose is to separate:

1. rule evaluation;
2. Prometheus-to-Alertmanager delivery;
3. Alertmanager-to-receiver delivery.

Step 11: Trigger the Alert Safely

Use a test Node Exporter target.

One controlled method:

sudo systemctl stop node_exporter

Do this only on a host where stopping monitoring does not affect production response.

Because the rule has:

for: 2m

the state progression should be:

1. expression becomes true;
2. alert becomes Pending;
3. after two continuous minutes, alert becomes Firing;
4. Prometheus sends it to Alertmanager;
5. Alertmanager routes it to the test receiver.

Restart the exporter after verification:

sudo systemctl start node_exporter

The alert should eventually resolve, and send_resolved: true allows the test receiver to receive a resolved notification.

Step 12: Verify Each Layer Separately

Prometheus Rules

Confirm:

• rule is loaded;
• state moves Pending → Firing;
• labels/annotations look correct.

Prometheus Alerts

Confirm the Firing alert instance contains:

• alertname;
• instance;
• job;
• severity;
• team.

Alertmanager

Confirm the alert appears in Alertmanager and maps to the intended receiver.

Test receiver

Confirm a notification arrives.

If one layer succeeds and the next fails, you now know where to troubleshoot.

Verify the Setup

The alerting pipeline is complete when:

• promtool check rules linux-alerts.yml succeeds;
• promtool check config prometheus.yml succeeds;
• the rule appears in Prometheus;
• amtool check-config alertmanager.yml succeeds;
• strict-mode validation succeeds or documented migration work remains;
• Prometheus can reach Alertmanager;
• a controlled test reaches Pending;
• it reaches Firing after the configured for;
• Alertmanager receives the alert;
• the safe receiver receives the notification;
• restart/recovery resolves the alert.

Do not mark the alerting setup complete solely because Alertmanager's web UI loads.

Common Problems and Fixes

Problem: Rule file validates but does not appear

Cause: rule_files does not include the file, path is wrong, permissions block it, or Prometheus was not reloaded.

Fix: inspect the active Prometheus configuration and startup/reload logs.

Problem: Alert never leaves Pending

Cause: the expression becomes false before the for duration completes.

Fix: inspect the underlying metric over time. Decide whether the alert should require sustained failure or whether the test method is unstable.

Problem: Alert is Firing in Prometheus but absent in Alertmanager

Cause: Prometheus cannot reach the configured Alertmanager or the alertmanager target config is wrong.

Fix: inspect Prometheus notification logs/config and test TCP/HTTP reachability to port 9093.

Problem: Alert is in Alertmanager but no notification arrives

Cause: receiver configuration, routing, network access, credentials, or downstream webhook/integration failure.

Fix: inspect Alertmanager logs and receiver status. Do not change the Prometheus rule if Alertmanager already has the alert.

Problem: amtool warns about UTF-8 matchers

Cause: configuration relies on classic matcher syntax.

Fix: follow the suggested quoting changes and rerun:

amtool check-config alertmanager.yml \
  --enable-feature="utf8-strict-mode"

Current docs recommend new installations start compatible with strict mode.

Problem: Too many notifications arrive for multiple hosts

Cause: Alertmanager creates/routs alert groups based on label sets and route grouping configuration.

Fix: configure deliberate group_by, group_wait, group_interval, and repeat_interval based on incident behavior. Do not erase useful instance labels just to reduce message count.

Problem: Alert keeps firing after exporter recovery

Cause: scrape has not recovered, rule evaluation has not observed the recovery yet, or keep_firing_for is configured in another rule.

Fix: check up{job="node"} first. Current Prometheus rules also support keep_firing_for; if used, account for it in the expected recovery timeline.

Best Practices

Alert on symptoms when possible

Prometheus's own alerting practices recommend keeping alerts simple and focusing on symptoms associated with real user pain.

A Node Exporter-down alert is useful for monitoring-pipeline health, but it should not become your only server alert.

Use for to absorb small blips

Short transient failures should not automatically page a human unless your service objective requires it.

Keep labels actionable

Useful labels include team, service, severity, and environment. Avoid unbounded labels.

Put runbook context in annotations

A responder should know what failed and what to inspect next.

Validate in CI

Use:

promtool check rules
promtool check config
amtool check-config

before deploying configuration changes.

Separate alert evaluation from notification routing

Prometheus decides when an alert is active. Alertmanager decides where notifications go. Preserve that separation.

Prepare for UTF-8 strict matchers now

New Alertmanager configurations should not accumulate syntax that only works in classic fallback mode.

When This Setup Makes Sense

Prometheus rules plus Alertmanager are a strong fit when:

• Prometheus already owns metric evaluation;
• alert rules are managed as files/code;
• you want Alertmanager grouping, routing, silencing, and inhibition;
• the organization prefers alerts independent of Grafana.

Grafana-managed alerting may be preferable when Grafana is the central alerting control plane across several data sources.

Avoid defining the same alert independently in both systems unless duplicate evaluation and routing are intentional.

FAQ

What is the difference between Prometheus and Alertmanager?

Prometheus evaluates alert expressions. Alertmanager receives firing alerts and handles grouping, routing, silencing, inhibition, and notification delivery.

Why is my alert Pending instead of Firing?

The rule's for: duration has not completed continuously. If the expression becomes false before the duration ends, the alert does not reach Firing.

How do I validate Prometheus alert rules?

Use:

promtool check rules <rule-file>

and validate the full Prometheus configuration with:

promtool check config prometheus.yml

How do I validate Alertmanager configuration?

Use:

amtool check-config alertmanager.yml

For UTF-8 strict compatibility:

amtool check-config alertmanager.yml \
  --enable-feature="utf8-strict-mode"

Which Alertmanager version should I install?

Use the current official release for your maintained environment and record alertmanager --version. On September 11, 2026, the GitHub Releases page marks 0.32.1 as Latest, while the repository changelog already includes a 0.32.2 entry, so this tutorial avoids pretending the public release metadata is perfectly synchronized.

Conclusion

Reliable Prometheus alerting becomes easier to operate when you verify the pipeline one stage at a time.

Write an actionable rule, validate it with promtool, make Prometheus load it, connect Prometheus to Alertmanager, validate Alertmanager with amtool, and use a safe receiver to prove delivery.

Once that plumbing is trusted, replace the simple test rule with alerts based on real service symptoms and operating objectives rather than accumulating a large catalog of noisy thresholds.

References

1. Prometheus Documentation — "Alerting rules" — https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/ — accessed September 11, 2026.
2. Prometheus Documentation — "Alertmanager" — https://prometheus.io/docs/alerting/latest/alertmanager/ — accessed September 11, 2026.
3. Prometheus Documentation — "Alertmanager configuration" — https://prometheus.io/docs/alerting/latest/configuration/ — accessed September 11, 2026.
4. Prometheus Documentation — "Alerting based on metrics" — https://prometheus.io/docs/tutorials/alerting_based_on_metrics/ — accessed September 11, 2026.
5. Prometheus Documentation — "Alerting practices" — https://prometheus.io/docs/practices/alerting/ — accessed September 11, 2026.
6. Prometheus Documentation — "Prometheus configuration" — https://prometheus.io/docs/prometheus/latest/configuration/configuration/ — accessed September 11, 2026.
7. Prometheus GitHub — "Releases" — https://github.com/prometheus/prometheus/releases — accessed September 11, 2026.
8. Prometheus Alertmanager GitHub — "Releases" — https://github.com/prometheus/alertmanager/releases — accessed September 11, 2026.
9. Prometheus Alertmanager GitHub — "CHANGELOG.md" — https://github.com/prometheus/alertmanager/blob/main/CHANGELOG.md — accessed September 11, 2026.