How to Create Grafana Alerts for Server Metrics

Grafana can display Prometheus metrics in dashboards, but a dashboard does not notify anybody when a server becomes unhealthy. Grafana Alerting can evaluate a PromQL query on a schedule and route a notification when the resulting signal crosses a condition.

The first decision is architectural: Grafana-managed alert rules and Prometheus-native/data-source-managed rules are not the same thing. This tutorial deliberately creates a Grafana-managed rule using Prometheus as the data source. It does not edit a Prometheus rule file.

Last verified: September 10, 2026. Grafana's Prometheus alerting documentation was reviewed August 4, 2026 and currently documents both workflows. The navigation and alerting UI can evolve, so use the current /latest/ documentation if labels differ.

What You'll Accomplish

You will:

1. verify a Prometheus server metric in Grafana;
2. create a Grafana-managed alert rule;
3. use a CPU-utilization query suitable for alert evaluation;
4. configure an evaluation interval and pending period;
5. make an explicit decision for No Data and Error states;
6. configure and test a contact point; and
7. validate the alert without generating uncontrolled production noise.

Prerequisites

You need:

• Grafana with Alerting available;
• a configured Prometheus data source;
• Node Exporter or equivalent server metrics;
• permission to create alert rules;
• permission to create/use a contact point or notification policy;
• a safe email/webhook/other test destination.

The sibling dashboard tutorial is useful preparation because it verifies the metrics before they become alert inputs.

Quick Answer

Go to Alerting → Alert rules → New alert rule, choose a Grafana-managed rule, and query the Prometheus data source. For CPU utilization, a current Grafana-documented pattern is 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100). Keep the threshold separate from the PromQL when possible, set an evaluation interval and a pending period so a brief spike does not immediately page somebody, then configure No Data and Error handling explicitly. Route the rule to a tested contact point or notification policy. Validate the rule with a deliberately safe threshold/test target, then restore production thresholds before enabling normal on-call delivery.

Step 1: Choose Grafana-Managed Alerting Deliberately

Current Grafana documentation describes two Prometheus alerting workflows.

Grafana-managed alert rules

These rules are defined and evaluated by Grafana. You create and manage them in Grafana's Alerting UI while using Prometheus as the query data source.

Data source-managed rules

These are rules defined in the Prometheus rule files. Grafana can display Prometheus-managed rules, but current documentation states that for a Prometheus data source this view is read-only. To modify those rules, edit Prometheus configuration/rule files.

This tutorial uses Grafana-managed alerting because the goal is to create the rule entirely in Grafana.

If your organization manages alerts as code in Prometheus, use the later Prometheus/Alertmanager tutorial instead rather than duplicating the same alert in both systems.

Step 2: Start with a Metric You Already Trust

Do not create an alert from a query you have never inspected.

In Grafana Explore or the dashboard, evaluate:

node_cpu_seconds_total

Then confirm your host labels.

For a CPU utilization signal, Grafana's current Prometheus alerting documentation provides a query pattern equivalent to:

100 - (
  avg by (instance) (
    rate(node_cpu_seconds_total{mode="idle"}[5m])
  ) * 100
)

Why use [5m] here instead of $__rate_interval?

Grafana dashboards can use macros such as $__rate_interval.

Current Grafana query-editor documentation explicitly warns that alert queries do not support dashboard template variables. An alert should not depend on a dashboard variable that has no value during background rule evaluation.

A fixed five-minute rate window is an understandable example for this tutorial. It is not a universal alert setting. Tune it to your scrape interval and operational objective.

Step 3: Create a New Grafana-Managed Rule

Current Grafana documentation gives the core path:

1. Navigate to Alerting → Alert rules.
2. Select New alert rule.
3. Give the rule a descriptive name.
4. Select the Prometheus data source.
5. enter the PromQL query.
6. configure the alert condition;
7. set evaluation interval and pending period;
8. configure labels and notification behavior; and
9. save the rule.

Use a name such as:

Linux host sustained CPU utilization

Avoid names like CPU > 90 if you expect the threshold to change over time. The name should describe the symptom.

Step 4: Keep the Threshold Separate from the Query

Grafana's current Prometheus alerting guide demonstrates a useful pattern:

• Query A calculates the metric value.
• The Grafana condition compares the latest/reduced value with a threshold.

For example, Query A returns CPU utilization as a percentage. The condition then evaluates whether the value is above your chosen threshold.

This separation has two advantages:

1. you can inspect the raw signal independently of the threshold;
2. threshold changes do not require rewriting the PromQL expression.

Do not copy a threshold blindly

The official documentation uses example values such as 90% CPU. That does not mean 90% is correct for your server.

A useful alert threshold depends on:

• workload characteristics;
• number of cores;
• autoscaling behavior;
• latency impact;
• how long saturation must persist; and
• whether a responder can take a meaningful action.

A build worker at 95% CPU may be healthy. A latency-sensitive API at sustained saturation may not be.

Step 5: Set Evaluation Interval and Pending Period

Grafana evaluates the rule repeatedly.

The evaluation interval controls how often the rule is evaluated. The pending period requires the condition to remain true before transitioning to the alerting state.

These are separate from the PromQL rate window.

A conceptual flow is:

1. PromQL calculates CPU utilization using the last five minutes of samples.
2. Grafana evaluates the rule on its schedule.
3. If the threshold is breached, the alert becomes Pending.
4. If the breach persists for the pending period, it becomes Alerting/Firing according to the current state model.

A pending period can prevent a single short spike from paging the team, but an excessively long pending period can hide a real incident.

Choose timing from the response objective and normal metric variability.

Step 6: Decide What No Data Means

This is one of the most important alert-design decisions.

If Prometheus returns no series, several realities are possible:

• the server is down;
• Node Exporter is down;
• Prometheus cannot scrape;
• a label changed;
• the query is wrong;
• the data source is temporarily unavailable; or
• the host was intentionally retired.

Current Grafana-managed rules have explicit No Data behavior.

Do not automatically treat No Data as Healthy/Normal simply to suppress noise. That can hide telemetry failure.

Also do not automatically page the application owner for every scrape gap.

Decide who owns missing telemetry and whether a separate up == 0/target-health alert provides a clearer signal.

Step 7: Decide What Execution Error Means

An execution error can occur when:

• Prometheus is unreachable;
• a query times out;
• credentials fail;
• Grafana cannot execute the data-source request; or
• another evaluation problem occurs.

Current Grafana documentation notes that the default Error behavior can fire an alert, which may create false alarms for transient data-source errors.

The alternatives can include Error, Alerting, Normal, or Keep Last State depending on the current alerting configuration.

Make the decision explicit.

For a critical metric, silently treating a broken data source as Normal can be dangerous. For a noisy noncritical dashboard metric, paging on every transient query error can also be dangerous.

A separate monitoring path for the observability stack itself often gives cleaner ownership.

Step 8: Create and Test a Contact Point

Grafana uses contact points to define notification destinations such as email, Slack, webhooks, PagerDuty, and other integrations.

Current Grafana navigation documents this area under:

Alerts & IRM → Alerting → Notification configuration → Contact points

Create a dedicated test contact point first if possible.

Test the contact point independently

Grafana's current contact-point documentation provides a Test action for the Grafana Alertmanager:

1. edit or create the contact point;
2. select Test;
3. choose a predefined or custom test;
4. send the test notification.

This is valuable because it separates:

"Can Grafana deliver a notification?"

from:

"Does my alert query fire correctly?"

Verify delivery before debugging the rule.

Step 9: Route the Alert

Grafana can route an alert directly to a contact point or through notification policies.

Notification policies are useful when labels determine routing, grouping, and timing across many alert rules.

For a first rule, direct routing can be simpler if the current Grafana version/account supports it in the rule workflow.

For a shared production alerting environment, notification policies are often more maintainable because:

• rules can carry labels such as team="platform" and severity="warning";
• routing remains separate from query logic; and
• grouped notifications reduce duplicate messages.

Do not encode an email address into the alert query. Signal and delivery should remain separate concerns.

Step 10: Add Useful Labels and Annotations

Labels help identify and route an alert instance.

Examples:

• team="platform"
• service="linux-host"
• severity="warning"

Use only labels with stable operational meaning.

Annotations/messages should explain:

• what symptom occurred;
• which instance is affected;
• what a responder should inspect; and
• where the runbook is.

Avoid repeating raw dashboard data without interpretation.

Do not place secrets in labels or annotations. They can appear in notifications, logs, APIs, and incident systems.

Step 11: Validate the Rule Safely

Do not create CPU load on a production server just to test an alert.

Safer approaches include:

• temporarily use a threshold that the existing test metric can cross in a non-production environment;
• point a cloned rule at a test instance;
• use a deterministic test metric if your environment has one; or
• validate the notification channel independently with the contact-point Test action.

Observe:

• the query returns data;
• the threshold condition becomes true;
• the alert enters Pending if configured;
• it transitions as expected;
• the notification reaches the intended test destination; and
• it resolves/recoveries correctly.

Then restore the intended production threshold and routing.

This tutorial does not claim PerfMonitoring executed an alert in your Grafana instance.

Verify the Setup

A Grafana server-metric alert is complete when:

• the rule type is explicitly Grafana-managed;
• Prometheus query A returns expected data;
• the instance labels identify the intended servers;
• the threshold reflects a real operational boundary;
• evaluation interval is documented;
• pending period is intentional;
• No Data handling is intentional;
• Error handling is intentional;
• a contact point can send a test notification;
• labels route to the intended owner; and
• a controlled rule test behaves as expected.

Do not call the alert finished merely because it appears in the Alert rules list.

Common Problems and Fixes

Problem: The dashboard query works but the alert query fails

Cause: the dashboard query uses template variables or macros not supported in alert evaluation.

Fix: replace dashboard variables with explicit labels and use a fixed rate range where appropriate. Current Grafana docs specifically warn that alert queries do not support template variables.

Problem: The rule fires when Prometheus briefly fails

Cause: execution errors are configured to alert, or No Data is interpreted as a failure for the same rule.

Fix: decide whether telemetry availability deserves a separate alert. Tune Error/No Data behavior deliberately rather than suppressing every failure.

Problem: The alert stays Pending

Cause: the condition has not remained true for the full pending period, or the signal crosses back below threshold between evaluations.

Fix: inspect evaluation history and the actual metric. If the symptom is intermittent, decide whether sustained or burst behavior is operationally important before changing the pending period.

Problem: The rule shows No Data

Cause: Prometheus is not returning a matching series.

Fix: run the exact query in Explore without the alert condition. Check data source, time range, label filters, scrape target, and metric name.

Problem: The contact point test works, but the rule sends nothing

Cause: the rule may not be firing, routing labels do not match the notification policy, or it uses a different Alertmanager/contact point.

Fix: verify alert state first, then inspect notification-policy matching and the selected Alertmanager.

Problem: Several hosts produce too many notifications

Cause: one alert rule can create multiple alert instances by label, and notification routing/grouping is not configured appropriately.

Fix: preserve useful per-instance labels but group related notifications through a notification policy. Do not erase the instance label simply to reduce messages.

Best Practices

Alert on actionable symptoms

High CPU is useful only when somebody can interpret or act on it. Consider whether latency, saturation, queue depth, or service error rate is closer to user impact.

Require persistence for noisy infrastructure metrics

CPU changes quickly. A pending period can distinguish a sustained problem from a short workload burst.

Monitor telemetry health separately

A server alert and a "Prometheus cannot scrape this server" alert represent different failure modes and often different remediation.

Route with stable labels

Use team/service/severity labels consistently so notification policies remain understandable.

Keep dashboards and alerts related but not identical

A dashboard query can be exploratory and variable-driven. An alert query must be deterministic in the background and should have explicit scope.

Test notification delivery

A correct rule with a broken contact point is not operational alerting.

When This Setup Makes Sense

Grafana-managed alerting is a good fit when:

• Grafana is already the operational alerting control plane;
• teams want to manage rules in Grafana rather than Prometheus files;
• multiple data sources need a common alerting workflow; or
• notification policies/contact points are centrally managed in Grafana.

Prometheus-native rules plus Alertmanager may be a better fit when:

• alerts are managed as code alongside Prometheus;
• you want Prometheus to own rule evaluation independently of Grafana; or
• existing infrastructure already relies on Alertmanager.

Do not configure the same symptom in both systems unless duplicate evaluation/notifications are intentional.

FAQ

What is a Grafana-managed alert rule?

It is a rule stored/evaluated by Grafana. The rule can query Prometheus, but it is not the same as a Prometheus rule file.

Can Grafana edit Prometheus-native rules?

Current Grafana documentation states that Prometheus data-source-managed rules are visible read-only in Grafana. Modify those rules in Prometheus configuration/rule files.

What should Grafana do when Prometheus returns no data?

There is no universal answer. Decide whether missing data represents the monitored service failing, the monitoring pipeline failing, a query/label problem, or an expected host lifecycle event. Configure No Data accordingly.

Why is my alert stuck in Pending?

The threshold must remain breached for the configured pending period. If the metric recovers before that period ends, the alert will not progress to firing.

Can I test a Grafana contact point without forcing the alert?

Yes. Current Grafana Alertmanager contact-point configuration includes a Test action that can send a predefined or custom test notification.

Conclusion

A reliable Grafana alert is more than a PromQL expression plus a threshold. It has a clear evaluation owner, stable query scope, deliberate pending behavior, explicit No Data/Error semantics, and a verified notification route.

For Prometheus server metrics, first verify the signal in a dashboard or Explore. Then create a Grafana-managed rule, keep the threshold distinct from the metric query, test the contact point, and validate the rule safely.

If your organization wants Prometheus itself to evaluate rules and Alertmanager to route them, use the separate Prometheus alerting tutorial rather than mixing the two workflows.

References

1. Grafana Labs Documentation — "Prometheus alerting" — https://grafana.com/docs/grafana/latest/datasources/prometheus/alerting/ — accessed September 10, 2026; last reviewed by Grafana August 4, 2026.
2. Grafana Labs Documentation — "Prometheus query editor" — https://grafana.com/docs/grafana/latest/datasources/prometheus/query-editor/ — accessed September 10, 2026.
3. Grafana Labs Documentation — "Configure contact points" — https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/ — accessed September 10, 2026.
4. Grafana Labs Documentation — "Configure notification policies" — https://grafana.com/docs/grafana/latest/alerting/configure-notifications/create-notification-policy/ — accessed September 10, 2026.
5. Grafana Labs Documentation — "Introduction to Grafana Alerting" — https://grafana.com/docs/grafana/latest/alerting/fundamentals/ — accessed September 10, 2026.