github
/
prometheus
mirror of https://github.com/prometheus/prometheus
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Clarify what happens when a rule group takes too long to execute 

Namely, call out that all subsequent evaluations will be skipped until the initial evaluation completes.

Signed-off-by: Jennifer Villa <jvilla2013@gmail.com>
pull/12909/head
Jennifer Villa 2023-09-30 10:23:54 -04:00 committed by GitHub
parent a38179c4e1
commit 5027863c40
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 4 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
docs/configuration/recording_rules.md
View File

@ -147,3 +147,7 @@ by the rule are discarded, and if it's an alerting rule, _all_ alerts for
the rule, active, pending, or inactive, are cleared as well. The event will be
recorded as an error in the evaluation, and as such no stale markers are
written.
# Failed rule evaluations due to slow evaluation
If a rule group hasn't finished evaluating before the next evaluation is supposed to start, the next evaluation will be skipped. Subsequent evaluations of the rule group will continue to be skipped until the initial evaluation either completes or times out. When this happens, there will be a gap in the recording rule metric. The `rule_group_iterations_missed_total` metric will be incremented for each missed iteration.