Browse Source

Don't increment prometheus_tsdb_compactions_failed_total when context is canceled (#10772)

When restarting Prometheus I sometimes see:

caller=db.go:832 level=error component=tsdb msg="compaction failed" err="compact head: persist head block: 2 errors: populate block: context canceled; context canceled"

And prometheus_tsdb_compactions_failed_total metric gets incremented.
This makes it more difficult to write alerts based on
prometheus_tsdb_compactions_failed_total metric since any restart can
trigger it.

Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>
pull/10855/head^2
Łukasz Mierzwa 2 years ago committed by GitHub
parent
commit
d65f037def
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
  1. 4
      tsdb/db.go

4
tsdb/db.go

@ -888,7 +888,9 @@ func (db *DB) Compact() (returnErr error) {
db.cmtx.Lock()
defer db.cmtx.Unlock()
defer func() {
if returnErr != nil {
if returnErr != nil && !errors.Is(returnErr, context.Canceled) {
// If we got an error because context was canceled then we're most likely
// shutting down TSDB and we don't need to report this on metrics
db.metrics.compactionsFailed.Inc()
}
}()

Loading…
Cancel
Save