Add differentiating label for alerts that use nodeCriticalSeverity

This affects the following alerts:

- NodeFilesystemAlmostOutOfFiles
- NodeFilesystemSpaceFillingUp
- NodeFilesystemAlmostOutOfSpace
- NodeFilesystemFilesFillingUp

Each alert is actually two alerting rules, one for the warning level,
and one for the critical level. However, the critical one can be set to
emit as a warning by using $._config.nodeCriticalSeverity. When this is
true, both alerting rules will generate alerts with an identical set of
labels, which is an error when both alerts fire.

We add an extra label to these alerting rules in order to differentiate
between the two levels when $._config.nodeCriticalSeverity is set to
`warning`.

Signed-off-by: Justin Lei <lei.justin@gmail.com>
pull/3189/head
Justin Lei 6 days ago
parent 49d177bf95
commit 5d713dc40a

@ -18,6 +18,7 @@
'for': '1h', 'for': '1h',
labels: { labels: {
severity: 'warning', severity: 'warning',
window: '24h',
}, },
annotations: { annotations: {
summary: 'Filesystem is predicted to run out of space within the next 24 hours.', summary: 'Filesystem is predicted to run out of space within the next 24 hours.',
@ -38,6 +39,7 @@
'for': '1h', 'for': '1h',
labels: { labels: {
severity: '%(nodeCriticalSeverity)s' % $._config, severity: '%(nodeCriticalSeverity)s' % $._config,
window: '4h',
}, },
annotations: { annotations: {
summary: 'Filesystem is predicted to run out of space within the next 4 hours.', summary: 'Filesystem is predicted to run out of space within the next 4 hours.',
@ -56,6 +58,7 @@
'for': '30m', 'for': '30m',
labels: { labels: {
severity: 'warning', severity: 'warning',
threshold: '%(fsSpaceAvailableWarningThreshold)d%%' % $._config,
}, },
annotations: { annotations: {
summary: 'Filesystem has less than %(fsSpaceAvailableWarningThreshold)d%% space left.' % $._config, summary: 'Filesystem has less than %(fsSpaceAvailableWarningThreshold)d%% space left.' % $._config,
@ -74,6 +77,7 @@
'for': '30m', 'for': '30m',
labels: { labels: {
severity: '%(nodeCriticalSeverity)s' % $._config, severity: '%(nodeCriticalSeverity)s' % $._config,
threshold: '%(fsSpaceAvailableCriticalThreshold)d%%' % $._config,
}, },
annotations: { annotations: {
summary: 'Filesystem has less than %(fsSpaceAvailableCriticalThreshold)d%% space left.' % $._config, summary: 'Filesystem has less than %(fsSpaceAvailableCriticalThreshold)d%% space left.' % $._config,
@ -94,6 +98,7 @@
'for': '1h', 'for': '1h',
labels: { labels: {
severity: 'warning', severity: 'warning',
window: '24h',
}, },
annotations: { annotations: {
summary: 'Filesystem is predicted to run out of inodes within the next 24 hours.', summary: 'Filesystem is predicted to run out of inodes within the next 24 hours.',
@ -114,6 +119,7 @@
'for': '1h', 'for': '1h',
labels: { labels: {
severity: '%(nodeCriticalSeverity)s' % $._config, severity: '%(nodeCriticalSeverity)s' % $._config,
window: '4h',
}, },
annotations: { annotations: {
summary: 'Filesystem is predicted to run out of inodes within the next 4 hours.', summary: 'Filesystem is predicted to run out of inodes within the next 4 hours.',
@ -132,6 +138,7 @@
'for': '1h', 'for': '1h',
labels: { labels: {
severity: 'warning', severity: 'warning',
threshold: '5%',
}, },
annotations: { annotations: {
summary: 'Filesystem has less than 5% inodes left.', summary: 'Filesystem has less than 5% inodes left.',
@ -150,6 +157,7 @@
'for': '1h', 'for': '1h',
labels: { labels: {
severity: '%(nodeCriticalSeverity)s' % $._config, severity: '%(nodeCriticalSeverity)s' % $._config,
threshold: '3%',
}, },
annotations: { annotations: {
summary: 'Filesystem has less than 3% inodes left.', summary: 'Filesystem has less than 3% inodes left.',

Loading…
Cancel
Save