mirror of https://github.com/prometheus/prometheus
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Tree:
50b5837a63
add-body-size-limit-global
add-chunkiterator-interface
alexg-content-negotiation
alexg-content-negotiation-2
alexg-content-negotiation-dont-advertise
alexg-remove-HEAD-1
alexg/fix-fallback-scrape
alexg/settable-user-agent
allow-swappable-engine
arct2
arthursens/appendct-prwv2
arthursens/ct-in-wal
beorn7/lookback
beorn7/metrics
beorn7/polymorphism
bisect-series-base
bp/append-docs
bwplotka/2_0_cmp
bwplotka/2_18_cmp
bwplotka/config-watch
bwplotka/nhcb-wal-wbl
bwplotka/prw-initial
callum-filter-benchmark
callum-live-segment-test
callum-opportunistic-labels-cleanup
callum-remote-proto-2
callum-rw-e2e-dashboard-1
callum-rw-intern-unique
callum-rw-intern-v3
callum-rw-min-format
callum-rw-split-intern
callum-simplify-ts-filtering
callum-track-series-segment
callum-vt20
callum-watcher-poll-backoff
catch-sighup-early
cedwards/nhcb-wal-wbl
cedwards/ooo-native-histograms-promql-tests
csmarchbanks/arrow-native-histograms
csmarchbanks/arrow-responses
csmarchbanks/native-histogram-compact-format
csmarchbanks/native-histogram-json-format
csmarchbanks/native-histogram-verbose-format
csmarchbanks/optmize-arrow-responses
cut-2.54.2
dependabot/go_modules/github.com/fsnotify/fsnotify-1.8.0
dependabot/go_modules/k8s-io-029cbc0f93
dependabot/npm_and_yarn/web/ui/codemirror/language-6.10.3
dependabot/npm_and_yarn/web/ui/eslint-plugin-react-hooks-5.1.0-rc.0
dependabot/npm_and_yarn/web/ui/eslint/compat-1.2.2
dependabot/npm_and_yarn/web/ui/eslint/js-9.14.0
dependabot/npm_and_yarn/web/ui/globals-15.11.0
dependabot/npm_and_yarn/web/ui/lezer/common-1.2.3
dependabot/npm_and_yarn/web/ui/mantine/code-highlight-7.13.4
dependabot/npm_and_yarn/web/ui/react-router-dom-6.27.0
dependabot/npm_and_yarn/web/ui/reduxjs/toolkit-2.3.0
dependabot/npm_and_yarn/web/ui/rollup/plugin-node-resolve-15.3.0
dependabot/npm_and_yarn/web/ui/sanitize-html-2.13.1
dependabot/npm_and_yarn/web/ui/tanstack/react-query-5.60.6
dependabot/npm_and_yarn/web/ui/testing-library/jest-dom-6.6.3
dependabot/npm_and_yarn/web/ui/types/jest-29.5.14
dependabot/npm_and_yarn/web/ui/types/lodash-4.17.13
dependabot/npm_and_yarn/web/ui/types/react-18.3.12
dependabot/npm_and_yarn/web/ui/typescript-eslint/eslint-plugin-8.12.2
dependabot/npm_and_yarn/web/ui/uiw/react-codemirror-4.23.6
dependabot/npm_and_yarn/web/ui/uplot-1.6.31
dependabot/npm_and_yarn/web/ui/vitejs/plugin-react-4.3.3
exemplar-storage-separate-mvp
expand
f17-test
faster-exemplars
feature/new-prom-ui
feature/shutdownscrape
fionaliao/3.0-migration-guide-fixes
fix-2.21-iter-latency
fix-2.21-latency
fix-sym-7373
fix-sym-7373-2
fix-sym-7373-3
fix/binop-output-labels
for-cortex-1-4-1
histogram-charts
histogram-json-experiment
imprct
integrate-react
join
julius/y-axis-zero
jvp/enable-otlp-endpoint-by-default
jvp/metadata-poc
jvp/range-duration-with-open-intervals
jvp/refactor-append-to-have-hints
jvp/refactor-append-to-have-hints-and-single-appendcall
jvp/refactor-append-to-have-hints-as-value
keep-multiple-separators
krajo/engine-immutable-histograms
krajo/testing-pr14831
labels-utf8-fix
lazy-unsymbolize
lbl-optimize
main
manik1
manik2
mantine-ui-uplot
mem/fix_8551
merge-main
multi-err-cause
native-histogram-charts
nav-settings
nexucis/sd-optimisation
nexucis/sofisticat-query
nhcb
njpm/fix-mockIndex
njpm/metadata-remote-write-wiring-2-update
njpm/rw2-main-sync-conflicts
njpm/rw2-sync-main-conflicts
njpm/tsdb-utf8-mixed-querying
ooo-nh-fix
otel-3
owilliams/utf8
owilliams/utf8-02-mimir
owilliams/utf8-03-reads
partial-response
post-processing
pre-ooo
prom-common-update-0.43
promselect
proposalprocess
prw-shard-test
pull/14654/head
react-travis-test-branch
redesign-parser-interface
rel-2.55-backport-14947
release-0.16
release-0.18
release-0.20
release-1.1
release-1.2
release-1.3
release-1.4
release-1.5
release-1.6
release-1.7
release-1.8
release-1.8.2
release-2.0
release-2.1
release-2.10
release-2.11
release-2.12
release-2.13
release-2.14
release-2.15
release-2.16
release-2.17
release-2.18
release-2.19
release-2.2
release-2.20
release-2.21
release-2.22
release-2.23
release-2.24
release-2.25
release-2.26
release-2.27
release-2.28
release-2.29
release-2.3
release-2.30
release-2.30-merge
release-2.31
release-2.32
release-2.33
release-2.34
release-2.35
release-2.36
release-2.37
release-2.38
release-2.39
release-2.4
release-2.40
release-2.41
release-2.42
release-2.43
release-2.43+stringlabels
release-2.44
release-2.45
release-2.46
release-2.47
release-2.48
release-2.49
release-2.5
release-2.50
release-2.51
release-2.51+dedupelabels
release-2.52
release-2.53
release-2.54
release-2.54+dedupelabels
release-2.55
release-2.6
release-2.7
release-2.8
release-2.9
release-3.0
release-docs-formatting-fix
release-v2.50
remote-write-2.0
remote-write-checkpoint
remove-alertmanager
remove-circleci-windows-tests
resharding
revert-10878-jvp/replay-wait-for-idle-dont-always-sleep
revert-13498-alexg-content-negotiation-2
revert-14513-fix-histogram-overlap-compaction-bug
revert-14809-patch-mem-issue
revert-6605-master
revert-6907-prom/rules_race_condition
revert-8489-append
richih/examples
richih/remove_indirect_deps
rmremote_storage
rr
rr-proto
rule-page-followups
rw20-config
rw20-gen
rw20-helpers-gen
rw20-nhcb
scrape-bytes-scraped
segfault-chunk-iter
series-hint
shutdown-select-bug
superq/betteralign_promql
superq/drop_vendor
superq/fieldalignment
superq/help_link
superq/test
superq/update_storage_docs
superq/v2
test-rw-compression
textparsebench-manik1
trim-3.0changelog-for-2.55
tsdb-import
tsdb-import-csv
update-2.53-to-correct-place
v2.54.0+dedupelabels
vtproto-test
wip-play-prombench
write-req-content-size
0.1.0
0.10.0
0.11.0
0.11.1
0.12.0
0.13.0
0.13.0rc2
0.13.1
0.13.2
0.13.3
0.13.4
0.14.0
0.14.0rc1
0.14.0rc2
0.14.0rc3
0.15.0
0.15.0rc1
0.15.0rc2
0.15.0rc3
0.15.1
0.16.0
0.16.0rc1
0.16.0rc2
0.16.1
0.16.2
0.17.0
0.17.0rc1
0.17.0rc2
0.18.0
0.18.0rc1
0.19.0
0.19.1
0.19.2
0.19.3
0.2.0
0.2.1
0.20.0
0.3.0
0.4.0
0.5.0
0.6.0
0.7.0
0.8.0
0.9.0
0.9.0rc1
0.9.0rc2
0.9.0rc3
0.9.0rc4
0.9.0rc5
all-tests-pass
checkout
dev
discovery-handle-discoverer-updates
v0.300.0
v0.300.0-beta.0
v0.300.0-beta.1
v0.300.0-rc.0
v0.300.0-rc.1
v0.35.0
v0.35.0-rc1
v0.36.0
v0.36.0-rc0
v0.36.1
v0.36.2
v0.37.0
v0.37.0-rc.0
v0.37.0-rc.1
v0.37.1
v0.37.2
v0.37.3
v0.37.4
v0.37.5
v0.37.6
v0.37.7
v0.37.8
v0.37.9
v0.38.0
v0.38.0-rc.0
v0.39.0
v0.39.1
v0.39.2
v0.40.0
v0.40.0-rc.0
v0.40.1
v0.40.2
v0.40.3
v0.40.4
v0.40.5
v0.40.6
v0.40.7
v0.41.0
v0.41.0-rc.0
v0.42.0
v0.43.0
v0.43.0-rc.0
v0.43.0-rc.1
v0.43.1
v0.44.0
v0.44.0-rc.0
v0.44.0-rc.1
v0.44.0-rc.2
v0.45.0
v0.45.0-rc.0
v0.45.0-rc.1
v0.45.1
v0.45.2
v0.45.3
v0.45.4
v0.45.5
v0.45.6
v0.46.0
v0.46.0-rc.0
v0.47.0
v0.47.0-rc.0
v0.47.1
v0.47.2
v0.48.0
v0.48.0-rc.0
v0.48.0-rc.1
v0.48.0-rc.2
v0.48.1
v0.49.0
v0.49.0-rc.0
v0.49.0-rc.1
v0.49.0-rc.2
v0.49.1
v0.50.0
v0.50.0-rc.0
v0.50.0-rc.1
v0.50.1
v0.51.0
v0.51.1
v0.51.2
v0.52.0
v0.52.0-rc.0
v0.52.0-rc.1
v0.52.1
v0.53.0
v0.53.0-rc.0
v0.53.0-rc.1
v0.53.1
v0.53.2
v0.53.3
v0.54.0
v0.54.0-rc.0
v0.54.1
v0.55.0
v0.55.1
v1.0.0
v1.0.0-rc.0
v1.0.1
v1.0.2
v1.1.0
v1.1.1
v1.1.2
v1.1.3
v1.2.0
v1.2.1
v1.2.2
v1.2.3
v1.3.0
v1.3.0-beta.0
v1.3.1
v1.4.0
v1.4.1
v1.5.0
v1.5.1
v1.5.2
v1.5.3
v1.6.0
v1.6.1
v1.6.2
v1.6.3
v1.7.0
v1.7.1
v1.7.2
v1.8.0
v1.8.1
v1.8.2
v1.99.0
v1.99.0-retract
v2.0.0
v2.0.0-alpha.0
v2.0.0-alpha.1
v2.0.0-alpha.2
v2.0.0-alpha.3
v2.0.0-beta.0
v2.0.0-beta.1
v2.0.0-beta.2
v2.0.0-beta.3
v2.0.0-beta.4
v2.0.0-beta.5
v2.0.0-rc.0
v2.0.0-rc.1
v2.0.0-rc.2
v2.0.0-rc.3
v2.1.0
v2.10.0
v2.10.0-rc.0
v2.11.0
v2.11.0-rc.0
v2.11.1
v2.11.2
v2.12.0
v2.12.0-rc.0
v2.13.0
v2.13.0-rc.0
v2.13.1
v2.14.0
v2.14.0-rc.0
v2.15.0
v2.15.0-rc.0
v2.15.1
v2.15.2
v2.16.0
v2.16.0-rc.0
v2.16.0-rc.1
v2.17.0
v2.17.0-rc.0
v2.17.0-rc.1
v2.17.0-rc.2
v2.17.0-rc.3
v2.17.0-rc.4
v2.17.1
v2.17.2
v2.18.0
v2.18.0-rc.0
v2.18.0-rc.1
v2.18.1
v2.18.2
v2.19.0
v2.19.0-rc.0
v2.19.1
v2.19.2
v2.19.3
v2.2.0
v2.2.0-rc.0
v2.2.0-rc.1
v2.2.1
v2.20.0
v2.20.0-rc.0
v2.20.0-rc.1
v2.20.1
v2.21.0
v2.21.0-rc.0
v2.21.0-rc.1
v2.22.0
v2.22.0-rc.0
v2.22.1
v2.22.2
v2.23.0
v2.23.0-rc.0
v2.24.0
v2.24.0-rc.0
v2.24.1
v2.25.0
v2.25.0-rc.0
v2.25.1
v2.25.2
v2.26.0
v2.26.0-rc.0
v2.26.1
v2.27.0
v2.27.0-rc.0
v2.27.1
v2.28.0
v2.28.0-rc.0
v2.28.1
v2.29.0
v2.29.0-rc.0
v2.29.0-rc.1
v2.29.0-rc.2
v2.29.1
v2.29.2
v2.3.0
v2.3.1
v2.3.2
v2.30.0
v2.30.0-rc.0
v2.30.1
v2.30.2
v2.30.3
v2.30.4
v2.31.0
v2.31.0-rc.0
v2.31.0-rc.1
v2.31.1
v2.31.2
v2.32.0
v2.32.0-beta.0
v2.32.0-rc.0
v2.32.0-rc.1
v2.32.1
v2.33.0
v2.33.0-rc.0
v2.33.0-rc.1
v2.33.1
v2.33.2
v2.33.3
v2.33.4
v2.33.5
v2.34.0
v2.34.0-rc.0
v2.34.0-rc.1
v2.34.0-rc.2
v2.35.0
v2.35.0-rc0
v2.35.0-rc1
v2.35.0-retract
v2.36.0
v2.36.0-rc0
v2.36.1
v2.36.2
v2.37.0
v2.37.0-rc.0
v2.37.0-rc.1
v2.37.1
v2.37.2
v2.37.3
v2.37.4
v2.37.5
v2.37.6
v2.37.7
v2.37.8
v2.37.9
v2.38.0
v2.38.0-rc.0
v2.39.0
v2.39.0-rc.0
v2.39.1
v2.39.2
v2.4.0
v2.4.0-rc.0
v2.4.1
v2.4.2
v2.4.3
v2.40.0
v2.40.0-rc.0
v2.40.1
v2.40.2
v2.40.3
v2.40.4
v2.40.5
v2.40.6
v2.40.7
v2.41.0
v2.41.0-rc.0
v2.42.0
v2.42.0-rc.0
v2.43.0
v2.43.0+stringlabels
v2.43.0-rc.0
v2.43.0-rc.1
v2.43.0-rc.1+stringlabels
v2.43.1
v2.43.1+stringlabels
v2.44.0
v2.44.0-rc.0
v2.44.0-rc.1
v2.44.0-rc.2
v2.45.0
v2.45.0-rc.0
v2.45.0-rc.1
v2.45.1
v2.45.2
v2.45.3
v2.45.4
v2.45.5
v2.45.6
v2.46.0
v2.46.0-rc.0
v2.47.0
v2.47.0-rc.0
v2.47.1
v2.47.2
v2.48.0
v2.48.0-rc.0
v2.48.0-rc.1
v2.48.0-rc.2
v2.48.1
v2.49.0
v2.49.0-rc.0
v2.49.0-rc.1
v2.49.0-rc.2
v2.49.1
v2.5.0
v2.5.0-rc.0
v2.5.0-rc.1
v2.5.0-rc.2
v2.50.0
v2.50.0-rc.0
v2.50.0-rc.1
v2.50.1
v2.51.0
v2.51.0+dedupelabels
v2.51.0-rc.0
v2.51.0-rc.0+dedupelabels
v2.51.1
v2.51.2
v2.52.0
v2.52.0-rc.0
v2.52.0-rc.1
v2.53.0
v2.53.0-rc.0
v2.53.0-rc.1
v2.53.1
v2.53.2
v2.53.3
v2.54.0
v2.54.0+dedupelabels
v2.54.0-rc.0
v2.54.0-rc.0+dedupelabels
v2.54.0-rc.1
v2.54.1
v2.55.0
v2.55.0-rc.0
v2.55.0-rc.1
v2.55.1
v2.6.0
v2.6.0-rc.0
v2.6.0-rc.1
v2.6.1
v2.7.0
v2.7.0-rc.0
v2.7.0-rc.1
v2.7.0-rc.2
v2.7.1
v2.7.2
v2.8.0
v2.8.0-rc.0
v2.8.1
v2.9.0
v2.9.0-rc.0
v2.9.1
v2.9.2
v3.0.0
v3.0.0-beta.0
v3.0.0-beta.1
v3.0.0-rc.0
v3.0.0-rc.1
${ noResults }
prometheus/rules
Matt Bostock
926a5ab3dd
On one relatively large Prometheus instance (1.7M series), I noticed
that upgrades were frequently resulting in Prometheus undergoing crash
recovery on start-up.
On closer examination, I found that Prometheus was panicking on
shutdown.
It seems that our configuration management (or misconfiguration thereof)
is reloading Prometheus then immediately restarting it, which I suspect
is causing this race:
Sep 21 15:12:42 host systemd[1]: Reloading prometheus monitoring system.
Sep 21 15:12:42 host prometheus[18734]: time="2016-09-21T15:12:42Z" level=info msg="Loading configuration file /etc/prometheus/config.yaml" source="main.go:221"
Sep 21 15:12:42 host systemd[1]: Reloaded prometheus monitoring system.
Sep 21 15:12:44 host systemd[1]: Stopping prometheus monitoring system...
Sep 21 15:12:44 host prometheus[18734]: time="2016-09-21T15:12:44Z" level=warning msg="Received SIGTERM, exiting gracefully..." source="main.go:203"
Sep 21 15:12:44 host prometheus[18734]: time="2016-09-21T15:12:44Z" level=info msg="See you next time!" source="main.go:210"
Sep 21 15:12:44 host prometheus[18734]: time="2016-09-21T15:12:44Z" level=info msg="Stopping target manager..." source="targetmanager.go:90"
Sep 21 15:12:52 host prometheus[18734]: time="2016-09-21T15:12:52Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548"
Sep 21 15:12:56 host prometheus[18734]: time="2016-09-21T15:12:56Z" level=warning msg="Error on ingesting out-of-order samples" numDropped=1 source="scrape.go:467"
Sep 21 15:12:56 host prometheus[18734]: time="2016-09-21T15:12:56Z" level=error msg="Error adding file watch for \"/etc/prometheus/targets\": no such file or directory" source="file.go:84"
Sep 21 15:12:56 host prometheus[18734]: time="2016-09-21T15:12:56Z" level=error msg="Error adding file watch for \"/etc/prometheus/targets\": no such file or directory" source="file.go:84"
Sep 21 15:13:01 host prometheus[18734]: time="2016-09-21T15:13:01Z" level=info msg="Stopping rule manager..." source="manager.go:366"
Sep 21 15:13:01 host prometheus[18734]: time="2016-09-21T15:13:01Z" level=info msg="Rule manager stopped." source="manager.go:372"
Sep 21 15:13:01 host prometheus[18734]: time="2016-09-21T15:13:01Z" level=info msg="Stopping notification handler..." source="notifier.go:325"
Sep 21 15:13:01 host prometheus[18734]: time="2016-09-21T15:13:01Z" level=info msg="Stopping local storage..." source="storage.go:381"
Sep 21 15:13:01 host prometheus[18734]: time="2016-09-21T15:13:01Z" level=info msg="Stopping maintenance loop..." source="storage.go:383"
Sep 21 15:13:01 host prometheus[18734]: panic: close of closed channel
Sep 21 15:13:01 host prometheus[18734]: goroutine 7686074 [running]:
Sep 21 15:13:01 host prometheus[18734]: panic(0xba57a0, 0xc60c42b500)
Sep 21 15:13:01 host prometheus[18734]: /usr/local/go/src/runtime/panic.go:500 +0x1a1
Sep 21 15:13:01 host prometheus[18734]: github.com/prometheus/prometheus/rules.(*Manager).ApplyConfig.func1(0xc6645a9901, 0xc420271ef0, 0xc420338ed0, 0xc60c42b4f0, 0xc6645a9900)
Sep 21 15:13:01 host prometheus[18734]: /home/build/packages/prometheus/tmp/build/gopath/src/github.com/prometheus/prometheus/rules/manager.go:412 +0x3c
Sep 21 15:13:01 host prometheus[18734]: created by github.com/prometheus/prometheus/rules.(*Manager).ApplyConfig
Sep 21 15:13:01 host prometheus[18734]: /home/build/packages/prometheus/tmp/build/gopath/src/github.com/prometheus/prometheus/rules/manager.go:423 +0x56b
Sep 21 15:13:03 host systemd[1]: prometheus.service: main process exited, code=exited, status=2/INVALIDARGUMENT
|
8 years ago | |
|---|---|---|
| .. | ||
| alerting.go | storage: Contextify storage interfaces. | 8 years ago |
| alerting_test.go |
…
|
|
| manager.go | rules/manager.go: Fix race between reload and stop | 8 years ago |
| manager_test.go | storage: Contextify storage interfaces. | 8 years ago |
| recording.go | storage: Contextify storage interfaces. | 8 years ago |
| recording_test.go | storage: Contextify storage interfaces. | 8 years ago |