prometheus

Commit Graph

Author	SHA1	Message	Date
Ben Ye	1e4e37144d	Fixed wrongly handled not ready TSDB on web and API. (#7182 ) * fix federate endpoint panic Signed-off-by: yeya24 <yb532204897@gmail.com> * Fixed all cases of not ready TSDB being wrongly handled. * Fixed issue for federation. * Ensured this will never happen again thanks to interfaces * Fixes same issue for stats. * Added tests for readiness. * Fixed bug in stats. It was: status.MaxTime = db.Head().MaxTime() status.MinTime = db.Head().MaxTime() Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed Brian's comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed Brian's comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 years ago
Goutham Veeramachaneni	84b4d079c8	Make sure deleted intervals are excluded from Seek (#6980 ) Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	5 years ago
Julien Pivotto	fc3fb3265a	Merge pull request #7145 from prometheus/release-2.17 Backport release 2.17 into master	5 years ago
Julien Pivotto	ed1852ab95	TSDB: Isolation: avoid creating appenderId's without appender (#7135 ) Prior to this commit we could have situations where we are creating an appenderId but never creating an appender to go with it, therefore blocking the low watermak. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
ZouYu	2b7437d60e	Fix some warnings: 'redundant type from array, slice, or map composite literal' (#7109 ) Signed-off-by: ZouYu <zouy.fnst@cn.fujitsu.com>	5 years ago
Marek Slabicki	8224ddec23	Capitalizing first letter of all log lines (#7043 ) Signed-off-by: Marek Slabicki <thaniri@gmail.com>	5 years ago
Brian Brazil	cd73b3d33e	Reduce how much old WAL we keep around. (#7098 ) Previously we were keeping up to around 6 hours of WAL around by removing 1/3 every hours. This was excessive, so switch to removing 2/3 which will up to around 3 hours of WAL around. This will roughly halve the size of the WAL and halve startup time for those who are I/O bound. This may increase the checkpoint size for those with certain churn patterns, but by much less than we're saving from the segments. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	5 years ago
Brad Walker	3348930df5	Replace fileutil.ReadDir with ioutil.ReadDir (#7029 ) (#7033 ) * tsdb: Replace fileutil.ReadDir with ioutil.ReadDir (#7029) Signed-off-by: Brad Walker <brad@bradmwalker.com> * tsdb: Remove fileutil.ReadDir (#7029) Signed-off-by: Brad Walker <brad@bradmwalker.com>	5 years ago
MengZeLee	a7982ffc0f	Fix typo (#7068 ) Fix typo. Signed-off-by: MengZn <adnt587@gmail.com>	5 years ago
Brian Brazil	7646cbca32	Use .UTC everywhere we use time.Unix (#7066 ) time.Unix attaches the local timezone, which can then leak out (e.g. in the alert json). While this is harmless, we should be consistent. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	5 years ago
Julien Pivotto	9057decce2	Merge pull request #7060 from prometheus/release-2.17 Release 2.17	5 years ago
Julien Pivotto	ceef10cee4	Reset comment Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Julien Pivotto	73228b1b68	Those links should not be reverted Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Julien Pivotto	653f343547	Revert head posting optimization This reverts commit `52630ad0c7`. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Bartlomiej Plotka	d5c33877f9	storage: Added Chunks{Queryable/Querier/SeriesSet/Series/Iteratable. Added generic Merge{SeriesSet/Querier} implementation. (#7005 ) * storage: Added Chunks{Queryable/Querier/SeriesSet/Series/Iteratable. Added generic Merge{SeriesSet/Querier} implementation. ## Rationales: In many places (e.g. chunk Remote read, Thanos Receive fetching chunk from TSDB), we operate on encoded chunks not samples. This means that we unnecessary decode/encode, wasting CPU, time and memory. This PR adds chunk iterator interfaces and makes the merge code to be reused between both seriesSets I will make the use of it in following PR inside tsdb itself. For now fanout implements it and mergers. All merges now also allows passing series mergers. This opens doors for custom deduplications other than TSDB vertical ones (e.g. offline one we have in Thanos). ## Changes * Added Chunk versions of all iterating methods. It all starts in Querier/ChunkQuerier. The plan is that Storage will implement both chunked and samples. * Added Seek to chunks.Iterator interface for iterating over chunks. * NewMergeChunkQuerier was added; Both this and NewMergeQuerier are now using generigMergeQuerier to share the code. Generic code was added. * Improved tests. * Added some TODO for further simplifications in next PRs. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed Brian's comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Moved s/Labeled/SeriesLabels as per Krasi suggestion. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Addressed Krasi's comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Second iteration of Krasi comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> * Another round of comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 years ago
Ben Kochie	fac7a4a050	Merge pull request #7037 from prometheus/bjk/golint Enable golint in CI	5 years ago
Ben Kochie	269e7c8091	Fix golint issues. Signed-off-by: Ben Kochie <superq@gmail.com>	5 years ago
Ganesh Vernekar	6fdc852813	Fix TestHeadDeleteSimple to test reloaded Head too (#7021 ) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Ganesh Vernekar	e64a149984	Close Head in DBReadOnly.FlushWAL (#7022 ) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
zhulongcheng	e813f60fd6	tsdb: fix sequence check for WAL segments (#7032 ) Signed-off-by: zhulongcheng <zhulongcheng.dev@gmail.com>	5 years ago
zhulongcheng	dbb8f5861d	tsdb: add tombstonesHeaderSize constant (#7028 ) Signed-off-by: zhulongcheng <zhulongcheng.dev@gmail.com>	5 years ago
Julien Pivotto	f1984bb007	Merge pull request #7025 from prometheus/release-2.17 Merge release 2.17 into master	5 years ago
beorn7	526cff39b9	Fix tests that were broken by #7009 Signed-off-by: beorn7 <beorn@grafana.com>	5 years ago
Bartlomiej Plotka	c4eefd1b3a	storage: Removed SelectSorted method; Simplified interface; Added requirement for remote read to sort response. This is technically BREAKING CHANGE, but it was like this from the beginning: I just notice that we rely in Prometheus on remote read being sorted. This is because we use selected data from remote reads in MergeSeriesSet which rely on sorting. I found during work on https://github.com/prometheus/prometheus/pull/5882 that we do so many repetitions because of this, for not good reason. I think I found a good balance between convenience and readability with just one method. Smaller the interface = better. Also I don't know what TestSelectSorted was testing, but now it's testing sorting. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 years ago
Callum Styan	f802f1e8ca	Fix bug with WAL watcher and Live Reader metrics usage. (#6998 ) * Fix bug with WAL watcher and Live Reader metrics usage. Calling NewXMetrics when creating a Watcher or LiveReader results in a registration error, which we're ignoring, and as a result other than the first Watcher/Reader created, we had no metrics for either. So we would only have metrics like Watcher Records Read for the first remote write config in a users config file. Signed-off-by: Callum Styan <callumstyan@gmail.com>	5 years ago
Bartlomiej Plotka	8fa4ada9ae	Merge pull request #7010 from prometheus/beorn7/fix-test Fix tests that were broken by #7009	5 years ago
Ganesh Vernekar	e50fdbc70c	Live m-mapping of chunks on disk (#6830 ) * Live m-mapping of chunks on disk Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Part 2 Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Part 3 Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Part 4 Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Attempt to fix windows bug Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
beorn7	c0ecbb38af	Fix tests that were broken by #7009 Signed-off-by: beorn7 <beorn@grafana.com>	5 years ago
Björn Rabenstein	1da83305be	Merge pull request #7009 from prometheus/release-2.17 Merge release-2.17 into master	5 years ago
zhulongcheng	5f5c7a4477	tsdb: sort checkpoints by segment number (#6987 ) Signed-off-by: zhulongcheng <zhulongcheng.dev@gmail.com>	5 years ago
Julien Pivotto	8907ba6235	Make TSDB use storage errors This fixes #6992, which was introduced by #6777. There was an intermediate component which translated TSDB errors into storage errors, but that component was deleted and this bug went unnoticed, until we were watching at the Prombench results. Without this, scrape will fail instead of dropping samples or using "Add" when the series have been garbage collected. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Julien Pivotto	0f9e78bd88	tsdb: fix races around head chunks (#6985 ) * tsdb: fix races around head chunks Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Björn Rabenstein	d80b0810c1	Move crucial actions to defer (#6918 ) With defer having less of a performance penalty, there is no reason not to do those crucial operations via defer. Context: With isolation in place, if we forget to Commit/Rollback, the low watermark will get stuck forever. The current code should not have any bugs, but moving to defer helps to avoid future bugs. This is also moving the `closeAppend` in the `Commit` implementation itself to defer. If logging to the WAL fails, we would have missed the `closeAppend`. Signed-off-by: beorn7 <beorn@grafana.com>	5 years ago
Bartlomiej Plotka	9d9c45588e	Addressed Goutham's review. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 years ago
Bartlomiej Plotka	cd9516316a	Addressed Brian's comments. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 years ago
Bartlomiej Plotka	fe802f29c9	storage: Removed SelectSorted method; Simplified interface; Added requirement for remote read to sort response. This is technically BREAKING CHANGE, but it was like this from the beginning: I just notice that we rely in Prometheus on remote read being sorted. This is because we use selected data from remote reads in MergeSeriesSet which rely on sorting. I found during work on https://github.com/prometheus/prometheus/pull/5882 that we do so many repetitions because of this, for not good reason. I think I found a good balance between convenience and readability with just one method. Smaller the interface = better. Also I don't know what TestSelectSorted was testing, but now it's testing sorting. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	5 years ago
Björn Rabenstein	6029fa0b1b	Merge pull request #6951 from prometheus/beorn7/tsdb tsdb: Do a full rollback upon commit error	5 years ago
beorn7	f6f4fd6556	tsdb: Do a full rollback upon commit error I think the previous behavior is problematic as it will leave `memSeries` around that still have `pendingCommit` set to `true`. The only case where this can happen in this code path is a failure to write to the WAL, in which case we are probably in trouble anyway. I believe, however, we should still try to do the right thing and do the full rollback. This will implicitly try to write to the WAL again, but this time without samples, which may even succeed. (But we propagate the previous error in any case.) This also adds `a.head.putSeriesBuffer(a.sampleSeries)` to Rollback, which was previously missing. Signed-off-by: beorn7 <beorn@grafana.com>	5 years ago
李国忠	261cbab8e9	remove Unused parameter 'reg' in wal.Open function (#6941 ) Signed-off-by: fuling <fuling.lgz@alibaba-inc.com>	5 years ago
johncming	df14bca643	tsdb: fix typo for wrong metric name (#6938 ) Signed-off-by: johncming <johncming@yahoo.com>	5 years ago
beorn7	0193b746b1	Defer call to iso.closeAppend This is taken from #6918. Since we probably won't merge #6918 before the relase, we have to do this bit of it as it fixes an actual bug (iso.closeAppend is not called if the append fails because of an error logging to the WAL). Signed-off-by: beorn7 <beorn@grafana.com>	5 years ago
Ganesh Vernekar	2cc32c332d	Log WAL replay duration Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
Julien Pivotto	ed623f69e2	tsdb: don't allow ingesting empty labelsets (#6891 ) * tsdb: don't allow ingesting empty labelsets When we ingest an empty labelset in the head, further blocks can not be compacted, with the error: ``` level=error ts=2020-02-27T21:26:58.379Z caller=db.go:659 component=tsdb msg="compaction failed" err="persist head block: write compaction: add series: out-of-order series added with label set \"{}\" / prev: \"{}\"" ``` We should therefore reject those invalid empty labelsets upfront. This can be reproduced with the following: ``` cat << END > prometheus.yml scrape_configs: - job_name: 'prometheus' scrape_interval: 1s basic_auth: username: test password: test metric_relabel_configs: - regex: ".*" action: labeldrop static_configs: - targets: - 127.0.1.1:9090 END ./prometheus --storage.tsdb.min-block-duration=1m ``` And wait a few minutes. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
beorn7	d9af11e636	Do not attempt isolation for appendID == 0 Signed-off-by: beorn7 <beorn@grafana.com>	5 years ago
beorn7	7f30b0984d	Implement isolation This has been ported from https://github.com/prometheus/tsdb/pull/306. Original implementation by @brian-brazil, explained in detail in the 2nd half of this talk: https://promcon.io/2017-munich/talks/staleness-in-prometheus-2-0/ The implementation was then processed by @gouthamve into the PR linked above. Relevant slide deck: https://docs.google.com/presentation/d/1-ICg7PEmDHYcITykD2SR2xwg56Tzf4gr8zfz1OerY5Y/edit?usp=drivesdk Signed-off-by: beorn7 <beorn@grafana.com> Co-authored-by: Brian Brazil <brian.brazil@robustperception.io> Co-authored-by: Goutham Veeramachaneni <gouthamve@gmail.com>	5 years ago
beorn7	6b8181370f	Fix punctuation nits Signed-off-by: beorn7 <beorn@grafana.com>	5 years ago
Peter Štibraný	1d396b96dc	Specify that returned samples must be ordered by timestamp. (#6877 ) Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	5 years ago
Marco Pracucci	c391b6ca43	Use a cryptographically random generator for ULID Signed-off-by: Marco Pracucci <marco@pracucci.com>	5 years ago
Julien Pivotto	52630ad0c7	Make head Postings only return series in time range benchmark old ns/op new ns/op delta BenchmarkQuerierSelect/Head/1of1000000-8 405805161 120436132 -70.32% BenchmarkQuerierSelect/Head/10of1000000-8 403079620 120624292 -70.07% BenchmarkQuerierSelect/Head/100of1000000-8 404678647 120923522 -70.12% BenchmarkQuerierSelect/Head/1000of1000000-8 403145813 118636563 -70.57% BenchmarkQuerierSelect/Head/10000of1000000-8 405020046 125716206 -68.96% BenchmarkQuerierSelect/Head/100000of1000000-8 426305002 175808499 -58.76% BenchmarkQuerierSelect/Head/1000000of1000000-8 619002108 567013003 -8.40% BenchmarkQuerierSelect/SortedHead/1of1000000-8 1276316086 120281094 -90.58% BenchmarkQuerierSelect/SortedHead/10of1000000-8 1282631170 121836526 -90.50% BenchmarkQuerierSelect/SortedHead/100of1000000-8 1325824787 121174967 -90.86% BenchmarkQuerierSelect/SortedHead/1000of1000000-8 1271386268 121025117 -90.48% BenchmarkQuerierSelect/SortedHead/10000of1000000-8 1280223345 130838948 -89.78% BenchmarkQuerierSelect/SortedHead/100000of1000000-8 1271401620 243635515 -80.84% BenchmarkQuerierSelect/SortedHead/1000000of1000000-8 1360256090 1307744674 -3.86% BenchmarkQuerierSelect/Block/1of1000000-8 748183120 707888498 -5.39% BenchmarkQuerierSelect/Block/10of1000000-8 741084129 716317249 -3.34% BenchmarkQuerierSelect/Block/100of1000000-8 722157273 735624256 +1.86% BenchmarkQuerierSelect/Block/1000of1000000-8 727587744 731981838 +0.60% BenchmarkQuerierSelect/Block/10000of1000000-8 727518578 726860308 -0.09% BenchmarkQuerierSelect/Block/100000of1000000-8 765577046 757382386 -1.07% BenchmarkQuerierSelect/Block/1000000of1000000-8 1126722881 1084779083 -3.72% benchmark old allocs new allocs delta BenchmarkQuerierSelect/Head/1of1000000-8 4000018 24 -100.00% BenchmarkQuerierSelect/Head/10of1000000-8 4000036 82 -100.00% BenchmarkQuerierSelect/Head/100of1000000-8 4000216 625 -99.98% BenchmarkQuerierSelect/Head/1000of1000000-8 4002016 6028 -99.85% BenchmarkQuerierSelect/Head/10000of1000000-8 4020016 60037 -98.51% BenchmarkQuerierSelect/Head/100000of1000000-8 4200016 600047 -85.71% BenchmarkQuerierSelect/Head/1000000of1000000-8 6000016 6000016 +0.00% BenchmarkQuerierSelect/SortedHead/1of1000000-8 4000055 28 -100.00% BenchmarkQuerierSelect/SortedHead/10of1000000-8 4000073 87 -100.00% BenchmarkQuerierSelect/SortedHead/100of1000000-8 4000253 630 -99.98% BenchmarkQuerierSelect/SortedHead/1000of1000000-8 4002053 6036 -99.85% BenchmarkQuerierSelect/SortedHead/10000of1000000-8 4020053 60054 -98.51% BenchmarkQuerierSelect/SortedHead/100000of1000000-8 4200053 600074 -85.71% BenchmarkQuerierSelect/SortedHead/1000000of1000000-8 6000053 6000053 +0.00% BenchmarkQuerierSelect/Block/1of1000000-8 6000021 6000021 +0.00% BenchmarkQuerierSelect/Block/10of1000000-8 6000057 6000057 +0.00% BenchmarkQuerierSelect/Block/100of1000000-8 6000417 6000417 +0.00% BenchmarkQuerierSelect/Block/1000of1000000-8 6004017 6004017 +0.00% BenchmarkQuerierSelect/Block/10000of1000000-8 6040017 6040017 +0.00% BenchmarkQuerierSelect/Block/100000of1000000-8 6400017 6400017 +0.00% BenchmarkQuerierSelect/Block/1000000of1000000-8 10000018 10000018 +0.00% benchmark old bytes new bytes delta BenchmarkQuerierSelect/Head/1of1000000-8 176001177 1392 -100.00% BenchmarkQuerierSelect/Head/10of1000000-8 176002329 4368 -100.00% BenchmarkQuerierSelect/Head/100of1000000-8 176013849 33520 -99.98% BenchmarkQuerierSelect/Head/1000of1000000-8 176129056 321456 -99.82% BenchmarkQuerierSelect/Head/10000of1000000-8 177281049 3427376 -98.07% BenchmarkQuerierSelect/Head/100000of1000000-8 188801049 35055408 -81.43% BenchmarkQuerierSelect/Head/1000000of1000000-8 304001059 304001049 -0.00% BenchmarkQuerierSelect/SortedHead/1of1000000-8 229192188 2488 -100.00% BenchmarkQuerierSelect/SortedHead/10of1000000-8 229193340 5568 -100.00% BenchmarkQuerierSelect/SortedHead/100of1000000-8 229204860 35536 -99.98% BenchmarkQuerierSelect/SortedHead/1000of1000000-8 229320060 345104 -99.85% BenchmarkQuerierSelect/SortedHead/10000of1000000-8 230472060 3894672 -98.31% BenchmarkQuerierSelect/SortedHead/100000of1000000-8 241992060 40511632 -83.26% BenchmarkQuerierSelect/SortedHead/1000000of1000000-8 357192060 357192060 +0.00% BenchmarkQuerierSelect/Block/1of1000000-8 227201516 227201506 -0.00% BenchmarkQuerierSelect/Block/10of1000000-8 227203057 227203041 -0.00% BenchmarkQuerierSelect/Block/100of1000000-8 227217161 227217165 +0.00% BenchmarkQuerierSelect/Block/1000of1000000-8 227358279 227358289 +0.00% BenchmarkQuerierSelect/Block/10000of1000000-8 228769485 228769475 -0.00% BenchmarkQuerierSelect/Block/100000of1000000-8 242881487 242881477 -0.00% BenchmarkQuerierSelect/Block/1000000of1000000-8 384001705 384001705 +0.00% Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	5 years ago
Brian Brazil	cebe36c7d5	Make head Postings only return series in time range Series() will fetch all the metadata for a series, even if it's going to be filtered later due to time ranges. For 1M series we save ~1.1s if you only needed some of the data, but take an extra ~.2s if you did want everything. benchmark old ns/op new ns/op delta BenchmarkHeadSeries/1of1000000-4 1443715987 131553480 -90.89% BenchmarkHeadSeries/10of1000000-4 1433394040 130730596 -90.88% BenchmarkHeadSeries/100of1000000-4 1437444672 131360813 -90.86% BenchmarkHeadSeries/1000of1000000-4 1438958659 132573137 -90.79% BenchmarkHeadSeries/10000of1000000-4 1438061766 145742377 -89.87% BenchmarkHeadSeries/100000of1000000-4 1455060948 281659416 -80.64% BenchmarkHeadSeries/1000000of1000000-4 1633524504 1803550153 +10.41% benchmark old allocs new allocs delta BenchmarkHeadSeries/1of1000000-4 4000055 28 -100.00% BenchmarkHeadSeries/10of1000000-4 4000073 87 -100.00% BenchmarkHeadSeries/100of1000000-4 4000253 630 -99.98% BenchmarkHeadSeries/1000of1000000-4 4002053 6036 -99.85% BenchmarkHeadSeries/10000of1000000-4 4020053 60054 -98.51% BenchmarkHeadSeries/100000of1000000-4 4200053 600074 -85.71% BenchmarkHeadSeries/1000000of1000000-4 6000053 6000094 +0.00% benchmark old bytes new bytes delta BenchmarkHeadSeries/1of1000000-4 229192184 2488 -100.00% BenchmarkHeadSeries/10of1000000-4 229193336 5568 -100.00% BenchmarkHeadSeries/100of1000000-4 229204856 35536 -99.98% BenchmarkHeadSeries/1000of1000000-4 229320056 345104 -99.85% BenchmarkHeadSeries/10000of1000000-4 230472056 3894673 -98.31% BenchmarkHeadSeries/100000of1000000-4 241992056 40511632 -83.26% BenchmarkHeadSeries/1000000of1000000-4 357192056 402380440 +12.65% Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	5 years ago

1 2 3 4

156 Commits (bc1d7d7f270bf0c4553d3a972bcba72a3acf712a)