prometheus

Commit Graph

Author	SHA1	Message	Date
Ganesh Vernekar	2836e5513f	Add support to query unmerged, unsorted chunks (#299 ) * Add support to query unmerged, unsorted chunks Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix unrelated lint issue Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	2 years ago
Ganesh Vernekar	00b379c3a5	Include out of order samples in BenchmarkQueries (#286 ) Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	2 years ago
Ganesh Vernekar	ff0dc75758	Replay WBL even if OOO Time Window is 0 (#296 ) * Replay WBL even if OOO Time Window is 0 Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Apply feedback Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	2 years ago
Peter Štibraný	ae49ab5ea8	Merge remote-tracking branch 'upstream/main' into update-upstream-prometheus	2 years ago
Ganesh Vernekar	a632c73352	Simplify how OutOfOrderTimeWindow works (#285 ) * Simplify how OutOfOrderTimeWindow works Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Update test Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	2 years ago
Ganesh Vernekar	c6f3d4ab33	Remove temporary patch for out-of-order (#283 ) * Remove temporary patch for out-of-order Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Remove ooo_wbl patch and fix tests Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	2 years ago
Jesus Vazquez	6cfe44d7fd	WaitUntilIdle optimize idling time (#10878 ) Relates to @bboreham optimization in https://github.com/prometheus/prometheus/pull/10859 Bryan did reduce the sleep time improving the deltas on the benchmark by quite a lot. However I've been working on a similar implementation for out of order and I noticed that we actually get into this method thousands of times. @ywwg had the brilliant idea of not always sleeping before the select but actually make it a case in the select so we only sleep if we need to. The benchmark deltas are amazing ``` ❯ benchstat old_implementation.txt new_implementation_using_time_after.txt name old time/op new time/op delta LoadWAL/batches=10,seriesPerBatch=100,samplesPerSeries=7200,exemplarsPerSeries=0,mmappedChunkT=0-8 521ms ±25% 253ms ± 6% -51.47% (p=0.008 n=5+5) LoadWAL/batches=10,seriesPerBatch=100,samplesPerSeries=7200,exemplarsPerSeries=36,mmappedChunkT=0-8 773ms ± 3% 369ms ±31% -52.23% (p=0.008 n=5+5) LoadWAL/batches=10,seriesPerBatch=100,samplesPerSeries=7200,exemplarsPerSeries=72,mmappedChunkT=0-8 592ms ±28% 297ms ±28% -49.80% (p=0.008 n=5+5) LoadWAL/batches=10,seriesPerBatch=100,samplesPerSeries=7200,exemplarsPerSeries=360,mmappedChunkT=0-8 547ms ± 2% 999ms ±187% ~ (p=0.690 n=5+5) LoadWAL/batches=10,seriesPerBatch=10000,samplesPerSeries=50,exemplarsPerSeries=0,mmappedChunkT=0-8 11.3s ± 4% 1.3s ±44% -88.48% (p=0.008 n=5+5) LoadWAL/batches=10,seriesPerBatch=10000,samplesPerSeries=50,exemplarsPerSeries=2,mmappedChunkT=0-8 11.1s ± 1% 1.2s ±20% -89.08% (p=0.008 n=5+5) LoadWAL/batches=10,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=0,mmappedChunkT=0-8 1.24s ± 3% 0.18s ± 7% -85.76% (p=0.008 n=5+5) LoadWAL/batches=10,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=2,mmappedChunkT=0-8 1.24s ± 2% 0.18s ± 5% -85.24% (p=0.008 n=5+5) LoadWAL/batches=10,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=5,mmappedChunkT=0-8 1.23s ± 5% 0.27s ±33% -77.73% (p=0.008 n=5+5) LoadWAL/batches=10,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=24,mmappedChunkT=0-8 1.28s ± 1% 0.36s ± 7% -71.51% (p=0.008 n=5+5) LoadWAL/batches=100,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=0,mmappedChunkT=3800-8 12.1s ± 1% 3.1s ± 6% -74.33% (p=0.008 n=5+5) LoadWAL/batches=100,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=2,mmappedChunkT=3800-8 12.1s ± 1% 3.4s ± 4% -71.94% (p=0.008 n=5+5) LoadWAL/batches=100,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=5,mmappedChunkT=3800-8 12.1s ± 1% 3.8s ±17% -68.35% (p=0.008 n=5+5) LoadWAL/batches=100,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=24,mmappedChunkT=3800-8 12.4s ± 1% 4.0s ±18% -67.71% (p=0.008 n=5+5) ``` Benchmarked on Linux ``` goos: linux goarch: amd64 pkg: github.com/prometheus/prometheus/tsdb cpu: 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz ``` Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>	2 years ago
Julien Pivotto	bacd776356	Merge pull request #10907 from damnever/fix/panic Fix panic if series is not found when deleting series	2 years ago
Peter Štibraný	ffc60d8397	Reduce chunk write queue memory usage 2 (#10874 ) * Job queue This PR reimplements chan chunkWriteJob with custom buffered queue that should use less memory, because it doesn't preallocate entire buffer for maximum queue size at once. Instead it allocates individual "segments" with smaller size. As elements are added to the queue, they fill individual segments. When elements are removed from the queue (and segments), empty segments can be thrown away. This doesn't change memory usage of the queue when it's full, but should decrease its memory footprint when it's empty (queue will keep max 1 segment in such case). Signed-off-by: Peter Štibraný <pstibrany@gmail.com> * Modify test to work with low resolution timer. Signed-off-by: Peter Štibraný <pstibrany@gmail.com> * Improve comments. Signed-off-by: Peter Štibraný <pstibrany@gmail.com>	2 years ago
Ganesh Vernekar	5e8406a1d4	Avoid gaps in in-order data after restart with out-of-order enabled (#277 ) * Avoid gaps in in-order data after restart with out-of-order enabled Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix tests, do the temporary patch only if OOO is enabled Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Avoid Peter's confusion Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Use latest OutOfOrderTimeWindow Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	2 years ago
Jesus Vazquez	1446b53d87	Merge pull request #276 from grafana/jvp/rename-oooallowance-to-oootimewindow Rename OutOfOrderAllowance to OutOfOrderTimeWindow	2 years ago
Ganesh Vernekar	abde1e0ba1	Update MinOOOTime and MaxOOOTime properly after restart (#275 ) Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	2 years ago
Jesus Vazquez	e70e769889	Rename OutOfOrderAllowance to OutOfOrderTimeWindow After review Allowance is perhaps a bit misleading so we've decided to replace it with a more common term like TimeWindow.	2 years ago
Xiaochao Dong (@damnever)	6b042da2d8	Fix panic if series is not found when deleting series Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>	2 years ago
Ganesh Vernekar	df59320886	Add out-of-order sample support to the TSDB (#269 ) This implementation is based on this design doc: https://docs.google.com/document/d/1Kppm7qL9C-BJB1j6yb6-9ObG3AbdZnFUBYPNNWwDBYM/edit?usp=sharing This commit adds support to accept out-of-order ("OOO") sample into the TSDB up to a configurable time allowance. If OOO is enabled, overlapping querying are automatically enabled. Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Jesus Vazquez <jesus.vazquez@grafana.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Dieter Plaetinck <dieter@grafana.com>	2 years ago
Steve Azzopardi	04fe2c9522	fix(tsdb): inc mmap corruption counter on mmap out of sequence error (#10406 ) What --- When we see out of sequence chunks increase the chunk corruption counter to indicate that one of the chunks was corrupted. Reference: https://github.com/prometheus/prometheus/pull/10406#issuecomment-1142595527 Signed-off-by: Steve Azzopardi <steveazz@outlook.com>	2 years ago
Peter Štibraný	03a2313f7a	Reduce chunk write queue memory usage (#10873 ) * dont waste space on the chunkRefMap * add time factor * add comments * better readability * add instrumentation and more comments * formatting * uppercase comments * Address review feedback. Renamed "free" to "shrink" everywhere, updated comments and threshold to 1000. * double space Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> Co-authored-by: Peter Štibraný <pstibrany@gmail.com> Co-authored-by: Mauro Stettler <mauro.stettler@gmail.com>	2 years ago
Bryan Boreham	9f77d23889	tsdb: commit data periodically in CreateBlock (#10788 ) To avoid building up data in memory, commit and make a new appender periodically. The number `commitAfter = 10000` was chosen arbitrarily; testing with 10x more or less gives slightly worse results. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Łukasz Mierzwa	d65f037def	Don't increment prometheus_tsdb_compactions_failed_total when context is canceled (#10772 ) When restarting Prometheus I sometimes see: caller=db.go:832 level=error component=tsdb msg="compaction failed" err="compact head: persist head block: 2 errors: populate block: context canceled; context canceled" And prometheus_tsdb_compactions_failed_total metric gets incremented. This makes it more difficult to write alerts based on prometheus_tsdb_compactions_failed_total metric since any restart can trigger it. Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	2 years ago
Peter Štibraný	cf7aeb59a7	Merge remote-tracking branch 'upstream/main' into update-prometheus	2 years ago
Jesus Vazquez	06f1d3c349	Merge pull request #251 from grafana/codesome/ooopatch Add an option to enable overlapping compaction separately with overlapping queries	2 years ago
Bryan Boreham	542b9ecdbd	tsdb: reduce sleep time when reading WAL (#10859 ) The code sleeps for a short time to allow goroutines to finish, however it seems the duration can be reduced a lot, speeding up the reading process. I checked using some WAL data from production, and the queue is almost always empty at the time we enter `waitForIdle()` so there is no danger of spinning in the tight loop. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Ganesh Vernekar	0eb828c179	Add an option to enable overlapping compaction separately with overlapping queries Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	2 years ago
Peter Štibraný	d051065441	Remove use of io/ioutil.	3 years ago
Peter Štibraný	9d51bf50db	Merge upstream Prometheus	3 years ago
Peter Štibraný	55236be04a	Fix comments. (#248 )	3 years ago
Bryan Boreham	9f79a6f4b5	tsdb: faster CRC check by avoiding allocations (#10789 ) Instead of creating a new hashing object every time, call `crc32.Checksum` which computes the answer without allocations. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Peter Štibraný	1e2d2fb2d8	Job queue (#247 ) This PR reimplements chan chunkWriteJob with custom buffered queue that should use less memory, because it doesn't preallocate entire buffer for maximum queue size at once. Instead it allocates individual "segments" with smaller size. As elements are added to the queue, they fill individual segments. When elements are removed from the queue (and segments), empty segments can be thrown away. This doesn't change memory usage of the queue when it's full, but should decrease its memory footprint when it's empty (queue will keep max 1 segment in such case).	3 years ago
Mauro Stettler	459f59935c	Reduce chunk write queue memory usage (#131 ) * dont waste space on the chunkRefMap * add time factor * add comments * better readability * add instrumentation and more comments * formatting * uppercase comments * Address review feedback. Renamed "free" to "shrink" everywhere, updated comments and threshold to 1000. * double space Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com> Co-authored-by: Peter Štibraný <pstibrany@gmail.com>	3 years ago
Matej Gera	1dd247f68b	Remote Write: Rename confusing `walDir` parameter to `dir` (#10464 ) * Rename walDir parameter to dir Signed-off-by: Matej Gera <matejgera@gmail.com> * Improve NewQueueManager comment Signed-off-by: Matej Gera <matejgera@gmail.com>	3 years ago
David Leadbeater	57f4aab27d	Update godoc links and remove note about TSDB versioning (#10754 ) Signed-off-by: David Leadbeater <dgl@dgl.cx>	3 years ago
maizige	10b677b826	fix typo (#10696 ) Update doc comment Signed-off-by: gemaizi <864321211@qq.com>	3 years ago
Filip Petkovski	d3cb39044e	Fix typo in symbol table size exceeded error message (#10746 ) This commit fixes a typo when reporting an error that the the symbols table size has been exceeded. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>	3 years ago
Julien Pivotto	6e3a0efe40	Make necessary change to compile promql parser to wasm (#10683 ) Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>	3 years ago
Matthias Rampke	78f2645787	test(tsdb): break up repeated test to avoid timeout (#10671 ) On macOS, the TestTombstoneCleanRetentionLimitsRace performs very poorly. It takes more than a second to write out one block, and as it writes 400 of them, we run into the 10-minute test timeout frequently. While this doesn't fix the actual performance issue, breaking each iteration into a subtest makes the test pass reliably (because each iteration comfortably finishes in under a minute). Related report: https://groups.google.com/g/prometheus-developers/c/jxQ6Ayg6VJ4/m/03H_DS9PDAAJ Signed-off-by: Matthias Rampke <matthias@prometheus.io>	3 years ago
Łukasz Mierzwa	88f9b248b4	Correctly format error message (#10669 ) Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	3 years ago
Bryan Boreham	4b9f248e85	unit tests: make all Labels sorted alphabetically (#10532 ) "Labels is a sorted set of labels. Order has to be guaranteed upon instantiation." says the comment, so fix all the tests that break this rule. For `BenchmarkLabelValuesWithMatchers()` and `BenchmarkHeadLabelValuesWithMatchers()` the amount of work done changes significantly if you put the labels in order, because all series refs get neatly partitioned by the `tens` label, so I renamed the labels to maintain the previous behaviour. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Matthieu MOREL	e2ede285a2	refactor: move from io/ioutil to io and os packages (#10528 ) * refactor: move from io/ioutil to io and os packages * use fs.DirEntry instead of os.FileInfo after os.ReadDir Signed-off-by: MOREL Matthieu <matthieu.morel@cnp.fr>	3 years ago
Mauro Stettler	64e6c171c2	Merge pull request #216 from grafana/merge_upstream add option to use the new chunk disk mapper from upstream	3 years ago
Mauro Stettler	55cbbafe38	update comment Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>	3 years ago
Oleg Zaytsev	9d66af50a8	Fix TestMemSeries_append_atVariableRate Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	3 years ago
Oleg Zaytsev	ac87e2d4d6	Merge remote-tracking branch 'prometheus/main' into update-prometheus	3 years ago
Oleg Zaytsev	af0f6da5cb	Fix chunk overflow appending samples at a variable rate (#10607 ) * Add a test with variable samples rate append This test overflows the chunk created in memseries, and the total amount of samples in the (only) mmapped chunk is 29, instead of the 65565 appended ones. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Cut new chunk when rate prediction was wrong When appending samples at a slow rate, and then appending at a higher rate, the prediction we made to cut a new chunk is no longer valid. Sometimes this can even cause an overflow in the chunk, if more samples than uint16 can hold are appended. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Improve comment on 2samplesPerChunk Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> Assert that all chunks have less than 240 samples Also, trigger new chunk at 240, not at more than 240 Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	3 years ago
Mauro Stettler	00f1b1556c	undo renaming Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>	3 years ago
Mauro Stettler	2eef3e76b8	check return status of cutnewfile Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>	3 years ago
Mauro Stettler	3fa636a799	fix unit test Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>	3 years ago
Mauro Stettler	04114aa2e6	add old chunk disk mapper back Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>	3 years ago
Mauro Stettler	bd50b04fed	remove old chunk disk mapper initialization and replace it with the upstream one Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>	3 years ago
Paschalis Tsilias	40c1efe8bc	tsdb/agent: Ignore duplicate exemplars (#10595 ) * tsdb/agent: Ignore duplicate exemplars Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Make each exemplar unique in TestCommit Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Re-Trigger CI for Windows and UI-related steps Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Change test comment to properly re-trigger pipeline Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com> * Defer Close() calls for test agent and segment reader Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>	3 years ago
Jesus Vazquez	7106db9303	Merge remote-tracking branch 'upstream/main' into jvp/merge-prometheus-main	3 years ago

1 2 3 4 5 ...

566 Commits (2836e5513f1bc591535a859f5d41154a75e7c6bc)