prometheus

Commit Graph

Author	SHA1	Message	Date
🌲 Harry 🌊 John 🏔	d5f6887294	Pass limit param as hint to storage.Querier Signed-off-by: 🌲 Harry 🌊 John 🏔 <johrry@amazon.com>	5 months ago
Oleg Zaytsev	fd1a89b7c8	Pass affected labels to `MemPostings.Delete()` (#14307 ) * Pass affected labels to MemPostings.Delete As suggested by @bboreham, we can track the labels of the deleted series and avoid iterating through all the label/value combinations. This looks much faster on the MemPostings.Delete call. We don't have a benchmark on stripeSeries.gc() where we'll pay the price of iterating the labels of each one of the deleted series. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	5 months ago
Ben Ye	0e6fca8e76	add unit test Signed-off-by: Ben Ye <benye@amazon.com>	6 months ago
Ben Ye	e7db2e30a4	fix check context cancellation not incrementing count Signed-off-by: Ben Ye <benye@amazon.com>	6 months ago
Ben Ye	5a218708f1	tsdb: Extend compactor interface to allow compactions to create multiple output blocks (#14143 ) * add hook to allow head compaction to create multiple output blocks Signed-off-by: Ben Ye <benye@amazon.com> * change Compact interface; remove BlockPopulator changes Signed-off-by: Ben Ye <benye@amazon.com> * rebase main Signed-off-by: Ben Ye <benye@amazon.com> * fix lint Signed-off-by: Ben Ye <benye@amazon.com> * fix unit test Signed-off-by: Ben Ye <benye@amazon.com> * address feedbacks; add unit test Signed-off-by: Ben Ye <benye@amazon.com> * Apply suggestions from code review Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Update tsdb/compact_test.go Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> --------- Signed-off-by: Ben Ye <benye@amazon.com> Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>	6 months ago
Sebastian Rabenhorst	05380aa0ac	agent db: make rejecting ooo samples configurable (#14094 ) feat: Make OOO ingestion time window configurable for Prometheus Agent. Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>	6 months ago
Oleg Zaytsev	64a9abb8be	Change LabelValuesFor() to accept index.Postings (#14280 ) The only call we have to LabelValuesFor() has an index.Postings, and we expand it to pass to this method, which will iterate over the values. That's a waste of resources: we can iterate on the index.Postings directly. If there's any downstream implementation that has a slice of series, they can always do an index.ListPostings from them: doing that is cheaper than expanding an abstract index.Postings. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	6 months ago
Bryan Boreham	c5d923aa7c	Merge pull request #14279 from colega/fix-label-names-for-not-found headIndexReader.LabelNamesFor: skip not found series	6 months ago
Oleg Zaytsev	10a3c7220b	`MemPostings.PostingsForLabelMatching()`: don't hold the mutex while matching (#14286 ) * MemPostings.PostingsForLabelMatching: let mutex go This changes the `MemPostings.PostingsForLabelMatching` implementation to stop holding the read mutex while matching the label values. We've seen that this method can be slow when the matcher is expensive, that's why we even added a context expiration check. However, there are critical process that might be waiting on this mutex: writes (adding new series) and compaction (deleting the garbage-collected ones), so we should avoid holding it for a long period of time. Given that we've copied the values to a slice anyway, there's no need to hold the lock while matching. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	6 months ago
Oleg Zaytsev	2dc177d8af	`MemPostings.Delete()`: reduce locking/unlocking (#13286 ) * MemPostings: reduce locking/unlocking MemPostings.Delete is called from Head.gc(), i.e. it gets the IDs of the series that have churned. I'd assume that many label values aren't affected by that churn at all, so it doesn't make sense to touch the lock while checking them. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	6 months ago
Oleg Zaytsev	d0d361da53	headIndexReader.LabelNamesFor: skip not found series It's quite common during the compaction cycle to hold series IDs for series that aren't in the TSDB head anymore. We shouldn't fail if that happens, as the caller has no way to figure out which one of the IDs doesn't exist. Fixes https://github.com/prometheus/prometheus/issues/14278 Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	6 months ago
Ben Ye	8a08f452b6	tsdb: Allow passing a custom compactor to override the default one (#14113 ) * expose hook in tsdb to allow customizing compactor Signed-off-by: Ben Ye <benye@amazon.com> * address comment Signed-off-by: Ben Ye <benye@amazon.com> --------- Signed-off-by: Ben Ye <benye@amazon.com>	6 months ago
Bryan Boreham	42b546a43d	tsdb: add details to duplicate sample error (#13277 ) Now the error will include the timestamp and the existing and new values. When you are trying to track down the source of this error, it can be useful to see that the values are close, or alternating, or something else. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	6 months ago
Arve Knudsen	b8b9015e38	tsdb/index: Fix TestReader_PostingsForLabelMatchingHonorsContextCancel Fix number of series in TestReader_PostingsForLabelMatchingHonorsContextCancel (off by one). Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	6 months ago
Bryan Boreham	3ee52abb53	[ENHANCEMENT] TSDB: Save map lookup on validation Goes faster. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	6 months ago
Bryan Boreham	7d98487447	[ENHANCEMENT] TSDB: let Resize re-use buffer This saves having to zero the buffer every time. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	6 months ago
Bryan Boreham	c0bb156eca	[ENHANCEMENT] TSDB: Eliminate pointer when storing exemplars Saves memory and effort. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	6 months ago
Bryan Boreham	3eb5581877	[ENHANCEMENT] TSDB: Reduce map lookups on exemplar index In many cases we already have a pointer to the entry. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	6 months ago
Bryan Boreham	f0c50b5a66	[Test] TSDB: BenchmarkResizeExemplar multiple per series One exemplar per series is not a typical workload. Make it the same as `BenchmarkAddExemplar`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	6 months ago
Bryan Boreham	929fbf860e	[Test] TSDB: let BenchmarkAddExemplar reuse slots Test with different amounts of capacity and exemplars, so that sometimes new exemplars are evicting older exemplars. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	6 months ago
Ben Ye	6683895620	optimize regex matching for empty label values in posting match (#14075 ) Also update tests. Signed-off-by: Ben Ye <benye@amazon.com>	6 months ago
Arve Knudsen	b2396c0c8f	Upgrade to golangci-lint v1.59.0 Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	6 months ago
Alan Protasio	8894d65cd6	Fix head stats and hooks when replaying a corrupted snapshot (#14079 ) * Fixing head stats and hooks when replaying a corrupted snapshot Signed-off-by: alanprot <alanprot@gmail.com> * Fixing create/removed series metrics Signed-off-by: alanprot <alanprot@gmail.com> * Refactoring to have common code between gc and flush method Signed-off-by: alanprot <alanprot@gmail.com> * Update tsdb/head.go Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> * refactor Signed-off-by: alanprot <alanprot@gmail.com> * Update tsdb/head_test.go Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> * Update tsdb/head_test.go Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> --------- Signed-off-by: alanprot <alanprot@gmail.com> Signed-off-by: Alan Protasio <alanprot@gmail.com> Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>	6 months ago
Björn Rabenstein	3119b8a055	Merge pull request #13218 from machine424/ro-promtool Make DBReadOnly more RO	6 months ago
Oleg Zaytsev	fe9cb5a803	Check context every 128 labels instead of 100 (#14118 ) Follow up on https://github.com/prometheus/prometheus/pull/14096 As promised, I bring a benchmark, which shows a very small improvement if context is checked every 128 iterations of label instead of every 100. It's much easier for a computer to check modulo 128 than modulo 100. This is a very small 0-2% improvement but I'd say this is one of the hottest paths of the app so this is still relevant. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	6 months ago
Arve Knudsen	5ca56eeb6b	tsdb/index: Refactor Reader tests (#14071 ) tsdb/index: Refactor Reader tests Co-authored-by: Björn Rabenstein <github@rabenste.in> Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: Björn Rabenstein <github@rabenste.in>	7 months ago
Oleksandr Redko	f10c3454e9	Enable perfsprint linter and fix up code Signed-off-by: Oleksandr Redko <oleksandr.red+github@gmail.com>	7 months ago
György Krajcsovits	b215a41be4	tsdb/index/postings: fix missing lock unlock Followup to #14096 Unfortunately the previous PR introduced this bug by not releasing the lock before returning. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	7 months ago
George Krajcsovits	fdaafdb041	tsdb: check for context cancel before regex matching postings (#14096 ) * tsdb: check for context cancel before regex matching postings Regex matching can be heavy if the regex takes a lot of cycles to evaluate and we can get stuck evaluating postings for a long time without this fix. The constant checkContextEveryNIterations=100 may be changed later. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	7 months ago
Arve Knudsen	5c4310aa37	[ENHANCEMENT] TSDB: Optimize querying with regexp matchers Add method `PostingsForLabelMatching` to `tsdb.IndexReader`, to obtain postings for labels with a certain name and values accepted by a provided callback, and use it from `tsdb.PostingsForMatchers`. The intention is to optimize regexp matcher paths, especially not having to load all label values before matching on them. Plus tests, and refactor some `tsdb/index.Reader` methods. Benchmarking shows memory reduction up to ~100%, and speedup of up to ~50%. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>	7 months ago
Arve Knudsen	d699dc3c77	Fix language in docs and comments (#14041 ) Fix language in docs and comments --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: Björn Rabenstein <github@rabenste.in>	7 months ago
Arve Knudsen	108a6bc9f6	tsdb/chunkenc.Pool: Refactor Get and Put Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	7 months ago
Alan Protasio	d15869af32	Avoid creating new slices for labels values on postings for matchers (#13958 ) * Avoid creating new slices for labels values on postings for matchers Signed-off-by: alanprot <alanprot@gmail.com> * refactor Signed-off-by: alanprot <alanprot@gmail.com> --------- Signed-off-by: alanprot <alanprot@gmail.com>	7 months ago
Arthur Silva Sens	b5b5e1e5ae	Merge pull request #13919 from GiedriusS/dont_forget_to_unregister tsdb/wlog: unregister metrics on WL close	7 months ago
Giedrius Statkevičius	bdf490726a	tsdb/wlog: add test for metrics unregistering Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>	7 months ago
machine424	c5a1cc9148	chore(tsdb): add a sandboxDir to DBReadOnly, the directory can be used for transient file writes. use it in loadDataAsQueryable to make sure the RO Head doesn't truncate or cut new chunks in data/chunks_head/. add a -sandbox-dir-root flag to "promtool tsdb dump/dump-openmetrics" to control the root of that sandbox dirrectory. Signed-off-by: machine424 <ayoubmrini424@gmail.com>	8 months ago
Giedrius Statkevičius	3b8fe00767	tsdb/wlog: unregister metrics on WL close Thanos can create and destroy TSDBs dynamically, and once a TSDB disappears its files are deleted. Calculating the size of the WAL then fails with errors like: ``` msg: "Failed to calculate size of "wal" dir", "err": "lstat /tsdbdir/wal: no such file or directory", "caller": "wlog.go:271" ``` Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>	8 months ago
Matthieu MOREL	6f595c6762	golangci-lint: enable whitespace linter (#13905 ) Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	8 months ago
Jonathan Halterman	633224886a	Write out of order hint when initially creating meta file (#13894 ) Signed-off-by: Jonathan Halterman <jonathan@grafana.com> Signed-off-by: Jonathan Halterman <jhalterman@gmail.com> Co-authored-by: Jesus Vazquez <jesusvazquez@users.noreply.github.com>	8 months ago
Łukasz Mierzwa	277f04f0c4	Stop compactions if there's a block to write (#13754 ) * Stop compactions if there's a block to write db.Compact() checks if there's a block to write with HEAD chunks before calling db.compactBlocks(). This is to ensure that if we need to write a block then it happens ASAP, otherwise memory usage might keep growing. But what can also happen is that we don't need to write any block, we start db.compactBlocks(), compaction takes hours, and in the meantime HEAD needs to write out chunks to a block. This can be especially problematic if, for example, you run Thanos sidecar that's uploading block, which requires that compactions are disabled. Then you disable Thanos sidecar and re-enable compactions. When db.compactBlocks() is finally called it might have a huge number of blocks to compact, which might take a very long time, during which HEAD cannot write out chunks to a new block. In such case memory usage will keep growing until either: - compactions are finally finished and HEAD can write a block - we run out of memory and Prometheus gets OOM-killed This change adds a check for pending HEAD block writes inside db.compactBlocks(), so that we bail out early if there are still compactions to run, but we also need to write a new block. Also add a test for compactBlocks. --------- Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com> Signed-off-by: Lukasz Mierzwa <lukasz@cloudflare.com>	8 months ago
Bryan Boreham	fc567a1df8	Merge pull request #13889 from komisan19/refactor/utilize_standard_functions_max/min refactor: utilize standard functions max/min in promtool and tsdb	8 months ago
Julien	8eb9228af8	Merge pull request #13864 from yeya24/expose-compactor-metrics Expose compactor metrics	8 months ago
Jonathan Halterman	113938aeb8	Log out of order when writing a block (#13888 ) Signed-off-by: Jonathan Halterman <jonathan@grafana.com>	8 months ago
komisan19	0249e080b4	refactor: utilize standard functions max/min Signed-off-by: komisan19 <18901496+komisan19@users.noreply.github.com>	8 months ago
Nicolas Takashi	8125634086	[refactor] moving mergedOOOChunks Iterator (#13881 ) Signed-off-by: Nicolas Takashi <nicolas.tcs@hotmail.com>	8 months ago
carehabit	a672662073	all: fix some typos (#13863 ) Signed-off-by: carehabit <shenyuting@outlook.com>	8 months ago
Ben Ye	ded35ef20d	expose compactor metrics Signed-off-by: Ben Ye <benye@amazon.com>	8 months ago
Nicolas Takashi	0b762db154	[refactor] moving mergedOOOChunks to ooo_head_read Signed-off-by: Nicolas Takashi <nicolas.tcs@hotmail.com>	8 months ago
Arve Knudsen	35aab01de0	tsdb/wlog.Checkpoint: Handle also float histograms Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	8 months ago
Bartlomiej Plotka	de05d5e11b	Merge pull request #13776 from aknuds1/arve/checkpoint-histogram-samples tsdb/wlog.Checkpoint: Fix counting of histogram samples	8 months ago

1 2 3 4 5 ...

1039 Commits (d5f68872943062d4535ef8cffc53b4613a541d11)