prometheus

Commit Graph

Author	SHA1	Message	Date
Oleg Zaytsev	726ed124e4	Replace `ListPostings.Seek`'s binary search call by the generic `slices.BinarySearch` (#14393 ) Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	5 months ago
Oleg Zaytsev	fd1a89b7c8	Pass affected labels to `MemPostings.Delete()` (#14307 ) * Pass affected labels to MemPostings.Delete As suggested by @bboreham, we can track the labels of the deleted series and avoid iterating through all the label/value combinations. This looks much faster on the MemPostings.Delete call. We don't have a benchmark on stripeSeries.gc() where we'll pay the price of iterating the labels of each one of the deleted series. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	5 months ago
Ben Ye	0e6fca8e76	add unit test Signed-off-by: Ben Ye <benye@amazon.com>	5 months ago
Ben Ye	e7db2e30a4	fix check context cancellation not incrementing count Signed-off-by: Ben Ye <benye@amazon.com>	5 months ago
Oleg Zaytsev	64a9abb8be	Change LabelValuesFor() to accept index.Postings (#14280 ) The only call we have to LabelValuesFor() has an index.Postings, and we expand it to pass to this method, which will iterate over the values. That's a waste of resources: we can iterate on the index.Postings directly. If there's any downstream implementation that has a slice of series, they can always do an index.ListPostings from them: doing that is cheaper than expanding an abstract index.Postings. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	6 months ago
Oleg Zaytsev	10a3c7220b	`MemPostings.PostingsForLabelMatching()`: don't hold the mutex while matching (#14286 ) * MemPostings.PostingsForLabelMatching: let mutex go This changes the `MemPostings.PostingsForLabelMatching` implementation to stop holding the read mutex while matching the label values. We've seen that this method can be slow when the matcher is expensive, that's why we even added a context expiration check. However, there are critical process that might be waiting on this mutex: writes (adding new series) and compaction (deleting the garbage-collected ones), so we should avoid holding it for a long period of time. Given that we've copied the values to a slice anyway, there's no need to hold the lock while matching. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	6 months ago
Oleg Zaytsev	2dc177d8af	`MemPostings.Delete()`: reduce locking/unlocking (#13286 ) * MemPostings: reduce locking/unlocking MemPostings.Delete is called from Head.gc(), i.e. it gets the IDs of the series that have churned. I'd assume that many label values aren't affected by that churn at all, so it doesn't make sense to touch the lock while checking them. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	6 months ago
Arve Knudsen	b8b9015e38	tsdb/index: Fix TestReader_PostingsForLabelMatchingHonorsContextCancel Fix number of series in TestReader_PostingsForLabelMatchingHonorsContextCancel (off by one). Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	6 months ago
Oleg Zaytsev	fe9cb5a803	Check context every 128 labels instead of 100 (#14118 ) Follow up on https://github.com/prometheus/prometheus/pull/14096 As promised, I bring a benchmark, which shows a very small improvement if context is checked every 128 iterations of label instead of every 100. It's much easier for a computer to check modulo 128 than modulo 100. This is a very small 0-2% improvement but I'd say this is one of the hottest paths of the app so this is still relevant. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	6 months ago
Arve Knudsen	5ca56eeb6b	tsdb/index: Refactor Reader tests (#14071 ) tsdb/index: Refactor Reader tests Co-authored-by: Björn Rabenstein <github@rabenste.in> Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: Björn Rabenstein <github@rabenste.in>	6 months ago
Oleksandr Redko	f10c3454e9	Enable perfsprint linter and fix up code Signed-off-by: Oleksandr Redko <oleksandr.red+github@gmail.com>	6 months ago
György Krajcsovits	b215a41be4	tsdb/index/postings: fix missing lock unlock Followup to #14096 Unfortunately the previous PR introduced this bug by not releasing the lock before returning. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	6 months ago
George Krajcsovits	fdaafdb041	tsdb: check for context cancel before regex matching postings (#14096 ) * tsdb: check for context cancel before regex matching postings Regex matching can be heavy if the regex takes a lot of cycles to evaluate and we can get stuck evaluating postings for a long time without this fix. The constant checkContextEveryNIterations=100 may be changed later. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	6 months ago
Arve Knudsen	5c4310aa37	[ENHANCEMENT] TSDB: Optimize querying with regexp matchers Add method `PostingsForLabelMatching` to `tsdb.IndexReader`, to obtain postings for labels with a certain name and values accepted by a provided callback, and use it from `tsdb.PostingsForMatchers`. The intention is to optimize regexp matcher paths, especially not having to load all label values before matching on them. Plus tests, and refactor some `tsdb/index.Reader` methods. Benchmarking shows memory reduction up to ~100%, and speedup of up to ~50%. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>	7 months ago
Arve Knudsen	d699dc3c77	Fix language in docs and comments (#14041 ) Fix language in docs and comments --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: Björn Rabenstein <github@rabenste.in>	7 months ago
Matthieu MOREL	6f595c6762	golangci-lint: enable whitespace linter (#13905 ) Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	8 months ago
carrychair	856f6e49c8	fix function and struct name Signed-off-by: carrychair <linghuchong404@gmail.com>	9 months ago
machine424	f477e0539a	Move from golang.org/x/exp/slices into slices now that we only support Go >= 1.21 Prevent adding back golang.org/x/exp/slices. Signed-off-by: machine424 <ayoubmrini424@gmail.com>	9 months ago
Bryan Boreham	93b72ec5dd	tsdb: create SymbolTables for labels as required Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	9 months ago
Bryan Boreham	17f48f2b3b	Tests: use replacement DeepEquals in more places Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	10 months ago
Peter Štibraný	e2b9cfeeeb	Enforce chunks ordering when writing index. (#8085 ) Document conditions on chunks. Add check on chunk time ordering. Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>	10 months ago
Bryan Boreham	98c4889029	Merge pull request #9298 from Creatone/creatone/use-testify tests: Move from t.Errorf and others.	10 months ago
Mikhail Fesenko	419dd265cc	Fix strange code, add messages to code brought in #8106 (#13509 ) Signed-off-by: Mikhail Fesenko <proggga@gmail.com>	10 months ago
Mikhail Fesenko	5f2c3a5d3e	Small improvements, add const, remove copypasta (#8106 ) Signed-off-by: Mikhail Fesenko <proggga@gmail.com> Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com>	10 months ago
Paweł Szulik	5961f78186	Refactor tsdb tests to use testify. Signed-off-by: Paweł Szulik <paul.szulik@gmail.com>	10 months ago
Marco Pracucci	501bc6419e	Add ShardedPostings() support to TSDB (#10421 ) This PR is a reference implementation of the proposal described in #10420. In addition to what described in #10420, in this PR I've introduced labels.StableHash(). The idea is to offer an hashing function which doesn't change over time, and that's used by query sharding in order to get a stable behaviour over time. The implementation of labels.StableHash() is the hashing function used by Prometheus before stringlabels, and what's used by Grafana Mimir for query sharding (because built before stringlabels was a thing). Follow up work As mentioned in #10420, if this PR is accepted I'm also open to upload another foundamental piece used by Grafana Mimir query sharding to accelerate the query execution: an optional, configurable and fast in-memory cache for the series hashes. Signed-off-by: Marco Pracucci <marco@pracucci.com>	10 months ago
Giedrius Statkevičius	61b4080a14	tsdb/{index,compact}: allow using custom postings encoding format (#13242 ) * tsdb/{index,compact}: allow using custom postings encoding format We would like to experiment with a different postings encoding format in Thanos so in this change I am proposing adding another argument to `NewWriter` which would allow users to change the format if needed. Also, wire the leveled compactor so that it would be possible to change the format there too. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * tsdb/compact: use a struct for leveled compactor options As discussed on Slack, let's use a struct for the options in leveled compactor. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * tsdb: make changes after Bryan's review - Make changes less intrusive - Turn the postings encoder type into a function - Add NewWriterWithEncoder() Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>	11 months ago
Bryan Boreham	d0c2d9c0b9	Merge pull request #12878 from bboreham/loser-tree postings: use Loser Tree for merge	12 months ago
Matthieu MOREL	8f6cf3aabb	tsdb: use Go standard errors Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	12 months ago
Bryan Boreham	ab3a47b489	postings: use Loser Tree for merge It's faster. Note change to test - instead of requiring that the data structure is identical to `EmptyPostings()`, check that calling `Next()` returns false, which implies it was empty. Also the check for context cancellation during initialization was removed. Initialization should be a small portion of the work done during merge, so it's not worth plumbing a context argument through. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	12 months ago
Bryan Boreham	ee700151a3	tsdb/index: add benchmark for Postings.Merge Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	12 months ago
Matthieu MOREL	9c4782f1cc	golangci-lint: enable testifylint linter (#13254 ) Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	12 months ago
Julien Pivotto	90ed7b08dc	Merge pull request #13124 from mmorel-35/patch-5 tsdb/index: use Go standard errors package	1 year ago
Matthieu MOREL	2972cc5e8f	tsdb/index: use Go standard errors package Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	1 year ago
songjiayang	443867f1aa	symbolCacheEntry field type alignment, thus saving 8 bytes. Signed-off-by: songjiayang <songjiayang1@gmail.com>	1 year ago
Arve Knudsen	ae9221e152	tsdb/index.Symbols: Drop context argument from Lookup method (#13058 ) Drop context argument from tsdb/index.Symbols.Lookup since lookup should be fast and the context checking is a performance hit. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	1 year ago
Oleksandr Redko	fa90ca46e5	ci(lint): enable godot; append dot at the end of comments Signed-off-by: Oleksandr Redko <Oleksandr_Redko@epam.com>	1 year ago
George Krajcsovits	7d7b9eacff	Fix int32 overflow issues (#12978 ) On a 32 bit architecture the size of int is 32 bits. Thus converting from int64, uint64 can overflow it and flip the sign. Try for yourself in playground: package main import "fmt" func main() { x := int64(0x1F0000001) y := int64(1) z := int32(x - y) // numerically this is 0x1F0000000 fmt.Printf("%v\n", z) } Prints -268435456 as if x was smaller. Followup to #12650 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	1 year ago
Oleg Zaytsev	fe90dcccff	Revert ListPostings change (#12955 ) Reverts change from https://github.com/prometheus/prometheus/pull/12906 The benchmarks show that it's slower when intersecting, which is a common usage for ListPostings (when intersecting matchers from Head) (old is before #12906, new is #12906): │ old │ new │ │ sec/op │ sec/op vs base │ Intersect/LongPostings1-16 20.54µ ± 1% 21.11µ ± 1% +2.76% (p=0.000 n=20) Intersect/LongPostings2-16 51.03m ± 1% 52.40m ± 2% +2.69% (p=0.000 n=20) Intersect/ManyPostings-16 194.2m ± 3% 332.1m ± 1% +71.00% (p=0.000 n=20) geomean 5.882m 7.161m +21.74% Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	1 year ago
Oleg Zaytsev	5bd8c8c561	Clarify Postings.At() contract (#12921 ) It's implicit, but should be explicit. It is invalid to call At() after a failed call to Next() or Seek(). Following up on https://github.com/prometheus/prometheus/pull/12906 Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	1 year ago
Oleg Zaytsev	1492031ef2	Optimize ListPostings Next() (#12906 ) The Next() call of ListPostings() was updating two values, while we can just update the position. This is up to 30% faster for high number of Postings. goos: linux goarch: amd64 pkg: github.com/prometheus/prometheus/tsdb/index cpu: 11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz │ old │ new │ │ sec/op │ sec/op vs base │ ListPostings/count=100-16 819.2n ± 0% 732.6n ± 0% -10.58% (p=0.000 n=20) ListPostings/count=1000-16 2.685µ ± 1% 2.017µ ± 0% -24.88% (p=0.000 n=20) ListPostings/count=10000-16 21.43µ ± 1% 14.81µ ± 0% -30.91% (p=0.000 n=20) ListPostings/count=100000-16 209.4µ ± 1% 143.3µ ± 0% -31.55% (p=0.000 n=20) ListPostings/count=1000000-16 2.086m ± 1% 1.436m ± 1% -31.18% (p=0.000 n=20) geomean 29.02µ 21.41µ -26.22% We're talking about microseconds here, but they just keep adding. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	1 year ago
Goutham Veeramachaneni	86729d4d7b	Update exp package (#12650 )	1 year ago
Arve Knudsen	156222cc50	Add context argument to LabelQuerier.LabelValues (#12665 ) Add context argument to LabelQuerier.LabelValues and LabelQuerier.SortedLabelValues. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	1 year ago
Arve Knudsen	a964349e97	Add context argument to LabelQuerier.LabelNames (#12666 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	1 year ago
Arve Knudsen	4451ba10b4	Add context argument to IndexReader.Postings (#12667 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	1 year ago
Julien Pivotto	1f5934e7be	Merge pull request #10623 from songjiayang/update-index make sure response error when TOC parse failed	1 year ago
Julien Pivotto	0f85e4f41d	Merge pull request #12539 from bboreham/slices-sorts Replace sort.Slice with faster slices.SortFunc	1 year ago
Bryan Boreham	ce153e3fff	Replace sort.Sort with faster slices.SortFunc The generic version is more efficient. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	1 year ago
Bryan Boreham	5255bf06ad	Replace sort.Slice with faster slices.SortFunc The generic version is more efficient. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	1 year ago
Marco Pracucci	35069910f5	Fix infinite loop in index Writer when a series contains duplicated label names Signed-off-by: Marco Pracucci <marco@pracucci.com>	1 year ago

1 2 3

137 Commits (6ca5b9cd40c5bd4e64fe3b281463a73f1035141b)