prometheus

Commit Graph

Author	SHA1	Message	Date
Giedrius Statkevičius	61b4080a14	tsdb/{index,compact}: allow using custom postings encoding format (#13242 ) * tsdb/{index,compact}: allow using custom postings encoding format We would like to experiment with a different postings encoding format in Thanos so in this change I am proposing adding another argument to `NewWriter` which would allow users to change the format if needed. Also, wire the leveled compactor so that it would be possible to change the format there too. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * tsdb/compact: use a struct for leveled compactor options As discussed on Slack, let's use a struct for the options in leveled compactor. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> * tsdb: make changes after Bryan's review - Make changes less intrusive - Turn the postings encoder type into a function - Add NewWriterWithEncoder() Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com> --------- Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>	11 months ago
Bryan Boreham	d0c2d9c0b9	Merge pull request #12878 from bboreham/loser-tree postings: use Loser Tree for merge	12 months ago
Matthieu MOREL	8f6cf3aabb	tsdb: use Go standard errors Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	12 months ago
Bryan Boreham	ab3a47b489	postings: use Loser Tree for merge It's faster. Note change to test - instead of requiring that the data structure is identical to `EmptyPostings()`, check that calling `Next()` returns false, which implies it was empty. Also the check for context cancellation during initialization was removed. Initialization should be a small portion of the work done during merge, so it's not worth plumbing a context argument through. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	12 months ago
Bryan Boreham	ee700151a3	tsdb/index: add benchmark for Postings.Merge Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	12 months ago
Matthieu MOREL	9c4782f1cc	golangci-lint: enable testifylint linter (#13254 ) Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	12 months ago
Julien Pivotto	90ed7b08dc	Merge pull request #13124 from mmorel-35/patch-5 tsdb/index: use Go standard errors package	1 year ago
Matthieu MOREL	2972cc5e8f	tsdb/index: use Go standard errors package Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	1 year ago
songjiayang	443867f1aa	symbolCacheEntry field type alignment, thus saving 8 bytes. Signed-off-by: songjiayang <songjiayang1@gmail.com>	1 year ago
Arve Knudsen	ae9221e152	tsdb/index.Symbols: Drop context argument from Lookup method (#13058 ) Drop context argument from tsdb/index.Symbols.Lookup since lookup should be fast and the context checking is a performance hit. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	1 year ago
Oleksandr Redko	fa90ca46e5	ci(lint): enable godot; append dot at the end of comments Signed-off-by: Oleksandr Redko <Oleksandr_Redko@epam.com>	1 year ago
George Krajcsovits	7d7b9eacff	Fix int32 overflow issues (#12978 ) On a 32 bit architecture the size of int is 32 bits. Thus converting from int64, uint64 can overflow it and flip the sign. Try for yourself in playground: package main import "fmt" func main() { x := int64(0x1F0000001) y := int64(1) z := int32(x - y) // numerically this is 0x1F0000000 fmt.Printf("%v\n", z) } Prints -268435456 as if x was smaller. Followup to #12650 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	1 year ago
Oleg Zaytsev	fe90dcccff	Revert ListPostings change (#12955 ) Reverts change from https://github.com/prometheus/prometheus/pull/12906 The benchmarks show that it's slower when intersecting, which is a common usage for ListPostings (when intersecting matchers from Head) (old is before #12906, new is #12906): │ old │ new │ │ sec/op │ sec/op vs base │ Intersect/LongPostings1-16 20.54µ ± 1% 21.11µ ± 1% +2.76% (p=0.000 n=20) Intersect/LongPostings2-16 51.03m ± 1% 52.40m ± 2% +2.69% (p=0.000 n=20) Intersect/ManyPostings-16 194.2m ± 3% 332.1m ± 1% +71.00% (p=0.000 n=20) geomean 5.882m 7.161m +21.74% Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	1 year ago
Oleg Zaytsev	5bd8c8c561	Clarify Postings.At() contract (#12921 ) It's implicit, but should be explicit. It is invalid to call At() after a failed call to Next() or Seek(). Following up on https://github.com/prometheus/prometheus/pull/12906 Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	1 year ago
Oleg Zaytsev	1492031ef2	Optimize ListPostings Next() (#12906 ) The Next() call of ListPostings() was updating two values, while we can just update the position. This is up to 30% faster for high number of Postings. goos: linux goarch: amd64 pkg: github.com/prometheus/prometheus/tsdb/index cpu: 11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz │ old │ new │ │ sec/op │ sec/op vs base │ ListPostings/count=100-16 819.2n ± 0% 732.6n ± 0% -10.58% (p=0.000 n=20) ListPostings/count=1000-16 2.685µ ± 1% 2.017µ ± 0% -24.88% (p=0.000 n=20) ListPostings/count=10000-16 21.43µ ± 1% 14.81µ ± 0% -30.91% (p=0.000 n=20) ListPostings/count=100000-16 209.4µ ± 1% 143.3µ ± 0% -31.55% (p=0.000 n=20) ListPostings/count=1000000-16 2.086m ± 1% 1.436m ± 1% -31.18% (p=0.000 n=20) geomean 29.02µ 21.41µ -26.22% We're talking about microseconds here, but they just keep adding. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	1 year ago
Goutham Veeramachaneni	86729d4d7b	Update exp package (#12650 )	1 year ago
Arve Knudsen	156222cc50	Add context argument to LabelQuerier.LabelValues (#12665 ) Add context argument to LabelQuerier.LabelValues and LabelQuerier.SortedLabelValues. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	1 year ago
Arve Knudsen	a964349e97	Add context argument to LabelQuerier.LabelNames (#12666 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	1 year ago
Arve Knudsen	4451ba10b4	Add context argument to IndexReader.Postings (#12667 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	1 year ago
Julien Pivotto	1f5934e7be	Merge pull request #10623 from songjiayang/update-index make sure response error when TOC parse failed	1 year ago
Julien Pivotto	0f85e4f41d	Merge pull request #12539 from bboreham/slices-sorts Replace sort.Slice with faster slices.SortFunc	1 year ago
Bryan Boreham	ce153e3fff	Replace sort.Sort with faster slices.SortFunc The generic version is more efficient. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	1 year ago
Bryan Boreham	5255bf06ad	Replace sort.Slice with faster slices.SortFunc The generic version is more efficient. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	1 year ago
Marco Pracucci	35069910f5	Fix infinite loop in index Writer when a series contains duplicated label names Signed-off-by: Marco Pracucci <marco@pracucci.com>	1 year ago
Baskar Shanmugam	905a0bd63a	Added 'limit' query parameter support to /api/v1/status/tsdb endpoint (#12336 ) * Added 'topN' query parameter support to /api/v1/status/tsdb endpoint Signed-off-by: Baskar Shanmugam <baskar.shanmugam.career@gmail.com> * Updated query parameter for tsdb status to 'limit' Signed-off-by: Baskar Shanmugam <baskar.shanmugam.career@gmail.com> * Corrected Stats() parameter name from topN to limit Signed-off-by: Baskar Shanmugam <baskar.shanmugam.career@gmail.com> * Fixed p.Stats CI failure Signed-off-by: Baskar Shanmugam <baskar.shanmugam.career@gmail.com> --------- Signed-off-by: Baskar Shanmugam <baskar.shanmugam.career@gmail.com>	2 years ago
Baskar Shanmugam	f731a90a7f	Fix LabelValueStats in posting stats (#12342 ) Problem: LabelValueStats - This will provide a list of the label names and memory used in bytes. It is calculated by adding the length of all values for a given label name. But internally Prometheus stores the name and the value independently for each series. Solution: MemPostings struct maintains the values to seriesRef map which is used to get the number of series which contains the label values. Using that LabelValueStats is calculated as: seriesCnt * len(value name) Signed-off-by: Baskar Shanmugam <baskar.shanmugam.career@gmail.com>	2 years ago
Matthieu MOREL	bae9a21200	Merge branch 'main' into linter/nilerr Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2 years ago
beorn7	5b53aa1108	style: Replace `else if` cascades with `switch` Wiser coders than myself have come to the conclusion that a `switch` statement is almost always superior to a statement that includes any `else if`. The exceptions that I have found in our codebase are just these two: * The `if else` is followed by an additional statement before the next condition (separated by a `;`). * The whole thing is within a `for` loop and `break` statements are used. In this case, using `switch` would require tagging the `for` loop, which probably tips the balance. Why are `switch` statements more readable? For one, fewer curly braces. But more importantly, the conditions all have the same alignment, so the whole thing follows the natural flow of going down a list of conditions. With `else if`, in contrast, all conditions but the first are "hidden" behind `} else if `, harder to spot and (for no good reason) presented differently from the first condition. I'm sure the aforemention wise coders can list even more reasons. In any case, I like it so much that I have found myself recommending it in code reviews. I would like to make it a habit in our code base, without making it a hard requirement that we would test on the CI. But for that, there has to be a role model, so this commit eliminates all `if else` occurrences, unless it is autogenerated code or fits one of the exceptions above. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Đurica Yuri Nikolić	b028112331	Making the number of CPU cores used for sorting postings lists editable (#12247 ) Signed-off-by: Yuri Nikolic <durica.nikolic@grafana.com>	2 years ago
Matthieu MOREL	fb3eb21230	enable gocritic, unconvert and unused linters Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2 years ago
Oleg Zaytsev	de93a279a0	Shortcut postings for matchers when empty postings are selected (#11813 ) * Add more benchmark cases * Add shortcuts for empty postings Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2 years ago
Ganesh Vernekar	fd89d7892c	Merge pull request #11809 from bboreham/dont-sort-postings-values tsdb: sort values for Postings only when required	2 years ago
György Krajcsovits	97626c9583	Fix comment Comment was not updated when code changed from labels to builder in #11717 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
Bryan Boreham	cf92cd2688	tsdb: sort values for Postings only when required In the head and in v1 postings on disk, it makes no difference whether postings are sorted. Only for v2 does the code step through in order. So, move the sorting to where it is required, and thus skip it entirely in the head. Label values in on-disk blocks are already sorted, but `slices.Sort` is very fast on already-sorted data so we don't bother checking. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	3da2c99ffd	tsdb/index: don't call ExpandPostings in a benchmark This allocates memory for all the returned values, which skews the result. We aren't trying to benchmark `ExpandPostings`, so just step through all the values without storing them to consume them. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	4931983ca9	tsdb/index: make BenchmarkIntersect do work on each loop Previously all the postings constructed were consumed on the first iteration, so subsequent iterations did no work. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	10b27dfb84	Simplify IndexReader.Series interface Instead of passing in a `ScratchBuilder` and `Labels`, just pass the builder and the caller can extract labels from it. In many cases the caller didn't use the Labels value anyway. Now in `Labels.ScratchBuilder` we need a slightly different API: one to assign what will be the result, instead of overwriting some other `Labels`. This is safer and easier to reason about. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	a5bdff414b	Update package tsdb/index tests for new labels.Labels type Note in one cases we needed an extra copy of labels in case they change. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	d3d96ec887	tsdb/index: use ScratchBuilder to create Labels This necessitates a change to the `tsdb.IndexReader` interface: `index.Reader` is used from multiple goroutines concurrently, so we can't have state in it. We do retain a `ScratchBuilder` in `blockBaseSeriesSet` which is iterator-like. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	927a14b0e9	Update package tsdb/index for new labels.Labels type Incomplete - needs further changes to `Decoder.Series()`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Oleg Zaytsev	8553a98267	Optimize postings offset table reading (#11535 ) * Add BenchmarkOpenBlock * Use specific types when reading offset table Instead of reading a generic-ish []string, we can read a generic type which would be specifically labels.Label. This avoid allocating a slice that escapes to the heap, making it both faster and more efficient in terms of memory management. * Update error message for unexpected number of keys * s/posting offset table/postings offset table/ * Remove useless lastKey assignment * Use two []bytes vars, simplify Applied PR feedback: removed generics, moved the label indices reading to that specific test as we're not using it in production anyway, we're just testing what we've just built. Also using two []bytes variables for name and value that use the backing buffer instead of using strings, this reduces allocations a lot as we only copy them when we store them (this is optimized by the compiler). * Fix the dumb bug Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> Co-authored-by: Marco Pracucci <marco@pracucci.com>	2 years ago
Bryan Boreham	3330d85ba8	Replace sort.Strings and sort.Ints with faster slices.Sort (#11318 ) Use new experimental package `golang.org/x/exp/slices`. slices.Sort works on values that are directly comparable, like ints, so avoids the overhad of an interface call to `.Less()`. Left tests unchanged, because they don't need the speed and it may be a cross-check that slices.Sort gives the same answer. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Bryan Boreham	7f2374b703	tsdb: faster postings sort with generic slices.Sort (#11054 ) Use new experimental package `golang.org/x/exp/slices`. Some of the speedup comes from comparing SeriesRef (which is an int64) directly rather than through an interface `.Less()` call; some comes from exp/slices using "pattern-defeating quicksort(pdqsort)". Signed-off-by: Bryan Boreham <bjboreham@gmail.com> Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
songjiayang	c2af0de522	make sure response error when TOC parse failed Signed-off-by: songjiayang <songjiayang1@gmail.com>	2 years ago
Filip Petkovski	d3cb39044e	Fix typo in symbol table size exceeded error message (#10746 ) This commit fixes a typo when reporting an error that the the symbols table size has been exceeded. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>	3 years ago
Matthieu MOREL	e2ede285a2	refactor: move from io/ioutil to io and os packages (#10528 ) * refactor: move from io/ioutil to io and os packages * use fs.DirEntry instead of os.FileInfo after os.ReadDir Signed-off-by: MOREL Matthieu <matthieu.morel@cnp.fr>	3 years ago
Bryan Boreham	2c1be4df7b	tsdb: more efficient sorting of postings read from WAL at startup (#10500 ) * tsdb: avoid slice-to-interface allocation in EnsureOrder This is pulling the `seriesRefSlice` out of the loop, so the compiler doesn't allocate a new one on the heap every time. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * tsdb: use pointer type in Pool for EnsureOrder As noted by staticcheck, Pool prefers the objects in the pool to have pointer type. This is a little more fiddly to code, but avoids allocation of a wrapper object every time a slice is put into the pool. Removed a comment that said fixing this has a performance penalty: not borne out by benchmarks. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Eng Zer Jun	3e67654d37	refactor: use `T.TempDir()` and `B.TempDir` to create temporary directory The directory created by `T.TempDir()` and `B.TempDir()` is automatically removed when the test and all its subtests complete. Reference: https://pkg.go.dev/testing#T.TempDir Reference: https://pkg.go.dev/testing#B.TempDir Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	3 years ago
Oleg Zaytsev	a83d46ee9c	Tidy postingsWithIndexHeap (#10123 ) Unexported postingsWithIndexHeap's methods that don't need to be exported, and added detailed comments. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	3 years ago
Oleg Zaytsev	701545286d	Pop intersected postings heap without popping (#10092 ) See this comment for detailed explanation: https://github.com/prometheus/prometheus/pull/9907#issuecomment-1002189932 TL;DR: if we don't call Pop() on the heap implementation, we don't need to return our param as an `interface{}` so we save an allocation. This would be popped for every label value, so it can be thousands of saved allocations here (see benchmarks). Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	3 years ago

1 2 3

111 Commits (88d07c43cb12299ee2f6f1279133d8057931f521)