prometheus

Commit Graph

Author	SHA1	Message	Date
George Krajcsovits	00ab05c3b9	Native histograms: fix spurios counter reset when merging recoded chunk to normal chunk (#14513 ) * chunkenc: allow missing empty buckets on histogram append Allow appending to chunks when the histogram to be added is missing some buckets, but the missing buckets are empty in the chunk. For example bucket at index 5 is present in the chunk, but its value is 0 and the new histogram doesn't have a bucket at index 5. This fixes an issue of merging chunks where one chunk was recoded to retroactively have some empty buckets in all the histograms and we are merging in a histogram that doesn't have the empty bucket (because it was not recoded yet). The operation alters the histogram that is being added, however this has already been the case when appending gauge histograms. Thus the test TestHistogramSeriesToChunks in storage package is changed to explicitly test what happened to the appended histogram - Compact(0) call is removed. The new expandIntSpansAndBuckets and expandFloatSpansAndBuckets functions are a merge of expandSpansForward and counterResetInAnyBucket and counterResetInAnyFloatBucket. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	4 months ago
Jeanette Tan	f028496133	Merge branch 'main' into nhcb Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	7 months ago
Arve Knudsen	108a6bc9f6	tsdb/chunkenc.Pool: Refactor Get and Put Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	7 months ago
zenador	4acbb7dea6	Add custom buckets to native histogram chunks encoding (#13706 ) * add custom bounds to chunks encoding * change custom buckets schema number * rename custom bounds to custom values Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	8 months ago
Filip Petkovski	583f3e587c	Optimize histogram iterators (#13340 ) Optimize histogram iterators Histogram iterators allocate new objects in the AtHistogram and AtFloatHistogram methods, which makes calculating rates over long ranges expensive. In #13215 we allowed an existing object to be reused when converting an integer histogram to a float histogram. This commit follows the same idea and allows injecting an existing object in the AtHistogram and AtFloatHistogram methods. When the injected value is nil, iterators allocate new histograms, otherwise they populate and return the injected object. The commit also adds a CopyTo method to Histogram and FloatHistogram which is used in the BufferedIterator to overwrite items in the ring instead of making new copies. Note that a specialized HPoint pool is needed for all of this to work (`matrixSelectorHPool`). --------- Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com> Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>	10 months ago
Fiona Liao	b8bcaef14d	Fix histogram append errors (#13201 ) * Fix histogram append errors We should check counterReset condition rather than okToAppend because if there's a counter reset, okToAppend is always set to false. Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>	1 year ago
Oleksandr Redko	8e5f0387a2	ci(lint): enable nolintlint and remove redundant comments (#12926 ) Signed-off-by: Oleksandr Redko <Oleksandr_Redko@epam.com>	1 year ago
György Krajcsovits	b2fa4d910a	Fix more counterResetInAnyBucket edgecases Case a) empty span is at the beginning of the spans. Case b) two consequtive empty spans with positive offsets. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	1 year ago
George Krajcsovits	b6f903b5f9	Fix handling of explicit counter reset header in histograms. (#12772 ) * Fix handling of explicit counter reset header in histograms. Explicit counter reset were being ignored. Also there was no unit test coverage. Add test case for the first sample in a chunk. Add test case for non first sample in chunk. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> --------- Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	1 year ago
George Krajcsovits	6cd2d1621f	Hide histogram chunk append and reset header internals (#12352 ) tsdb: Hide histogram chunk append and reset header internals Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>	1 year ago
György Krajcsovits	c6618729c9	Fix HistogramAppender.Appendable array out of bound error The code did not handle spans with 0 length properly. Spans with length zero are now skipped in the comparison. Span index check not done against length-1, since length is a unit32, thus subtracting 1 leads to 2^32, not -1. Fixes and unit tests for both integer and float histograms added. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2 years ago
Matthieu MOREL	bae9a21200	Merge branch 'main' into linter/nilerr Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2 years ago
beorn7	c3c7d44d84	lint: Adjust to the lint warnings raised by current versions of golint-ci We haven't updated golint-ci in our CI yet, but this commit prepares for that. There are a lot of new warnings, and it is mostly because the "revive" linter got updated. I agree with most of the new warnings, mostly around not naming unused function parameters (although it is justified in some cases for documentation purposes – while things like mocks are a good example where not naming the parameter is clearer). I'm pretty upset about the "empty block" warning to include `for` loops. It's such a common pattern to do something in the head of the `for` loop and then have an empty block. There is still an open issue about this: https://github.com/mgechev/revive/issues/810 I have disabled "revive" altogether in files where empty blocks are used excessively, and I have made the effort to add individual `// nolint:revive` where empty blocks are used just once or twice. It's borderline noisy, though, but let's go with it for now. I should mention that none of the "empty block" warnings for `for` loop bodies were legitimate. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Matthieu MOREL	fb3eb21230	enable gocritic, unconvert and unused linters Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2 years ago
beorn7	1cfc8f65a3	histograms: Return actually useful counter reset hints This is a bit more conservative than we could be. As long as a chunk isn't the first in a block, we can be pretty sure that the previous chunk won't disappear. However, the incremental gain of returning NotCounterReset in these cases is probably very small and might not be worth the code complications. Wwith this, we now also pay attention to an explicitly set counter reset during ingestion. While the case doesn't show up in practice yet, there could be scenarios where the metric source knows there was a counter reset even if it might not be visible from the values in the histogram. It is also useful for testing. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
beorn7	57c18420ab	histograms: General readability tweaks - Adjust doc comments to go1.19 style. - Break down some overly long lines. - Minor doc comment tweaks and fixes. - Some renaming. Some rationales for the last point: I have renamed “interjections” into “inserts”, mostly because it is shorter, and the word shows up a lot by now (and the concept is cryptic enough to not obfuscate it even more with abbreviations). I have also tried to find more descriptive naming for the “compare spans” functions. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Mingjie Shao	78d3c4e823	tsdb: Fixed typo in Histogram Signed-off-by: Mingjie Shao <com.jerryshao@jerryshao.com>	2 years ago
beorn7	6dcd03dbf3	tsdb: Add integer gauge histogram support This follows what #11783 has done for float gauge histograms. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Ganesh Vernekar	d7f5129042	tsdb: Add logic to determine appendable gauge float histograms This is to check if a gauge histogram can be appended to the given chunk. If not, it tells what changes to make to the chunk and the histogram if possible. Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	2 years ago
Ganesh Vernekar	6fd89a6fd2	Add chunk encoding for float histogram (#11716 ) Signed-off-by: Marc Tudurí <marctc@protonmail.com> Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Marc Tudurí <marctc@protonmail.com>	2 years ago
beorn7	5f366e9b62	histograms: Improve tests and fix exposed bugs This adds negative buckets and access of float histograms to TestHistogramChunkSameBuckets and TestHistogramChunkBucketChanges. It also exercises a specific pattern of reusing an iterator (one where no access has happened). This exposes two bugs (where entries for positive buckets where used where the corresponding entries for negative buckets should have been used). One was fixed in #11627 (not merged), which triggered the work in this commit. This commit fixes both issues, so #11627 can be closed. It also simplifies the code in the histogramIterator.Next method that aims to recycle existing slice capacity. Furthermore, this is on top of the release-2.40 branch because we should probably cut a bugfix release for this. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Ganesh Vernekar	b8b0d45d69	Fix reset of a histogram chunk iterator Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	2 years ago
Björn Rabenstein	503ffba49a	chunkenc: Slightly optimize xorWrite/xoRead (#11476 ) With these changes, the "happy path" when the leading and trailing number of bits don't need an update, fewer operations are needed. The change is probably very marginal (no change in the benchmark added here, but the benchmark also doesn't cover non-changing values), and an argument could me made that avoiding pointers also has its benefits. However, I think that reducing the number of return values improves readability. Which convinced me that I should at least propose this. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Ganesh Vernekar	507bfa46fd	Fix HistogramChunk's AtFloatHistogram() Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	2 years ago
Signed-off-by: Jesus Vazquez	3362bf6d79	Fix merge conflicts Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>	2 years ago
Ganesh Vernekar	b2d01cbc57	Remove unnecessary code in encoding/decoding histograms (#11252 ) * Remove unnecessary code in encoding/decoding histograms Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix review comments Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	2 years ago
beorn7	49be0784b4	tsdb: Fix chunk handling during histogram recoding Previously, the maxTime wasn't updated properly in case of a recoding happening. My apologies for reformatting many lines for line length. During the bug hunt, I tried to make things more readable in a reasonably wide editor window. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Björn Rabenstein	7e42acd3b1	tsdb: Rework iterators (#9877 ) - Pick At... method via return value of Next/Seek. - Do not clobber returned buckets. - Add partial FloatHistogram suppert. Note that the promql package is now _only_ dealing with FloatHistograms, following the idea that PromQL only knows float values. As a byproduct, I have removed the histogramSeries metric. In my understanding, series can have both float and histogram samples, so that metric doesn't make sense anymore. As another byproduct, I have converged the sampleBuf and the histogramSampleBuf in memSeries into one. The sample type stored in the sampleBuf has been extended to also contain histograms even before this commit. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	5d4db805ac	Merge branch 'main' into sparsehistogram	3 years ago
beorn7	4c28d9fac7	Move to histogram.Histogram pointers This is to avoid copying the many fields of a histogram.Histogram all the time. This also fixes a bunch of formerly broken tests. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	78ef9c6359	chunkenc: make xor reading more DRY Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	4a1b84f8b2	chunkenc: make xor writing more DRY Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	1a4e54cfbb	tsdb: Complete chunk format documentation This also tweaks and fixes a few things done previously. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	ed33aea392	Avoid redundant varint decoding in chunk appender construction Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	7093b089f2	Use more varbit in histogram chunks This adds bit buckets for larger numbers to varbit encoding and also an unsigned version of varbit encoding. Then, varbit encoding is used for all the histogram chunk data instead of varint. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Björn Rabenstein	7309c20e7e	Merge pull request #9500 from codesome/resettests Add unit test for counter reset header	3 years ago
Ganesh Vernekar	dcaf568279	Metadata -> Layout renaming Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Ganesh Vernekar	4e206c7c77	Fix reviews Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Ganesh Vernekar	85e6686f84	Add unit test for counter reset header Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Björn Rabenstein	311673d62e	Save on slice allocations in histogramIterator (#9494 ) Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	7a8bb8222c	Style cleanup of all the changes in sparsehistogram so far A lot of this code was hacked together, literally during a hackathon. This commit intends not to change the code substantially, but just make the code obey the usual style practices. A (possibly incomplete) list of areas: * Generally address linter warnings. * The `pgk` directory is deprecated as per dev-summit. No new packages should be added to it. I moved the new `pkg/histogram` package to `model` anticipating what's proposed in #9478. * Make the naming of the Sparse Histogram more consistent. Including abbreviations, there were just too many names for it: SparseHistogram, Histogram, Histo, hist, his, shs, h. The idea is to call it "Histogram" in general. Only add "Sparse" if it is needed to avoid confusion with conventional Histograms (which is rare because the TSDB really has no notion of conventional Histograms). Use abbreviations only in local scope, and then really abbreviate (not just removing three out of seven letters like in "Histo"). This is in the spirit of https://github.com/golang/go/wiki/CodeReviewComments#variable-names * Several other minor name changes. * A lot of formatting of doc comments. For one, following https://github.com/golang/go/wiki/CodeReviewComments#comment-sentences , but also layout question, anticipating how things will look like when rendered by `godoc` (even where `godoc` doesn't render them right now because they are for unexported types or not a doc comment at all but just a normal code comment - consistency is queen!). * Re-enabled `TestQueryLog` and `TestEndopints` (they pass now, leaving them disabled was presumably an oversight). * Bucket iterator for histogram.Histogram is now created with a method. * HistogramChunk.iterator now allows iterator recycling. (I think @dieterbe only commented it out because he was confused by the question in the comment.) * HistogramAppender.Append panics now because we decided to treat staleness marker differently. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago

41 Commits (6ca5b9cd40c5bd4e64fe3b281463a73f1035141b)