prometheus

Commit Graph

Author	SHA1	Message	Date
Matthieu MOREL	bae9a21200	Merge branch 'main' into linter/nilerr Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2 years ago
Ben Ye	fd3630b9a3	add ctx to QueryEngine interface Signed-off-by: Ben Ye <benye@amazon.com>	2 years ago
Matthieu MOREL	fb3eb21230	enable gocritic, unconvert and unused linters Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2 years ago
beorn7	c0879d64cf	promql: Separate `Point` into `FPoint` and `HPoint` In other words: Instead of having a “polymorphous” `Point` that can either contain a float value or a histogram value, use an `FPoint` for floats and an `HPoint` for histograms. This seemingly small change has a _lot_ of repercussions throughout the codebase. The idea here is to avoid the increase in size of `Point` arrays that happened after native histograms had been added. The higher-level data structures (`Sample`, `Series`, etc.) are still “polymorphous”. The same idea could be applied to them, but at each step the trade-offs needed to be evaluated. The idea with this change is to do the minimum necessary to get back to pre-histogram performance for functions that do not touch histograms. Here are comparisons for the `changes` function. The test data doesn't include histograms yet. Ideally, there would be no change in the benchmark result at all. First runtime v2.39 compared to directly prior to this commit: ``` name old time/op new time/op delta RangeQuery/expr=changes(a_one[1d]),steps=1-16 391µs ± 2% 542µs ± 1% +38.58% (p=0.000 n=9+8) RangeQuery/expr=changes(a_one[1d]),steps=10-16 452µs ± 2% 617µs ± 2% +36.48% (p=0.000 n=10+10) RangeQuery/expr=changes(a_one[1d]),steps=100-16 1.12ms ± 1% 1.36ms ± 2% +21.58% (p=0.000 n=8+10) RangeQuery/expr=changes(a_one[1d]),steps=1000-16 7.83ms ± 1% 8.94ms ± 1% +14.21% (p=0.000 n=10+10) RangeQuery/expr=changes(a_ten[1d]),steps=1-16 2.98ms ± 0% 3.30ms ± 1% +10.67% (p=0.000 n=9+10) RangeQuery/expr=changes(a_ten[1d]),steps=10-16 3.66ms ± 1% 4.10ms ± 1% +11.82% (p=0.000 n=10+10) RangeQuery/expr=changes(a_ten[1d]),steps=100-16 10.5ms ± 0% 11.8ms ± 1% +12.50% (p=0.000 n=8+10) RangeQuery/expr=changes(a_ten[1d]),steps=1000-16 77.6ms ± 1% 87.4ms ± 1% +12.63% (p=0.000 n=9+9) RangeQuery/expr=changes(a_hundred[1d]),steps=1-16 30.4ms ± 2% 32.8ms ± 1% +8.01% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=10-16 37.1ms ± 2% 40.6ms ± 2% +9.64% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=100-16 105ms ± 1% 117ms ± 1% +11.69% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=1000-16 783ms ± 3% 876ms ± 1% +11.83% (p=0.000 n=9+10) ``` And then runtime v2.39 compared to after this commit: ``` name old time/op new time/op delta RangeQuery/expr=changes(a_one[1d]),steps=1-16 391µs ± 2% 547µs ± 1% +39.84% (p=0.000 n=9+8) RangeQuery/expr=changes(a_one[1d]),steps=10-16 452µs ± 2% 616µs ± 2% +36.15% (p=0.000 n=10+10) RangeQuery/expr=changes(a_one[1d]),steps=100-16 1.12ms ± 1% 1.26ms ± 1% +12.20% (p=0.000 n=8+10) RangeQuery/expr=changes(a_one[1d]),steps=1000-16 7.83ms ± 1% 7.95ms ± 1% +1.59% (p=0.000 n=10+8) RangeQuery/expr=changes(a_ten[1d]),steps=1-16 2.98ms ± 0% 3.38ms ± 2% +13.49% (p=0.000 n=9+10) RangeQuery/expr=changes(a_ten[1d]),steps=10-16 3.66ms ± 1% 4.02ms ± 1% +9.80% (p=0.000 n=10+9) RangeQuery/expr=changes(a_ten[1d]),steps=100-16 10.5ms ± 0% 10.8ms ± 1% +3.08% (p=0.000 n=8+10) RangeQuery/expr=changes(a_ten[1d]),steps=1000-16 77.6ms ± 1% 78.1ms ± 1% +0.58% (p=0.035 n=9+10) RangeQuery/expr=changes(a_hundred[1d]),steps=1-16 30.4ms ± 2% 33.5ms ± 4% +10.18% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=10-16 37.1ms ± 2% 40.0ms ± 1% +7.98% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=100-16 105ms ± 1% 107ms ± 1% +1.92% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=1000-16 783ms ± 3% 775ms ± 1% -1.02% (p=0.019 n=9+9) ``` In summary, the runtime doesn't really improve with this change for queries with just a few steps. For queries with many steps, this commit essentially reinstates the old performance. This is good because the many-step queries are the one that matter most (longest absolute runtime). In terms of allocations, though, this commit doesn't make a dent at all (numbers not shown). The reason is that most of the allocations happen in the sampleRingIterator (in the storage package), which has to be addressed in a separate commit. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Trevor Whitney	dd94ebb87b	promql: set CounterResetHint after rate and sum Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>	2 years ago
Justin Lei	af1d9e01c7	Refactor tsdbutil for tests/native histograms (#11948 ) * Add float histograms to ChunkFromSamplesGeneric Signed-off-by: Justin Lei <justin.lei@grafana.com> * Add GenerateSamples functions to tsdbutil Signed-off-by: Justin Lei <justin.lei@grafana.com> PR responses Signed-off-by: Justin Lei <justin.lei@grafana.com> --------- Signed-off-by: Justin Lei <justin.lei@grafana.com>	2 years ago
beorn7	1cfc8f65a3	histograms: Return actually useful counter reset hints This is a bit more conservative than we could be. As long as a chunk isn't the first in a block, we can be pretty sure that the previous chunk won't disappear. However, the incremental gain of returning NotCounterReset in these cases is probably very small and might not be worth the code complications. Wwith this, we now also pay attention to an explicitly set counter reset during ingestion. While the case doesn't show up in practice yet, there could be scenarios where the metric source knows there was a counter reset even if it might not be visible from the values in the histogram. It is also useful for testing. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Ganesh Vernekar	3c2ea91a83	tsdb: Test gauge float histograms Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	2 years ago
Marc Tudurí	49f775d8a0	histograms: Add missing float histograms tests for PromQL (#11780 ) * test: TestSparseHistogramRate * test: TestSparseHistogram_HistogramQuantile * test: TestSparseHistogram_HistogramFraction * test: TestSparseHistogram_HistogramFraction * test: TestSparseHistogram_Sum_Count_AddOperator * test: TestSparseHistogram_HistogramCountAndSum * tests: fix TestSparseHistogram_HistogramCountAndSum * linter * refactor TestSparseHistogram_HistogramCountAndSum * wrap TestSparseHistogram_HistogramCountAndSum Signed-off-by: Marc Tuduri <marctc@protonmail.com>	2 years ago
Marc Tudurí	9474610baf	Support FloatHistogram in TSDB (#11522 ) Extends Appender.AppendHistogram function to accept the FloatHistogram. TSDB supports appending, querying, WAL replay, for this new type of histogram. Signed-off-by: Marc Tudurí <marctc@protonmail.com> Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>	2 years ago
Bryan Boreham	aa634e0b7e	Update package promql tests for new labels.Labels type Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Jesus Vazquez	e934d0f011	Merge 'main' into sparsehistogram Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>	2 years ago
Giedrius Statkevičius	a1d6ba59ac	promql: pass down subquery interval (#11163 ) If we are populating series for a subquery then set the interval parameter accordingly so that downstream users could use that information. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>	2 years ago
beorn7	a7c519930e	histograms: Add Compact method to the normal integer Histogram And use the new method to call to compact Histograms during parsing. This happens for both `Histogram` and `FloatHistogram`. In this way, if targets decide to optimize the exposition size by merging spans with empty buckets in between, we still get a normalized results. It will also normalize away any valid but weird representations like empty spans, spans with offset zero, and empty buckets at the start or end of a span. The implementation seemed easy at first as it just turns the `compactBuckets` helper into a generic function (which now got its own file). However, the integer Histograms have delta buckets instead of absolute buckets, which had to be treated specially in the generic `compactBuckets` function. To make sure it works, I have added plenty of explicit tests for `Histogram` in addition to the `FloatHistogram` tests. I have also updated the doc comment for the `Compact` method. Based on the insights now expressed in the doc comment, compacting with a maxEmptyBuckets > 0 is rarely useful. Therefore, this commit also sets the value to 0 in the two cases we were using 3 so far. We might still want to reconsider, so I don't want to remove the maxEmptyBuckets parameter right now. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Bryan Boreham	b01d29cf9e	promql: in tests use labels.FromStrings And a few cases of `EmptyLabels()`. Replacing code which assumes the internal structure of `Labels`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Ganesh Vernekar	71489d0e3d	Fix count() for histograms and add test case Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	2 years ago
Ganesh Vernekar	9325caa41c	Remove a TODO that is no longer valid (#11186 ) Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	2 years ago
beorn7	c9fd3c235d	Merge branch 'main' into sparsehistogram	2 years ago
Vilius Pranckaitis	4660656312	Allow setting custom lookback delta for instant queries (#9946 ) * Allow setting custom lookback delta for instant queries Signed-off-by: Vilius Pranckaitis <vpranckaitis@gmail.com>	2 years ago
Levi Harrison	77a7af4461	Add histogram validation (#11052 ) * Add histogram validation Signed-off-by: Levi Harrison <git@leviharrison.dev> * Correct negative offset validation Signed-off-by: Levi Harrison <git@leviharrison.dev> * Address review comments Signed-off-by: Levi Harrison <git@leviharrison.dev> * Validation benchmark Signed-off-by: Levi Harrison <git@leviharrison.dev> * Add more checks Signed-off-by: Levi Harrison <git@leviharrison.dev> * Attempt to fix tests Signed-off-by: Levi Harrison <git@leviharrison.dev> * Fix stuff Signed-off-by: Levi Harrison <git@leviharrison.dev>	2 years ago
Łukasz Mierzwa	54a3c3ba3f	Print query that caused a panic (#10995 ) We print the stacktrace of a panic when query causes one, but there's no information about the query itself, which makes it harder to debug and reproduce the issue. This adds the 'expr' string to the logged panic. Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	2 years ago
beorn7	9eafed0f79	promql: Add `histogram_count` and `histogram_sum` This follow a simple function-based approach to access the count and sum fields of a native Histogram. It might be more elegant to implement “accessors” via the dot operator, as considered in the brainstorming doc [1]. However, that would require the introduction of a whole new concept in PromQL. For the PoC, we should be fine with the function-based approch. Even the obvious inefficiencies (rate'ing a whole histogram twice when we only want to rate each the count and the sum once) could be optimized behind the scenes. Note that the function-based approach elegantly solves the problem of detecting counter resets in the sum of observations in the case of negative observations. (Since the whole native Histogram is rate'd, the counter reset is detected for the Histogram as a whole.) We will decide later if an “accessor” approach is really needed. It would change the example expression for average duration in functions.md from histogram_sum(rate(http_request_duration_seconds[10m])) / histogram_count(rate(http_request_duration_seconds[10m])) to rate(http_request_duration_seconds.sum[10m]) / rate(http_request_duration_seconds.count[10m]) [1]: https://docs.google.com/document/d/1ch6ru8GKg03N02jRjYriurt-CZqUVY09evPg6yKTA1s/edit Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
beorn7	a3a8f58bb3	promql: Add histogram_fraction function Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
beorn7	ffaabea91a	promql: Refine zero bucket treatment in histogramQuantile Essentially, this mirrors the existing behavior for negative buckets: If a histogram has only negative buckets, the upper bound of the zero bucket is assumed to be zero. Furthermore, it makes sure that the zero bucket boundaries are not modified if a histogram that has no buckets at all but samples in the zero bucket. Also, add an TODO to vet if we really want this behavior. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
beorn7	40ad5e284a	Merge branch 'main' into beorn7/sparsehistogram	3 years ago
Łukasz Mierzwa	08262454a3	Preallocate Labels in labels.Builder (#10749 ) This tries to avoid re-allocations of labels slice since we know possible max size Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	3 years ago
beorn7	3bc711e333	Merge branch 'main' into sparsehistogram	3 years ago
Matthieu MOREL	e2ede285a2	refactor: move from io/ioutil to io and os packages (#10528 ) * refactor: move from io/ioutil to io and os packages * use fs.DirEntry instead of os.FileInfo after os.ReadDir Signed-off-by: MOREL Matthieu <matthieu.morel@cnp.fr>	3 years ago
Alan Protasio	ce6a643ee8	Changing TotalQueryableSamples from int to int64 (#10549 ) * Changing TotalQueryableSamples from int to int64 Signed-off-by: Alan Protasio <approtas@amazon.com>	3 years ago
beorn7	4210aac74a	Merge branch 'main' into sparsehistogram	3 years ago
Andrew Bloomgarden	a64b9fe323	Report PeakSamples in query statistics This exactly corresponds to the statistic compared against MaxSamples during the course of query execution, so users can see how close their queries are to a limit. Co-authored-by: Harkishen Singh <harkishensingh@hotmail.com> Co-authored-by: Andrew Bloomgarden <blmgrdn@amazon.com> Signed-off-by: Andrew Bloomgarden <blmgrdn@amazon.com>	3 years ago
Alan Protasio	606ef33d91	Track and report Samples Queried per query We always track total samples queried and add those to the standard set of stats queries can report. We also allow optionally tracking per-step samples queried. This must be enabled both at the engine and query level to be tracked and rendered. The engine flag is exposed via a Prometheus feature flag, while the query flag is set when stats=all. Co-authored-by: Alan Protasio <approtas@amazon.com> Co-authored-by: Andrew Bloomgarden <blmgrdn@amazon.com> Co-authored-by: Harkishen Singh <harkishensingh@hotmail.com> Signed-off-by: Andrew Bloomgarden <blmgrdn@amazon.com>	3 years ago
Björn Rabenstein	ec80745884	Merge pull request #10075 from prometheus/beorn7/histogram model: Implement FloatHistogram.Compact	3 years ago
beorn7	3b4d6c3fdb	model: Implement FloatHistogram.Compact Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	e7592fe353	sparsehistogram: Address two TODOs Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	a6acdfe346	histograms: Doc comment and naming improvements Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Ganesh Vernekar	f580248759	Support + operator for sparse histograms (#9949 ) Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Ganesh Vernekar	187a767292	Implement sum() for sparse histograms (#9948 ) Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Ganesh Vernekar	4a43349aca	`histogram_quantile` for sparse histograms (#9935 ) * MergeFloatBucketIterator for []FloatBucketIterator Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * histogram_quantile for histograms Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix histogram_quantile Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Unit test and enhancements Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Iterators to iterate buckets in reverse and all buckets together including zero bucket Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Consider all buckets for histogram_quantile and fix the implementation Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Remove unneeded code Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix lint Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Björn Rabenstein	0e1b9dd308	Promql: Initial rate implementation for sparse histograms (#9926 ) Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	e4e24453fa	Merge branch 'main' into beorn7/merge2	3 years ago
Shihao Xia	0e82a96e2f	fix potential deadlock in test (#9010 ) * fix potential deadlock Signed-off-by: Shihao Xia <charlesxsh@hotmail.com> * fix deadlock Signed-off-by: Shihao Xia <charlesxsh@hotmail.com> * Update promql/engine_test.go Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com> Signed-off-by: Shihao Xia <charlesxsh@hotmail.com> Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>	3 years ago
beorn7	5d4db805ac	Merge branch 'main' into sparsehistogram	3 years ago
beorn7	9de3ab60df	promql: improve histogram support in engine.go Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	73858d7f82	storage: histogram support in memoized_iterator Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	9b30ca2598	promql: Support histogram in value string representation Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	4c28d9fac7	Move to histogram.Histogram pointers This is to avoid copying the many fields of a histogram.Histogram all the time. This also fixes a bunch of formerly broken tests. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	f1065e44a4	model: String method for histogram.Histogram This includes a regular bucket iterator and a string method for histogram.Bucket. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Thomas Jackson	f0003bc0ba	Don't drop ParenExpr when creating StepInvariantExpr (#9591 ) * Add test case to showcase the problem in #9590 Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com> * Don't unwrap ParenExpr in newStepInvariantExpr Fixes #9590 Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>	3 years ago
beorn7	c954cd9d1d	Move packages out of deprecated pkg directory This creates a new `model` directory and moves all data-model related packages over there: exemplar labels relabel rulefmt textparse timestamp value All the others are more or less utilities and have been moved to `util`: gate logging modetimevfs pool runtime Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago

1 2 3

128 Commits (c06c02b3b10b3d98bd7d59400d0a1b3762e90af3)