prometheus

Commit Graph

Author	SHA1	Message	Date
Neeraj Gartia	8aef821e10	eval_info command Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com>	2 months ago
Björn Rabenstein	5b9148e552	Merge pull request #14820 from charleskorn/promqltest-native-histogram-format promqltest: use test expression format for histograms in assertion failure messages and include reset hint in the test expression	2 months ago
Björn Rabenstein	df9916ef66	Merge pull request #14677 from prometheus/beorn7/histogram promql(native histograms): Introduce exponential interpolation	2 months ago
Jan Fajerski	96e5a94d29	promql: rename holt_winters to double_exponential_smoothing Signed-off-by: Jan Fajerski <jfajersk@redhat.com>	2 months ago
beorn7	6fcd225aee	promql(native histograms): Introduce exponential interpolation The linear interpolation (assuming that observations are uniformly distributed within a bucket) is a solid and simple assumption in lack of any other information. However, the exponential bucketing used by standard schemas of native histograms has been chosen to cover the whole range of observations in a way that bucket populations are spread out over buckets in a reasonably way for typical distributions encountered in real-world scenarios. This is the origin of the idea implemented here: If we divide a given bucket into two (or more) smaller exponential buckets, we "most naturally" expect that the samples in the original buckets will split among those smaller buckets in a more or less uniform fashion. With this assumption, we end up with an "exponential interpolation", which therefore appears to be a better match for histograms with exponential bucketing. This commit leaves the linear interpolation in place for NHCB, but changes the interpolation for exponential native histograms to exponential. This affects `histogram_quantile` and `histogram_fraction` (because the latter is more or less the inverse of the former). The zero bucket has to be treated specially because the assumption above would lead to an "interpolation to zero" (the bucket density approaches infinity around zero, and with the postulated uniform usage of buckets, we would end up with an estimate of zero for all quantiles ending up in the zero bucket). We simply fall back to linear interpolation within the zero bucket. At the same time, this commit makes the call to stick with the assumption that the zero bucket only contains positive observations for native histograms without negative buckets (and vice versa). (This is an assumption relevant for interpolation. It is a mostly academic point, as the zero bucket is supposed to be very small anyway. However, in cases where it _is_ relevantly broad, the assumption helps a lot in practice.) This commit also updates and completes the documentation to match both details about interpolation. As a more high level note: The approach here attempts to strike a balance between a more simplistic approach without any assumption, and a more involved approach with more sophisticated assumptions. I will shortly describe both for reference: The "zero assumption" approach would be to not interpolate at all, but _always_ return the harmonic mean of the bucket boundaries of the bucket the quantile ends up in. This has the advantage of minimizing the maximum possible relative error of the quantile estimation. (Depending on the exact definition of the relative error of an estimation, there is also an argument to return the arithmetic mean of the bucket boundaries.) While limiting the maximum possible relative error is a good property, this approach would throw away the information if a quantile is closer to the upper or lower end of the population within a bucket. This can be valuable trending information in a dashboard. With any kind of interpolation, the maximum possible error of a quantile estimation increases to the full width of a bucket (i.e. it more than doubles for the harmonic mean approach, and precisely doubles for the arithmetic mean approach). However, in return the _expectation value_ of the error decreases. The increase of the theoretical maximum only has practical relevance for pathologic distributions. For example, if there are thousand observations within a bucket, they could _all_ be at the upper bound of the bucket. If the quantile calculation picks the 1st observation in the bucket as the relevant one, an interpolation will yield a value close to the lower bucket boundary, while the true quantile value is close to the upper boundary. The "fancy interpolation" approach would be one that analyses the _actual_ distribution of samples in the histogram. A lot of statistics could be applied based on the information we have available in the histogram. This would include the population of neighboring (or even all) buckets in the histogram. In general, the resolution of a native histogram should be quite high, and therefore, those "fancy" approaches would increase the computational cost quite a bit with very little practical benefits (i.e. just tiny corrections of the estimated quantile value). The results are also much harder to reason with. Signed-off-by: beorn7 <beorn@grafana.com>	2 months ago
Jan Fajerski	91608c002f	Merge branch 'main' into release-3.0-beta.0 Conflicts: scrape/scrape_test.go Pick both changes.	3 months ago
Björn Rabenstein	8114f9a281	Merge pull request #14821 from charleskorn/nh-negative-multiplication-division promql: correctly handle unary negation of native histograms and add tests for multiplication and division of native histograms by negative scalars	3 months ago
Charles Korn	e8c7482137	Return negative counts when multiplied or divided by a negative value Signed-off-by: Charles Korn <charles.korn@grafana.com>	3 months ago
Jan Fajerski	fe4289b502	Merge branch 'main' into HEAD	3 months ago
Arve Knudsen	6ef1eb4e68	Merge pull request #14064 from aknuds1/arve/close-engine Release PromQL engine resources on close, from tests	3 months ago
Charles Korn	113de6301c	Add failing test cases for unary negation and multiplication and division with negative scalars Signed-off-by: Charles Korn <charles.korn@grafana.com>	3 months ago
Charles Korn	9b451abec7	Make positive and negative bucket counts different in existing test cases Signed-off-by: Charles Korn <charles.korn@grafana.com>	3 months ago
Charles Korn	90dc1b45db	promqltest: use test expression format for histograms in assertion failure messages Signed-off-by: Charles Korn <charles.korn@grafana.com>	3 months ago
Arve Knudsen	70bb219d33	Merge pull request #14680 from charleskorn/promqltest-scalar promqltest: improve error messages when unexpected type is returned, and fix inverted expected and actual values in error message	3 months ago
Arve Knudsen	5dfbcc390e	Merge remote-tracking branch 'prometheus/main' into arve/close-engine Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 months ago
Jan Fajerski	956245b25b	promqltest: adjust eval times and range selector In order to fix new tests for changes added in https://github.com/prometheus/prometheus/pull/13904. Signed-off-by: Jan Fajerski <jfajersk@redhat.com>	3 months ago
Jan Fajerski	00315ce15e	Merge branch 'main' into 3.0-main-sync-24-08-30 using -Xours Signed-off-by: Jan Fajerski <jfajersk@redhat.com>	3 months ago
Neeraj Gartia	8c7bf39d96	Moves TestNativeHistogram_MulDivOperator to promql testing framework (#14688 ) PromQL: add test for mul and div operator Also, remove the converted test from the engine_test.go file. This also includes an extension of the test framework to allow NaN/Inf in histogram buckets. --------- Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com>	3 months ago
Jorge Creixell	e9e3d64b7c	PromQL engine: Delay deletion of __name__ label to the end of the query evaluation (#14477 ) PromQL engine: Delay deletion of __name__ label to the end of the query evaluation - This change allows optionally preserving the `__name__` label via the `label_replace` and `label_join` functions, and helps prevent the dreaded "vector cannot contain metrics with the same labelset" error. - The implementation extends the `Series` and `Sample` structs with a boolean flag indicating whether the `__name__` label should be deleted at the end of the query evaluation. - The `label_replace` and `label_join` functions can still access the value of the `__name__` label, even if it has been previously marked for deletion. If `__name__` is used as target label, it won't be dropped at the end of the query evaluation. - Fixes https://github.com/prometheus/prometheus/issues/11397 - See https://github.com/jcreixell/prometheus/pull/2 for previous discussion, including the decision to create this PR and benchmark it before considering other alternatives (like refactoring `labels.Labels`). - See https://github.com/jcreixell/prometheus/pull/1 for an alternative implementation using a special label instead of boolean flags. - Note: a feature flag `promql-delayed-name-removal` has been added as it changes the behavior of some "weird" queries (see https://github.com/prometheus/prometheus/issues/11397#issuecomment-1451998792) Example (this always fails, as `__name__` is being dropped by `count_over_time`): ``` count_over_time({__name__!=""}[1m]) => Error executing query: vector cannot contain metrics with the same labelset ``` Before: ``` label_replace(count_over_time({__name__!=""}[1m]), "__name__", "count_$1", "__name__", "(.+)") => Error executing query: vector cannot contain metrics with the same labelset ``` After: ``` label_replace(count_over_time({__name__!=""}[1m]), "__name__", "count_$1", "__name__", "(.+)") => count_go_gc_cycles_automatic_gc_cycles_total{instance="localhost:9090", job="prometheus"} 1 count_go_gc_cycles_forced_gc_cycles_total{instance="localhost:9090", job="prometheus"} 1 ... ``` Signed-off-by: Jorge Creixell <jcreixell@gmail.com> --------- Signed-off-by: Jorge Creixell <jcreixell@gmail.com> Signed-off-by: Björn Rabenstein <github@rabenste.in>	3 months ago
Arve Knudsen	99204f23ee	Merge remote-tracking branch 'prometheus/main' into arve/close-engine Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 months ago
Björn Rabenstein	849215d90c	Merge pull request #14585 from fatsheep9146/covert-TestNativeHistogram_Sum_Count_Add_AvgOperator-to-framework convert TestNativeHistogram_Sum_Count_Add_AvgOperator into testing framework	3 months ago
Arve Knudsen	c9a460d570	Merge remote-tracking branch 'prometheus/main' into arve/close-engine Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 months ago
Ziqi Zhao	87eab0aaad	fix golang lint Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>	3 months ago
Ziqi Zhao	c7c4a5c347	add helper function to compare native histograms in testing Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>	3 months ago
Jan Fajerski	7c8c748399	promql tests: adjust range query intervals Some test queries need their interval adjusted to account for https://github.com/prometheus/prometheus/pull/13904. Otherwise the queries don't return enough samples. promql/engine_test.go:TestHistogramCopyFromIteratorRegression needed the same, but also the result needed a fix since `increase` interpolates over the full range. Signed-off-by: Jan Fajerski <jfajersk@redhat.com>	3 months ago
Björn Rabenstein	7fad1ec8ee	Merge pull request #14655 from suntala/suntala/sort-by-label-enhancement promql: Fall back to full label sets when sorting by label	3 months ago
Jan Fajerski	5138922b0d	Merge branch 'main' into 3.0-main-sync-24-08-21	3 months ago
Ziqi Zhao	8f828d45c1	convert TestNativeHistogram_Sum_Count_Add_AvgOperator into testing framework Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>	3 months ago
Charles Korn	6548421f61	promqltest: improve error messages when unexpected type is returned, and fix inverted expected and actual values in error message when scalar value does not match expected Signed-off-by: Charles Korn <charles.korn@grafana.com>	3 months ago
Charles Korn	52818a97e2	Merge branch 'main' into sum-and-avg-over-mixed-custom-exponential-histograms # Conflicts: # promql/promqltest/testdata/native_histograms.test	3 months ago
György Krajcsovits	386fc8b9f6	Update from review comments. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	3 months ago
György Krajcsovits	6aee5b4b38	fix typo Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	3 months ago
György Krajcsovits	06a8886b94	Native histograms: define behavior when rate is null. Histogram quantile returns NaN in this case, which might be surprising, so add a unit test that clarifies that this is intentional. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	4 months ago
suntala	fd2f44af7f	Fall back to comparing by label set when sorting by label desc Co-authored-by: Aleks Fazlieva <britishrum@users.noreply.github.com> Signed-off-by: suntala <arati.rana@grafana.com>	4 months ago
suntala	94ad489328	Fall back to comparing by label set when sorting by label Co-authored-by: Aleks Fazlieva <britishrum@users.noreply.github.com> Signed-off-by: suntala <arati.rana@grafana.com>	4 months ago
Charles Korn	f992f81bd0	Merge branch 'main' into sum-and-avg-over-mixed-custom-exponential-histograms Signed-off-by: Charles Korn <charleskorn@users.noreply.github.com>	4 months ago
Charles Korn	5cfdde327c	Address PR feedback: add extra test case Signed-off-by: Charles Korn <charles.korn@grafana.com>	4 months ago
Charles Korn	82bb35fabb	Address PR feedback: fix typo and rename variable Signed-off-by: Charles Korn <charles.korn@grafana.com>	4 months ago
Charles Korn	f07b3ae67b	Fix issue where `avg` over mixed exponential and custom buckets, or incompatible custom buckets, produces incorrect results or panics Signed-off-by: Charles Korn <charles.korn@grafana.com>	4 months ago
Charles Korn	5ee94f49a2	Fix issue where `sum` over mixed exponential and custom buckets, or incompatible custom buckets, produces incorrect results Signed-off-by: Charles Korn <charles.korn@grafana.com>	4 months ago
Charles Korn	424cefcf5e	Fix "cannot reduce resolution to custom buckets schema" panic in `rate` over native histograms with mix of custom and exponential buckets Signed-off-by: Charles Korn <charles.korn@grafana.com>	4 months ago
Charles Korn	f6e4b775e2	Check for errors first Signed-off-by: Charles Korn <charles.korn@grafana.com>	4 months ago
Björn Rabenstein	ee5bba07c0	Merge pull request #14413 from prometheus/beorn7/promql promql: more Kahan summation (avg) and less incremental mean calculation (avg, avg_over_time)	4 months ago
Charles Korn	aadec25faf	promql: Fix issue where some native histogram-related annotations are not emitted by `rate` (#14575 ) Signed-off-by: Charles Korn <charles.korn@grafana.com>	4 months ago
Arve Knudsen	7c873004c7	Merge remote-tracking branch 'prometheus/main' into arve/close-engine	4 months ago
Jan Fajerski	adf5d6bce1	Merge branch 'main' into 3.0-main-sync-24-07-18 Signed-off-by: Jan Fajerski <jfajersk@redhat.com> Conflicts: VERSION pick 3.0.0 promql/promqltest/testdata/histograms.test pick changes from `c39776c5b5`, but adjust 5m range selectors to 10m to account for https://github.com/prometheus/prometheus/pull/13904. Fixes: promql/promqltest/testdata/functions.test promql/promqltest/testdata/staleness.test Tests added in https://github.com/prometheus/prometheus/pull/9138 need to be adjusted to account for https://github.com/prometheus/prometheus/pull/13904.	4 months ago
Björn Rabenstein	ca7062cf49	Merge pull request #14464 from prometheus/beorn7/histogram promql: Add NHCB tests	4 months ago
beorn7	c39776c5b5	promql: Add NHCB tests This adds equivalent NHCB tests to the existing classic histogram tests. Signed-off-by: beorn7 <beorn@grafana.com>	4 months ago
Arve Knudsen	fbc9eddfaf	Refactor engine creation in tests Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	4 months ago
Arve Knudsen	fec6adadcd	Merge remote-tracking branch 'prometheus/main' into arve/close-engine	4 months ago

1 2

95 Commits (3a8131b3f16bed599786e9fcf7a080f408945038)