prometheus

Commit Graph

Author	SHA1	Message	Date
Arve Knudsen	c2bbabb4a7	promql.Engine: Refactor vector selector evaluation into a method (#14900 ) * PromQL.Engine: Refactor Matrix expansion into a method Add utility method promql.evaluator.expandSeriesToMatrix, for expanding a slice of storage.Series into a promql.Matrix. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Rename to generateMatrix Rename evaluator.expandSeriesToMatrix into generateMatrix, while also dropping the start, end, interval arguments since they are evaluator fields. Write more extensive method documentation. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> * Rename to evalVectorSelector Rename to evalVectorSelector after discussing with @michahoffmann. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2 months ago
Arthur Silva Sens	6bcb064d93	Merge pull request #14950 from Maniktherana/fuzz-om-minor-change chore: remove unused code	2 months ago
Manik Rana	3fba6bcdfa	chore: remove unused code Signed-off-by: Manik Rana <manikrana54@gmail.com>	2 months ago
Björn Rabenstein	5b9148e552	Merge pull request #14820 from charleskorn/promqltest-native-histogram-format promqltest: use test expression format for histograms in assertion failure messages and include reset hint in the test expression	2 months ago
Björn Rabenstein	df9916ef66	Merge pull request #14677 from prometheus/beorn7/histogram promql(native histograms): Introduce exponential interpolation	2 months ago
Jan Fajerski	96e5a94d29	promql: rename holt_winters to double_exponential_smoothing Signed-off-by: Jan Fajerski <jfajersk@redhat.com>	2 months ago
beorn7	6fcd225aee	promql(native histograms): Introduce exponential interpolation The linear interpolation (assuming that observations are uniformly distributed within a bucket) is a solid and simple assumption in lack of any other information. However, the exponential bucketing used by standard schemas of native histograms has been chosen to cover the whole range of observations in a way that bucket populations are spread out over buckets in a reasonably way for typical distributions encountered in real-world scenarios. This is the origin of the idea implemented here: If we divide a given bucket into two (or more) smaller exponential buckets, we "most naturally" expect that the samples in the original buckets will split among those smaller buckets in a more or less uniform fashion. With this assumption, we end up with an "exponential interpolation", which therefore appears to be a better match for histograms with exponential bucketing. This commit leaves the linear interpolation in place for NHCB, but changes the interpolation for exponential native histograms to exponential. This affects `histogram_quantile` and `histogram_fraction` (because the latter is more or less the inverse of the former). The zero bucket has to be treated specially because the assumption above would lead to an "interpolation to zero" (the bucket density approaches infinity around zero, and with the postulated uniform usage of buckets, we would end up with an estimate of zero for all quantiles ending up in the zero bucket). We simply fall back to linear interpolation within the zero bucket. At the same time, this commit makes the call to stick with the assumption that the zero bucket only contains positive observations for native histograms without negative buckets (and vice versa). (This is an assumption relevant for interpolation. It is a mostly academic point, as the zero bucket is supposed to be very small anyway. However, in cases where it _is_ relevantly broad, the assumption helps a lot in practice.) This commit also updates and completes the documentation to match both details about interpolation. As a more high level note: The approach here attempts to strike a balance between a more simplistic approach without any assumption, and a more involved approach with more sophisticated assumptions. I will shortly describe both for reference: The "zero assumption" approach would be to not interpolate at all, but _always_ return the harmonic mean of the bucket boundaries of the bucket the quantile ends up in. This has the advantage of minimizing the maximum possible relative error of the quantile estimation. (Depending on the exact definition of the relative error of an estimation, there is also an argument to return the arithmetic mean of the bucket boundaries.) While limiting the maximum possible relative error is a good property, this approach would throw away the information if a quantile is closer to the upper or lower end of the population within a bucket. This can be valuable trending information in a dashboard. With any kind of interpolation, the maximum possible error of a quantile estimation increases to the full width of a bucket (i.e. it more than doubles for the harmonic mean approach, and precisely doubles for the arithmetic mean approach). However, in return the _expectation value_ of the error decreases. The increase of the theoretical maximum only has practical relevance for pathologic distributions. For example, if there are thousand observations within a bucket, they could _all_ be at the upper bound of the bucket. If the quantile calculation picks the 1st observation in the bucket as the relevant one, an interpolation will yield a value close to the lower bucket boundary, while the true quantile value is close to the upper boundary. The "fancy interpolation" approach would be one that analyses the _actual_ distribution of samples in the histogram. A lot of statistics could be applied based on the information we have available in the histogram. This would include the population of neighboring (or even all) buckets in the histogram. In general, the resolution of a native histogram should be quite high, and therefore, those "fancy" approaches would increase the computational cost quite a bit with very little practical benefits (i.e. just tiny corrections of the estimated quantile value). The results are also much harder to reason with. Signed-off-by: beorn7 <beorn@grafana.com>	2 months ago
Jan Fajerski	15cea39136	promql: put holt_winters behind experimental feature flag Signed-off-by: Jan Fajerski <jfajersk@redhat.com>	2 months ago
Joshua Hesketh	b6107cc888	Make rate possible non-counter annotation consistent (#14910 ) * Make rate possible non-counter annotation consistent Previously a PossibleNonCounterInfo annotation would be left in cases where a range-vector selects 1 float data point, even if no more points are selected in order to calculate a rate. This change ensures an output float exists before emitting such an annotation. This fixes an inconsistency where a series with mixed data (ie, a float and a native histogram) would emit an annotation without any points. For example, ``` load 1m series{label="a"} 1 {{schema:1 sum:10 count:5 buckets:[1 2 3]}} eval instant at 1m rate(series[1m1s]) ``` Would have a PossibleNonCounterInfo annotation. Wheras ``` load 1m series{label="a"} {{schema:1 sum:10 count:5 buckets:[1 2 3]}} {{schema:1 sum:15 count:10 buckets:[1 2 3]}} eval instant at 1m rate(series[1m1s]) ``` Would not. --------- Signed-off-by: Joshua Hesketh <josh@nitrotech.org>	2 months ago
Mario Fernandez	5814920601	Fix: optimize .* regexp performance Shortcut for `.*` matches newlines as well. Add preamble change ^(?s: Add test dotAll flag por al regex Add and fix regex tests Signed-off-by: Mario Fernandez <mariofer@redhat.com>	2 months ago
Nathan Baulch	50cd453c8f	chore: Fix typos (#14868 ) * Fix typos --------- Signed-off-by: Nathan Baulch <nathan.baulch@gmail.com>	3 months ago
Jan Fajerski	91608c002f	Merge branch 'main' into release-3.0-beta.0 Conflicts: scrape/scrape_test.go Pick both changes.	3 months ago
Björn Rabenstein	8114f9a281	Merge pull request #14821 from charleskorn/nh-negative-multiplication-division promql: correctly handle unary negation of native histograms and add tests for multiplication and division of native histograms by negative scalars	3 months ago
Jan Fajerski	fa318711f4	Merge branch 'main' into 3.0-main-sync-24-09-09 Conflicts: cmd/prometheus/main.go docs/command-line/prometheus.md docs/feature_flags.md web/ui/build_ui.sh web/web.go Resolved by dropping the UTF-8 feature flag and adding the `auto-reload-config` feature flag. For the new web ui pick all changes from `main`.	3 months ago
Charles Korn	e8c7482137	Return negative counts when multiplied or divided by a negative value Signed-off-by: Charles Korn <charles.korn@grafana.com>	3 months ago
Charles Korn	2bdb3452d1	Modify parser for native histograms to allow negative values where supported Signed-off-by: Charles Korn <charles.korn@grafana.com>	3 months ago
Charles Korn	9852855084	Implement unary negation for native histograms Signed-off-by: Charles Korn <charles.korn@grafana.com>	3 months ago
Arve Knudsen	db5e48dc33	promql.Engine.Close: No-op if nil (#14861 ) Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 months ago
Björn Rabenstein	d2b659540e	Merge pull request #14819 from prometheus/beorn7/doc promql: Improve the rate extrapolation explanation	3 months ago
Jan Fajerski	befcfadf78	Fix merge conflicts Fix call to newTestEngine(t) in promql/engine_test.go:3214. `agent` feature-flag it's own cmdline flag now. Remove `scrape.name-escaping-scheme` argument. Signed-off-by: Jan Fajerski <jfajersk@redhat.com>	3 months ago
Jan Fajerski	fe4289b502	Merge branch 'main' into HEAD	3 months ago
Bryan Boreham	485523eed2	Merge pull request #14816 from bboreham/improve-promql-tracing Improve promql tracing	3 months ago
Arve Knudsen	6ef1eb4e68	Merge pull request #14064 from aknuds1/arve/close-engine Release PromQL engine resources on close, from tests	3 months ago
Charles Korn	113de6301c	Add failing test cases for unary negation and multiplication and division with negative scalars Signed-off-by: Charles Korn <charles.korn@grafana.com>	3 months ago
Charles Korn	9b451abec7	Make positive and negative bucket counts different in existing test cases Signed-off-by: Charles Korn <charles.korn@grafana.com>	3 months ago
Charles Korn	90dc1b45db	promqltest: use test expression format for histograms in assertion failure messages Signed-off-by: Charles Korn <charles.korn@grafana.com>	3 months ago
Charles Korn	e67358d203	histogram: include counter reset hint in test expression output Signed-off-by: Charles Korn <charles.korn@grafana.com>	3 months ago
beorn7	2304be9dd5	promql: Improve the rate extrapolation explanation Signed-off-by: beorn7 <beorn@grafana.com>	3 months ago
Bryan Boreham	abb0502685	[ENHANCEMENT] PromQL: Add detail to tracing spans For aggregates, operators, calls, show what operation is performed. Also add an event when series are expanded, typically time spent accessing TSDB. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 months ago
Bryan Boreham	54989ce6fd	PromQL: add short string method To be used in tracing or logging. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 months ago
Bryan Boreham	8742077498	[BUGFIX] PromQL: pass Context so spans parent correctly Assigning to `evaluator.ctx` in `eval()` broke the parent-child relationship. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 months ago
Arve Knudsen	70bb219d33	Merge pull request #14680 from charleskorn/promqltest-scalar promqltest: improve error messages when unexpected type is returned, and fix inverted expected and actual values in error message	3 months ago
Arve Knudsen	5dfbcc390e	Merge remote-tracking branch 'prometheus/main' into arve/close-engine Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 months ago
Jan Fajerski	956245b25b	promqltest: adjust eval times and range selector In order to fix new tests for changes added in https://github.com/prometheus/prometheus/pull/13904. Signed-off-by: Jan Fajerski <jfajersk@redhat.com>	3 months ago
Jan Fajerski	00315ce15e	Merge branch 'main' into 3.0-main-sync-24-08-30 using -Xours Signed-off-by: Jan Fajerski <jfajersk@redhat.com>	3 months ago
Neeraj Gartia	8c7bf39d96	Moves TestNativeHistogram_MulDivOperator to promql testing framework (#14688 ) PromQL: add test for mul and div operator Also, remove the converted test from the engine_test.go file. This also includes an extension of the test framework to allow NaN/Inf in histogram buckets. --------- Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com>	3 months ago
Jorge Creixell	e9e3d64b7c	PromQL engine: Delay deletion of __name__ label to the end of the query evaluation (#14477 ) PromQL engine: Delay deletion of __name__ label to the end of the query evaluation - This change allows optionally preserving the `__name__` label via the `label_replace` and `label_join` functions, and helps prevent the dreaded "vector cannot contain metrics with the same labelset" error. - The implementation extends the `Series` and `Sample` structs with a boolean flag indicating whether the `__name__` label should be deleted at the end of the query evaluation. - The `label_replace` and `label_join` functions can still access the value of the `__name__` label, even if it has been previously marked for deletion. If `__name__` is used as target label, it won't be dropped at the end of the query evaluation. - Fixes https://github.com/prometheus/prometheus/issues/11397 - See https://github.com/jcreixell/prometheus/pull/2 for previous discussion, including the decision to create this PR and benchmark it before considering other alternatives (like refactoring `labels.Labels`). - See https://github.com/jcreixell/prometheus/pull/1 for an alternative implementation using a special label instead of boolean flags. - Note: a feature flag `promql-delayed-name-removal` has been added as it changes the behavior of some "weird" queries (see https://github.com/prometheus/prometheus/issues/11397#issuecomment-1451998792) Example (this always fails, as `__name__` is being dropped by `count_over_time`): ``` count_over_time({__name__!=""}[1m]) => Error executing query: vector cannot contain metrics with the same labelset ``` Before: ``` label_replace(count_over_time({__name__!=""}[1m]), "__name__", "count_$1", "__name__", "(.+)") => Error executing query: vector cannot contain metrics with the same labelset ``` After: ``` label_replace(count_over_time({__name__!=""}[1m]), "__name__", "count_$1", "__name__", "(.+)") => count_go_gc_cycles_automatic_gc_cycles_total{instance="localhost:9090", job="prometheus"} 1 count_go_gc_cycles_forced_gc_cycles_total{instance="localhost:9090", job="prometheus"} 1 ... ``` Signed-off-by: Jorge Creixell <jcreixell@gmail.com> --------- Signed-off-by: Jorge Creixell <jcreixell@gmail.com> Signed-off-by: Björn Rabenstein <github@rabenste.in>	3 months ago
Arve Knudsen	99204f23ee	Merge remote-tracking branch 'prometheus/main' into arve/close-engine Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 months ago
Björn Rabenstein	849215d90c	Merge pull request #14585 from fatsheep9146/covert-TestNativeHistogram_Sum_Count_Add_AvgOperator-to-framework convert TestNativeHistogram_Sum_Count_Add_AvgOperator into testing framework	3 months ago
Owen Williams	9da75328ea	fix(utf8): ensure correct validation when legacy mode turned on (#14736 ) fix(utf8): ensure correct validation when legacy mode turned on This depends on the included update of the prometheus/common dependency. --------- Signed-off-by: Owen Williams <owen.williams@grafana.com>	3 months ago
Arve Knudsen	c9a460d570	Merge remote-tracking branch 'prometheus/main' into arve/close-engine Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 months ago
Ziqi Zhao	87eab0aaad	fix golang lint Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>	3 months ago
Ziqi Zhao	c7c4a5c347	add helper function to compare native histograms in testing Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>	3 months ago
beorn7	0f760f63dd	lint: Revamp our linting rules, mostly around doc comments Several things done here: - Set `max-issues-per-linter` to 0 so that we actually see all linter warnings and not just 50 per linter. (As we also set `max-same-issues` to 0, I assume this was the intention from the beginning.) - Stop using the golangci-lint default excludes (by setting `exclude-use-default: false`. Those are too generous and don't match our style conventions. (I have re-added some of the excludes explicitly in this commit. See below.) - Re-add the `errcheck` exclusion we have used so far via the defaults. - Exclude the signature requirement `govet` has for `Seek` methods because we use non-standard `Seek` methods a lot. (But we keep other requirements, while the default excludes completely disabled the check for common method segnatures.) - Exclude warnings about missing doc comments on exported symbols. (We used to be pretty adamant about doc comments, but stopped that at some point in the past. By now, we have about 500 missing doc comments. We may consider reintroducing this check, but that's outside of the scope of this commit. The default excludes of golangci-lint essentially ignore doc comments completely.) - By stop using the default excludes, we now get warnings back on malformed doc comments. That's the most impactful change in this commit. It does not enforce doc comments (again), but _if_ there is a doc comment, it has to have the recommended form. (Most of the changes in this commit are fixing this form.) - Improve wording/spelling of some comments in .golangci.yml, and remove an outdated comment. - Leave `package-comments` inactive, but add a TODO asking if we should change that. - Add a new sub-linter `comment-spacings` (and fix corresponding comments), which avoids missing spaces after the leading `//`. Signed-off-by: beorn7 <beorn@grafana.com>	3 months ago
Jan Fajerski	7c8c748399	promql tests: adjust range query intervals Some test queries need their interval adjusted to account for https://github.com/prometheus/prometheus/pull/13904. Otherwise the queries don't return enough samples. promql/engine_test.go:TestHistogramCopyFromIteratorRegression needed the same, but also the result needed a fix since `increase` interpolates over the full range. Signed-off-by: Jan Fajerski <jfajersk@redhat.com>	3 months ago
Björn Rabenstein	7fad1ec8ee	Merge pull request #14655 from suntala/suntala/sort-by-label-enhancement promql: Fall back to full label sets when sorting by label	3 months ago
Jan Fajerski	5138922b0d	Merge branch 'main' into 3.0-main-sync-24-08-21	3 months ago
Ziqi Zhao	8f828d45c1	convert TestNativeHistogram_Sum_Count_Add_AvgOperator into testing framework Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>	3 months ago
suntala	8e82ac8d2b	Add comments to the sort by label functions Signed-off-by: suntala <arati.rana@grafana.com>	3 months ago
Arve Knudsen	3a78e76282	Upgrade golangci-lint to v1.60.1 Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	3 months ago

1 2 3 4 5 ...

1118 Commits (be0f10054e6310a993e9e73dd97607071749ac76)