Commit Graph

907 Commits (cd72ebb05f79bdd4ddd96a65e3759e2310b173d8)

Author SHA1 Message Date
Charles Korn cd72ebb05f
promql: include more details in error message when creating test query fails or an unexpected series is returned (#13847)
* promql: include more details in error message when creating test query fails

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Include more details when an unexpected metric is returned

Signed-off-by: Charles Korn <charles.korn@grafana.com>

---------

Signed-off-by: Charles Korn <charles.korn@grafana.com>
2024-04-03 10:57:08 +02:00
Julius Volz 9b7de47787
Remove unused Dmn field on EvalNodeHelper (#13877)
https://github.com/prometheus/prometheus/pull/13446 removed the last usage of
this field, but didn't remove the field.

Signed-off-by: Julius Volz <julius.volz@gmail.com>
2024-04-02 18:45:46 +02:00
beorn7 65b4696b88 promql: Remove leftover debug output
Signed-off-by: beorn7 <beorn@grafana.com>
2024-03-27 19:02:27 +01:00
beorn7 2c1f9558b2 promql: Fix histogram comparison in test framework
The definition of histograms in the test framework may create
histograms in a non-compact form. Since histogram comparison relies on
exact equality of the bucket layout, we have to compact the histograms
created by the test framework language before comparing them to
histograms returned from the PromQL engine.

Signed-off-by: beorn7 <beorn@grafana.com>
2024-03-27 19:00:16 +01:00
Björn Rabenstein b9a2a4e329
Merge pull request #13852 from prometheus/fix-hist-std-dev-var-negative
Fix hist std dev var negative
2024-03-27 17:58:03 +01:00
Jeanette Tan 4f2df329bd improve handling of empty buckets with infinite bounds in histogram std dev/var
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2024-03-27 17:06:12 +01:00
Jeanette Tan 22d0f4f114 improve handling of negative bounds in histogram std dev/var
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2024-03-27 17:06:12 +01:00
Domantas 435f330d0b
[BUGFIX] labels: don't modify original labels in DropMetricName (#13845)
Restrict the capacity of first argument to `append()` to force an allocation.
This is for the slice implementation only.

Signed-off-by: Domantas Jadenkus <djadenkus@gmail.com>
2024-03-27 10:35:17 +00:00
Charles Korn 5cc97a1820
[tests]: extend test scripting language to support range queries (#13825)
* Extract method to make it easier to test.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Remove superfluous interface definition.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Add test cases for existing instant query functionality.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Add support for testing range queries

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Expand test coverage for instant queries and clarify error when a float is returned but a histogram is expected (or vice versa)

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Improve error message formatting

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Add test case for instant query command with invalid timestamp

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Fix linting warning.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Remove superfluous print statement and expected result

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Fix linting warning.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Add note about ordered range eval commands.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Check that matrix results are always sorted by labels.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

---------

Signed-off-by: Charles Korn <charles.korn@grafana.com>
2024-03-26 11:22:22 +00:00
Bryan Boreham 080d440bf8 Merge remote-tracking branch 'origin/main' into pr/13461 2024-03-25 12:14:26 +00:00
Bryan Boreham 773170f372
Merge pull request #13822 from dgl/promtool-test-errors
promtool: Avoid using testify for user rule tests
2024-03-23 09:42:34 +01:00
Jeanette Tan 9d32754bc0 add unit tests with all negative values for histogram_stddev and var
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2024-03-22 03:42:50 +08:00
David Leadbeater 7ec4a11472 promtool: Avoid using testify for user rule tests
Using testify outside of unit tests results in panics rather than a
useful error for the user.

Fixes #13703

Signed-off-by: David Leadbeater <dgl@dgl.cx>
2024-03-21 22:08:10 +11:00
tdakkota f6834c347a
promql: validate `label_join` destination label
Signed-off-by: tdakkota <tanc13@yandex.ru>
2024-03-20 22:02:10 +03:00
Bartlomiej Plotka 312e3fd728
Merge pull request #13713 from charleskorn/query-engine-interface
rules: allow using alternative PromQL engines for rule evaluation by callers using Prometheus as a lib.
2024-03-13 14:45:42 +01:00
Björn Rabenstein af3618fd35
Merge pull request #13667 from prometheus/beorn7/promql
Improve TestQueryStatistics and fix bugs exposed by it
2024-03-12 16:17:11 +01:00
Charles Korn 26262a1eb7
Remove unnecessary `SetQueryLogger` method on `QueryEngine` interface
Signed-off-by: Charles Korn <charles.korn@grafana.com>
2024-03-12 22:01:01 +11:00
carrychair 856f6e49c8 fix function and struct name
Signed-off-by: carrychair <linghuchong404@gmail.com>
2024-03-09 17:53:17 +08:00
beorn7 7f912db15a promql: Fix limiting of extrapolation to negative values
This is a bit tough to explain, but I'll try:

`rate` & friends have a sophisticated extrapolation algorithm.
Usually, we extrapolate the result to the total interval specified in
the range selector. However, if the first sample within the range is
too far away from the beginning of the interval, or if the last sample
within the range is too far away from the end of the interval, we
assume the series has just started half a sampling interval before the
first sample or after the last sample, respectively, and shorten the
extrapolation interval correspondingly. We calculate the sampling
interval by looking at the average time between samples within the
range, and we define "too far away" as "more than 110% of that
sampling interval".

However, if this algorithm leads to an extrapolated starting value
that is negative, we limit the start of the extrapolation interval to
the point where the extrapolated starting value is zero.

At least that was the intention.

What we actually implemented is the following: If extrapolating all
the way to the beginning of the total interval would lead to an
extrapolated negative value, we would only extrapolate to the zero
point as above, even if the algorithm above would have selected a
starting point that is just half a sampling interval before the first
sample and that starting point would not have an extrapolated negative
value. In other word: What was meant as a _limitation_ of the
extrapolation interval yielded a _longer_ extrapolation interval in
this case.

There is an exception to the case just described: If the increase of
the extrapolation interval is more than 110% of the sampling interval,
we suddenly drop back to only extrapolate to half a sampling interval.

This behavior can be nicely seen in the testcounter_zero_cutoff test,
where the rate goes up all the way to 0.7 and then jumps back to 0.6.

This commit changes the behavior to what was (presumably) intended
from the beginning: The extension of the extrapolation interval is
only limited if actually needed to prevent extrapolation to negative
values, but the "limitation" never leads to _more_ extrapolation
anymore.

The difference is subtle, and probably it never bothered anyone.
However, if you calculate a rate of a classic histograms, the old
behavior might create non-monotonic histograms as a result (because of
the jumps you can see nicely in the old version of the
testcounter_zero_cutoff test). With this fix, that doesn't happen
anymore.

Signed-off-by: beorn7 <beorn@grafana.com>
2024-03-07 01:20:33 +01:00
Charles Korn 4e77e8e5ef
Allow using alternative PromQL engines for rule evaluation
Signed-off-by: Charles Korn <charles.korn@grafana.com>
2024-03-06 14:54:33 +11:00
beorn7 f48c7a5503 promql: Add histograms to TestQueryStatistics
Also, fix the bugs exposed by the tests.

Signed-off-by: beorn7 <beorn@grafana.com>
2024-02-29 19:02:40 +01:00
beorn7 f46dd34982 promql: Add code comment
Signed-off-by: beorn7 <beorn@grafana.com>
2024-02-29 19:02:40 +01:00
beorn7 7d364c0451 promql: remove redundant line
Signed-off-by: beorn7 <beorn@grafana.com>
2024-02-29 19:02:40 +01:00
Björn Rabenstein d5f0a240fa
Merge pull request #13674 from prometheus/owilliams/dupeok
fix: restore ability to match __name__ multiple times in selector
2024-02-29 17:49:15 +01:00
Björn Rabenstein 9187bcbdd5
Merge pull request #13536 from bboreham/faster-label-replace
promql: faster range-query of label_replace and label_join
2024-02-29 17:03:00 +01:00
Owen Williams e01e7d36e2 fix: restore ability to match __name__ multiple times in selector
Add tests to cover this case.

Signed-off-by: Owen Williams <owen.williams@grafana.com>
2024-02-29 10:46:31 -05:00
machine424 f477e0539a
Move from golang.org/x/exp/slices into slices now that we only support Go >= 1.21
Prevent adding back golang.org/x/exp/slices.

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2024-02-28 14:54:53 +01:00
Björn Rabenstein 11932cd345
Merge pull request #13637 from bboreham/agg-warning
PromQL: improve warning for mixed values in aggregations
2024-02-28 13:52:30 +01:00
Owen Williams ac51a8024c tests(utf8): confirm that other quote marks are handled correctly in promql
Signed-off-by: Owen Williams <owen.williams@grafana.com>
2024-02-27 15:47:47 -05:00
Bryan Boreham 0347148628 promql: fuzz test needs symbol table for parser
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-26 11:45:25 +00:00
Bryan Boreham 22890b1eb3 PromQL: improve warning for mixed values in aggregations
Aggregations discard the metric name, so don't try to
include it in the error message.

Add a test that generates this warning.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-25 19:57:06 +00:00
Owen Williams a28d7865ad UTF-8: Add support for parsing UTF8 metric and label names
This adds support for the new grammar of `{"metric_name", "l1"="val"}` to promql and some of the exposition formats.
This grammar will also be valid for non-UTF-8 names.
UTF-8 names will not be considered valid unless model.NameValidationScheme is changed.

This does not update the go expfmt parser in text_parse.go, which will be addressed by https://github.com/prometheus/common/issues/554/.

Part of https://github.com/prometheus/prometheus/issues/13095

Signed-off-by: Owen Williams <owen.williams@grafana.com>
2024-02-15 14:34:37 -05:00
Bryan Boreham ff6c83269c
Merge pull request #13452 from bboreham/go-cmp
Tests: Use DeepEqual replacement using go-cmp, which is more flexible
2024-02-12 15:40:08 +01:00
Bryan Boreham 17f48f2b3b Tests: use replacement DeepEquals in more places
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-08 19:32:33 +00:00
Bryan Boreham 39af788dbd Tests: use replacement DeepEquals using go-cmp
Use DeepEqual replacement using go-cmp, which is more flexible.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-08 19:30:20 +00:00
beorn7 86d7618d84 promql: Fix wrongly scoped range vectors
Fixes #11708.

If a range vector is fixen in time with the @ modifier, it gets still
moved around for different steps in a range query. Since no additional
points are retrieved from the TSDB, this leads to steadily emptying
the range, leading to the weird behavior described in isse #11708.

This only happens for functions listed in `AtModifierUnsafeFunctions`,
and the only of those that takes a range vector is `predict_linear`,
which is the reason why we see it only for this particular function.

Signed-off-by: beorn7 <beorn@grafana.com>
2024-02-07 18:07:51 +01:00
beorn7 384ab025e0 promql: Expose issue #11708
The shorter range in the test makes early points drop out of the range
in range queries, exposing the issue.

Signed-off-by: beorn7 <beorn@grafana.com>
2024-02-07 18:06:05 +01:00
Bryan Boreham 98c4889029
Merge pull request #9298 from Creatone/creatone/use-testify
tests: Move from t.Errorf and others.
2024-02-04 16:27:57 +01:00
Bryan Boreham d3c1f0d8e0 promql: can now remove regex field from EvalNodeHelper
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-04 11:32:26 +01:00
Bryan Boreham fdd5b85e06 promql: faster range-query of label_replace and label_join
These functions act on the labels only, so don't need to go step by step
over the samples in a range query.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-04 11:12:55 +01:00
Faustas Butkus 6feffeb92e
promql: add histogram_avg function (#13467)
Add histogram_avg function

---------

Signed-off-by: Faustas Butkus <faustas.butkus@gmail.com>
Signed-off-by: Björn Rabenstein <github@rabenste.in>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
2024-02-01 18:28:42 +01:00
Alan Protasio c006c57efc
Proposal to improve FPointSlice and HPointSlice allocation. (#13448)
* Reusing points slice from previous series when the slice is under utilized
* Adding comments on the bench test

Signed-off-by: Alan Protasio <alanprot@gmail.com>
2024-02-01 16:22:38 +00:00
Paweł Szulik 1a47c7d59b Refactor lexer tests to use testify.
Signed-off-by: Paweł Szulik <paul.szulik@gmail.com>
2024-02-01 13:51:31 +00:00
Filip Petkovski a577a0a542
Fix last_over_time for native histograms
The last_over_time retains a histogram sample without making a copy.
This sample is now coming from the buffered iterator used for windowing functions,
and can be reused for reading subsequent samples as the iterator progresses.

I would propose copying the sample in the last_over_time function, similar to
how it is done for rate, sum_over_time and others.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2024-01-26 15:02:40 +01:00
Marco Pracucci f639d7794c
Fix TestParseExpressions
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-01-25 14:57:43 +01:00
Bryan Boreham 74b73d1e2c
Labels: Add DropMetricName function, used in PromQL (#13446)
This function is called very frequently when executing PromQL functions,
and we can do it much more efficiently inside Labels.

In the common case that `__name__` comes first in the labels, we simply
re-point to start at the next label, which is nearly free.

`DropMetricName` is now so cheap I removed the cache - benchmarks show
everything still goes faster.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-01-25 11:48:49 +01:00
Filip Petkovski 583f3e587c
Optimize histogram iterators (#13340)
Optimize histogram iterators

Histogram iterators allocate new objects in the AtHistogram and
AtFloatHistogram methods, which makes calculating rates over long
ranges expensive.

In #13215 we allowed an existing object to be reused
when converting an integer histogram to a float histogram. This commit follows
the same idea and allows injecting an existing object in the AtHistogram and
AtFloatHistogram methods. When the injected value is nil, iterators allocate
new histograms, otherwise they populate and return the injected object.

The commit also adds a CopyTo method to Histogram and FloatHistogram which
is used in the BufferedIterator to overwrite items in the ring instead of making
new copies.

Note that a specialized HPoint pool is needed for all of this to work 
(`matrixSelectorHPool`).

---------

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
2024-01-23 17:02:14 +01:00
Ben Ye d778591fd3 add more context cancellation check at evaluation time
Signed-off-by: Ben Ye <benye@amazon.com>
2024-01-21 14:19:39 -08:00
Björn Rabenstein bfbb13cf36
Merge pull request #13267 from linasm/simplify-native-histogram-math
promql: simplify Native Histogram arithmetics
2024-01-18 13:50:59 +01:00
Julien Pivotto 4f941bbf69
Merge pull request #13416 from tylitianrui/feat/remove_obsolete_build_tag
remove  obsolete build tag
2024-01-17 18:12:21 +01:00