Commit Graph

1091 Commits (91d80252c3e528728b0f88d254dd720f6be07cb8)

Author SHA1 Message Date
Arve Knudsen de16f5e387
[FEATURE] PromQL: Add experimental info function MVP (#14495)
The `info` function is an experiment to improve UX
around including labels from info metrics.
`info` has to be enabled via the feature flag `--enable-feature=promql-experimental-functions`.

This MVP of info simplifies the implementation by assuming:
* Only support for the target_info metric
* That target_info's identifying labels are job and instance

Also:
* Encode info samples' original timestamp as sample value
* Deduce info series select hints from top-most VectorSelector

---------

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Ying WANG <ying.wang@grafana.com>
Co-authored-by: Augustin Husson <augustin.husson@amadeus.com>
Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
Co-authored-by: Bryan Boreham <bjboreham@gmail.com>
2024-10-16 13:52:11 +01:00
Arve Knudsen e05e97cdd7 evaluator.rangeEval: Split out gatherVector method
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-10-16 14:01:03 +02:00
Arve Knudsen f7b396a1dc promql.Engine: Refactor vector selector evaluation into a method (#14900)
New method is named `evalVectorSelector`.

---------

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-10-15 14:57:54 +01:00
Bryan Boreham 7f99d2930d [BUGFIX] PromQL: make sort_by_label stable
Go's sorting functions can re-order equal elements, so the strategy of
sorting by the fallback ordering first does not always work.
Pulling the fallback into the main comparison function is more reliable
and more efficient.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-09-26 11:12:33 +01:00
Björn Rabenstein 1639450172 Merge pull request #14821 from charleskorn/nh-negative-multiplication-division
promql: correctly handle unary negation of native histograms and add tests for multiplication and division of native histograms by negative scalars
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-09-19 14:07:37 +01:00
Arve Knudsen db5e48dc33
promql.Engine.Close: No-op if nil (#14861)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-09-08 14:39:13 +02:00
Björn Rabenstein d2b659540e
Merge pull request #14819 from prometheus/beorn7/doc
promql: Improve the rate extrapolation explanation
2024-09-05 12:21:03 +02:00
Bryan Boreham 485523eed2
Merge pull request #14816 from bboreham/improve-promql-tracing
Improve promql tracing
2024-09-04 14:32:22 +01:00
Arve Knudsen 6ef1eb4e68
Merge pull request #14064 from aknuds1/arve/close-engine
Release PromQL engine resources on close, from tests
2024-09-04 12:07:35 +02:00
beorn7 2304be9dd5 promql: Improve the rate extrapolation explanation
Signed-off-by: beorn7 <beorn@grafana.com>
2024-09-03 22:11:14 +02:00
Bryan Boreham abb0502685 [ENHANCEMENT] PromQL: Add detail to tracing spans
For aggregates, operators, calls, show what operation is performed.
Also add an event when series are expanded, typically time spent
accessing TSDB.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-09-03 14:15:58 +01:00
Bryan Boreham 54989ce6fd PromQL: add short string method
To be used in tracing or logging.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-09-03 14:15:58 +01:00
Bryan Boreham 8742077498 [BUGFIX] PromQL: pass Context so spans parent correctly
Assigning to `evaluator.ctx` in `eval()` broke the parent-child relationship.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-09-03 14:15:29 +01:00
Arve Knudsen 70bb219d33
Merge pull request #14680 from charleskorn/promqltest-scalar
promqltest: improve error messages when unexpected type is returned, and fix inverted expected and actual values in error message
2024-09-03 10:38:04 +02:00
Arve Knudsen 5dfbcc390e Merge remote-tracking branch 'prometheus/main' into arve/close-engine
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-09-02 16:26:59 +02:00
Neeraj Gartia 8c7bf39d96
Moves TestNativeHistogram_MulDivOperator to promql testing framework (#14688)
PromQL: add test for mul and div operator

Also, remove the converted test from the engine_test.go file.

This also includes an extension of the test framework to allow NaN/Inf in histogram buckets.

---------

Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com>
2024-08-29 16:42:35 +02:00
Jorge Creixell e9e3d64b7c
PromQL engine: Delay deletion of __name__ label to the end of the query evaluation (#14477)
PromQL engine: Delay deletion of __name__ label to the end of the query evaluation

  - This change allows optionally preserving the `__name__` label via the `label_replace` and `label_join` functions, and helps prevent the dreaded "vector cannot contain metrics with the same labelset" error.
  - The implementation extends the `Series` and `Sample` structs with a boolean flag indicating whether the `__name__` label should be deleted at the end of the query evaluation.
  - The `label_replace` and `label_join` functions can still access the value of the `__name__` label, even if it has been previously marked for deletion. If  `__name__` is used as target label, it won't be dropped at the end of the query evaluation.
  - Fixes https://github.com/prometheus/prometheus/issues/11397
  - See https://github.com/jcreixell/prometheus/pull/2 for previous discussion, including the decision to create this PR and benchmark it before considering other alternatives (like refactoring `labels.Labels`).
  - See https://github.com/jcreixell/prometheus/pull/1 for an alternative implementation using a special label instead of boolean flags.
  - Note: a feature flag  `promql-delayed-name-removal` has been added as it changes the behavior of some "weird" queries (see https://github.com/prometheus/prometheus/issues/11397#issuecomment-1451998792)

Example (this always fails, as `__name__` is being dropped by `count_over_time`):

```
count_over_time({__name__!=""}[1m])

=> Error executing query: vector cannot contain metrics with the same labelset
```

Before:

```
label_replace(count_over_time({__name__!=""}[1m]), "__name__", "count_$1", "__name__", "(.+)")

=> Error executing query: vector cannot contain metrics with the same labelset
```

After:

```
label_replace(count_over_time({__name__!=""}[1m]), "__name__", "count_$1", "__name__", "(.+)")

=>
count_go_gc_cycles_automatic_gc_cycles_total{instance="localhost:9090", job="prometheus"} 1
count_go_gc_cycles_forced_gc_cycles_total{instance="localhost:9090", job="prometheus"} 1
...
```

Signed-off-by: Jorge Creixell <jcreixell@gmail.com>

---------

Signed-off-by: Jorge Creixell <jcreixell@gmail.com>
Signed-off-by: Björn Rabenstein <github@rabenste.in>
2024-08-29 15:50:39 +02:00
Arve Knudsen 99204f23ee Merge remote-tracking branch 'prometheus/main' into arve/close-engine
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-08-29 09:52:54 +02:00
Björn Rabenstein 849215d90c
Merge pull request #14585 from fatsheep9146/covert-TestNativeHistogram_Sum_Count_Add_AvgOperator-to-framework
convert TestNativeHistogram_Sum_Count_Add_AvgOperator into testing framework
2024-08-28 17:21:32 +02:00
Owen Williams 9da75328ea
fix(utf8): ensure correct validation when legacy mode turned on (#14736)
fix(utf8): ensure correct validation when legacy mode turned on

This depends on the included update of the prometheus/common dependency.

---------

Signed-off-by: Owen Williams <owen.williams@grafana.com>
2024-08-28 17:15:42 +02:00
Arve Knudsen c9a460d570 Merge remote-tracking branch 'prometheus/main' into arve/close-engine
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-08-26 12:17:10 +02:00
Ziqi Zhao 87eab0aaad fix golang lint
Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
2024-08-23 09:32:04 +08:00
Ziqi Zhao c7c4a5c347 add helper function to compare native histograms in testing
Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
2024-08-23 09:28:31 +08:00
beorn7 0f760f63dd lint: Revamp our linting rules, mostly around doc comments
Several things done here:

- Set `max-issues-per-linter` to 0 so that we actually see all linter
  warnings and not just 50 per linter. (As we also set
  `max-same-issues` to 0, I assume this was the intention from the
  beginning.)

- Stop using the golangci-lint default excludes (by setting
  `exclude-use-default: false`. Those are too generous and don't match
  our style conventions. (I have re-added some of the excludes
  explicitly in this commit. See below.)

- Re-add the `errcheck` exclusion we have used so far via the
  defaults.

- Exclude the signature requirement `govet` has for `Seek` methods
  because we use non-standard `Seek` methods a lot. (But we keep other
  requirements, while the default excludes completely disabled the
  check for common method segnatures.)

- Exclude warnings about missing doc comments on exported symbols. (We
  used to be pretty adamant about doc comments, but stopped that at
  some point in the past. By now, we have about 500 missing doc
  comments. We may consider reintroducing this check, but that's
  outside of the scope of this commit. The default excludes of
  golangci-lint essentially ignore doc comments completely.)

- By stop using the default excludes, we now get warnings back on
  malformed doc comments. That's the most impactful change in this
  commit. It does not enforce doc comments (again), but _if_ there is
  a doc comment, it has to have the recommended form. (Most of the
  changes in this commit are fixing this form.)

- Improve wording/spelling of some comments in .golangci.yml, and
  remove an outdated comment.

- Leave `package-comments` inactive, but add a TODO asking if we
  should change that.

- Add a new sub-linter `comment-spacings` (and fix corresponding
  comments), which avoids missing spaces after the leading `//`.

Signed-off-by: beorn7 <beorn@grafana.com>
2024-08-22 17:36:11 +02:00
Björn Rabenstein 7fad1ec8ee
Merge pull request #14655 from suntala/suntala/sort-by-label-enhancement
promql: Fall back to full label sets when sorting by label
2024-08-21 12:28:55 +02:00
Ziqi Zhao 8f828d45c1 convert TestNativeHistogram_Sum_Count_Add_AvgOperator into testing framework
Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
2024-08-21 09:24:50 +08:00
suntala 8e82ac8d2b Add comments to the sort by label functions
Signed-off-by: suntala <arati.rana@grafana.com>
2024-08-20 20:41:55 +02:00
Arve Knudsen 3a78e76282 Upgrade golangci-lint to v1.60.1
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-08-18 12:13:25 +02:00
Charles Korn 6548421f61
promqltest: improve error messages when unexpected type is returned, and fix inverted expected and actual values in error message when scalar value does not match expected
Signed-off-by: Charles Korn <charles.korn@grafana.com>
2024-08-16 13:37:05 +10:00
Björn Rabenstein 1daf7cdd62
Merge pull request #14626 from cuiweiyuan/main
chore: fix some function names
2024-08-15 11:46:21 +02:00
cuiweiyuan 1800af54f0 chore: fix some function names
Signed-off-by: cuiweiyuan <cuiweiyuan@aliyun.com>
2024-08-15 13:57:21 +08:00
Charles Korn 52818a97e2
Merge branch 'main' into sum-and-avg-over-mixed-custom-exponential-histograms
# Conflicts:
#	promql/promqltest/testdata/native_histograms.test
2024-08-14 07:52:08 +10:00
Björn Rabenstein c2bc6cfe97
Merge pull request #14621 from charleskorn/panic-message
promql: clarify error message logged when panic occurs during query evaluation
2024-08-13 23:02:43 +02:00
George Krajcsovits fe5354df88
Merge pull request #14658 from krajorama/zero-rate-nh-quantile
Native histograms: define behavior when rate is null.
2024-08-13 16:05:43 +02:00
György Krajcsovits 386fc8b9f6 Update from review comments.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-08-13 15:26:07 +02:00
Arve Knudsen 0503d4f372 PromQL: Fix comment regarding non-nil histogram pointer
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-08-13 08:55:24 +02:00
György Krajcsovits 6aee5b4b38 fix typo
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-08-12 12:04:45 +02:00
György Krajcsovits 06a8886b94 Native histograms: define behavior when rate is null.
Histogram quantile returns NaN in this case, which might be
surprising, so add a unit test that clarifies that this is
intentional.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-08-12 10:40:21 +02:00
suntala fd2f44af7f Fall back to comparing by label set when sorting by label desc
Co-authored-by: Aleks Fazlieva <britishrum@users.noreply.github.com>
Signed-off-by: suntala <arati.rana@grafana.com>
2024-08-11 21:44:03 +02:00
suntala 94ad489328 Fall back to comparing by label set when sorting by label
Co-authored-by: Aleks Fazlieva <britishrum@users.noreply.github.com>
Signed-off-by: suntala <arati.rana@grafana.com>
2024-08-11 21:44:03 +02:00
Charles Korn f992f81bd0
Merge branch 'main' into sum-and-avg-over-mixed-custom-exponential-histograms
Signed-off-by: Charles Korn <charleskorn@users.noreply.github.com>
2024-08-09 13:58:54 +10:00
Charles Korn 5cfdde327c
Address PR feedback: add extra test case
Signed-off-by: Charles Korn <charles.korn@grafana.com>
2024-08-09 13:57:37 +10:00
Charles Korn 82bb35fabb
Address PR feedback: fix typo and rename variable
Signed-off-by: Charles Korn <charles.korn@grafana.com>
2024-08-09 13:51:31 +10:00
Björn Rabenstein 41f8a0223c
Merge pull request #14609 from charleskorn/rate-over-mixed-custom-exponential-histograms
promql: fix "cannot reduce resolution to custom buckets schema" panic in `rate` over native histograms with mix of custom and exponential buckets
2024-08-08 14:03:41 +02:00
Manik Rana 02c465bf58
textparse: Implement `CreatedTimestamp()` in `openmetricsparse.go` (#14356)
* feat: initial implement of createedTimestamp() with tests

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* feat: return ct after finding it

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* chore: remove unneeded test

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* chore: add comments

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* feat: multiple changes

- implement changes from pair programming session
- use newParse.val()
- advance parser p if ct is found

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* fix: check if err from p.Next()

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* feat: advance parser and parse histograms + summary

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* fix: restore previous tests

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* fix: retore failing tests

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* chore: remove unneeded comments

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* fix: return nil when mtype doesn't match

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* chore: update go fmt version

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* chore: cleanup

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* fix: comments

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* feat: document deepcopyparser

Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com>
Signed-off-by: Manik Rana <Manikrana54@gmail.com>
Signed-off-by: Manik Rana <manikrana54@gmail.com>

* chore: lint

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* fix: cover edgecase of `gauge_created` in CreatedTimestamp()

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* refac: readability updates

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* refac: dedeuplicate labeldiff checks

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* tests: add tests for new label functions

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* feat: document CreatedTimestamp func

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* refac: optimize `CreatedTimestamp()`

 - Use refactored CreatedTimestamp function with bug fixes
 - Remove unused code in labels.go
 - Improve code documentation

 Signed-off-by: Manik Rana <manikrana54@gmail.com>

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* chore: add tests and lint fixes

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* chore: remove mName

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* chore: lint

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* chore: comments

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* tests: add tests for CT parse failures and deepCopy

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* refac: edit expectCT struct

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* chore: lint

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* chore: add new label in deepCopy

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* fix: use p.builder in deepCopy

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* fix: add NewMetricsParserWithOpts

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* chore: lint

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* chore: comments

Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com>
Signed-off-by: Manik Rana <Manikrana54@gmail.com>

* chore: comments

Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com>
Signed-off-by: Manik Rana <Manikrana54@gmail.com>

* chore: rename var

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* fix: add condition for OM fuzzing

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* fix: build tags

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* refac: default skipCT to false

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* refac: rename skipCT to skipCTSeries

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* chore: formatting

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* chore: comments and readability updates

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* chore: comments

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Manik Rana <Manikrana54@gmail.com>

* refac: remove NewOpenMetricsParserWithOpts

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* chore: lint

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* refac: extract skipCTSeries logic from parseMetricSuffix

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* refac: inline create a NewOpenMetricsParser

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* chore: comments

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* chore: comments

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Manik Rana <Manikrana54@gmail.com>

* refac: improve error handling

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* fix: return error instead of nil

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* fix: remove skipCT check from tBraceOpen

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* Pair programming with Manik, Arthur and Daniel.

Signed-off-by: bwplotka <bwplotka@gmail.com>

* chore: comments and use helper funcs

Signed-off-by: Manik Rana <manikrana54@gmail.com>

* chore: lint

Signed-off-by: Manik Rana <manikrana54@gmail.com>

---------

Signed-off-by: Manik Rana <manikrana54@gmail.com>
Signed-off-by: Manik Rana <Manikrana54@gmail.com>
Signed-off-by: bwplotka <bwplotka@gmail.com>
Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com>
Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
2024-08-08 12:35:35 +01:00
Charles Korn f91009aa2e
promql: clarify error message when panic occurs during query evaluation
Signed-off-by: Charles Korn <charles.korn@grafana.com>
2024-08-08 09:11:38 +10:00
Björn Rabenstein 27579c9148
Merge pull request #14605 from krajorama/fix-staleness-pool-corrupt
Fix histogram pool poisoning bug chunkenc.Iterator
2024-08-07 21:02:08 +02:00
George Krajcsovits 17b0b788da
Update promql/engine.go
Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>
2024-08-07 20:15:46 +02:00
Charles Korn 0f4bc87b4f
Make linter happy
Signed-off-by: Charles Korn <charles.korn@grafana.com>
2024-08-07 15:35:06 +10:00
Charles Korn f07b3ae67b
Fix issue where `avg` over mixed exponential and custom buckets, or incompatible custom buckets, produces incorrect results or panics
Signed-off-by: Charles Korn <charles.korn@grafana.com>
2024-08-07 15:32:35 +10:00