Commit Graph

13155 Commits (3a908d8e088763e6df7f7a5f74345ada30c9b0bb)

Author SHA1 Message Date
beorn7 3a908d8e08 promql: Improve Kahan usage in avg_over_time
The calculation of the mean value in avg_over_time is performed in an
incremental fashion. This introduces additional numerical errors that
even Kahan summation cannot compensate, but at least we can use the
Kahan-corrected mean value when we use the intermediate mean value in
the calculation.

Signed-off-by: beorn7 <beorn@grafana.com>
2024-07-10 19:20:24 +02:00
beorn7 c46074f4dd promql: make avg aggregation more precise and less expensive
The basic idea here is that the previous code was always doing
incremental calculation of the mean value, which is more costly and
can be less precise. It protects against overflows, but in most cases,
an overflow doesn't happen anyway.

The other idea applied here is to expand on #14074, where Kahan
summation was applied to sum().

With this commit, the average is calculated in a conventional way
(adding everything up and divide in the end) as long as the sum isn't
overflowing float64. This is combined with Kahan summation so that the
avg aggregation, in most cases, is really equivalent to the sum
aggregation with a following division (which is the user's expectation
as avg is supposed to be syntactic sugar for sum with a following
divison).

If the sum hits ±Inf, the calculation reverts to incremental
calculation of the mean value. Kahan summation is also applied here,
although it cannot fully compensate for the numerical errors
introduced by the incremental mean calculation. (The tests added in
this commit would fail if incremental mean calculation was always
used.)

Signed-off-by: beorn7 <beorn@grafana.com>
2024-07-10 19:20:24 +02:00
beorn7 b5b04ddbe3 promql: add avg aggregation benchmark
Signed-off-by: beorn7 <beorn@grafana.com>
2024-07-04 18:59:30 +02:00
beorn7 44d8c1d182 nit: add period at end of sentence
Signed-off-by: beorn7 <beorn@grafana.com>
2024-07-04 17:31:39 +02:00
beorn7 9a837b7f3c promql: Make groupedAggregation.groupCount a float64
It's always used as such. Let's avoid the countless conversions.

Signed-off-by: beorn7 <beorn@grafana.com>
2024-07-04 17:31:34 +02:00
Manik Rana c9bc1c2be0
Update histogram math (#13680)
ui: Fix many edge cases of rendering a histogram

---------

Signed-off-by: Manik Rana <manikrana54@gmail.com>
2024-07-04 14:01:09 +02:00
JuanJo Ciarlante c94c5b64c3
feat: add limitk() and limit_ratio() operators (#12503)
* rebase 2024-07-01, picks previous renaming to `limitk()` and `limit_ratio()`

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* gofumpt -d -extra

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* more lint fixes

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* more lint fixes+

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* put limitk() and limit_ratio() behind --enable-feature=promql-experimental-functions

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* EnableExperimentalFunctions for TestConcurrentRangeQueries() also

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* use testutil.RequireEqual to fix tests, WIP equivalent thingie for require.Contains

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* lint fix

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* moar linting

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* rebase 2024-06-19

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* re-add limit(2, metric) testing for N=2 common series subset

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* move `ratio = param` to default switch case, for better readability

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* gofumpt -d -extra util/testutil/cmp.go

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* early break when reaching k elems in limitk(), should have always been so (!)

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* small typo fix

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* no-change small break-loop rearrange for readability

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* remove IsNan(ratio) condition in switch-case, already handled as input validation

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* no-change adding some comments

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* no-change simplify fullMatrix() helper functions used for tests

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* add `limitk(-1, metric)` testcase, which is handled as any k < 1 case

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* engine_test.go: no-change create `requireCommonSeries() helper func (moving code into it) for readability

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* rebase 2024-06-21

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* engine_test.go: HAPPY NOW about its code -> reorg, create and use simpleRangeQuery() function, less lines and more readable ftW \o/

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* move limitk(), limit_ratio() testing to promql/promqltest/testdata/limit.test

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* remove stale leftover after moving tests from engine_test.go to testdata/

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* fix flaky `limit_ratio(0.5, ...)` test case

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* Update promql/engine.go

Co-authored-by: Julius Volz <julius.volz@gmail.com>
Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* Update promql/engine.go

Co-authored-by: Julius Volz <julius.volz@gmail.com>
Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* Update promql/engine.go

Co-authored-by: Julius Volz <julius.volz@gmail.com>
Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* fix AddRatioSample() implementation to use a single conditional (instead of switch/case + fallback return)

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* docs/querying/operators.md: document r < 0

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* add negative limit_ratio() example to docs/querying/examples.md

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* move more extensive docu examples to docs/querying/operators.md

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* typo

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* small docu fix for poor-mans-normality-check, add it to limit.test ;)

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* limit.test: expand "Poor man's normality check" to whole eval range

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* restore mistakenly removed existing small comment

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* expand poors-man-normality-check case(s)

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* Revert "expand poors-man-normality-check case(s)"

This reverts commit f69e1603b2ebe69c0a100197cfbcf6f81644b564, indeed too
flaky 0:)

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* remove humor from docs/querying/operators.md

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* fix signoff

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* add web/ui missing changes

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* expand limit_ratio test cases, cross-fingering they'll not be flaky

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* remove flaky test

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* add missing warnings.Merge(ws) in instant-query return shortcut

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* add missing LimitK||LimitRatio case to codemirror-promql/src/parser/parser.ts

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* fix ui-lint

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

* actually fix returned warnings :]

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>

---------

Signed-off-by: JuanJo Ciarlante <juanjosec@gmail.com>
Co-authored-by: Julius Volz <julius.volz@gmail.com>
2024-07-03 22:18:57 +02:00
Bryan Boreham 82a8c6abe2
[ENHANCEMENT] Optimize regexps with multiple prefixes (#13843)
For example `foo.*|bar.*|baz.*`. Instead of checking each one in turn,
we build a map of prefixes, then check the smaller set that could match
the string supplied.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Improve testing and readability

Address review comments on #13843

Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-07-03 18:45:36 +01:00
Carrie Edwards ba948e94fb
Merge pull request #13699 from prometheus/cedwards/tsdb-test-helpers
Refactor tsdb tests to use test util functions to test different sample types
2024-07-03 09:50:29 -07:00
Carrie Edwards ee909b340b Fix test
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
2024-07-03 09:28:38 -07:00
Carrie Edwards 00499006c0 Testutil refactoring
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
2024-07-03 09:28:38 -07:00
Carrie Edwards 55f53330b2 Use storage.ExpandSamples instead of samplesFromIterator
Co-authored by: Fiona Liao <fiona.liao@grafana.com>:

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
2024-07-03 09:28:38 -07:00
Carrie Edwards 06550883c1 Clean up of tests and test utils
Co-authored by: Fiona Liao <fiona.liao@grafana.com>:

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
2024-07-03 09:28:38 -07:00
Carrie Edwards cb76884352 Fix linting
Co-authored by: Fiona Liao <fiona.liao@grafana.com>:

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
2024-07-03 09:28:38 -07:00
Carrie Edwards 45a32a29ef Update tsdb tests to use test utils.
Co-authored-by: Fiona Liao <fiona.liao@grafana.com>
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
2024-07-03 09:28:38 -07:00
Carrie Edwards 60917f628b Add test utilities for testing different sample types.
Co-authored-by: Fiona Liao <fiona.liao@grafana.com>
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
2024-07-03 09:28:38 -07:00
Bryan Boreham 134e8dc7af
TSDB: Simplify OOO Select by copying the head chunk (#14396)
Instead of carrying around extra fields in `Meta` structs which let us
approximate what was in the chunk at the time, take a copy of the chunk.

This simplifies lots of code, and lets us correct a couple of tests which
were embedding the wrong answer.

We can also remove boundedIterator, which was only used to constrain
the OOO head chunk.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-07-03 15:08:07 +01:00
Ganesh Vernekar 3d54bcc018
Merge pull request #14362 from charleskorn/charleskorn/sum-infinity 2024-07-03 01:05:03 -04:00
Oleg Zaytsev 726ed124e4
Replace `ListPostings.Seek`'s binary search call by the generic `slices.BinarySearch` (#14393)
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2024-07-02 14:51:05 +01:00
Bartlomiej Plotka 9cf4b969dd
Merge pull request #14388 from prometheus/dependabot/go_modules/documentation/examples/remote_storage/github.com/prometheus/common-0.55.0
build(deps): bump github.com/prometheus/common from 0.54.0 to 0.55.0 in /documentation/examples/remote_storage
2024-07-02 12:30:50 +02:00
dependabot[bot] c53fdade42
build(deps): bump github.com/prometheus/common
Bumps [github.com/prometheus/common](https://github.com/prometheus/common) from 0.54.0 to 0.55.0.
- [Release notes](https://github.com/prometheus/common/releases)
- [Changelog](https://github.com/prometheus/common/blob/main/RELEASE.md)
- [Commits](https://github.com/prometheus/common/compare/v0.54.0...v0.55.0)

---
updated-dependencies:
- dependency-name: github.com/prometheus/common
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-07-01 23:29:21 +00:00
Arve Knudsen a39e7e287f
Merge pull request #14371 from aknuds1/arve/cleanup-otlp-histograms
OTLP: Document histogram conversion methods / clean up code a bit
2024-07-01 05:06:08 -07:00
Arve Knudsen 9595b174e5 otlp: Document regular and exponential histogram conversions
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-07-01 13:35:40 +02:00
Arve Knudsen 24b78bef32 otlp: Clean up exponential histogram code slightly
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-07-01 11:07:07 +02:00
Bryan Boreham 32f4bf5f68
Merge pull request #14368 from pracucci/optimize-literal-prefix-string-matcher
FastRegexMatcher: small optimization for the literal prefix case
2024-07-01 09:59:51 +01:00
Marco Pracucci ec31acaf02
FastRegexMatcher: small optimization for the literal prefix case
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-07-01 10:12:50 +02:00
Paulo Dias f4b1fcb73e
discovery: add support for gathering flavor name in Openstack discovery (#14312)
* feat: add support for gathering flavor name in Openstack discovery

Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>

* Update instance.go

Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com>
Signed-off-by: Paulo Dias <44772900+paulojmdias@users.noreply.github.com>

* Update configuration.md

Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com>
Signed-off-by: Paulo Dias <44772900+paulojmdias@users.noreply.github.com>

* fix: fix linting

Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>

* fix: fix instance type

Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>

* Update docs/configuration/configuration.md

Co-authored-by: Simon Pasquier <spasquie@redhat.com>
Signed-off-by: Paulo Dias <44772900+paulojmdias@users.noreply.github.com>

---------

Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>
Signed-off-by: Paulo Dias <44772900+paulojmdias@users.noreply.github.com>
Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com>
Co-authored-by: Simon Pasquier <spasquie@redhat.com>
2024-06-30 19:18:18 +02:00
Bryan Boreham 675d02cd0b
Merge pull request #14333 from bboreham/faster-dedupelabels
[PERF] Labels: faster encoding for -tags dedupelabels
2024-06-30 12:26:48 +01:00
Bartlomiej Plotka 4f2558de42
Merge pull request #14366 from rapphil/rapphil-fix-panic-ruler
Make rules Manager Update method  no-op after Close
2024-06-29 21:42:54 +02:00
Raphael Silva e0c9b2ee19 Fix linting errors
Signed-off-by: Raphael Silva <rapphil@gmail.com>
2024-06-28 23:44:08 +00:00
Raphael Silva cd5a7b5020 Make rules Manager Update method no-op after Close
This has to be done because Close and Update methods are accessed concurrently.

Signed-off-by: Raphael Silva <rapphil@gmail.com>
2024-06-28 23:39:46 +00:00
Rick Rackow 9290d1308d
fix(docs/querying): explain `ceil` behaviour more explicitly with examples (#11987)
* fix(docs/querying): explain `ceil` behaviour more explicitly with examples

Signed-off-by: Rick Rackow <rick.rackow@gmail.com>

* fix(docs/querying): explain `floor` behaviour more explicitly with examples

Signed-off-by: Rick Rackow <rick.rackow@paymenttools.com>

---------

Signed-off-by: Rick Rackow <rick.rackow@gmail.com>
Signed-off-by: Rick Rackow <rick.rackow@paymenttools.com>
Co-authored-by: Rick Rackow <rick.rackow@paymenttools.com>
2024-06-28 22:18:04 +02:00
Charles Korn fd6bdf5230
Fix issue where summation of +/- infinity returns NaN instead of infinity
Signed-off-by: Charles Korn <charles.korn@grafana.com>
2024-06-28 11:26:54 +10:00
Björn Rabenstein 2e58d46522
Merge pull request #13662 from prometheus/nhcb
Native histograms custom buckets storage
2024-06-27 21:44:20 +02:00
Björn Rabenstein 19da579799
Merge pull request #12821 from rexagod/12769
parser: support underscores
2024-06-27 17:04:18 +02:00
Bartlomiej Plotka 7f5ae0462d
MAINTAINERS.md: Added Nico and Alex as remote/storage owners. (#14352)
Would love to propose @npazosmendez and @alexgreenbank to help us own remote storage (especially write). 

Nico & Alex are not yet Prometheus maintainer, but I think that’s fine.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2024-06-27 15:56:40 +01:00
Pranshu Srivastava 3c9e3ee552
parser: support underscores
Support underscores in numbers, namely, decimals, hexadecimals, and
exponentials.

Fixes #12769
Signed-off-by: Pranshu Srivastava <rexagod@gmail.com>

Signed-off-by: Pranshu Srivastava <rexagod@gmail.com>
2024-06-27 19:20:52 +05:30
Björn Rabenstein cb7306155b
Merge pull request #14004 from liam-howe-maersk/implement-config-marshal
configuration: Implement IsZero for relabel.Regex to remove default regex
2024-06-27 12:13:14 +02:00
Bryan Boreham c5040c5ea9
Merge pull request #10490 from DrAuYueng/fix-docker-sd-service-missing
[ENHANCEMENT] Docker SD: add MatchFirstNetwork for containers with multiple networks

Fixes docker sd service misssing in shared mode and deduplicate targets by network
2024-06-26 12:33:50 +01:00
Charles Korn 2dd07fbb1b
notifier: optionally drain queued notifications before shutting down (#14290)
* Add draining of queued notifications to `notifier.Manager`

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Update docs

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Address PR feedback

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Add more logging

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Address offline feedback: remove timeout

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Ensure stopping takes priority over further processing, make tests more robust

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Make channel unbuffered

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Update docs

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Fix race in test

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Remove unnecessary context

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Make Stop safe to call multiple times

Signed-off-by: Charles Korn <charles.korn@grafana.com>

---------

Signed-off-by: Charles Korn <charles.korn@grafana.com>
2024-06-26 11:32:04 +01:00
Bryan Boreham f24ce00320
Merge pull request #13434 from bboreham/elide-queriers
storage: don't wrap single querier in merge-queriers
2024-06-25 17:47:09 +01:00
Arve Knudsen 1abeebacb8
Merge pull request #14340 from aknuds1/arve/fix-lint
Upgrade to golangci-lint v1.59.1, fix errors
2024-06-25 18:01:25 +02:00
Daniel Mellado 1b5f65002a Bump go-retryablehttp to fix basic auth creds leak
This PR updates go-retryablehttp to version 0.7.7, even if it's used as
an indirect import. Versions previous to that can didn't sanitize urls,
discussed at HDCSEC-2024-12 [1]

[1] https://discuss.hashicorp.com/t/hcsec-2024-12-go-retryablehttp-can-leak-basic-auth-credentials-to-log-files/68027

Signed-off-by: Daniel Mellado <dmellado@redhat.com>
2024-06-25 17:34:44 +02:00
Jan-Otto Kröpke 99355443c7
remote write handler: reject samples with future timestamps (#14304)
* fix(remote_write): reject samples with future timestamps

* increase check to +10 minutes to allow for clock drift

---------

Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>
Signed-off-by: Jan-Otto Kröpke <github@jkroepke.de>
Co-authored-by: Bryan Boreham <bjboreham@gmail.com>
2024-06-25 12:25:39 +01:00
Bryan Boreham 348f7f8d0c
Merge pull request #14341 from charleskorn/charleskorn/cleanup-pending-read
Fix issue where pending OOO read can be left dangling if creating querier fails
2024-06-25 09:23:54 +01:00
Ben Ye 246b7c6a5c
TSDB: Change block populator to accept postings index function (#14213)
Signed-off-by: Ben Ye <benye@amazon.com>
2024-06-25 09:21:48 +01:00
Ben Ye 5585a3c7e5
tsdb: expose hook to customize block querier (#14114)
* expose hook for block querier

Signed-off-by: Ben Ye <benye@amazon.com>

* update comment

Signed-off-by: Ben Ye <benye@amazon.com>

* use defined type

Signed-off-by: Ben Ye <benye@amazon.com>

---------

Signed-off-by: Ben Ye <benye@amazon.com>
2024-06-25 09:47:06 +02:00
Charles Korn 2c5e88748e
Fix issue where pending OOO read can be left dangling if creating querier fails
Signed-off-by: Charles Korn <charles.korn@grafana.com>
2024-06-25 14:22:44 +10:00
Arve Knudsen 0395b04419 golangci-lint: Upgrade to v1.59.1
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-06-24 16:14:22 -07:00
Arve Knudsen d902116b41 Fix various linting errors
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-06-24 16:11:53 -07:00