Commit Graph

37 Commits (cfb54fa6a3e4349398875d4d5d48025e36489356)

Author SHA1 Message Date
Neeraj Gartia d4b1f9eb33
Corrects the behaviour of binary opperators between histogram and float (#14726)
promql: corrects binary operators functioning for mixed sample with histogram and float

For invalid pairings of sample types, an annotation is added now.

Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com>

---------

Signed-off-by: Neeraj Gartia <neerajgartia211002@gmail.com>
2024-10-15 14:44:36 +02:00
Charles Korn e8c7482137
Return negative counts when multiplied or divided by a negative value
Signed-off-by: Charles Korn <charles.korn@grafana.com>
2024-09-09 14:37:59 +10:00
Jeanette Tan 5e4e93c316 fix lint
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2024-06-07 19:24:05 +08:00
Jeanette Tan 14f8dded39 Merge branch 'main' into nhcb
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2024-06-07 19:17:14 +08:00
Oleksandr Redko f10c3454e9 Enable perfsprint linter and fix up code
Signed-off-by: Oleksandr Redko <oleksandr.red+github@gmail.com>
2024-05-15 17:51:05 +03:00
zenador 4acbb7dea6
Add custom buckets to native histogram chunks encoding (#13706)
* add custom bounds to chunks encoding
* change custom buckets schema number
* rename custom bounds to custom values

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2024-03-22 14:36:39 +01:00
György Krajcsovits 5d0a0a7542 Add custom buckets to native histogram model (#13592)
* add custom buckets to native histogram model
* simple copy for custom bounds
* return errors for unsupported add/sub operations
* add test cases for string and update appendhistogram in scrape to account for new schema
* check fields which are supposed to be unused but may affect results in equals
* allow appending custom buckets histograms regardless of max schema

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2024-02-28 14:06:43 +01:00
Filip Petkovski 583f3e587c
Optimize histogram iterators (#13340)
Optimize histogram iterators

Histogram iterators allocate new objects in the AtHistogram and
AtFloatHistogram methods, which makes calculating rates over long
ranges expensive.

In #13215 we allowed an existing object to be reused
when converting an integer histogram to a float histogram. This commit follows
the same idea and allows injecting an existing object in the AtHistogram and
AtFloatHistogram methods. When the injected value is nil, iterators allocate
new histograms, otherwise they populate and return the injected object.

The commit also adds a CopyTo method to Histogram and FloatHistogram which
is used in the BufferedIterator to overwrite items in the ring instead of making
new copies.

Note that a specialized HPoint pool is needed for all of this to work 
(`matrixSelectorHPool`).

---------

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
2024-01-23 17:02:14 +01:00
Björn Rabenstein 85078b968f
Merge pull request #13208 from linasm/panic-free-float-histogram-add-sub
FloatHistogram.Add/Sub: handle any schema change
2023-12-07 20:12:21 +01:00
Linas Medziunas bd895baefc FloatHistogram.Add/Sub: handle any schema change
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-12-07 20:50:54 +02:00
Matthieu MOREL 9c4782f1cc
golangci-lint: enable testifylint linter (#13254)
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-12-07 11:35:01 +00:00
Linas Medziunas 49613823f8 Reuse slices in [Float]Histogram.ReduceResolution
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-28 15:45:31 +02:00
Linas Medziunas f99ecc376e Fix FloatHistogram.Add/Sub mutating its argument
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-26 09:26:34 +02:00
Ziqi Zhao e250f09b5d
change origin schema in `ReduceResolution` method of histogram and float histogram (#13116)
* change origin schema in ReduceResolution method of histogram and float histogram

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>

---------

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
2023-11-10 14:33:34 +01:00
Linas Medžiūnas e3e22abadb
Merge branch 'main' into linasm/optimize-floatBucketIterator2
Signed-off-by: Linas Medžiūnas <linasm@users.noreply.github.com>
2023-10-24 16:27:10 +03:00
Marc Tuduri 1ce066e51c
More periods
Signed-off-by: Marc Tuduri <marctc@protonmail.com>
2023-10-18 11:53:50 +02:00
Marc Tuduri af7c31ee10
PR feedback
Signed-off-by: Marc Tuduri <marctc@protonmail.com>
2023-10-18 11:53:50 +02:00
Marc Tuduri 8fededf6ad
promql(histograms): Change sample total calculation for histograms
Signed-off-by: Marc Tuduri <marctc@protonmail.com>
2023-10-18 11:51:11 +02:00
Linas Medziunas ec823d9daf Update comments, bitwise comparison of float buckets
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-10-09 16:09:46 +03:00
Linas Medziunas c5c5c569fa Histogram performance: optimize floatBucketIterator
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-10-09 09:40:59 +03:00
Linas Medziunas dfb6292600 Compare FloatHistogram.[Zero]Count float values as binary
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-09-28 09:06:54 +03:00
Gregor Zeitlinger f01718262a
Unit tests for native histograms (#12668)
promql: Extend testing framework to support native histograms

This includes both the internal testing framework as well as the rules unit test feature of promtool.

This also adds a bunch of basic tests. Many of the code level tests can now be converted to tests within the framework, and more tests can be added easily.

---------

Signed-off-by: Harold Dost <h.dost@criteo.com>
Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
Signed-off-by: Stephen Lang <stephen.lang@grafana.com>
Co-authored-by: Harold Dost <h.dost@criteo.com>
Co-authored-by: Stephen Lang <stephen.lang@grafana.com>
Co-authored-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
2023-08-25 23:35:42 +02:00
beorn7 9aadd54786 histogram: Fix bounds of buckets returned by floatBucketIterator
The bounds weren't really used so far, so no actual bug in the code so
far. But it's obviously confusing if the bounds returned by a
floatBucketIterator with a target schema different from the original
schema are wrong.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-07-19 18:19:18 +02:00
Carrie Edwards a462f7fa21 Add function for iterating through all buckets in reverse to find max bucket
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
2023-07-11 21:51:20 +08:00
zenador 191bf9055b
Handle more arithmetic operators for native histograms (#12262)
Handle more arithmetic operators and aggregators for native histograms

This includes operators for multiplication (formerly known as scaling), division, and subtraction. Plus aggregations for average and the avg_over_time function.

Stdvar and stddev will (for now) ignore histograms properly (rather than counting them but adding a 0 for them).

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-05-16 21:15:20 +02:00
Björn Rabenstein dccfb9db4e
histogram: Remove code replication via generics (#11361)
* histogram: Simplify iterators

We don't really need currLower and currUpper and can calculate it when
needed (as already done for the floatBucketIterator). The calculation
is cheap, while keeping those extra variables around costs RAM
(potentially a lot with many iterators).

* histogram: Convert Bucket/FloatBucket to one generic type

* histogram: Move some bucket iterator code into generic base iterator

* histogram: Remove cumulative iterator for FloatHistogram

We added it in the past for completeness (Histogram has one), but it
has never been used. Plus, even the cumulative iterator for Histogram
is only there for test reasons.

We can always add it back, and then maybe even using generics.

Signed-off-by: beorn7 <beorn@grafana.com>
2022-10-03 16:45:27 +05:30
beorn7 a7c519930e histograms: Add Compact method to the normal integer Histogram
And use the new method to call to compact Histograms during
parsing. This happens for both `Histogram` and `FloatHistogram`. In
this way, if targets decide to optimize the exposition size by merging
spans with empty buckets in between, we still get a normalized
results. It will also normalize away any valid but weird
representations like empty spans, spans with offset zero, and empty
buckets at the start or end of a span.

The implementation seemed easy at first as it just turns the
`compactBuckets` helper into a generic function (which now got its own
file). However, the integer Histograms have delta buckets instead of
absolute buckets, which had to be treated specially in the generic
`compactBuckets` function. To make sure it works, I have added plenty
of explicit tests for `Histogram` in addition to the `FloatHistogram`
tests.

I have also updated the doc comment for the `Compact` method.

Based on the insights now expressed in the doc comment, compacting
with a maxEmptyBuckets > 0 is rarely useful. Therefore, this commit
also sets the value to 0 in the two cases we were using 3 so far. We
might still want to reconsider, so I don't want to remove the
maxEmptyBuckets parameter right now.

Signed-off-by: beorn7 <beorn@grafana.com>
2022-09-27 13:04:16 +02:00
beorn7 2d233cf95e Histogram: Fix allFloatBucketIterator
If the zero threshold overlaps with the highest negative bucket and/or
the lowest positive bucket, its upper or lower boundary, respectively,
has to be adjusted. In valid histograms, only ever the highest
negative bucket and/or the lowest positive bucket may overlap with the
zero bucket. This is assumed in this code to simplify the checks.

Signed-off-by: beorn7 <beorn@grafana.com>
2022-05-03 16:24:11 +02:00
beorn7 56db51c826 Histgram: Fix Compact for spans of only empty buckets
Signed-off-by: beorn7 <beorn@grafana.com>
2022-04-12 00:37:50 +02:00
beorn7 15583af9bb Histogram: Fix crash when compacting only empty buckets
Signed-off-by: beorn7 <beorn@grafana.com>
2022-04-07 16:24:48 +02:00
beorn7 9fbcf14e5c histogram: Handle changes of the ZeroThreshold and the Schema
Signed-off-by: beorn7 <beorn@grafana.com>
2022-03-17 18:05:31 +01:00
Björn Rabenstein ec80745884
Merge pull request #10075 from prometheus/beorn7/histogram
model: Implement FloatHistogram.Compact
2022-01-05 16:09:39 +01:00
beorn7 3b4d6c3fdb model: Implement FloatHistogram.Compact
Signed-off-by: beorn7 <beorn@grafana.com>
2022-01-05 14:34:03 +01:00
beorn7 e7592fe353 sparsehistogram: Address two TODOs
Signed-off-by: beorn7 <beorn@grafana.com>
2022-01-04 12:48:59 +01:00
beorn7 a6acdfe346 histograms: Doc comment and naming improvements
Signed-off-by: beorn7 <beorn@grafana.com>
2021-12-15 16:50:37 +01:00
Ganesh Vernekar 4a43349aca
`histogram_quantile` for sparse histograms (#9935)
* MergeFloatBucketIterator for []FloatBucketIterator

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* histogram_quantile for histograms

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix histogram_quantile

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Unit test and enhancements

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Iterators to iterate buckets in reverse and all buckets together including zero bucket

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Consider all buckets for histogram_quantile and fix the implementation

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Remove unneeded code

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix lint

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2021-12-06 19:17:22 +05:30
Björn Rabenstein 0e1b9dd308
Promql: Initial rate implementation for sparse histograms (#9926)
Signed-off-by: beorn7 <beorn@grafana.com>
2021-12-06 18:19:18 +05:30