* modify unit test framework to automatically generate native histograms with custom buckets from classic histogram series
* add very basic tests for classic histogram converted into native histogram with custom bounds
* fix histogram_quantile for native histograms with custom buckets
* make loading with nhcb explicit
* evaluate native histograms with custom buckets on queries with explicit keyword
* use regex replacer
* use temp histogram struct for automatically loading converted nhcb
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>
The strings produced by these tests can run to thousands of characters,
which makes test logs difficult to read.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Restrict the capacity of first argument to `append()` to force an allocation.
This is for the slice implementation only.
Signed-off-by: Domantas Jadenkus <djadenkus@gmail.com>
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Restrict the capacity of first argument to `append()` to force an allocation.
This is for the slice implementation only.
Signed-off-by: Domantas Jadenkus <djadenkus@gmail.com>
I was bored on a train and I spent some amount of time trying to scratch
some nanoseconds off the Labels.Compare when running with stringlabels.
I would be ashamed to admit the real amount of time I spent on it.
The worst thing is, I can't really explain why this is performing so
much better, and someone should re-run the benchmarks on their machine
to confirm that it's not something related to general relativity because
the train is moving. I also added some extra real-life benchmark cases
with longer labelsets (these aren't the longest we have in production,
but kubernetes labelsets are fairly common in Prometheus so I thought it
would be nice to have them).
My benchmarks show this diff:
goos: darwin
goarch: arm64
pkg: github.com/prometheus/prometheus/model/labels
│ old │ new │
│ sec/op │ sec/op vs base │
Labels_Compare/equal 5.898n ± 0% 5.875n ± 1% -0.40% (p=0.037 n=10)
Labels_Compare/not_equal 11.78n ± 2% 11.01n ± 1% -6.54% (p=0.000 n=10)
Labels_Compare/different_sizes 4.959n ± 1% 4.906n ± 2% -1.05% (p=0.050 n=10)
Labels_Compare/lots 21.32n ± 0% 17.54n ± 5% -17.75% (p=0.000 n=10)
Labels_Compare/real_long_equal 15.06n ± 1% 14.92n ± 0% -0.93% (p=0.000 n=10)
Labels_Compare/real_long_different_end 25.20n ± 0% 24.43n ± 0% -3.04% (p=0.000 n=10)
geomean 11.86n 11.25n -5.16%
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Use a stack buffer to reduce memory allocations.
`Write(AppendQuote(AvailableBuffer` does not allocate or copy when
the buffer has sufficient space.
Also add a benchmark, with some refactoring.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
The docs suggest the Next method returns a bool, but that's not the case (`Entry` is an int).
```
// Next advances the parser to the next sample. It returns false if no
// more samples were read or an error occurred.
Next() (Entry, error)
```
The docs were first added in d80a3de235 in 2017. Back then the signature was
indeed `func (p *Parser) Next() bool`. But then it got refactored in 76a4a46cb0
and the signature changed with it, yet docs stayed the same - and eventually made their way into the `Parser` interface.
However, the Protobuf parser does have the right wording: 5de2df752f
```
// Next advances the parser to the next "sample" (emulating the behavior of a
// text format parser). It returns (EntryInvalid, io.EOF) if no samples were
// read.
```
Changing all other implementations (and the interface itself) to match this doc.
Signed-off-by: Ondrej Kokes <ondrej.kokes@gmail.com>
* add custom buckets to native histogram model
* simple copy for custom bounds
* return errors for unsupported add/sub operations
* add test cases for string and update appendhistogram in scrape to account for new schema
* check fields which are supposed to be unused but may affect results in equals
* allow appending custom buckets histograms regardless of max schema
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Fix is to add json tags to `Metadata` struct. Absence of these tags
causes Go to use the field name, which starts with an upper-case
letter and breaks the protocol.
Extend tests to verify the JSON response.
Signed-off-by: ismail simsek <ismailsimsek09@gmail.com>
Fix is to add json tags to `Metadata` struct. Absence of these tags
causes Go to use the field name, which starts with an upper-case
letter and breaks the protocol.
Extend tests to verify the JSON response.
Signed-off-by: ismail simsek <ismailsimsek09@gmail.com>
The individual strings for label names and values are held in a table,
and each Labels value is a run of varint-encoded indexes into that table.
When creating new labels, a sync.Mutex is locked around reads and writes.
When reading labels, there is no locking because the table of strings
used by those labels is immutable.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
This adds support for the new grammar of `{"metric_name", "l1"="val"}` to promql and some of the exposition formats.
This grammar will also be valid for non-UTF-8 names.
UTF-8 names will not be considered valid unless model.NameValidationScheme is changed.
This does not update the go expfmt parser in text_parse.go, which will be addressed by https://github.com/prometheus/common/issues/554/.
Part of https://github.com/prometheus/prometheus/issues/13095
Signed-off-by: Owen Williams <owen.williams@grafana.com>
scrape: support parsing exemplars from native histogram
---------
Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
Signed-off-by: Björn Rabenstein <github@rabenste.in>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
The current implementation of `InternStrings` will only save memory
when the whole set of labels is identical to one already seen, and this
cannot happen in the one place it is called from in Prometheus,
remote-write, which already detects identical series.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
This PR is a reference implementation of the proposal described in #10420.
In addition to what described in #10420, in this PR I've introduced labels.StableHash(). The idea is to offer an hashing function which doesn't change over time, and that's used by query sharding in order to get a stable behaviour over time. The implementation of labels.StableHash() is the hashing function used by Prometheus before stringlabels, and what's used by Grafana Mimir for query sharding (because built before stringlabels was a thing).
Follow up work
As mentioned in #10420, if this PR is accepted I'm also open to upload another foundamental piece used by Grafana Mimir query sharding to accelerate the query execution: an optional, configurable and fast in-memory cache for the series hashes.
Signed-off-by: Marco Pracucci <marco@pracucci.com>
This function is called very frequently when executing PromQL functions,
and we can do it much more efficiently inside Labels.
In the common case that `__name__` comes first in the labels, we simply
re-point to start at the next label, which is nearly free.
`DropMetricName` is now so cheap I removed the cache - benchmarks show
everything still goes faster.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Optimize histogram iterators
Histogram iterators allocate new objects in the AtHistogram and
AtFloatHistogram methods, which makes calculating rates over long
ranges expensive.
In #13215 we allowed an existing object to be reused
when converting an integer histogram to a float histogram. This commit follows
the same idea and allows injecting an existing object in the AtHistogram and
AtFloatHistogram methods. When the injected value is nil, iterators allocate
new histograms, otherwise they populate and return the injected object.
The commit also adds a CopyTo method to Histogram and FloatHistogram which
is used in the BufferedIterator to overwrite items in the ring instead of making
new copies.
Note that a specialized HPoint pool is needed for all of this to work
(`matrixSelectorHPool`).
---------
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Add warnings for histogramRate applied with isCounter not matching counter/gauge histogram
---------
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
The slices package is added to the standard library in Go 1.21;
we need to import from the exp area to maintain compatibility with Go 1.20.
Signed-off-by: tyltr <tylitianrui@126.com>
Liam Howe