Commit Graph

311 Commits (89ecb3a3f2e48a66260a9af1a834ed8617b86f27)

Author SHA1 Message Date
beorn7 1c3bd04bea histograms: Modify test to expose bug #12552
Signed-off-by: beorn7 <beorn@grafana.com>
2023-07-12 18:17:35 +02:00
Douglas Camata 47819fef01
Merge branch 'main' of github.com:prometheus/prometheus into document-and-test-relabel-memory-reuse
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
2023-07-12 15:01:29 +02:00
Ziqi Zhao 42d9169ba1 enhance histogram_quantile to get min/max value
Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
2023-07-12 04:29:54 +08:00
Carrie Edwards a462f7fa21 Add function for iterating through all buckets in reverse to find max bucket
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
2023-07-11 21:51:20 +08:00
Bryan Boreham e1115ae58d
labels: improve Get method for stringlabels build (#12485)
Inline one call to `decodeString`, and skip decoding the value string
until we find a match for the name.
Do a quick check on the first character in each string,
and exit early if we've gone past - labels are sorted in order.

Also improve tests and benchmark:
* labels: test Get with varying lengths - it's not typical for Prometheus labels to all be the same length.
* extend benchmark with label not found

---------

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-06-26 18:35:22 +01:00
Bryan Boreham 87d08abe11
labels: faster Compare function when using -tags stringlabels (#12451)
Instead of unpacking every individual string, we skip to the point
where there is a difference, going 8 bytes at a time where possible.

Add benchmark for Compare; extend tests too.

---------

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com>
2023-06-20 20:58:47 +01:00
Oleg Zaytsev 6a18962cfa
mv labels_string.go labels_stringlabels.go (#12328)
This is a minor cosmetical change, but my IDE (and I guess many of them)
nests `labels_string.go` under `labels.go` because it assumes it's the
file generated by the `stringer` tool, which follows that naming
pattern.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2023-06-13 09:38:00 +01:00
Michael Hoffmann 344c8ff97c
feat: dont compile regex matcher if we know its a literal (#12434)
labels: dont compile regex matcher if we know its a literal

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

Co-authored-by: Sharad <sharadgaur@gmail.com>
2023-06-07 21:54:30 +01:00
zenador 191bf9055b
Handle more arithmetic operators for native histograms (#12262)
Handle more arithmetic operators and aggregators for native histograms

This includes operators for multiplication (formerly known as scaling), division, and subtraction. Plus aggregations for average and the avg_over_time function.

Stdvar and stddev will (for now) ignore histograms properly (rather than counting them but adding a 0 for them).

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-05-16 21:15:20 +02:00
Bryan Boreham a073e04a9b
Merge pull request #12366 from prometheus/release-2.44
Merge release 2.44 back to main
2023-05-16 18:06:29 +01:00
beorn7 9e500345f3 textparse/scrape: Add option to scrape both classic and native histograms
So far, if a target exposes a histogram with both classic and native
buckets, a native-histogram enabled Prometheus would ignore the
classic buckets. With the new scrape config option
`scrape_classic_histograms` set, both buckets will be ingested,
creating all the series of a classic histogram in parallel to the
native histogram series. For example, a histogram `foo` would create a
native histogram series `foo` and classic series called `foo_sum`,
`foo_count`, and `foo_bucket`.

This feature can be used in a migration strategy from classic to
native histograms, where it is desired to have a transition period
during which both native and classic histograms are present.

Note that two bugs in classic histogram parsing were found and fixed
as a byproduct of testing the new feature:

1. Series created from classic _gauge_ histograms didn't get the
   _sum/_count/_bucket prefix set.
2. Values of classic _float_ histograms weren't parsed properly.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-05-13 01:32:25 +02:00
Bryan Boreham 7a48a266b6
labels: respect Set after Del in Builder (#12322)
* labels: respect Set after Del in Builder

The implementations are not symmetric between `Set()` and `Del()`, so
we must be careful. Add tests for this, both in labels and in relabel
where the issue was reported.

Also make the slice implementation consistent re `slices.Contains`.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-05-03 11:59:27 +01:00
cui fliter 276ca6a883 fix some comments
Signed-off-by: cui fliter <imcusg@gmail.com>
2023-04-25 14:19:16 +08:00
Matthieu MOREL bae9a21200
Merge branch 'main' into linter/nilerr
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-04-19 19:56:39 +02:00
beorn7 5b53aa1108 style: Replace `else if` cascades with `switch`
Wiser coders than myself have come to the conclusion that a `switch`
statement is almost always superior to a statement that includes any
`else if`.

The exceptions that I have found in our codebase are just these two:

* The `if else` is followed by an additional statement before the next
  condition (separated by a `;`).
* The whole thing is within a `for` loop and `break` statements are
  used. In this case, using `switch` would require tagging the `for`
  loop, which probably tips the balance.

Why are `switch` statements more readable?

For one, fewer curly braces. But more importantly, the conditions all
have the same alignment, so the whole thing follows the natural flow
of going down a list of conditions. With `else if`, in contrast, all
conditions but the first are "hidden" behind `} else if `, harder to
spot and (for no good reason) presented differently from the first
condition.

I'm sure the aforemention wise coders can list even more reasons.

In any case, I like it so much that I have found myself recommending
it in code reviews. I would like to make it a habit in our code base,
without making it a hard requirement that we would test on the CI. But
for that, there has to be a role model, so this commit eliminates all
`if else` occurrences, unless it is autogenerated code or fits one of
the exceptions above.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-04-19 17:22:31 +02:00
beorn7 c3c7d44d84 lint: Adjust to the lint warnings raised by current versions of golint-ci
We haven't updated golint-ci in our CI yet, but this commit prepares
for that.

There are a lot of new warnings, and it is mostly because the "revive"
linter got updated. I agree with most of the new warnings, mostly
around not naming unused function parameters (although it is justified
in some cases for documentation purposes – while things like mocks are
a good example where not naming the parameter is clearer).

I'm pretty upset about the "empty block" warning to include `for`
loops. It's such a common pattern to do something in the head of the
`for` loop and then have an empty block. There is still an open issue
about this: https://github.com/mgechev/revive/issues/810 I have
disabled "revive" altogether in files where empty blocks are used
excessively, and I have made the effort to add individual
`// nolint:revive` where empty blocks are used just once or twice.
It's borderline noisy, though, but let's go with it for now.

I should mention that none of the "empty block" warnings for `for`
loop bodies were legitimate.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-04-19 17:10:10 +02:00
Bryan Boreham 1801cd4196 labels: small optimization to stringlabels
Add a fast path for the common case that a string is less than 127 bytes
long, to skip a shift and the loop.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-04-14 15:06:57 +00:00
Matthieu MOREL fb3eb21230 enable gocritic, unconvert and unused linters
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-04-13 19:20:22 +00:00
Bryan Boreham 10cc60af01 labels: add ScratchBuilder.Overwrite for slice implementation
This is a method used by some downstream projects; it was created to
optimize the implementation in `labels_string.go` but we should have one
for both implementations so the same code works with either.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-04-13 11:07:54 +00:00
Ganesh Vernekar 5588cab8b2
Merge pull request #12173 from bboreham/builder-no-empty-labels
labels: simplify call to get Labels from Builder
2023-04-04 12:02:55 +05:30
Bryan Boreham e917202766 labels: make sure estimated size is not negative
Deleted labels are remembered, even if they were not in `base` or were
removed from `add`, so `base+add-del` could go negative.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-04-02 11:17:09 +01:00
Bryan Boreham ee1157c14a labels: shrink stack arrays in Builder.Range
Go spends some time initializing all the elements of these arrays to
zero, so reduce the size from 1024 to 128. This is still much bigger
than we ever expect for a set of labels.

(If someone does have more than 128 labels it will still work, but via
heap allocation.)

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-03-22 17:14:43 +00:00
Bryan Boreham b987afa7ef labels: simplify call to get Labels from Builder
It took a `Labels` where the memory could be re-used, but in practice
this hardly ever benefitted. Especially after converting `relabel.Process`
to `relabel.ProcessBuilder`.

Comparing the parameter to `nil` was a bug; `EmptyLabels` is not `nil`
so the slice was reallocated multiple times by `append`.

Lastly `Builder.Labels()` now estimates that the final size will depend
on labels added and deleted.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-03-22 17:05:20 +00:00
Bryan Boreham 3743d87c56 labels: cope with mutating Builder during Range call
Although we had a different slice, the underlying memory was the same so
any changes meant we could skip some values.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-03-16 13:28:15 +00:00
Bryan Boreham 3c4ab7a069 labels: add test for Builder.Range
Including mutating the Builder being Ranged over.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-03-16 13:25:55 +00:00
Björn Rabenstein 847093479b
Merge pull request #11978 from trevorwhitney/set-counter-hint
Set `CounterResetHint` and use in recording rules
2023-03-14 21:52:41 +01:00
Trevor Whitney e3513d1dd2
Change nested ifs to a switch
Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>
2023-03-14 14:22:20 -06:00
Trevor Whitney b4e324f637
Handle valid cases of mismatched hints when adding
Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>
2023-03-14 14:22:14 -06:00
Trevor Whitney dd94ebb87b
promql: set CounterResetHint after rate and sum
Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>
2023-03-14 14:21:59 -06:00
Julien Pivotto 5583c77b3a
Merge pull request #12095 from damnever/unnecessary-sort
Remove unnecessary sort
2023-03-09 13:12:02 +01:00
Xiaochao Dong (@damnever) 36fc1158b5 Remove unnecessary sort
Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
2023-03-08 15:36:02 +08:00
Bryan Boreham 11d019ed5a relabel: keep intermediate results in labels.Builder
Save work converting between Builder and Labels.

Also expose ProcessBuilder, so callers can supply a Builder.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-03-07 17:21:37 +00:00
Bryan Boreham d740abf0c6 model/labels: add Get and Range to Builder
This lets relabelling work on a `Builder` rather than converting to and
from `Labels` on every rule.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-03-07 17:20:54 +00:00
Bryan Boreham ff993b279a
Merge pull request #12073 from bboreham/slices-sort2
labels: use slices.Sort for better performance
2023-03-07 09:31:50 +00:00
Bryan Boreham 38c6d3da9f labels: use slices.Sort for better performance
The difference is modest, but we've used `slices.Sort` in lots of other
places so why not here.

name     old time/op    new time/op    delta
Builder    1.04µs ± 3%    0.95µs ± 3%   -8.27%  (p=0.008 n=5+5)

name     old alloc/op   new alloc/op   delta
Builder      312B ± 0%      288B ± 0%   -7.69%  (p=0.008 n=5+5)

name     old allocs/op  new allocs/op  delta
Builder      2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.008 n=5+5)

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-03-06 18:22:49 +00:00
Bryan Boreham a07a0be024 Add benchmark for labels.Builder
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-03-06 18:21:58 +00:00
Bryan Boreham 30297f0d9b stringlabels: size buffer for added labels
This makes the buffer the correct size for the common case that labels
have only been added. It will be too large for the case that labels are
changed, but the current buffer resize logic in `appendLabelTo` doubles
the buffer, so a small over-estimate is better.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-03-06 16:29:21 +00:00
Jonatan Ivanov 9225d118dc
Add Exemplars support for all time series
This change removes restrictions to allow adding exemplars
to all time series. It also contains some improvements in test values
so that it is easier to track what is tested.
The advantage of doing this is having a little less error-prone tests:
"yy" is not really descriptive but "counter-test" can give people
a better idea about what is tested so it is harder to make mistakes.

Closes gh-11982

Signed-off-by: Jonatan Ivanov <jonatan.ivanov@gmail.com>
2023-03-03 17:09:17 -08:00
Renning Bruns 5ec1b4baaf
make hashmod a lot easier to read and a little faster (#11551)
Previous code was effectively doing BigEndian.Uint64, so call that and save time.
An md5.Sum result is always 16 bytes. The first 8 are not used in the result, just as before.

Signed-off-by: Renning Bruns <ren@renmail.net>
2023-02-28 17:36:58 +00:00
Bryan Boreham 35026fb26d
Merge pull request #11746 from prometheus/remove-microbenchmarks
These benchmarks were testing things related to what Prometheus does, but not testing actual Prometheus code. 
Moved the label-copying benchmark into the labels package.
2023-02-23 12:33:24 +01:00
Levi Harrison 64ff6bece6
Merge pull request #11682 from dgrisonnet/parsing-errors
Improve the Prometheus parser error outputs to be more comprehensive
2023-02-22 17:35:27 -05:00
Bryan Boreham f03b8d0968 Add benchmark copying labels
Taken from previous tsdb/test/BenchmarkLabelsClone.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-02-22 16:36:45 +00:00
Damien Grisonnet 0fee615719 model/textparse: improve openmetrics error outputs
Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>
2023-02-22 17:19:01 +01:00
Bryan Boreham 6136ae67e0 labels: shrink by making internals a single string
This commit adds an alternate implementation for `labels.Labels`, behind
a build tag `stringlabels`.

Instead of storing label names and values as individual strings, they
are all concatenated into one string in this format:

    [len][name0][len][value0][len][name1][len][value1]...

The lengths are varint encoded so usually a single byte.

The previous `[]string` had 24 bytes of overhead for the slice and 16
for each label name and value; this one has 16 bytes overhead plus 1
for each name and value.

In `ScratchBuilder.Overwrite` and `Labels.Hash` we use an unsafe
conversion from string to byte slice. `Overwrite` is explicitly unsafe,
but for `Hash` this is a pure performance hack.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-02-22 15:34:23 +00:00
Damien Grisonnet e3b5416d05 model/textparse: improve error outputs
Parsing errors in the Prometheus HTTP format parser are very hard to
investigate since they only approximately indicate what is going wrong
in the parser and don't provide any information about the incorrect
input. As such it is very hard to tell what is wrong in the format
exposed by the application.

Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>
2023-02-03 16:54:57 +01:00
Julien Pivotto 8e500dbd39 Add rulefmt tests
Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>
2023-01-19 10:44:35 +01:00
Julien Pivotto ce55e5074d Add 'keep_firing_for' field to alerting rules
This commit adds a new 'keep_firing_for' field to Prometheus alerting
rules. The 'resolve_delay' field specifies the minimum amount of time
that an alert should remain firing, even if the expression does not
return any results.

This feature was discussed at a previous dev summit, and it was
determined that a feature like this would be useful in order to allow
the expression time to stabilize and prevent confusing resolved messages
from being propagated through Alertmanager.

This approach is simpler than having two PromQL queries, as was
sometimes discussed, and it should be easy to implement.

This commit does not include tests for the 'resolve_delay' field.  This
is intentional, as the purpose of this commit is to gather comments on
the proposed design of the 'resolve_delay' field before implementing
tests. Once the design of the 'resolve_delay' field has been finalized,
a follow-up commit will be submitted with tests."

See https://github.com/prometheus/prometheus/issues/11570

Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>
2023-01-13 12:11:39 +01:00
beorn7 3e5ad99c33 textparse: Minor doc comment improvement
Signed-off-by: beorn7 <beorn@grafana.com>
2023-01-11 00:30:55 +01:00
beorn7 3f977e89ab textparse: Add tests for float and gauge histograms
Signed-off-by: beorn7 <beorn@grafana.com>
2023-01-10 18:42:05 +01:00
beorn7 c26891043f textparse: Set CounterResetHint
If a (float or integer) histogram is a gauge histogram, set the
CounterResetHint accordingly. (The default value is fine for the
normal counter histograms.)

Signed-off-by: beorn7 <beorn@grafana.com>
2023-01-10 16:25:23 +01:00
Björn Rabenstein f07adbd45f
Merge pull request #11814 from prometheus/beorn7/protobuf
textparse: Add gauge histogram support
2023-01-10 15:18:44 +01:00
Ganesh Vernekar 57bcbf1888
Merge pull request #11783 from codesome/gauge-histogram
tsdb: Add gauge histogram support
2023-01-10 19:06:08 +05:30
Ganesh Vernekar a87e7e9e33
tsdb: Add counter reset hint to histograms and support in WAL
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-01-10 17:41:53 +05:30
Ganesh Vernekar c94a41c4b2
Merge pull request #11785 from Fish-pro/erroris
Use errors.Is to check for a specific error
2023-01-10 14:56:14 +05:30
beorn7 b5d4a94e9d textparse: Add gauge histogram support to protobuf parsing
With this commit, the parser stops to see a gauge histogram (whether
native or conventional) as an unexpected metric type. It ingests it
normally, it even sets the `GaugeHistogram` type in the metadata (as
it has already done for a conventional gauge histogram scraped using
OpenMetrics), but it otherwise treats it as a normal counter-like
histogram.

Once #11783 is merged, though, it should be very easy to utilize the
type information.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-01-05 15:39:10 +01:00
beorn7 e9d9bb1b08 textparse: Handle unknown metric types in protobuf gracefully
So far, the parser hasn't validated that the type is valid in the
`Next()` call. Later, in the `Series()` call, however, it assumes that
we will only see valid types and therefore panics with `encountered
unexpected metric type, this is a bug`.

This commit fixes said bug by adding validation to the `Next()` call.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-01-05 15:21:18 +01:00
Damien Grisonnet 96b9d8cebb model/textparse: associate correct token to errors
In some cases, the Prometheus HTTP format parser was not returning the
right token in the error output which made debugging impossible.

Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>
2023-01-02 13:57:27 +01:00
Fish-pro 6ed71a229e Use errors.Is to check for a specific error
Signed-off-by: Fish-pro <zechun.chen@daocloud.io>
2022-12-29 23:23:07 +08:00
Marc Tudurí 9474610baf
Support FloatHistogram in TSDB (#11522)
Extends Appender.AppendHistogram function to accept the FloatHistogram. TSDB supports appending, querying, WAL replay, for this new type of histogram.

Signed-off-by: Marc Tudurí <marctc@protonmail.com>
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-12-28 14:25:07 +05:30
Bryan Boreham 10b27dfb84 Simplify IndexReader.Series interface
Instead of passing in a `ScratchBuilder` and `Labels`, just pass the
builder and the caller can extract labels from it. In many cases the
caller didn't use the Labels value anyway.

Now in `Labels.ScratchBuilder` we need a slightly different API: one
to assign what will be the result, instead of overwriting some other
`Labels`. This is safer and easier to reason about.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-19 15:22:09 +00:00
Bryan Boreham 8d350d9e0c Update package model/textparse tests for new labels.Labels type
We don't want to touch the result labels now we create them differently.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-19 15:22:09 +00:00
Bryan Boreham 1f04899ae3 Update package model/textparse for new labels.Labels type
Parse metrics using labels.ScratchBuilder, so we reduce assumptions about
internals of Labels.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-19 15:22:09 +00:00
Bryan Boreham fe9fe0e1e5 Update package model/relabel tests for new labels.Labels type
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-19 15:22:09 +00:00
Bryan Boreham 8ad7b64c0f Update package model/relabel for new labels.Labels type
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-19 15:22:09 +00:00
Bryan Boreham b10fd9aea3 model/labels: add a basic test for ScratchBuilder
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-19 15:22:09 +00:00
Bryan Boreham cbf432d2ac Update package labels tests for new labels.Labels type
Re-did the FromStrings test to avoid assumptions about how it works.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-19 15:22:09 +00:00
Bryan Boreham 617bee60f1 labels: use ScratchBuilder in ReadLabels
Instead of relying on being able to append to it like a slice.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-19 15:22:09 +00:00
Bryan Boreham 2b8b8d9ac7 labels: new methods to work without access to internals
Without changing the definition of `labels.Labels`, add methods which
enable code using it to work without knowledge of the internals.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-19 15:22:09 +00:00
Bryan Boreham ea7345a09c labels: improve comment on Builder.Set
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-19 15:22:09 +00:00
Bryan Boreham a19b369f9e labels: avoid lint warning on New()
This code is a bit cleaner.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-19 15:22:09 +00:00
Julien Pivotto bb323db613
Merge pull request #11074 from damnever/fix/datamodelvalidation
Validate the metric name and label names
2022-12-08 14:31:12 +01:00
Xiaochao Dong (@damnever) 9979024a30 Report error if the series contains invalid metric names or labels during scrape
Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
2022-12-08 20:01:20 +08:00
Bryan Boreham 8d4140a06e labels: note that Hash may change
For performance reasons we may use a different implementation of Hash()
in future, so note this so callers can be warned.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-11-28 16:17:32 +00:00
Julien Pivotto 005ede70de relabel: add keepequal/dropequal relabel action
Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>
2022-11-10 14:41:02 +01:00
Ganesh Vernekar bde500e690
Merge pull request #11420 from jesusvazquez/jvp/update-sparsehistogram-with-main-v2
Merge branch 'main' into sparsehistogram
2022-10-12 11:45:50 +05:30
Douglas Camata 28a66e183d Update relabel.Process comment
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
2022-10-10 11:31:23 +02:00
beorn7 bf0847073d histogram: Modify getBound to deal properly with infinity
The bucket receiving math.MaxFloat64 observations now has
math.MaxFloat64 as upper bound, while the bucket after it (the last
possible bucket) has +Inf.

This also adds a test for getBound and moves the getBound code to
generic.go (where it should have been in the first place).

Signed-off-by: beorn7 <beorn@grafana.com>
2022-10-06 17:40:03 +02:00
Jesus Vazquez e934d0f011 Merge 'main' into sparsehistogram
Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>
2022-10-05 22:14:49 +02:00
Björn Rabenstein dccfb9db4e
histogram: Remove code replication via generics (#11361)
* histogram: Simplify iterators

We don't really need currLower and currUpper and can calculate it when
needed (as already done for the floatBucketIterator). The calculation
is cheap, while keeping those extra variables around costs RAM
(potentially a lot with many iterators).

* histogram: Convert Bucket/FloatBucket to one generic type

* histogram: Move some bucket iterator code into generic base iterator

* histogram: Remove cumulative iterator for FloatHistogram

We added it in the past for completeness (Histogram has one), but it
has never been used. Plus, even the cumulative iterator for Histogram
is only there for test reasons.

We can always add it back, and then maybe even using generics.

Signed-off-by: beorn7 <beorn@grafana.com>
2022-10-03 16:45:27 +05:30
Bryan Boreham 8120af22e2 benchmarks: SetBytes takes bytes per operation
Where the code was multiplying bytes by number of operations, this
resulted in absurdly high throughput numbers.

Also, in `BenchmarkParse()`, don't run the `expfmt` case twice.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-10-02 22:11:33 +02:00
beorn7 a7c519930e histograms: Add Compact method to the normal integer Histogram
And use the new method to call to compact Histograms during
parsing. This happens for both `Histogram` and `FloatHistogram`. In
this way, if targets decide to optimize the exposition size by merging
spans with empty buckets in between, we still get a normalized
results. It will also normalize away any valid but weird
representations like empty spans, spans with offset zero, and empty
buckets at the start or end of a span.

The implementation seemed easy at first as it just turns the
`compactBuckets` helper into a generic function (which now got its own
file). However, the integer Histograms have delta buckets instead of
absolute buckets, which had to be treated specially in the generic
`compactBuckets` function. To make sure it works, I have added plenty
of explicit tests for `Histogram` in addition to the `FloatHistogram`
tests.

I have also updated the doc comment for the `Compact` method.

Based on the insights now expressed in the doc comment, compacting
with a maxEmptyBuckets > 0 is rarely useful. Therefore, this commit
also sets the value to 0 in the two cases we were using 3 so far. We
might still want to reconsider, so I don't want to remove the
maxEmptyBuckets parameter right now.

Signed-off-by: beorn7 <beorn@grafana.com>
2022-09-27 13:04:16 +02:00
Ganesh Vernekar 2474c6fb2c
Error on amending histograms on append (#11308)
* Error on amending histograms on append

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Rename Matches to Equals

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-09-19 13:10:30 +05:30
Bryan Boreham 5421c778ba labels: in tests use labels.FromStrings
Replacing code which assumes the internal structure of `Labels`.

Add a convenience function `EmptyLabels()` which is more efficient than
calling `New()`.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-09-09 13:34:49 +02:00
Julien Pivotto 96d5a32659
Update go to 1.19, set min version to 1.18 (#11279)
* Update go to 1.19, set min version to 1.18

Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>

* Update golangci-lint

Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>

Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>
2022-09-07 11:30:48 +02:00
Cosrider bef6556ca5
delete redundant alias (#11180)
Signed-off-by: Cosrider <cosrider7@gmail.com>

Signed-off-by: Cosrider <cosrider7@gmail.com>
2022-08-31 15:50:38 +02:00
Marc Tudurí f7df3b86ba
histograms: parse float histograms from proto definition (#11149)
* histograms: parse float histograms from proto definition

Signed-off-by: Marc Tuduri <marctc@protonmail.com>

* Improve comment

Signed-off-by: Marc Tuduri <marctc@protonmail.com>

* Ignore float buckets

Signed-off-by: Marc Tuduri <marctc@protonmail.com>

* Refactor Histogram() function

Signed-off-by: Marc Tuduri <marctc@protonmail.com>

* Fix test_float_histogram

Signed-off-by: Marc Tuduri <marctc@protonmail.com>

* Update model/textparse/protobufparse.go

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Signed-off-by: Marc Tudurí <marctc@protonmail.com>

* Update protobufparse.go

Signed-off-by: Marc Tudurí <marctc@protonmail.com>

* Update scrape.go

Signed-off-by: Marc Tudurí <marctc@protonmail.com>

* Update scrape/scrape.go

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Signed-off-by: Marc Tudurí <marctc@protonmail.com>

Signed-off-by: Marc Tuduri <marctc@protonmail.com>
Signed-off-by: Marc Tudurí <marctc@protonmail.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2022-08-25 20:37:41 +05:30
Bryan Boreham 8b863c42dd
Optimise relabeling by re-using memory (#11147)
* model/relabel: Add benchmark

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* model/relabel: re-use Builder across relabels

Saves memory allocations.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* labels.Builder: allow re-use of result slice

This reduces memory allocations where the caller has a suitable slice available.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* model/relabel: re-use source values slice

To reduce memory allocations.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Unwind one change causing test failures

Restore original behaviour in PopulateLabels, where we must not overwrite the input set.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* relabel: simplify values optimisation

Use a stack-based array for up to 16 source labels, which will be the
vast majority of cases.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* lint

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-08-19 15:27:52 +05:30
beorn7 c9fd3c235d Merge branch 'main' into sparsehistogram 2022-08-10 17:54:37 +02:00
Bryan Boreham a7f19b5775 labels: add a test for JSON and YAML marshalling
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-08-03 11:18:02 +02:00
Bryan Boreham 10699c37a3 labels: test BytesWithoutLabels does not remove __name__ by default
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-08-03 11:18:02 +02:00
Bryan Boreham d46ef0aa8e labels: tweak BenchmarkLabels_Get()
So the benchmark works without requiring `Labels` internals to be a slice.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-08-03 11:18:02 +02:00
Bryan Boreham 24ebff9c4a labels: don't test that Hash() works on unordered labels
Labels are required to be ordered.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-08-03 11:18:02 +02:00
Bryan Boreham fb2d883f1e labels: test Compare without knowing the exact result
Only the sign of the result is important

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-08-03 11:18:02 +02:00
beorn7 89fb7117c3 textparse: Remove TODO that is actually done
Signed-off-by: beorn7 <beorn@grafana.com>
2022-07-20 19:08:36 +02:00
beorn7 c40b105efd histograms: Move to new exposition protobuf format
This is an incompatible protobuf change. Instrumented targets must
include https://github.com/prometheus/client_golang/pull/1092 to make
this work.

Signed-off-by: beorn7 <beorn@grafana.com>
2022-07-19 18:11:33 +02:00
Paschalis Tsilias d1122e0743
Introduce TSDB changes for appending metadata to the WAL (#10972)
* Append metadata to the WAL

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Remove extra whitespace; Reword some docstrings and comments

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Use RLock() for hasNewMetadata check

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Use single byte for metric type in RefMetadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Update proposed WAL format for single-byte type metadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Implementa MetadataAppender interface for the Agent

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Address first round of review comments

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Amend description of metadata in wal.md

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Correct key used to retrieve metadata from cache

When we're setting metadata entries in the scrapeCace, we're using the
p.Help(), p.Unit(), p.Type() helpers, which retrieve the series name and
use it as the cache key. When checking for cache entries though, we used
p.Series() as the key, which included the metric name _with_ its labels.
That meant that we were never actually hitting the cache. We're fixing
this by utiling the __name__ internal label for correctly getting the
cache entries after they've been set by setHelp(), setType() or
setUnit().

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Put feature behind a feature flag

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix AppendMetadata docstring

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Reorder WAL format document

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Change error message of AppendMetadata; Fix access of s.meta in AppendMetadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Reuse temporary buffer in Metadata encoder

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Only keep latest metadata for each refID during checkpointing

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix test that's referencing decoding metadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Avoid creating metadata block if no new metadata are present

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Add tests for corrupt metadata block and relevant record type

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix CR comments

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Extract logic about changing metadata in an anonymous function

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Implement new proposed WAL format and amend relevant tests

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Use 'const' for metadata field names

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Apply metadata to head memSeries in Commit, not in AppendMetadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Add docstring and rename extracted helper in scrape.go

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Add tests for tsdb-related cases

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix linter issues vol1

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix linter issues vol2

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix Windows test by closing WAL reader files

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Use switch instead of two if statements in metadata decoding

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix review comments around TestMetadata* tests

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Add code for replaying WAL; test correctness of in-memory data after a replay

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Remove scrape-loop related code from PR

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Address first round of comments

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Simplify tests by sorting slices before comparison

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix test to use separate transactions

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Empty out buffer and record slices after encoding latest metadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix linting issue

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Update calculation for DroppedMetadata metric

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Rename MetadataAppender interface and AppendMetadata method to MetadataUpdater/UpdateMetadata

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Reuse buffer when encoding latest metadata for each series

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Fix review comments; Check all returned error values using two helpers

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Simplify use of helpers

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Satisfy linter

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>
2022-07-19 10:58:52 +02:00
beorn7 28f028e938 Merge branch 'main' into sparsehistogram 2022-07-12 19:07:13 +02:00
Julien Pivotto c637705403
Merge pull request #10893 from prymitive/unwrap_errors
Implement Unwrap() on errors returned from rulefmt
2022-06-30 11:30:21 +02:00
Matthieu MOREL c2b4de3611
refactor (package model): move from github.com/pkg/errors to 'errors' and 'fmt' packages (#10747)
Signed-off-by: Matthieu MOREL <mmorel-35@users.noreply.github.com>
Co-Authored-By: Julien Pivotto <roidelapluie@gmail.com>

Co-authored-by: Julien Pivotto <roidelapluie@gmail.com>
2022-06-27 21:29:19 +02:00
Łukasz Mierzwa 648b12d8c5 Implement Unwrap() on errors returned from rulefmt
I'd like to unwrap errors returned from rulefmt but both Error and WrappedError types are missing Unwrap() method.

Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>
2022-06-20 17:01:52 +01:00
beorn7 095b6c93dd Merge branch 'main' into sparsehistogram 2022-06-14 14:27:35 +02:00
Paschalis Tsilias 4f3791024a
Drop extra string held in relabel.Regexp struct (#10846)
* Drop extra string held in relabel.Regexp struct

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>

* Use slice operations instead of TrimPrefix/TrimSuffix; Override String() method

Signed-off-by: Paschalis Tsilias <paschalist0@gmail.com>
2022-06-13 15:00:48 +02:00
beorn7 40ad5e284a Merge branch 'main' into beorn7/sparsehistogram 2022-06-09 20:50:30 +02:00
Bryan Boreham 2e2c014d52
Labels: optimise creation of signature with/without labels (#10667)
* Labels: create signature with/without labels

Instead of creating a new Labels slice then converting to signature,
go directly to the signature and save time.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Labels: refactor Builder tests

Have one test with a range of cases, and have them check the final
output rather than checking the internal structure of the Builder.

Also add a couple of cases where the value is "", which should be
interpreted as 'delete'.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Labels: add 'Keep' function to Builder

This lets us replace `Labels.WithLabels` with the more general `Builder`.

In `engine.resultMetric()` we can call `Keep()` instead of checking
and calling `Del()`.

Avoid calling `Sort()` in `Builder.Labels()` if we didn't add anything,
so that `Keep()` has the same performance as `WithLabels()`.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-06-07 10:08:27 +05:30
Łukasz Mierzwa b11062bfcc
Don't count added labels when initializing slice (#10756)
This was added in #10749 but based on feedback it might be over-allocating so worth removing.

Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>
2022-06-04 08:45:02 +05:30
Łukasz Mierzwa 08262454a3
Preallocate Labels in labels.Builder (#10749)
This tries to avoid re-allocations of labels slice since we know possible max size

Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>
2022-05-25 16:22:47 +02:00
Łukasz Mierzwa 89de30a0b7
Avoid reallocating map in MatchLabels (#10715)
We know the max size of our map so we can create it with that information and avoid extra allocations

Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>
2022-05-20 10:18:31 +02:00
beorn7 3bc711e333 Merge branch 'main' into sparsehistogram 2022-05-04 13:37:13 +02:00
beorn7 2d233cf95e Histogram: Fix allFloatBucketIterator
If the zero threshold overlaps with the highest negative bucket and/or
the lowest positive bucket, its upper or lower boundary, respectively,
has to be adjusted. In valid histograms, only ever the highest
negative bucket and/or the lowest positive bucket may overlap with the
zero bucket. This is assumed in this code to simplify the checks.

Signed-off-by: beorn7 <beorn@grafana.com>
2022-05-03 16:24:11 +02:00
Julien Pivotto 71dbb4d091
Add lowercase and uppercase relabel action (#10641)
Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>
2022-05-03 10:09:53 +02:00
Matthieu MOREL e2ede285a2
refactor: move from io/ioutil to io and os packages (#10528)
* refactor: move from io/ioutil to io and os packages
* use fs.DirEntry instead of os.FileInfo after os.ReadDir

Signed-off-by: MOREL Matthieu <matthieu.morel@cnp.fr>
2022-04-27 11:24:36 +02:00
beorn7 56db51c826 Histgram: Fix Compact for spans of only empty buckets
Signed-off-by: beorn7 <beorn@grafana.com>
2022-04-12 00:37:50 +02:00
beorn7 15583af9bb Histogram: Fix crash when compacting only empty buckets
Signed-off-by: beorn7 <beorn@grafana.com>
2022-04-07 16:24:48 +02:00
beorn7 08efde4de5 Adjust to newest gofumpt
Signed-off-by: beorn7 <beorn@grafana.com>
2022-04-05 18:35:01 +02:00
beorn7 7ee1836ef5 Merge branch 'main' into sparsehistogram 2022-04-05 18:31:19 +02:00
Peter Štibraný cd18da3605
Fix OpenMetrics parser to sort uppercase labels correctly. (#10510)
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
2022-03-31 20:18:53 +02:00
beorn7 f9c411604d Fix spelling errors
Signed-off-by: beorn7 <beorn@grafana.com>
2022-03-22 16:02:13 +01:00
beorn7 4210aac74a Merge branch 'main' into sparsehistogram 2022-03-22 14:47:42 +01:00
beorn7 9fbcf14e5c histogram: Handle changes of the ZeroThreshold and the Schema
Signed-off-by: beorn7 <beorn@grafana.com>
2022-03-17 18:05:31 +01:00
Nick Pillitteri 53ac9d6d66
Compare equality of label.Label structs directly (#10427)
* labels.Equal benchmark for equal, not equal, and differing lengths

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>

* Compare equality of label.Label structs directly

Compare the structs using `==` instead of the name and value
of each label. This is functionally equivalent and about ~10%
faster in my testing.

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>

* Use longer more realistic names and values in benchmark

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
2022-03-15 00:30:04 +01:00
Levi Harrison a9e2ffb717
Specify type in metadata parser errors (#10269) 2022-03-08 08:04:11 -05:00
Łukasz Mierzwa da23c4649a
Enable misspell check in golangci-lint (#10393)
Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>
2022-03-03 18:11:19 +01:00
Matthias Loibl afdd1357e0
Merge pull request #10267 from roidelapluie/contenttype
Improve content-type error handling
2022-02-14 11:57:09 +01:00
Julien Pivotto 9a2e93228e
Switch to grafana/regexp everywhere (#10268)
Let's have a consistent library for regexp.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2022-02-13 00:58:27 +01:00
Bryan Boreham 579331446a
Allow downstream projects to use faster regexp (#10251)
* Add benchmark for FastRegexMatcher

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Use modified regexp package with optimisations

See https://github.com/grafana/regexp/tree/speedup#readme

Includes the following changes proposed upstream:
* [regexp: allow patterns with no alternates to be one-pass](https://go-review.googlesource.com/c/go/+/353711)
* [regexp: speed up onepass prefix check](https://go-review.googlesource.com/c/go/+/354909)
* [regexp: handle prefix string with fold-case](https://go-review.googlesource.com/c/go/+/358756)
* [regexp: avoid copying each instruction executed](https://go-review.googlesource.com/c/go/+/355789)
* [regexp: allow prefix string anchored at beginning](https://go-review.googlesource.com/c/go/+/377294)

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Use regexp code identical to upstream Go

Change `grafana/regexp` import to use `main` branch.

This means Prometheus is not using the proposed optimisations, but
downstream users of Prometheus code are able to `replace` the library
with the `speedup` branch which does.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-02-08 11:03:20 +01:00
Julien Pivotto f695df843f Improve content-type error handling
- Call err everywhere
- Change log message to underscore-separated field

Followup on #10186

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2022-02-08 11:02:51 +01:00
Matheus Pimenta 8d8ce641a4
error for invalid media type should not be completely swallowed (#10186)
* error for invalid media type should not be completely swallowed

Signed-off-by: Matheus Pimenta <matheuscscp@gmail.com>
2022-02-08 10:57:56 +01:00
Björn Rabenstein ec80745884
Merge pull request #10075 from prometheus/beorn7/histogram
model: Implement FloatHistogram.Compact
2022-01-05 16:09:39 +01:00
beorn7 3b4d6c3fdb model: Implement FloatHistogram.Compact
Signed-off-by: beorn7 <beorn@grafana.com>
2022-01-05 14:34:03 +01:00
beorn7 e7592fe353 sparsehistogram: Address two TODOs
Signed-off-by: beorn7 <beorn@grafana.com>
2022-01-04 12:48:59 +01:00
beorn7 27f865ec49 histograms: avoid initialization tracking in reverseFloatBucketIterator
Signed-off-by: beorn7 <beorn@grafana.com>
2021-12-15 16:56:33 +01:00
beorn7 a6acdfe346 histograms: Doc comment and naming improvements
Signed-off-by: beorn7 <beorn@grafana.com>
2021-12-15 16:50:37 +01:00
Ganesh Vernekar 4a43349aca
`histogram_quantile` for sparse histograms (#9935)
* MergeFloatBucketIterator for []FloatBucketIterator

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* histogram_quantile for histograms

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix histogram_quantile

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Unit test and enhancements

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Iterators to iterate buckets in reverse and all buckets together including zero bucket

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Consider all buckets for histogram_quantile and fix the implementation

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Remove unneeded code

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix lint

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2021-12-06 19:17:22 +05:30
Björn Rabenstein 0e1b9dd308
Promql: Initial rate implementation for sparse histograms (#9926)
Signed-off-by: beorn7 <beorn@grafana.com>
2021-12-06 18:19:18 +05:30
Björn Rabenstein 7e42acd3b1
tsdb: Rework iterators (#9877)
- Pick At... method via return value of Next/Seek.
- Do not clobber returned buckets.
- Add partial FloatHistogram suppert.

Note that the promql package is now _only_ dealing with
FloatHistograms, following the idea that PromQL only knows float
values.

As a byproduct, I have removed the histogramSeries metric. In my
understanding, series can have both float and histogram samples, so
that metric doesn't make sense anymore.

As another byproduct, I have converged the sampleBuf and the
histogramSampleBuf in memSeries into one. The sample type stored in
the sampleBuf has been extended to also contain histograms even before
this commit.

Signed-off-by: beorn7 <beorn@grafana.com>
2021-11-29 13:24:23 +05:30
beorn7 6a820a646c histogram: Add FloatHistogram
Including a few adjustments for normal Histogram, too, e.g. use
pointer receiver to avoid the large copy on method calls.

Signed-off-by: beorn7 <beorn@grafana.com>
2021-11-23 19:40:49 +01:00
beorn7 5d4db805ac Merge branch 'main' into sparsehistogram 2021-11-17 19:57:31 +01:00
beorn7 73858d7f82 storage: histogram support in memoized_iterator
Signed-off-by: beorn7 <beorn@grafana.com>
2021-11-15 21:55:58 +01:00
ozairasim 52159840eb
Expose Error function for WrappedError (#9662) 2021-11-12 17:38:59 -05:00
beorn7 f1065e44a4 model: String method for histogram.Histogram
This includes a regular bucket iterator and a string method for
histogram.Bucket.

Signed-off-by: beorn7 <beorn@grafana.com>
2021-11-11 17:29:22 +01:00
beorn7 c954cd9d1d Move packages out of deprecated pkg directory
This creates a new `model` directory and moves all data-model related
packages over there:
  exemplar labels relabel rulefmt textparse timestamp value

All the others are more or less utilities and have been moved to `util`:
  gate logging modetimevfs pool runtime

Signed-off-by: beorn7 <beorn@grafana.com>
2021-11-09 08:03:10 +01:00
beorn7 7a8bb8222c Style cleanup of all the changes in sparsehistogram so far
A lot of this code was hacked together, literally during a
hackathon. This commit intends not to change the code substantially,
but just make the code obey the usual style practices.

A (possibly incomplete) list of areas:

* Generally address linter warnings.

* The `pgk` directory is deprecated as per dev-summit. No new packages should
  be added to it. I moved the new `pkg/histogram` package to `model`
  anticipating what's proposed in #9478.

* Make the naming of the Sparse Histogram more consistent. Including
  abbreviations, there were just too many names for it: SparseHistogram,
  Histogram, Histo, hist, his, shs, h. The idea is to call it "Histogram" in
  general. Only add "Sparse" if it is needed to avoid confusion with
  conventional Histograms (which is rare because the TSDB really has no notion
  of conventional Histograms). Use abbreviations only in local scope, and then
  really abbreviate (not just removing three out of seven letters like in
  "Histo"). This is in the spirit of
  https://github.com/golang/go/wiki/CodeReviewComments#variable-names

* Several other minor name changes.

* A lot of formatting of doc comments. For one, following
  https://github.com/golang/go/wiki/CodeReviewComments#comment-sentences
  , but also layout question, anticipating how things will look like
  when rendered by `godoc` (even where `godoc` doesn't render them
  right now because they are for unexported types or not a doc comment
  at all but just a normal code comment - consistency is queen!).

* Re-enabled `TestQueryLog` and `TestEndopints` (they pass now,
  leaving them disabled was presumably an oversight).

* Bucket iterator for histogram.Histogram is now created with a
  method.

* HistogramChunk.iterator now allows iterator recycling. (I think
  @dieterbe only commented it out because he was confused by the
  question in the comment.)

* HistogramAppender.Append panics now because we decided to treat
  staleness marker differently.

Signed-off-by: beorn7 <beorn@grafana.com>
2021-10-11 13:02:03 +02:00
Bjoern Rabenstein 006b5517e2 Simplify makefiles.
This removes the dependancy on C leveldb and snappy.
It also takes care of fewer dependencies as they would
anyway not work on any non-Debian, non-Brew system.

Change-Id: Ia70dce1ba8a816a003587927e0b3a3f8ad2fd28c
2014-11-25 17:10:39 +01:00
Julius Volz ae30453214 Add label names -> label values index.
Change-Id: Ie39b4044558afc4d1aa937de7dcf8df61f821fb4
2014-03-28 15:16:37 +01:00
Julius Volz 1eee448bc1 Store samples in custom binary encoding.
This has been shown to provide immense decoding speed benefits.

See also:

https://groups.google.com/forum/#!topic/prometheus-developers/FeGl_qzGrYs

Change-Id: I7d45b4650e44ddecaa91dad9d7fdb3cd0b9f15fe
2014-03-09 22:31:38 +01:00
Julius Volz a310afd28f Update "build" -> ".build" in remaining Makefiles. 2013-06-26 16:09:37 +02:00
Matt T. Proud 30b1cf80b5 WIP - Snapshot of Moving to Client Model. 2013-06-25 15:52:42 +02:00
Julius Volz 0226d1ac7a Implement alerts dashboard and expression console links. 2013-06-13 22:35:40 +02:00
Matt T. Proud f895acb9ef Trailing build system cleanups.
The .gitignore files had problems, and the build steps could be
simplified.
2013-06-13 15:38:03 +02:00
Julius Volz fc97e688c6 Improve printing of rules and expressions. 2013-06-11 11:39:31 +02:00