prometheus

Commit Graph

Author	SHA1	Message	Date
Jeanette Tan	9a8bd8eac6	Fix possible non-counter warning for empty names and native histograms Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	1 year ago
Julius Volz	191c24a0ed	Fix: Exempt "_bucket" suffix from PossibleNonCounterInfo warning (#12982 ) Related to PR #12152 Signed-off-by: Julius Volz <julius.volz@gmail.com> Signed-off-by: Levi Harrison <git@leviharrison.dev>	1 year ago
Jeanette Tan	0cbf0c1c68	Revise according to code review Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	1 year ago
Jeanette Tan	feaa93da77	Add warning when monotonicity is forced in the input to histogram_quantile Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	1 year ago
zenador	69edd8709b	Add warnings (and annotations) to PromQL query results (#12152 ) Return annotations (warnings and infos) from PromQL queries This generalizes the warnings we have already used before (but only for problems with remote read) as "annotations". Annotations can be warnings or infos (the latter could be false positives). We do not treat them different in the API for now and return them all as "warnings". It would be easy to distinguish them and return infos separately, should that appear useful in the future. The new annotations are then used to create a lot of warnings or infos during PromQL evaluations. Partially these are things we have wanted for a long time (e.g. inform the user that they have applied `rate` to a metric that doesn't look like a counter), but the new native histograms have created even more needs for those annotations (e.g. if a query tries to aggregate float numbers with histograms). The annotations added here are not yet complete. A prominent example would be a warning about a range too short for a rate calculation. But such a warnings is more tricky to create with good fidelity and we will tackle it later. Another TODO is to take annotations into account when evaluating recording rules. --------- Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	1 year ago
zenador	54aaa2bd7e	Add `histogram_stdvar` and `histogram_stddev` functions (#12614 ) * Add new function: histogram_stdvar and histogram_stddev Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	1 year ago
zenador	191bf9055b	Handle more arithmetic operators for native histograms (#12262 ) Handle more arithmetic operators and aggregators for native histograms This includes operators for multiplication (formerly known as scaling), division, and subtraction. Plus aggregations for average and the avg_over_time function. Stdvar and stddev will (for now) ignore histograms properly (rather than counting them but adding a 0 for them). Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>	2 years ago
Matthieu MOREL	bae9a21200	Merge branch 'main' into linter/nilerr Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2 years ago
beorn7	5b53aa1108	style: Replace `else if` cascades with `switch` Wiser coders than myself have come to the conclusion that a `switch` statement is almost always superior to a statement that includes any `else if`. The exceptions that I have found in our codebase are just these two: * The `if else` is followed by an additional statement before the next condition (separated by a `;`). * The whole thing is within a `for` loop and `break` statements are used. In this case, using `switch` would require tagging the `for` loop, which probably tips the balance. Why are `switch` statements more readable? For one, fewer curly braces. But more importantly, the conditions all have the same alignment, so the whole thing follows the natural flow of going down a list of conditions. With `else if`, in contrast, all conditions but the first are "hidden" behind `} else if `, harder to spot and (for no good reason) presented differently from the first condition. I'm sure the aforemention wise coders can list even more reasons. In any case, I like it so much that I have found myself recommending it in code reviews. I would like to make it a habit in our code base, without making it a hard requirement that we would test on the CI. But for that, there has to be a role model, so this commit eliminates all `if else` occurrences, unless it is autogenerated code or fits one of the exceptions above. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
beorn7	c3c7d44d84	lint: Adjust to the lint warnings raised by current versions of golint-ci We haven't updated golint-ci in our CI yet, but this commit prepares for that. There are a lot of new warnings, and it is mostly because the "revive" linter got updated. I agree with most of the new warnings, mostly around not naming unused function parameters (although it is justified in some cases for documentation purposes – while things like mocks are a good example where not naming the parameter is clearer). I'm pretty upset about the "empty block" warning to include `for` loops. It's such a common pattern to do something in the head of the `for` loop and then have an empty block. There is still an open issue about this: https://github.com/mgechev/revive/issues/810 I have disabled "revive" altogether in files where empty blocks are used excessively, and I have made the effort to add individual `// nolint:revive` where empty blocks are used just once or twice. It's borderline noisy, though, but let's go with it for now. I should mention that none of the "empty block" warnings for `for` loop bodies were legitimate. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Matthieu MOREL	fb3eb21230	enable gocritic, unconvert and unused linters Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2 years ago
beorn7	817a2396cb	Name float values as "floats", not as "values" In the past, every sample value was a float, so it was fine to call a variable holding such a float "value" or "sample". With native histograms, a sample might have a histogram value. And a histogram value is still a value. Calling a float value just "value" or "sample" or "V" is therefore misleading. Over the last few commits, I already renamed many variables, but this cleans up a few more places where the changes are more invasive. Note that we do not to attempt naming in the JSON APIs or in the protobufs. That would be quite a disruption. However, internally, we can call variables as we want, and we should go with the option of avoiding misunderstandings. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
beorn7	c0879d64cf	promql: Separate `Point` into `FPoint` and `HPoint` In other words: Instead of having a “polymorphous” `Point` that can either contain a float value or a histogram value, use an `FPoint` for floats and an `HPoint` for histograms. This seemingly small change has a _lot_ of repercussions throughout the codebase. The idea here is to avoid the increase in size of `Point` arrays that happened after native histograms had been added. The higher-level data structures (`Sample`, `Series`, etc.) are still “polymorphous”. The same idea could be applied to them, but at each step the trade-offs needed to be evaluated. The idea with this change is to do the minimum necessary to get back to pre-histogram performance for functions that do not touch histograms. Here are comparisons for the `changes` function. The test data doesn't include histograms yet. Ideally, there would be no change in the benchmark result at all. First runtime v2.39 compared to directly prior to this commit: ``` name old time/op new time/op delta RangeQuery/expr=changes(a_one[1d]),steps=1-16 391µs ± 2% 542µs ± 1% +38.58% (p=0.000 n=9+8) RangeQuery/expr=changes(a_one[1d]),steps=10-16 452µs ± 2% 617µs ± 2% +36.48% (p=0.000 n=10+10) RangeQuery/expr=changes(a_one[1d]),steps=100-16 1.12ms ± 1% 1.36ms ± 2% +21.58% (p=0.000 n=8+10) RangeQuery/expr=changes(a_one[1d]),steps=1000-16 7.83ms ± 1% 8.94ms ± 1% +14.21% (p=0.000 n=10+10) RangeQuery/expr=changes(a_ten[1d]),steps=1-16 2.98ms ± 0% 3.30ms ± 1% +10.67% (p=0.000 n=9+10) RangeQuery/expr=changes(a_ten[1d]),steps=10-16 3.66ms ± 1% 4.10ms ± 1% +11.82% (p=0.000 n=10+10) RangeQuery/expr=changes(a_ten[1d]),steps=100-16 10.5ms ± 0% 11.8ms ± 1% +12.50% (p=0.000 n=8+10) RangeQuery/expr=changes(a_ten[1d]),steps=1000-16 77.6ms ± 1% 87.4ms ± 1% +12.63% (p=0.000 n=9+9) RangeQuery/expr=changes(a_hundred[1d]),steps=1-16 30.4ms ± 2% 32.8ms ± 1% +8.01% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=10-16 37.1ms ± 2% 40.6ms ± 2% +9.64% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=100-16 105ms ± 1% 117ms ± 1% +11.69% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=1000-16 783ms ± 3% 876ms ± 1% +11.83% (p=0.000 n=9+10) ``` And then runtime v2.39 compared to after this commit: ``` name old time/op new time/op delta RangeQuery/expr=changes(a_one[1d]),steps=1-16 391µs ± 2% 547µs ± 1% +39.84% (p=0.000 n=9+8) RangeQuery/expr=changes(a_one[1d]),steps=10-16 452µs ± 2% 616µs ± 2% +36.15% (p=0.000 n=10+10) RangeQuery/expr=changes(a_one[1d]),steps=100-16 1.12ms ± 1% 1.26ms ± 1% +12.20% (p=0.000 n=8+10) RangeQuery/expr=changes(a_one[1d]),steps=1000-16 7.83ms ± 1% 7.95ms ± 1% +1.59% (p=0.000 n=10+8) RangeQuery/expr=changes(a_ten[1d]),steps=1-16 2.98ms ± 0% 3.38ms ± 2% +13.49% (p=0.000 n=9+10) RangeQuery/expr=changes(a_ten[1d]),steps=10-16 3.66ms ± 1% 4.02ms ± 1% +9.80% (p=0.000 n=10+9) RangeQuery/expr=changes(a_ten[1d]),steps=100-16 10.5ms ± 0% 10.8ms ± 1% +3.08% (p=0.000 n=8+10) RangeQuery/expr=changes(a_ten[1d]),steps=1000-16 77.6ms ± 1% 78.1ms ± 1% +0.58% (p=0.035 n=9+10) RangeQuery/expr=changes(a_hundred[1d]),steps=1-16 30.4ms ± 2% 33.5ms ± 4% +10.18% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=10-16 37.1ms ± 2% 40.0ms ± 1% +7.98% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=100-16 105ms ± 1% 107ms ± 1% +1.92% (p=0.000 n=10+10) RangeQuery/expr=changes(a_hundred[1d]),steps=1000-16 783ms ± 3% 775ms ± 1% -1.02% (p=0.019 n=9+9) ``` In summary, the runtime doesn't really improve with this change for queries with just a few steps. For queries with many steps, this commit essentially reinstates the old performance. This is good because the many-step queries are the one that matter most (longest absolute runtime). In terms of allocations, though, this commit doesn't make a dent at all (numbers not shown). The reason is that most of the allocations happen in the sampleRingIterator (in the storage package), which has to be addressed in a separate commit. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Bryan Boreham	b987afa7ef	labels: simplify call to get Labels from Builder It took a `Labels` where the memory could be re-used, but in practice this hardly ever benefitted. Especially after converting `relabel.Process` to `relabel.ProcessBuilder`. Comparing the parameter to `nil` was a bug; `EmptyLabels` is not `nil` so the slice was reallocated multiple times by `append`. Lastly `Builder.Labels()` now estimates that the final size will depend on labels added and deleted. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Trevor Whitney	dd94ebb87b	promql: set CounterResetHint after rate and sum Signed-off-by: Trevor Whitney <trevorjwhitney@gmail.com>	2 years ago
Bryan Boreham	56fefcd812	Update package promql for new labels.Labels type We use `labels.Builder` to parse metrics, to avoid depending on the internal implementation. This is not efficient, but the feature is only used in tests. It wasn't efficient previously either - calling `Sort()` after adding each label. `createLabelsForAbsentFunction` also uses a Builder now, and gets an extra `map` to replace the previous `Has()` usage. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> Fix up promql to compile with changes to Labels	2 years ago
Bryan Boreham	6bdecf377c	Switch from 'sanity' to more inclusive lanuage (#9376 ) * Switch from 'sanity' to more inclusive lanuage "Removing ableist language in code is important; it helps to create and maintain an environment that welcomes all developers of all backgrounds, while emphasizing that we as developers select the most articulate, precise, descriptive language we can rather than relying on metaphors. The phrase sanity check is ableist, and unnecessarily references mental health in our code bases. It denotes that people with mental illnesses are inferior, wrong, or incorrect, and the phrase sanity continues to be used by employers and other individuals to discriminate against these people." From https://gist.github.com/seanmhanson/fe370c2d8bd2b3228680e38899baf5cc Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Jesus Vazquez	e934d0f011	Merge 'main' into sparsehistogram Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>	2 years ago
beorn7	a7c519930e	histograms: Add Compact method to the normal integer Histogram And use the new method to call to compact Histograms during parsing. This happens for both `Histogram` and `FloatHistogram`. In this way, if targets decide to optimize the exposition size by merging spans with empty buckets in between, we still get a normalized results. It will also normalize away any valid but weird representations like empty spans, spans with offset zero, and empty buckets at the start or end of a span. The implementation seemed easy at first as it just turns the `compactBuckets` helper into a generic function (which now got its own file). However, the integer Histograms have delta buckets instead of absolute buckets, which had to be treated specially in the generic `compactBuckets` function. To make sure it works, I have added plenty of explicit tests for `Histogram` in addition to the `FloatHistogram` tests. I have also updated the doc comment for the `Compact` method. Based on the insights now expressed in the doc comment, compacting with a maxEmptyBuckets > 0 is rarely useful. Therefore, this commit also sets the value to 0 in the two cases we were using 3 so far. We might still want to reconsider, so I don't want to remove the maxEmptyBuckets parameter right now. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Julien Pivotto	96d5a32659	Update go to 1.19, set min version to 1.18 (#11279 ) * Update go to 1.19, set min version to 1.18 Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu> * Update golangci-lint Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu> Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>	2 years ago
Bryan Boreham	8b863c42dd	Optimise relabeling by re-using memory (#11147 ) * model/relabel: Add benchmark Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * model/relabel: re-use Builder across relabels Saves memory allocations. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * labels.Builder: allow re-use of result slice This reduces memory allocations where the caller has a suitable slice available. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * model/relabel: re-use source values slice To reduce memory allocations. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Unwind one change causing test failures Restore original behaviour in PopulateLabels, where we must not overwrite the input set. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * relabel: simplify values optimisation Use a stack-based array for up to 16 source labels, which will be the vast majority of cases. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * lint Signed-off-by: Bryan Boreham <bjboreham@gmail.com> Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
beorn7	9eafed0f79	promql: Add `histogram_count` and `histogram_sum` This follow a simple function-based approach to access the count and sum fields of a native Histogram. It might be more elegant to implement “accessors” via the dot operator, as considered in the brainstorming doc [1]. However, that would require the introduction of a whole new concept in PromQL. For the PoC, we should be fine with the function-based approch. Even the obvious inefficiencies (rate'ing a whole histogram twice when we only want to rate each the count and the sum once) could be optimized behind the scenes. Note that the function-based approach elegantly solves the problem of detecting counter resets in the sum of observations in the case of negative observations. (Since the whole native Histogram is rate'd, the counter reset is detected for the Histogram as a whole.) We will decide later if an “accessor” approach is really needed. It would change the example expression for average duration in functions.md from histogram_sum(rate(http_request_duration_seconds[10m])) / histogram_count(rate(http_request_duration_seconds[10m])) to rate(http_request_duration_seconds.sum[10m]) / rate(http_request_duration_seconds.count[10m]) [1]: https://docs.google.com/document/d/1ch6ru8GKg03N02jRjYriurt-CZqUVY09evPg6yKTA1s/edit Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
beorn7	a3a8f58bb3	promql: Add histogram_fraction function Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
beorn7	40ad5e284a	Merge branch 'main' into beorn7/sparsehistogram	3 years ago
Matthieu MOREL	0906f2eafa	refactor (promql): move from github.com/pkg/errors to 'errors' and 'fmt' (#10817 ) Signed-off-by: Matthieu MOREL <mmorel-35@users.noreply.github.com> Co-authored-by: Matthieu MOREL <mmorel-35@users.noreply.github.com>	3 years ago
Bryan Boreham	2e2c014d52	Labels: optimise creation of signature with/without labels (#10667 ) * Labels: create signature with/without labels Instead of creating a new Labels slice then converting to signature, go directly to the signature and save time. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Labels: refactor Builder tests Have one test with a range of cases, and have them check the final output rather than checking the internal structure of the Builder. Also add a couple of cases where the value is "", which should be interpreted as 'delete'. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Labels: add 'Keep' function to Builder This lets us replace `Labels.WithLabels` with the more general `Builder`. In `engine.resultMetric()` we can call `Keep()` instead of checking and calling `Del()`. Avoid calling `Sort()` in `Builder.Labels()` if we didn't add anything, so that `Keep()` has the same performance as `WithLabels()`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Ivo Gosemann	e22b54e253	Adds day_of_year function to PromQL Signed-off-by: Ivo Gosemann <ivo.gosemann@sap.com>	3 years ago
beorn7	106e20cde5	Histogram: Fix and simplify histogram_quantile For conventional histograms, we need to gather all the individual bucket timeseries at a data point to do the quantile calculation. The code so far mirrored this behavior for the new native histograms. However, since a single data point contains all the buckets alreade, that's actually not needed. This PR simplifies the code while still detecting a mix of conventional and native histograms. The weird signature calculation for the conventional histograms is getting even weirder because of that. If this PR turns out to do the right thing, I will implement a proper fix for the signature calculation upstream. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	4210aac74a	Merge branch 'main' into sparsehistogram	3 years ago
beorn7	9fbcf14e5c	histogram: Handle changes of the ZeroThreshold and the Schema Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Julien Pivotto	9a2e93228e	Switch to grafana/regexp everywhere (#10268 ) Let's have a consistent library for regexp. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	3 years ago
David N Perkins	472456efde	Added issue link to the comment Signed-off-by: David N Perkins <David.N.Perkins@ibm.com>	3 years ago
David N Perkins	52662e8105	Added metric name to histogram quartile binning and associated unit test Signed-off-by: David N Perkins <David.N.Perkins@ibm.com>	3 years ago
beorn7	a6acdfe346	histograms: Doc comment and naming improvements Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Ganesh Vernekar	4a43349aca	`histogram_quantile` for sparse histograms (#9935 ) * MergeFloatBucketIterator for []FloatBucketIterator Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * histogram_quantile for histograms Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix histogram_quantile Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Unit test and enhancements Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Iterators to iterate buckets in reverse and all buckets together including zero bucket Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Consider all buckets for histogram_quantile and fix the implementation Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Remove unneeded code Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> * Fix lint Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	3 years ago
Björn Rabenstein	0e1b9dd308	Promql: Initial rate implementation for sparse histograms (#9926 ) Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Matthew	7ee52bfb86	feat: linreg => slope 0 for const values (#9728 ) * feat: linreg => slope 0 for const values Signed-off-by: mtfoley <mtfoley.mae@gmail.com>	3 years ago
Thomas Jackson	f0003bc0ba	Don't drop ParenExpr when creating StepInvariantExpr (#9591 ) * Add test case to showcase the problem in #9590 Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com> * Don't unwrap ParenExpr in newStepInvariantExpr Fixes #9590 Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>	3 years ago
beorn7	c954cd9d1d	Move packages out of deprecated pkg directory This creates a new `model` directory and moves all data-model related packages over there: exemplar labels relabel rulefmt textparse timestamp value All the others are more or less utilities and have been moved to `util`: gate logging modetimevfs pool runtime Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Mateusz Gozdek	1a6c2283a3	Format Go source files using 'gofumpt -w -s -extra' Part of #9557 Signed-off-by: Mateusz Gozdek <mgozdekof@gmail.com>	3 years ago
darshanime	42d786f1ac	use kahan summation for aggregation functions Signed-off-by: darshanime <deathbullet@gmail.com>	3 years ago
darshanime	694b872dee	address stylistic nits Signed-off-by: darshanime <deathbullet@gmail.com>	3 years ago
darshanime	a905354da3	use kahan for avg_over_time Signed-off-by: darshanime <deathbullet@gmail.com>	3 years ago
darshanime	0a9deb9597	use kahan summation for numerical stability Signed-off-by: darshanime <deathbullet@gmail.com>	3 years ago
Levi Harrison	4ac4e2ca4b	Remove parenthesis Signed-off-by: Levi Harrison <git@leviharrison.dev>	3 years ago
Levi Harrison	6faca22eec	Add inverse hyperbolic functions Signed-off-by: Levi Harrison <git@leviharrison.dev>	3 years ago
Levi Harrison	9c54ee0a6e	Changed function comment Signed-off-by: Levi Harrison <git@leviharrison.dev>	3 years ago
Levi Harrison	53d88fd147	Added hyperbolic trig functions Signed-off-by: Levi Harrison <git@leviharrison.dev>	3 years ago
Levi Harrison	9fc7ba33aa	Remove `atan2()` Signed-off-by: Levi Harrison <git@leviharrison.dev>	3 years ago
Levi Harrison	309e020c0a	Added functions Signed-off-by: Levi Harrison <git@leviharrison.dev>	3 years ago

1 2 3

150 Commits (0c1e447d681b5bd0e1bdb2e109d86a47dc962995)