prometheus

Commit Graph

Author	SHA1	Message	Date
Ganesh Vernekar	bde500e690	Merge pull request #11420 from jesusvazquez/jvp/update-sparsehistogram-with-main-v2 Merge branch 'main' into sparsehistogram	2 years ago
beorn7	bf0847073d	histogram: Modify getBound to deal properly with infinity The bucket receiving math.MaxFloat64 observations now has math.MaxFloat64 as upper bound, while the bucket after it (the last possible bucket) has +Inf. This also adds a test for getBound and moves the getBound code to generic.go (where it should have been in the first place). Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Jesus Vazquez	e934d0f011	Merge 'main' into sparsehistogram Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>	2 years ago
Björn Rabenstein	dccfb9db4e	histogram: Remove code replication via generics (#11361 ) * histogram: Simplify iterators We don't really need currLower and currUpper and can calculate it when needed (as already done for the floatBucketIterator). The calculation is cheap, while keeping those extra variables around costs RAM (potentially a lot with many iterators). * histogram: Convert Bucket/FloatBucket to one generic type * histogram: Move some bucket iterator code into generic base iterator * histogram: Remove cumulative iterator for FloatHistogram We added it in the past for completeness (Histogram has one), but it has never been used. Plus, even the cumulative iterator for Histogram is only there for test reasons. We can always add it back, and then maybe even using generics. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Giedrius Statkevičius	a1d6ba59ac	promql: pass down subquery interval (#11163 ) If we are populating series for a subquery then set the interval parameter accordingly so that downstream users could use that information. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>	2 years ago
Bryan Boreham	3330d85ba8	Replace sort.Strings and sort.Ints with faster slices.Sort (#11318 ) Use new experimental package `golang.org/x/exp/slices`. slices.Sort works on values that are directly comparable, like ints, so avoids the overhad of an interface call to `.Less()`. Left tests unchanged, because they don't need the speed and it may be a cross-check that slices.Sort gives the same answer. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
beorn7	a7c519930e	histograms: Add Compact method to the normal integer Histogram And use the new method to call to compact Histograms during parsing. This happens for both `Histogram` and `FloatHistogram`. In this way, if targets decide to optimize the exposition size by merging spans with empty buckets in between, we still get a normalized results. It will also normalize away any valid but weird representations like empty spans, spans with offset zero, and empty buckets at the start or end of a span. The implementation seemed easy at first as it just turns the `compactBuckets` helper into a generic function (which now got its own file). However, the integer Histograms have delta buckets instead of absolute buckets, which had to be treated specially in the generic `compactBuckets` function. To make sure it works, I have added plenty of explicit tests for `Histogram` in addition to the `FloatHistogram` tests. I have also updated the doc comment for the `Compact` method. Based on the insights now expressed in the doc comment, compacting with a maxEmptyBuckets > 0 is rarely useful. Therefore, this commit also sets the value to 0 in the two cases we were using 3 so far. We might still want to reconsider, so I don't want to remove the maxEmptyBuckets parameter right now. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
Bryan Boreham	b01d29cf9e	promql: in tests use labels.FromStrings And a few cases of `EmptyLabels()`. Replacing code which assumes the internal structure of `Labels`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Julien Pivotto	96d5a32659	Update go to 1.19, set min version to 1.18 (#11279 ) * Update go to 1.19, set min version to 1.18 Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu> * Update golangci-lint Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu> Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>	2 years ago
Ganesh Vernekar	71489d0e3d	Fix count() for histograms and add test case Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	2 years ago
Bryan Boreham	8b863c42dd	Optimise relabeling by re-using memory (#11147 ) * model/relabel: Add benchmark Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * model/relabel: re-use Builder across relabels Saves memory allocations. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * labels.Builder: allow re-use of result slice This reduces memory allocations where the caller has a suitable slice available. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * model/relabel: re-use source values slice To reduce memory allocations. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Unwind one change causing test failures Restore original behaviour in PopulateLabels, where we must not overwrite the input set. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * relabel: simplify values optimisation Use a stack-based array for up to 16 source labels, which will be the vast majority of cases. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * lint Signed-off-by: Bryan Boreham <bjboreham@gmail.com> Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2 years ago
Ganesh Vernekar	9325caa41c	Remove a TODO that is no longer valid (#11186 ) Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>	2 years ago
Cosrider	af79dce8ea	Close file descriptor in query logging (#11148 ) Signed-off-by: Cosrider <cosrider7@gmail.com> Signed-off-by: Cosrider <cosrider7@gmail.com>	2 years ago
beorn7	c9fd3c235d	Merge branch 'main' into sparsehistogram	2 years ago
Vilius Pranckaitis	4660656312	Allow setting custom lookback delta for instant queries (#9946 ) * Allow setting custom lookback delta for instant queries Signed-off-by: Vilius Pranckaitis <vpranckaitis@gmail.com>	2 years ago
Levi Harrison	77a7af4461	Add histogram validation (#11052 ) * Add histogram validation Signed-off-by: Levi Harrison <git@leviharrison.dev> * Correct negative offset validation Signed-off-by: Levi Harrison <git@leviharrison.dev> * Address review comments Signed-off-by: Levi Harrison <git@leviharrison.dev> * Validation benchmark Signed-off-by: Levi Harrison <git@leviharrison.dev> * Add more checks Signed-off-by: Levi Harrison <git@leviharrison.dev> * Attempt to fix tests Signed-off-by: Levi Harrison <git@leviharrison.dev> * Fix stuff Signed-off-by: Levi Harrison <git@leviharrison.dev>	2 years ago
Julien Pivotto	d41e5a5582	Prettifier: Add spaces with non-callable keywords (#11005 ) * Prettifier: Add spaces with non-callable keywords I prefer to have a difference between, on one side: functions calls, end(), start(), and on the other side with, without, ignoring, by and group_rrigt, group_left. The reasoning is that the former ones are not calls, while other are functions. Additionally, it matches the examples in our documentation. Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu> * Fix tests Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>	2 years ago
Łukasz Mierzwa	54a3c3ba3f	Print query that caused a panic (#10995 ) We print the stacktrace of a panic when query causes one, but there's no information about the query itself, which makes it harder to debug and reproduce the issue. This adds the 'expr' string to the logged panic. Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	2 years ago
beorn7	53982c3562	Merge branch 'main' into sparsehistogram	2 years ago
Oleg Zaytsev	d2abe9a58a	Shortcut `ContainsSameLabelset()` (#11000 ) * Shortcut Matrix.ContainsSameLabelset() It's quite often to execute this check on a Matrix that has zero or only one series. There's no need to allocate a map for those cases. There's also a one-liner for two-series case, so why not using it? Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Add license header Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Optimize Vector.ContainsSameLabelset Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>	2 years ago
beorn7	28f028e938	Merge branch 'main' into sparsehistogram	2 years ago
Harkishen Singh	44fcf876ca	Adds support for prettifying PromQL expression (#10544 ) * Implement Pretty() function for AST nodes. Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com> This commit adds .Pretty() for all nodes of PromQL AST. Each .Pretty() prettifies the node it belongs to, and under no circustance, the parent or child node is touch/prettified. Read more in the "Approach" part in `prettier.go` * Refactor functions between printer.go & prettier.go Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com> This commit removes redundancy between printer.go and prettier.go by taking out the common code into separate private functions. * Add more unit tests for Prettier. Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com> * Add support for spliting function calls with 1 arg & unary expressions. Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com> This commit does 2 things: 1. It adds support to split function calls that have 1 arg and exceeds the max_characters_per_line to multiple lines. 2. Splits Unary expressions that exceed the max_characters_per_line. This is done by formatting the child node and then removing the prefix indent, which is already applied before the unary operator.	2 years ago
beorn7	9eafed0f79	promql: Add `histogram_count` and `histogram_sum` This follow a simple function-based approach to access the count and sum fields of a native Histogram. It might be more elegant to implement “accessors” via the dot operator, as considered in the brainstorming doc [1]. However, that would require the introduction of a whole new concept in PromQL. For the PoC, we should be fine with the function-based approch. Even the obvious inefficiencies (rate'ing a whole histogram twice when we only want to rate each the count and the sum once) could be optimized behind the scenes. Note that the function-based approach elegantly solves the problem of detecting counter resets in the sum of observations in the case of negative observations. (Since the whole native Histogram is rate'd, the counter reset is detected for the Histogram as a whole.) We will decide later if an “accessor” approach is really needed. It would change the example expression for average duration in functions.md from histogram_sum(rate(http_request_duration_seconds[10m])) / histogram_count(rate(http_request_duration_seconds[10m])) to rate(http_request_duration_seconds.sum[10m]) / rate(http_request_duration_seconds.count[10m]) [1]: https://docs.google.com/document/d/1ch6ru8GKg03N02jRjYriurt-CZqUVY09evPg6yKTA1s/edit Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
beorn7	a3a8f58bb3	promql: Add histogram_fraction function Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
beorn7	ffaabea91a	promql: Refine zero bucket treatment in histogramQuantile Essentially, this mirrors the existing behavior for negative buckets: If a histogram has only negative buckets, the upper bound of the zero bucket is assumed to be zero. Furthermore, it makes sure that the zero bucket boundaries are not modified if a histogram that has no buckets at all but samples in the zero bucket. Also, add an TODO to vet if we really want this behavior. Signed-off-by: beorn7 <beorn@grafana.com>	2 years ago
beorn7	40ad5e284a	Merge branch 'main' into beorn7/sparsehistogram	3 years ago
Matthieu MOREL	0906f2eafa	refactor (promql): move from github.com/pkg/errors to 'errors' and 'fmt' (#10817 ) Signed-off-by: Matthieu MOREL <mmorel-35@users.noreply.github.com> Co-authored-by: Matthieu MOREL <mmorel-35@users.noreply.github.com>	3 years ago
Bryan Boreham	2e2c014d52	Labels: optimise creation of signature with/without labels (#10667 ) * Labels: create signature with/without labels Instead of creating a new Labels slice then converting to signature, go directly to the signature and save time. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Labels: refactor Builder tests Have one test with a range of cases, and have them check the final output rather than checking the internal structure of the Builder. Also add a couple of cases where the value is "", which should be interpreted as 'delete'. Signed-off-by: Bryan Boreham <bjboreham@gmail.com> * Labels: add 'Keep' function to Builder This lets us replace `Labels.WithLabels` with the more general `Builder`. In `engine.resultMetric()` we can call `Keep()` instead of checking and calling `Del()`. Avoid calling `Sort()` in `Builder.Labels()` if we didn't add anything, so that `Keep()` has the same performance as `WithLabels()`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	3 years ago
Łukasz Mierzwa	08262454a3	Preallocate Labels in labels.Builder (#10749 ) This tries to avoid re-allocations of labels slice since we know possible max size Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	3 years ago
Ivo Gosemann	e22b54e253	Adds day_of_year function to PromQL Signed-off-by: Ivo Gosemann <ivo.gosemann@sap.com>	3 years ago
beorn7	3bc711e333	Merge branch 'main' into sparsehistogram	3 years ago
beorn7	d16b314b72	Histogram: Do not render empty buckets in JSON output While empty buckets can make sense in the internal representation (by joining spans that would otherwise need more overhead for separate representation), there are no spans in the JSON rendering. Therefore, the JSON should not contain any empty buckets, since any buckets not included in the output counts as empty anyway. This changes both the inefficient MarshalJSON implementation as well as the jsoniter implementation. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Matthieu MOREL	e2ede285a2	refactor: move from io/ioutil to io and os packages (#10528 ) * refactor: move from io/ioutil to io and os packages * use fs.DirEntry instead of os.FileInfo after os.ReadDir Signed-off-by: MOREL Matthieu <matthieu.morel@cnp.fr>	3 years ago
beorn7	37bbc07118	Histogram: Add jsoniter marshaling This now even enables jsoniter marshaling of Points in an instant query (which previously used the traditional JSON marshaling). Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	99894f6afa	Histogram: Implement inefficient JSON rendering Format is according to example 4 in https://docs.google.com/document/d/1Efu0LX-fgNWix6ehfeCR0FzeWtHvftWFNoy7cYW9nqU/edit# This is inefficient as it doesn't use jsoniter. It actually prevents it for regular Points (with a conventional float Value), too. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Alan Protasio	ce6a643ee8	Changing TotalQueryableSamples from int to int64 (#10549 ) * Changing TotalQueryableSamples from int to int64 Signed-off-by: Alan Protasio <approtas@amazon.com>	3 years ago
beorn7	106e20cde5	Histogram: Fix and simplify histogram_quantile For conventional histograms, we need to gather all the individual bucket timeseries at a data point to do the quantile calculation. The code so far mirrored this behavior for the new native histograms. However, since a single data point contains all the buckets alreade, that's actually not needed. This PR simplifies the code while still detecting a mix of conventional and native histograms. The weird signature calculation for the conventional histograms is getting even weirder because of that. If this PR turns out to do the right thing, I will implement a proper fix for the signature calculation upstream. Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
beorn7	7ee1836ef5	Merge branch 'main' into sparsehistogram	3 years ago
Martina Ferrari	3e4bd4d913	Avoid literal integer overflows in 32 bit arches. This commit ensures 64-bit integers are used in various tests that other wise fail in 32-bit architectures. It also adds support for int64 and uint64 types in the template.convertToFloat function to support the test changes. Closes: 10481 Signed-off-by: Martina Ferrari <tina@debian.org>	3 years ago
beorn7	4210aac74a	Merge branch 'main' into sparsehistogram	3 years ago
Andrew Bloomgarden	a64b9fe323	Report PeakSamples in query statistics This exactly corresponds to the statistic compared against MaxSamples during the course of query execution, so users can see how close their queries are to a limit. Co-authored-by: Harkishen Singh <harkishensingh@hotmail.com> Co-authored-by: Andrew Bloomgarden <blmgrdn@amazon.com> Signed-off-by: Andrew Bloomgarden <blmgrdn@amazon.com>	3 years ago
Alan Protasio	606ef33d91	Track and report Samples Queried per query We always track total samples queried and add those to the standard set of stats queries can report. We also allow optionally tracking per-step samples queried. This must be enabled both at the engine and query level to be tracked and rendered. The engine flag is exposed via a Prometheus feature flag, while the query flag is set when stats=all. Co-authored-by: Alan Protasio <approtas@amazon.com> Co-authored-by: Andrew Bloomgarden <blmgrdn@amazon.com> Co-authored-by: Harkishen Singh <harkishensingh@hotmail.com> Signed-off-by: Andrew Bloomgarden <blmgrdn@amazon.com>	3 years ago
beorn7	9fbcf14e5c	histogram: Handle changes of the ZeroThreshold and the Schema Signed-off-by: beorn7 <beorn@grafana.com>	3 years ago
Łukasz Mierzwa	a4317bf0ec	Run gofumpt on all files (#10392 ) * Run gofumpt on all files Getting golangci-lint errors when building on my laptop, possibly because I have newer version of gofumpt then what it was formatted with. Run gofumpt -w -extra on all files as it will be needed in the future anyway. * Update golangci-lint to v1.44.2 v1.44.0 upgraded gofumpt so bumping version in CI will help keep formatting correct for everyone * Address golangci-lint error Getting 'error-strings: error strings should not be capitalized or end with punctuation or a newline' from revive here. Drop new line. Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>	3 years ago
Tobias Klausmann	b998636893	Improve error logging for missing config and QL dir (#10260 ) * Improve error logging for missing config and QL dir Currently, when Prometheus can't open its config file or the query logging dir under the data dir, it only logs what it has been given default or commandline/config. Depending on the environment this can be less than helpful, since the working directory may be unclear to the user. I have specifically kept the existing error messages as intact as possible to a) still log the parameter as given and b) cause as little disruption for log-parsers/-analyzers as possible. So in case of the config file or the data dir being non-absolute paths, I use os.GetWd to find the working dir and assemble an absolute path for error logging purposes. If GetWd fails, we just log "unknown", as recovering from an error there would be very complex measure, likely not worth the code/effort. Example errors: ``` $ ./prometheus ts=2022-02-06T16:00:53.034Z caller=main.go:445 level=error msg="Error loading config (--config.file=prometheus.yml)" fullpath=/home/klausman/src/prometheus/prometheus.yml err="open prometheus.yml: no such file or directory" $ touch prometheus.yml $ ./prometheus [...] ts=2022-02-06T16:01:00.992Z caller=query_logger.go:99 level=error component=activeQueryTracker msg="Error opening query log file" file=data/queries.active fullpath=/home/klausman/src/prometheus/data/queries.active err="open data/queries.active: permission denied" panic: Unable to create mmap-ed active query log [...] $ ``` Signed-off-by: Tobias Klausmann <klausman@schwarzvogel.de> * Replace our own logic with just using filepath.Abs() Signed-off-by: Tobias Klausmann <klausman@schwarzvogel.de> * Further simplification Signed-off-by: Tobias Klausmann <klausman@schwarzvogel.de> * Review edits Signed-off-by: Tobias Klausmann <klausman@schwarzvogel.de> * Review edits Signed-off-by: Tobias Klausmann <klausman@schwarzvogel.de> * Review edits Signed-off-by: Tobias Klausmann <klausman@schwarzvogel.de>	3 years ago
jyz0309	e40deb1086	address comment Signed-off-by: jyz0309 <45495947@qq.com>	3 years ago
jyz0309	02e032884a	add doc Signed-off-by: jyz0309 <45495947@qq.com>	3 years ago
jyz0309	7f32a5d0d6	add NaN case Signed-off-by: jyz0309 <45495947@qq.com>	3 years ago
Julien Pivotto	9a2e93228e	Switch to grafana/regexp everywhere (#10268 ) Let's have a consistent library for regexp. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	3 years ago
Matheus Pimenta	8d8ce641a4	error for invalid media type should not be completely swallowed (#10186 ) * error for invalid media type should not be completely swallowed Signed-off-by: Matheus Pimenta <matheuscscp@gmail.com>	3 years ago

1 2 3 4 5 ...

714 Commits (1eb327b99b8ef13e8cb43f5c8ea8abfe3893e623)