Commit Graph

29 Commits (452d8583157c4037ebacdb25c4885b50416f319c)

Author SHA1 Message Date
Bryan Boreham 80ac0d7c82 promql: add benchmark for match against blank string
Blank strings are not handled efficiently by tsdb.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-01-05 14:05:54 +00:00
Bryan Boreham dbd7021cc2
promql: add test for race conditions in query engine (#11743)
* promql: refactor BenchmarkRangeQuery so we can re-use test cases

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* promql: add test for race conditions in query engine

Note we skip large count_values queries -
`count_values` allocates a slice per unique value in the output, and
this test has unique values on every step of every series so it adds up
to a lot of slices. Add Go runtime overhead for checking `-race`, and
it chews up many gigabytes.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* TestConcurrentRangeQueries: wait before starting goroutine

Instead of starting 100 goroutines which just wait for the semaphore.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2022-12-20 17:58:29 +01:00
Alan Protasio 606ef33d91 Track and report Samples Queried per query
We always track total samples queried and add those to the standard set
of stats queries can report.

We also allow optionally tracking per-step samples queried. This must be
enabled both at the engine and query level to be tracked and rendered.
The engine flag is exposed via a Prometheus feature flag, while the
query flag is set when stats=all.

Co-authored-by: Alan Protasio <approtas@amazon.com>
Co-authored-by: Andrew Bloomgarden <blmgrdn@amazon.com>
Co-authored-by: Harkishen Singh <harkishensingh@hotmail.com>
Signed-off-by: Andrew Bloomgarden <blmgrdn@amazon.com>
2022-03-21 23:49:17 +01:00
beorn7 c954cd9d1d Move packages out of deprecated pkg directory
This creates a new `model` directory and moves all data-model related
packages over there:
  exemplar labels relabel rulefmt textparse timestamp value

All the others are more or less utilities and have been moved to `util`:
  gate logging modetimevfs pool runtime

Signed-off-by: beorn7 <beorn@grafana.com>
2021-11-09 08:03:10 +01:00
Dieter Plaetinck cda025b5b5
TSDB: demistify SeriesRefs and ChunkRefs (#9536)
* TSDB: demistify seriesRefs and ChunkRefs

The TSDB package contains many types of series and chunk references,
all shrouded in uint types.  Often the same uint value may
actually mean one of different types, in non-obvious ways.

This PR aims to clarify the code and help navigating to relevant docs,
usage, etc much quicker.

Concretely:

* Use appropriately named types and document their semantics and
  relations.
* Make multiplexing and demuxing of types explicit
  (on the boundaries between concrete implementations and generic
  interfaces).
* Casting between different types should be free.  None of the changes
  should have any impact on how the code runs.

TODO: Implement BlockSeriesRef where appropriate (for a future PR)

Signed-off-by: Dieter Plaetinck <dieter@grafana.com>

* feedback

Signed-off-by: Dieter Plaetinck <dieter@grafana.com>

* agent: demistify seriesRefs and ChunkRefs

Signed-off-by: Dieter Plaetinck <dieter@grafana.com>
2021-11-06 15:40:04 +05:30
Bryan Boreham 5a754bc043
Short-circuit vector binary ops (#9362)
In degenerate cases we can save the effort of building a map.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2021-09-21 17:37:36 +05:30
Bryan Boreham c4942ef3b7
Optimise query_range by computing join signatures just once (#9360)
* Add benchmark case for many-to-one join

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* query_range: compute join signatures just once

For an expression like `a + on(p,q) b`, extract the `p,q` part from each
series once, instead of re-computing at every step of the range.

Although there was a cache, computing the key by concatenating all
labels was expensive.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2021-09-21 15:58:39 +05:30
Bryan Boreham 7d105277fe
Optimise topk where k==1 (#9365)
* Add benchmark for query_range with topk

Modify sample data so values within a metric differ

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Optimise topk where k==1

In this case we don't need a heap to keep track of values; just a single
slot is fine.

Simplify the initialization of the heap: since all cases start off as a
single-item heap we can just assign the value directly.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Allow at least one slot in results for topk, quantile

k isn't set for quantile, but we need space to start collecting values

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2021-09-21 15:57:28 +05:30
Marco Pracucci 6719071a0f
Optimize aggregations in PromQL engine (#8594)
* Optimize aggregations in PromQL engine

Signed-off-by: Marco Pracucci <marco@pracucci.com>
2021-03-19 17:52:29 +01:00
Tom Wilkie 7369561305
Combine Appender.Add and AddFast into a single Append method. (#8489)
This moves the label lookup into TSDB, whilst still keeping the cached-ref optimisation for repeated Appends.

This makes the API easier to consume and implement.  In particular this change is motivated by the scrape-time-aggregation work, which I don't think is possible to implement without it as it needs access to label values.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
2021-02-18 17:37:00 +05:30
Annanay 7f98a744e5 Add context to Appender interface
Signed-off-by: Annanay <annanayagarwal@gmail.com>
2020-07-24 19:40:51 +05:30
Tobias Guggenmos 4835bbf376
Merge branch 'master' into split_parser 2020-02-19 15:18:13 +01:00
Bartlomiej Plotka 2cf637fbf5 Addressed comments.
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-02-17 18:03:57 +00:00
Tobias Guggenmos 273ea9093a Fix promql tests
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-17 16:09:23 +01:00
Tobias Guggenmos ff0ea1c1ac Fix more identifiers
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-02-17 16:07:53 +01:00
Julien Pivotto 9adad8ad30 Remove MaxConcurrent from the PromQL engine opts (#6712)
Since we use ActiveQueryTracker to check for concurrency in
d992c36b3a it does not make sense to keep
the MaxConcurrent value as an option of the PromQL engine.

This pull request removes it from the PromQL engine options, sets the
max concurrent metric to -1 if there is no active query tracker, and use
the value of the active query tracker otherwise.

It removes dead code and also will inform people who import the promql
package that we made that change, as it breaks the EngineOpts struct.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-01-28 20:38:49 +00:00
Julien Pivotto e0afec906f add absent_over_time (#6490)
* Implement absent_over_time

Fixes #2882

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-01-03 15:26:12 +00:00
Josh Soref 91d76c8023 Spelling (#6517)
* spelling: alertmanager

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: attributes

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: autocomplete

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: bootstrap

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: caught

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: chunkenc

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: compaction

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: corrupted

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: deletable

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: expected

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: fine-grained

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: initialized

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: iteration

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: javascript

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: multiple

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: number

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: overlapping

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: possible

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: postings

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: procedure

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: programmatic

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: queuing

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: querier

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: repairing

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: received

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: reproducible

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: retention

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: sample

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: segements

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: semantic

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: software [LICENSE]

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: staging

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: timestamp

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: unfortunately

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: uvarint

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: subsequently

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: ressamples

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-01-02 15:54:09 +01:00
Tobias Guggenmos a48acbcb23 PromQL: Fix parser benchmark (#6495)
The parser benchmarks called the `ParseMetric` function instead of the `ParseExpr` function, which resulted in parsing failing every time.

This means only the case of PromQL parser failure was benchmarked.

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-20 11:30:41 +00:00
Tobias Guggenmos b38b25e9e1 Add Benchmark for the PromQL Parser (#6355)
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-16 14:04:56 +00:00
Chris Marchbanks 0685eb5395
Refactor testutil.NewStorage into a new package
This avoids a circular dependency between the testutil and storage
packages.

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2019-08-08 19:43:04 -06:00
Chris Marchbanks ab3cdb9500 Allow promql benchmarks to query more samples (#4770)
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2018-10-22 18:02:22 +01:00
Callum Styan 9bca041285 WIP: keep track of samples per query, set a max # of samples (#4513)
* keep track of samples per query, set a max # of samples that can be in
memory at once

Signed-off-by: Callum Styan <callumstyan@gmail.com>
2018-10-02 12:59:19 +01:00
Alin Sinpalean 372e7652b7 Reuse (copy) overlapping matrix samples between range evaluation steps (#4315)
* Reuse (copy) overlapping matrix samples between range evaluation steps.

Signed-off-by: Alin Sinpalean <alin.sinpalean@gmail.com>
2018-07-18 11:14:02 +01:00
Julius Volz 219e477272 Fix some (valid) lint errors (#4287)
Signed-off-by: Julius Volz <julius.volz@gmail.com>
2018-07-18 05:07:33 +01:00
Alin Sinpalean 96fb0b2155 Optimize PromQL aggregations (#4248)
* Compute hash of label subsets without creating a LabelSet first.

Signed-off-by: Alin Sinpalean <alin.sinpalean@gmail.com>
2018-07-18 04:56:27 +01:00
Brian Brazil dd6781add2 Optimise PromQL (#3966)
* Move range logic to 'eval'

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make aggregegate range aware

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* PromQL is statically typed, so don't eval to find the type.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Extend rangewrapper to multiple exprs

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Start making function evaluation ranged

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make instant queries a special case of range queries

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Eliminate evalString

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Evaluate range vector functions one series at a time

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make unary operators range aware

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make binops range aware

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Pass time to range-aware functions.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make simple _over_time functions range aware

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Reduce allocs when working with matrix selectors

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Add basic benchmark for range evaluation

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Reuse objects for function arguments

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Do dropmetricname and allocating output vector only once.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Add range-aware support for range vector functions with params

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Optimise holt_winters, cut cpu and allocs by ~25%

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make rate&friends range aware

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make more functions range aware. Document calling convention.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make date functions range aware

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make simple math functions range aware

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Convert more functions to be range aware

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make more functions range aware

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Specialcase timestamp() with vector selector arg for range awareness

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Remove transition code for functions

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Remove the rest of the engine transition code

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Remove more obselete code

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Remove the last uses of the eval* functions

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Remove engine finalizers to prevent corruption

The finalizers set by matrixSelector were being called
just before the value they were retruning to the pool
was then being provided to the caller. Thus a concurrent query
could corrupt the data that the user has just been returned.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Add new benchmark suite for range functinos

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Migrate existing benchmarks to new system

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Expand promql benchmarks

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Simply test by removing unused range code

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* When testing instant queries, check range queries too.

To protect against subsequent steps in a range query being
affected by the previous steps, add a test that evaluates
an instant query that we know works again as a range query
with the tiimestamp we care about not being the first step.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Reuse ring for matrix iters. Put query results back in pool.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Reuse buffer when iterating over matrix selectors

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Unary minus should remove metric name

Cut down benchmarks for faster runs.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Reduce repetition in benchmark test cases

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Work series by series when doing normal vectorSelectors

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Optimise benchmark setup, cuts time by 60%

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Have rangeWrapper use an evalNodeHelper to cache across steps

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Use evalNodeHelper with functions

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Cache dropMetricName within a node evaluation.

This saves both the calculations and allocs done by dropMetricName
across steps.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Reuse input vectors in rangewrapper

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Reuse the point slices in the matrixes input/output by rangeWrapper

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make benchmark setup faster using AddFast

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Simplify benchmark code.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Add caching in VectorBinop

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Use xor to have one-level resultMetric hash key

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Add more benchmarks

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Call Query.Close in apiv1

This allows point slices allocated for the response data
to be reused by later queries, saving allocations.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Optimise histogram_quantile

It's now 5-10% faster with 97% less garbage generated for 1k steps

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make the input collection in rangeVector linear rather than quadratic

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Optimise label_replace, for 1k steps 15x fewer allocs and 3x faster

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Optimise label_join, 1.8x faster and 11x less memory for 1k steps

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Expand benchmarks, cleanup comments, simplify numSteps logic.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Address Fabian's comments

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Comments from Alin.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Address jrv's comments

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Remove dead code

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Address Simon's comments.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Rename populateIterators, pre-init some sizes

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Handle case where function has non-matrix args first

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Split rangeWrapper out to rangeEval function, improve comments

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Cleanup and make things more consistent

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Make EvalNodeHelper public

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>

* Fabian's comments.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2018-06-04 15:47:45 +02:00
Brian Brazil bf7d87aed2 Cleanup storage from all tests.
Fixed #3299
2018-03-09 07:53:35 +00:00
Tom Wilkie 835eb8c653 Add _test.go suffix to promql/{bench.go, test.go} to prevent importing the testing package in a normal binary. 2017-07-07 15:52:44 +01:00