Commit Graph

579 Commits (0a28f1ae9d33b71ed2240d8e000ca5d73f6874fd)

Author SHA1 Message Date
Callum Styan 8fd73b1d28
Add Exemplar Remote Write support (#8296)
* Write exemplars to the WAL and send them over remote write.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Update example for exemplars, print data in a more obvious format.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Add metrics for remote write of exemplars.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix incorrect slices passed to send in remote write.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* We need to unregister the new metrics.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address review comments

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Order of exemplar append vs write exemplar to WAL needs to change.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Several fixes to prevent sending uninitialized or incorrect samples with an exemplar. Fix dropping exemplar for missing series. Add tests for queue_manager sending exemplars

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Store both samples and exemplars in the same timeseries buffer to remove the alloc when building final request, keep sub-slices in separate buffers for re-use

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Condense sample/exemplar delivery tests to parameterized sub-tests

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Rename test methods for clarity now that they also handle exemplars

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Rename counter variable. Fix instances where metrics were not updated correctly

Signed-off-by: Martin Disibio <mdisibio@gmail.com>

* Add exemplars to LoadWAL benchmark

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* last exemplars timestamp metric needs to convert value to seconds with
ms precision

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Process exemplar records in a separate go routine when loading the WAL.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address review comments related to clarifying comments and variable
names. Also refactor sample/exemplar to enqueue prompb types.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Regenerate types proto with comments, update protoc version again.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Put remote write of exemplars behind a feature flag.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address some of Ganesh's review comments.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Move exemplar remote write feature flag to a config file field.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address Bartek's review comments.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Don't allocate exemplar buffers in queue_manager if we're not going to
send exemplars over remote write.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Add ValidateExemplar function, validate exemplars when appending to head
and log them all to WAL before adding them to exemplar storage.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address more reivew comments from Ganesh.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Add exemplar total label length check.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Address a few last review comments

Signed-off-by: Callum Styan <callumstyan@gmail.com>

Co-authored-by: Martin Disibio <mdisibio@gmail.com>
2021-05-06 13:53:52 -07:00
Julien Pivotto e36e5fa833
Merge pull request #8731 from yeya24/update
Improve grouping label match logic
2021-04-27 00:55:22 +02:00
Fiona Liao 9b83d8330a
Fix memSafeIterator.Seek() (#8748)
* Add range query test cases

This includes a couple of failing ones that double count some points due
to the iterator seek bug.

Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com>
Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>

* Add Seek() implementation for memSafeIterator

Previously, calling memSafeIterator.Seek() would call the Seek() method
on its embedded iterator. This was causing the embedded iterator and the
memSafeIterator to get out of sync because when the embedded Seek()
moved to the next element of the embedded iterator, memSafeIterator
didn't "know" about it. memSafeIterator has to "know" when the embedded
iterator has moved to be able to work out when it should be reading from
its buffer rather than the embedded iterator.

Used same logic as for xorIterator.Seek() (which in runtime is used as
the embedded iterator) - return false if the iterator has an error and
try to move to next element if the required time hasn't been reached, or
if no elements have been read yet. The memSafeIterator.Next() method is
being called so memSafeIterator.i is always accurate.

Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>

* Add tsdb package test

Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>

Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com>
2021-04-27 00:43:22 +02:00
yeya24 d698e062dc improve grouping label match logic
Signed-off-by: yeya24 <yb532204897@gmail.com>
2021-04-16 22:04:58 -04:00
Marco Pracucci 6719071a0f
Optimize aggregations in PromQL engine (#8594)
* Optimize aggregations in PromQL engine

Signed-off-by: Marco Pracucci <marco@pracucci.com>
2021-03-19 17:52:29 +01:00
Callum Styan 289ba11b79
Add circular in-memory exemplars storage (#6635)
* Add circular in-memory exemplars storage

Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Signed-off-by: Martin Disibio <mdisibio@gmail.com>

Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Co-authored-by: Tom Wilkie <tom.wilkie@gmail.com>
Co-authored-by: Martin Disibio <mdisibio@gmail.com>

* Fix some comments, clean up exemplar metrics struct and exemplar tests.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix exemplar query api null vs empty array issue.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

Co-authored-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Co-authored-by: Tom Wilkie <tom.wilkie@gmail.com>
Co-authored-by: Martin Disibio <mdisibio@gmail.com>
2021-03-16 15:17:45 +05:30
Marco Pracucci 7bbab380b6
Added tracing span to evaluator.eval()
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2021-03-15 15:05:47 +01:00
Marco Pracucci b92c03023d
Optimized vector selector
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2021-03-11 14:32:56 +01:00
Julien Pivotto ad5ed416ba
Merge pull request #8487 from pschou/dev_neg_offset
allow negative offset
2021-03-08 22:18:45 +01:00
Julien Pivotto 5742a18590 Fix subqueries with default resolution in promql unit tests
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-03-07 09:20:04 +01:00
schou 3bafe17870 add negative offset to printer
Signed-off-by: schou <pschou@users.noreply.github.com>
2021-02-23 21:16:28 -05:00
pschou f80b52be69
Merge branch 'main' into dev_neg_offset 2021-02-23 20:52:57 -05:00
schou efbf3af05d reduce addOffst to 2 parameters
Signed-off-by: schou <pschou@users.noreply.github.com>
2021-02-23 20:46:52 -05:00
schou 75d932a172 var init for bool
Signed-off-by: schou <pschou@users.noreply.github.com>
2021-02-23 20:26:35 -05:00
schou 22bfc11738 aggregate booleans for ease of reading
Signed-off-by: schou <pschou@users.noreply.github.com>
2021-02-23 20:26:35 -05:00
schou 22cd48868a adding feature flag, promql-negative-offset
Signed-off-by: schou <pschou@users.noreply.github.com>
2021-02-23 20:25:56 -05:00
schou 851acf86ee rebuild generated_parser.y.go
Signed-off-by: schou <pschou@users.noreply.github.com>
2021-02-23 20:25:56 -05:00
schou 0c577c547b add negative offset ability
Signed-off-by: schou <pschou@users.noreply.github.com>
2021-02-23 20:25:56 -05:00
songjiayang 776c2b8f42 Speed delta value without loop to calculate resultValue
Signed-off-by: songjiayang <songjiayang1@gmail.com>
2021-02-23 22:28:04 +08:00
pschou aff3c702ab
promql: Add sgn, clamp and last_over_time functions (#8457)
* Add sgn, clamp and last_over_time functions

Signed-off-by: schou <pschou@users.noreply.github.com>
2021-02-20 16:34:52 +01:00
Danny Kopping 42a0e0acad
Prevent lexer from seeking to next rune after lexing escape sequence. (#8517)
* Prevent lexer from seeking to next rune after lexing escape sequence.

Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
2021-02-19 07:38:05 +01:00
Tom Wilkie 7369561305
Combine Appender.Add and AddFast into a single Append method. (#8489)
This moves the label lookup into TSDB, whilst still keeping the cached-ref optimisation for repeated Appends.

This makes the API easier to consume and implement.  In particular this change is motivated by the scrape-time-aggregation work, which I don't think is possible to implement without it as it needs access to label values.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
2021-02-18 17:37:00 +05:30
Mauro Stettler 7715fe3219
Add matchers to LabelValues() call (#8400)
* Accept matchers in querier LabelValues()

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* create matcher to only select metrics which have searched label

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* test case for merge querier with matchers

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* test LabelValues with matchers on head

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* add test for LabelValues on block

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* formatting fix

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Add comments

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* add missing lock release

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* remove unused parameter

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Benchmarks for LabelValues() methods on block/head

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Better comment

Co-authored-by: Julien Pivotto <roidelapluie@gmail.com>
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* update comment

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* minor refactor make code cleaner

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* better comments

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* fix expected errors in test

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Deleting parameter which can only be empty

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* fix comments

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* remove unnecessary lock

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* only lookup label value if label name was looked up

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Return error when there is one

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Call .Get() on decoder before checking errors

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* only lock head.symMtx when necessary

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* remove unnecessary delete()

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* re-use code instead of duplicating it

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Consistently return error from LabelValueFor()

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* move helper func from util.go to querier.go

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Fix test expectation

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>

* ensure result de-duplication and sorting works

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* return named error from LabelValueFor()

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

Co-authored-by: Julien Pivotto <roidelapluie@gmail.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2021-02-09 23:08:35 +05:30
Ganesh Vernekar 86c71856e8
Add start() and end() pre-processors for @ modifier (#8425)
* Add start() and end() pre-processors for @ modifier

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix reviews

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>

* Fix review comments

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2021-02-09 21:33:16 +05:30
Marcelo E. Magallon 75d86c6747 Update golangci-lint to 1.36.0
In the previous version, 1.18.0, the "megacheck" linter paid attention
to the '//lint:ignore' comment, but that is no longer there.

Newer version pay attention to '//nolint:<linter>,<linter>,...'
comments, optionally followed by a "second" comment introduced by '//'.

Update the directives to use this style.

This is related to prometheus/blackbox_exporter#738 and
prometheus/blackbox_exporter#745.

Signed-off-by: Marcelo E. Magallon <marcelo.magallon@grafana.com>
2021-02-04 08:53:33 -06:00
Ganesh Vernekar ed8d31b059
Remove empty test
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2021-02-04 11:53:32 +05:30
Ganesh Vernekar b18fde996e
Fix timestamp() function for @ modifier
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2021-02-03 19:13:12 +05:30
Ganesh Vernekar b5dfa2414b
Fix flaky TestAtModifier (#8401)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2021-01-24 20:23:30 +05:30
Ganesh Vernekar 9199fcb8d1
'@ <timestamp>' modifier (#8121)
This commit adds `@ <timestamp>` modifier as per this design doc: https://docs.google.com/document/d/1uSbD3T2beM-iX4-Hp7V074bzBRiRNlqUdcWP6JTDQSs/edit.

An example query:

```
rate(process_cpu_seconds_total[1m]) 
  and
topk(7, rate(process_cpu_seconds_total[1h] @ 1234))
```

which ranks based on last 1h rate and w.r.t. unix timestamp 1234 but actually plots the 1m rate.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2021-01-20 16:27:39 +05:30
Julien Pivotto 34656bbf8e
PromQL parser: fastpath for non-empty matchers and metric name (#8345)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-01-07 11:38:38 +01:00
Ganesh Vernekar 063154eab7
Automate part of TestMaxQuerySamples (#8314)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2021-01-04 14:33:36 +05:30
Atibhi Agrawal 0dd6e0cb76
change string(model.metricnamelabel) to model.metricnamelabel (#8289)
Signed-off-by: aSquare14 <atibhi.a@gmail.com>
2020-12-15 20:17:31 +05:30
Brian Brazil bef9d4e182
Don't include rendered expression in parse errors. (#8177)
For a sufficiently complex expression, having to render
thousands of nodes for every subquery type error can take
a while - so don't do that.

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-11-12 15:25:52 +01:00
Ganesh Vernekar d30da66d77
Fix timestamp() method for vector selector inside paren (#8164)
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2020-11-09 18:21:50 +05:30
Julien Pivotto 6c56a1faaa
Testify: move to require (#8122)
* Testify: move to require

Moving testify to require to fail tests early in case of errors.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* More moves

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-10-29 09:43:23 +00:00
Julien Pivotto 1282d1b39c
Refactor test assertions (#8110)
* Refactor test assertions

This pull request gets rid of assert.True where possible to use
fine-grained assertions.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-10-27 11:06:53 +01:00
Julien Pivotto 4e5b1722b3
Move away from testutil, refactor imports (#8087)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-10-22 11:00:08 +02:00
Harold Dost bf7c771f54
promql: Organize legacy tests into appropriate files (#7989)
* Separate tests into:
** Aggregators
** Functions
** Operators
** Selectors
* Remove simple files and place tests into other files.
* Eliminate some of the _over_time tests since there are already plenty of
  edge cases present in the functions.test file.

Relates to #740

Signed-off-by: Harold Dost <harolddost@gmail.com>
2020-10-07 10:09:20 +01:00
Brian Brazil 924ce978b7
Don't do literal regex matching optimisation when case insensitive. (#8013)
Fixes #7994

Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
2020-10-06 13:16:26 +01:00
Harkishen Singh fc8e769d71
Use ASSIGN when using = inside braces (#7911)
* Fix EQL when using = inside braces.

Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>

* EQL => EQLC and ASSIGN => EQL

Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>

* Aligned yacc code.

Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>
2020-09-09 15:40:02 +05:30
Vijay Samuel 00ee73ef91
Export members of EvalNodeHelper to facilitate usage in external functions (#7860)
Signed-off-by: Vijay Samuel <vjsamuel@ebay.com>
2020-08-27 19:30:10 +01:00
johncming 267dd17fe7
promql/parser: use pos instead of origin. (#7853)
Signed-off-by: johncming <johncming@yahoo.com>
2020-08-26 15:26:26 +05:30
Julien Pivotto 6f9e7ff750
Drop metric name in bool comparison between two instant vectors (#7819)
* Drop metric name in bool comparison between two instant vectors

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-08-22 21:04:03 +02:00
Julien Pivotto 20ab94fedf
Hints: Separating out the range and offsets of PromQL subqueries (#7667)
Fix #7629

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-08-11 07:21:39 +01:00
Annanay Agarwal 118aeab02c
Make context key type public (#7748)
Signed-off-by: Annanay <annanayagarwal@gmail.com>
2020-08-05 09:51:36 +01:00
Julien Pivotto d867491364
Human-friendly durations in PromQL (#7713)
* Add support for user-friendly durations

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-08-04 21:12:41 +02:00
johncming 31929b83d5
promql: use explicit type declare instead of string. (#7716)
Signed-off-by: johncming <johncming@yahoo.com>
2020-08-02 09:57:38 +01:00
johncming 1c1b394e5e
promql: Swap order of parseBrokenJSON. (#7718)
Signed-off-by: johncming <johncming@yahoo.com>
2020-08-02 09:48:57 +01:00
Bartlomiej Plotka e6d7cc5fa4
tsdb: Added ChunkQueryable implementations to db; unified MergeSeriesSets and vertical to single struct. (#7069)
* tsdb: Added ChunkQueryable implementations to db; unified compactor, querier and fanout block iterating.

Chained to https://github.com/prometheus/prometheus/pull/7059

* NewMerge(Chunk)Querier now takies multiple primaries allowing tsdb DB code to use it.
* Added single SeriesEntry / ChunkEntry for all series implementations.
* Unified all vertical, and non vertical for compact and querying to single
merge series / chunk sets by reusing VerticalSeriesMergeFunc for overlapping algorithm (same logic as before)
* Added block (Base/Chunk/)Querier for block querying. We then use populateAndTomb(Base/Chunk/) to iterate over chunks or samples.
* Refactored endpoint tests and querier tests to include subtests.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed comments from Brian and Beorn.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed snapshot test and added chunk iterator support for DBReadOnly.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed race when iterating over Ats first.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed tests.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed populate block tests.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed endpoints test.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed test.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Added test & fixed case of head open chunk.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed DBReadOnly tests and bug producing 1 sample chunks.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Added cases for partial block overlap for multiple full chunks.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Added extra tests for chunk meta after compaction.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Fixed small vertical merge bug and added more tests for that.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-07-31 16:03:02 +01:00
Annanay 9bba8a6eae Merge branch 'master' into appender-context
Signed-off-by: Annanay <annanayagarwal@gmail.com>
2020-07-30 16:43:18 +05:30