Commit Graph

439 Commits (097fda1440385279029cd5fe23fdfca3311726c2)

Author SHA1 Message Date
Tobias Guggenmos 4aef43a9f5 Cleanup: Remove parser switching logic (#6583)
During the PromQL parser rewrite there was some logic put in place that allowed switching between the non generated and the generated parser. Since the parser is now fully generated this is not needed anymore.

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-01-08 14:59:25 +00:00
Tobias Guggenmos 2064abab40 Cleanup: Remove unused variable (#6581)
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-01-08 14:58:52 +00:00
Tobias Guggenmos 7c7746257c Cleanup: remove function that does not do anything (#6580)
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-01-08 14:58:21 +00:00
Tobias Guggenmos d8bc24349c PromQL: Fix unary operator precedence (#6579)
* PromQL: Fix unary operator precedence

Fixes #6575

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-01-08 14:07:10 +00:00
Julien Pivotto 9d9bc524e5 Add query log (#6520)
* Add query log, make stats logged in JSON like in the API

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-01-08 13:28:43 +00:00
Tobias Guggenmos 3d6cf1c289 PromQL: Make parser completely generated (#6548)
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2020-01-08 11:04:47 +00:00
Julien Pivotto 577e738986 Cleanup PromQL functions (#6551)
* Cleanup PromQL functions

The engine ensures, for Matrix functions, that functions are called with exactly one series at the time.
Therefore a lot of code can be inlined and we can directly assume the first element of the arguments exists and contains all the samples needed.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-01-06 10:33:36 +00:00
Julien Pivotto e0afec906f add absent_over_time (#6490)
* Implement absent_over_time

Fixes #2882

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-01-03 15:26:12 +00:00
Josh Soref 91d76c8023 Spelling (#6517)
* spelling: alertmanager

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: attributes

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: autocomplete

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: bootstrap

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: caught

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: chunkenc

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: compaction

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: corrupted

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: deletable

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: expected

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: fine-grained

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: initialized

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: iteration

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: javascript

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: multiple

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: number

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: overlapping

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: possible

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: postings

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: procedure

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: programmatic

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: queuing

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: querier

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: repairing

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: received

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: reproducible

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: retention

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: sample

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: segements

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: semantic

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: software [LICENSE]

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: staging

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: timestamp

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: unfortunately

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: uvarint

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: subsequently

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>

* spelling: ressamples

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
2020-01-02 15:54:09 +01:00
Tobias Guggenmos 213a8fe89a PromQL: Parse Series descriptions using the generated parser (#6494)
* Use generated parser for series descriptions

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-21 08:44:12 +00:00
Julien Pivotto f363a71ef7 promql: function: fix typo (#6501)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2019-12-20 22:30:55 +00:00
Julien Pivotto 4b3bab890d Fix absent output where two label matchers are identical (#6493)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2019-12-20 14:33:33 +00:00
Tobias Guggenmos a48acbcb23 PromQL: Fix parser benchmark (#6495)
The parser benchmarks called the `ParseMetric` function instead of the `ParseExpr` function, which resulted in parsing failing every time.

This means only the case of PromQL parser failure was benchmarked.

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-20 11:30:41 +00:00
Tobias Guggenmos 1e0cd28bf6 PromQL: Support trailing commas in grouping opts (#6480)
* PromQL: Support trailing commas in grouping opts

Fixes #6470

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-20 11:28:56 +00:00
Julien Pivotto f315c69129 promql: fix typo in value.go (#6491)
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2019-12-19 22:30:32 +00:00
Tobias Guggenmos db1258f2a5 PromQL: Refactor error message generation (#6481)
* Add parser method to produce errors messages about unexpected items
* PromQL: use parser.unexpected in generated parser

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-18 17:36:43 +00:00
Tobias Guggenmos 9e34f08ac3 PromQL: Parse grouping opts with the generated parser (#6472)
* PromQL: Parse grouping opts with the generated parser

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-18 14:18:52 +00:00
Tobias Guggenmos 53615412b4 PromQL: Parse Metrics using generated parser (#6466)
* Parse Metrics with the generated parser

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-16 16:38:18 +00:00
Tobias Guggenmos b38b25e9e1 Add Benchmark for the PromQL Parser (#6355)
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-16 14:04:56 +00:00
Tobias Guggenmos 8cb4a48e2e PromQL: Parse label sets using the generated parser (#6432)
* Add grammar for label_sets
* Parse label Sets using the generated parser
* Allow trailing commas for label sets and selectors
* Add test to trigger all possible error messages for label matchers

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-16 13:58:46 +00:00
Tobias Guggenmos 5c503d85f7 PromQL: export lexer (#6435)
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-09 19:03:31 +00:00
Tobias Guggenmos 35c1f31721 PromQL: Use more standart format for error positions (#6433)
The most common format (used by go, gcc and clang) for compiler error positions seems to be

`filename:line:char:` or `line:char:` if the filename is unknown.

This PR adapts the PromQL parser to use this convention.

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-09 16:39:03 +00:00
Tobias Guggenmos cd39ebe7c6 PromQL: export function list (#6434)
This PR exports the list of supported PromQL functions and their signatures.

The reason for that is that the PromQL language server likes to use that list.

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-09 15:33:43 +00:00
Mark Nevill b0a5c51b95 Return unused point slice to pool in Call and VectorSelector eval. (#6427)
Signed-off-by: Mark Nevill <mark.nevill@gmail.com>
2019-12-09 10:32:40 +00:00
Tobias Guggenmos 3bb715031f PromQL: Use generated parser to parse label matchers (#6410)
Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-12-05 16:16:12 +00:00
Garrett 5a9c4acfbf Pushdown aggregator group by through read hint (#6401)
* Pushdown aggregator group by through read hint

Implement https://github.com/prometheus/prometheus/issues/6400

* add temporal aggregation pushdown support

Signed-off-by: xiancli <xiancli@ebay.com>
2019-12-05 14:06:28 +00:00
Tobias Guggenmos 408574a6e1 promql: Allow injecting fake tokens into the generated parser (#6381)
* promql: Allow injecting fake tokens into the generated parser

Yacc grammars do not support having multiple start symbols.

To work around that restriction, it is possible to inject fake tokens into the lexer stream,
as described here https://www.gnu.org/software/bison/manual/html_node/Multiple-start_002dsymbols.html .

This is part of the parser rewrite effort described in #6256.

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-11-27 12:59:03 +00:00
Tobias Guggenmos bbd92b85da promql: Use capitalized names for item types (#6371)
For yacc generated parsers there is the convention to capitalize the names of item types provided by the lexer, which makes it easy to distinct lexer tokens (capitalized) from nonterminal symbols (not capitalized) in language grammars.

This convention is also followed by the (non generated) go compiler (see https://golang.org/pkg/go/token/#Token).

Part of the parser rewrite described in #6256.

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-11-26 13:29:42 +00:00
Tobias Guggenmos c229ed17e2 promql: Implement yyLexer interface (#6370)
This is the first step towards a generated lexer as described in #6256.

It adds methods to the parser struct, that make it implement the yyLexer interface required by a yacc generated parser, as described here: https://godoc.org/golang.org/x/tools/cmd/goyacc .

The yyLexer interface is implemented by the parser struct instead of the lexer struct for the following reasons:

* Both parsers have a lookahead that the lexer does not know about. This solution makes it possible to synchronize these lookaheads when switching parsers.
* The routines to handle parser errors are not accessible to the lexer.

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-11-26 13:28:36 +00:00
Bartosz Jakubski 80ba03c67d fix subquery with offset conversion to string (#6373)
Signed-off-by: Bartosz Jakubski <b.jakubski@gmail.com>
2019-11-26 06:45:51 +00:00
Tobias Guggenmos c63259b83c promql: Clean up parser struct (#6360)
* promql: Clean up parser struct

The parser struct used two have two somewhat misused fields:

peekCount int
token     [3]item

By reading the code carefully one notices, that peekCount always has the value 0 or 1 and that only the first element of token is ever accessed.

To make this clearer, this commit replaces the token array with a single variable and the peekCount int with a boolean.

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-11-25 12:29:14 +00:00
Tobias Guggenmos d0cff29749 PromQL: Reduce Code duplication in AST traversion (#6362)
* promql: Add Children function for easier AST traversion

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-11-25 11:41:59 +00:00
Tobias Guggenmos ac3932ea35 Improve PromQL parser performance by making it non-concurrent (#6356)
Before this commit, the PromQL parser ran in two goroutines:
* The lexer goroutine that splits the input into tokens and sent them over a channel to
* the parser goroutine which produces the abstract syntax tree

The Problem with this approach is that the parser spends more time on goroutine creation
and syncronisation than on actual parsing.

This commit removes that concurrency and replaces the channel by a slice based buffer.

Benchmarks show that this makes the up to 7 times faster than before.

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-11-21 18:43:09 +00:00
Harkishen Singh 37d666949c fixes space issue in duration range promql (#6295)
* fix space issue in duration range promql

Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>

* updated logic

Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>

* fixed lexer to skip over the spaces

Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>

* added unittests for updated lexer

Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>

* added unittests for updated lexer

Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>
2019-11-11 09:56:24 +01:00
Tobias Guggenmos fe80cf4734 promql: Eliminate dead code (#6215)
peek() already ensures to not return a ItemComment so checking for this
is redundant.

Signed-off-by: Tobias Guggenmos <tguggenm@redhat.com>
2019-10-25 10:18:20 +02:00
Krasi Georgiev 5b8ff9b8a4
Merge the 2.13 release branch (#6169)
* Include tsdb tool in builds (#6085) (#6089)

Add the tsdb tool to promu so that it's included in the release
tarballs.

Signed-off-by: Ben Kochie <superq@gmail.com>

* web/ui: fix for CVE-2019-10215 (#6098)

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

* cut 2.13 release (#6099)

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>

* Fix panic in ARM builds of Prometheus (#6110)

An extra sync.Pool was added during a refactor which caused some 64 bit,
atomically accessed variables to no longer be 64 bit aligned. By moving
all atomically accessed variables to the beginning of the struct they
are guaranteed to be 64 bit aligned.

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>

* promql: fix potential panic in the query logger (#6094)

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

* Cut release 2.13.1 (#6145)

Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>
2019-10-18 02:21:58 +03:00
Alex Dzyoba e3882629ba promql: Move tests to testutil (#6103)
* promql: Move tests to testutil

Signed-off-by: Alex Dzyoba <alex@dzyoba.com>

* promql: Match error type via errors.As in tests

Signed-off-by: Alex Dzyoba <alex@dzyoba.com>

* promql: Remove unused `expectedList` func from lex_test.go

Signed-off-by: Alex Dzyoba <alex@dzyoba.com>
2019-10-09 18:06:53 -06:00
AllenZMC 1e62435960 fix wrong spells in live_reader.go (#5899)
* fix wrong spells in live_reader.go
* fix wrong spells in lex.go

Signed-off-by: czm <zhongming.chang@daocloud.io>
2019-09-21 16:36:33 +03:00
yuxiaobo96 5f6a56a0bb Third correction (#6012)
Signed-off-by: yuxiaobo <yuxiaobogo@163.com>
2019-09-20 10:48:56 +01:00
johncming ac3cedb108 promql: Modify the order of return values. (#5940)
Signed-off-by: johncming <johncming@yahoo.com>
2019-08-24 02:39:15 +01:00
johncming d1b3a6e013 promql: Optimize the writing (#5941)
Signed-off-by: johncming <johncming@yahoo.com>
2019-08-24 02:38:48 +01:00
Krzysztof Kowalczyk 6e53980bbf fix fuzz targets (#5851)
Signed-off-by: Krzysztof Kowalczyk <kkowalczyk@gmail.com>
2019-08-12 20:47:57 +02:00
AllenZMC 3638e4ab18 fix mis-spelling in fuzz.go (#5848)
Signed-off-by: czm <zhongming.chang@daocloud.io>
2019-08-09 15:19:45 +01:00
Chris Marchbanks 529ccff07b
Remove all usages of stretchr/testify
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2019-08-08 19:49:27 -06:00
Chris Marchbanks 0685eb5395
Refactor testutil.NewStorage into a new package
This avoids a circular dependency between the testutil and storage
packages.

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2019-08-08 19:43:04 -06:00
AllenZMC 04cb37fea8 fix wrong spells in query_logger.go (#5843)
Signed-off-by: czm <zhongming.chang@daocloud.io>
2019-08-07 15:02:53 +01:00
ethan 8928094b56 func name ref correct "qauntile" -> "quantile" (#5834)
Signed-off-by: ethan <guangming.wang@daocloud.io>
2019-08-06 06:11:16 +01:00
AllenZMC 8d6ec5acdc fix word 'seriess' to 'series' (#5828)
Signed-off-by: czm <zhongming.chang@daocloud.io>
2019-08-04 16:52:51 +01:00
AllenZMC ead0933dd9 fix word 'substracting' to 'subtracting' (#5822)
Signed-off-by: czm <zhongming.chang@daocloud.io>
2019-08-01 15:44:38 +01:00
Advait Bhatwadekar 5d401f1e1b Added query logging for prometheus. Issue #1315 (#5794)
* Added query logging for prometheus.
Options added:
1) active.queries.filepath: Filename where queries will be recorded
2) active.queries.filesize: Size of the file where queries will be recorded.

Functionality added:
All active queries are now logged in a file. If prometheus crashes unexpectedly, these queries are also printed out on stdout in the rerun.

Queries are written concurrently to an mmaped file, and removed once they are done. Their positions in the file are reused. They are written in json format. However, due to dynamic nature of application, the json has an extra comma after the last query, and is missing an ending ']'. There may also null bytes in the tail of file.

Signed-off-by: Advait Bhatwadekar <advait123@ymail.com>
2019-07-31 16:12:43 +01:00