prometheus

Commit Graph

Author	SHA1	Message	Date
Tobias Schmidt	29ced0090f	Fix common english misspellings	8 years ago
Tobias Schmidt	04ae6196f2	Fix parsing of label names which are also keywords The current separation between lexer and parser is a bit fuzzy when it comes to operators, aggregators and other keywords. The lexer already tries to determine the type of a token, even though that type might change depending on the context. This led to the problematic behavior that no tokens known to the lexer could be used as label names, including operators (and, by, ...), aggregators (count, quantile, ...) or other keywords (for, offset, ...). This change additionally checks whether an identifier is one of these types. We might want to check whether the specific item identification should be moved from the lexer to the parser.	8 years ago
beorn7	fc6737b7fb	storage: improve index lookups tl;dr: This is not a fundamental solution to the indexing problem (like tindex is) but it at least avoids utilizing the intersection problem to the greatest possible amount. In more detail: Imagine the following query: nicely:aggregating:rule{job="foo",env="prod"} While it uses a nicely aggregating recording rule (which might have a very low cardinality), Prometheus still intersects the low number of fingerprints for `{__name__="nicely:aggregating:rule"}` with the many thousands of fingerprints matching `{job="foo"}` and with the millions of fingerprints matching `{env="prod"}`. This totally innocuous query is dead slow if the Prometheus server has a lot of time series with the `{env="prod"}` label. Ironically, if you make the query more complicated, it becomes blazingly fast: nicely:aggregating:rule{job=~"foo",env=~"prod"} Why so? Because Prometheus only intersects with non-Equal matchers if there are no Equal matchers. That's good in this case because it retrieves the few fingerprints for `{__name__="nicely:aggregating:rule"}` and then starts right ahead to retrieve the metric for those FPs and checking individually if they match the other matchers. This change is generalizing the idea of when to stop intersecting FPs and go into "retrieve metrics and check them individually against remaining matchers" mode: - First, sort all matchers by "expected cardinality". Matchers matching the empty string are always worst (and never used for intersections). Equal matchers are in general consider best, but by using some crude heuristics, we declare some better than others (instance labels or anything that looks like a recording rule). - Then go through the matchers until we hit a threshold of remaining FPs in the intersection. This threshold is higher if we are already in the non-Equal matcher area as intersection is even more expensive here. - Once the threshold has been reached (or we have run out of matchers that do not match the empty string), start with "retrieve metrics and check them individually against remaining matchers". A beefy server at SoundCloud was spending 67% of its CPU time in index lookups (fingerprintsForLabelPairs), serving mostly a dashboard that is exclusively built with recording rules. With this change, it spends only 35% in fingerprintsForLabelPairs. The CPU usage dropped from 26 cores to 18 cores. The median latency for query_range dropped from 14s to 50ms(!). As expected, higher percentile latency didn't improve that much because the new approach is _occasionally_ running into the worst case while the old one was _systematically_ doing so. The 99th percentile latency is now about as high as the median before (14s) while it was almost twice as high before (26s).	8 years ago
Brian Brazil	16690736ab	Add count_values() aggregator. This is useful for counting how many instances of a job are running a particular version/build. Fixes #622	9 years ago
Brian Brazil	7f23a4a099	Add type check on topk/bottomk parameter.	9 years ago
Brian Brazil	3e5136e36d	Make topk/bottomk aggregators.	9 years ago
Brian Brazil	3b89616d82	Allow on, ignoring, by and without wit empty laberls. This offers new semantics in allowing on() for matching two single-element vectors with no known common labels. Previosuly this was often done using on(dummy). This also allows making it explict that you meant to do an aggregation without labels via by(). Fixes #1597.	9 years ago
Brian Brazil	246a817300	Flip vector matching to be ignoring by default. This is a noop semantically.	9 years ago
Ali Reza	e7eba75690	remove keeping_extra because it's replaced with keep_common change all keepExtra label into keepCommon, and move action into removed list change incorrect token list	9 years ago
Brian Brazil	68e70d992a	Clarify error message around on(x) group_left(x)	9 years ago
Brian Brazil	7201c010c4	Rename On to MatchingLabels	9 years ago
Brian Brazil	d991f0cf47	For many-to-one matches, always copy label from one side. This is a breaking change for everyone using the machine roles labeling approach.	9 years ago
Brian Brazil	d1edfb25b3	Add support for OneToMany with IGNORING. The labels listed in the group_ modifier will be copied from the one side to the many side. It will be valid to specify no labels. This is intended to replace the existing ON/GROUP_* support.,	9 years ago
Brian Brazil	1d08c4fef0	Add 'ignoring' as modifier for binops. Where 'on' uses the given labels to match, 'ignoring' uses all other labels to match. group_left/right is not supported yet.	9 years ago
Tobias Schmidt	8cc86f25c0	Implement relative complement set operator "unless" The `unless` set operator can be used to return all vector elements from the LHS which do not match the elements on the RHS. A use case is to return all metrics for nodes which do not have a specific role: node_load1 unless on(instance) chef_role{role="app"}	9 years ago
Fabian Reinartz	ab3d7a0ec0	Remove old alerting syntax	9 years ago
Patrick Bogen	2062fbae0f	rewrite operator balancing to be recursive	9 years ago
Brian Brazil	9d0112d7cf	Add without aggregator modifier. This has the advantage that the user doesn't need to list all labels they want to keep (as with "by") but without having to worry about inconsistent labels as when there's only one time series (as with "keeping_common"). Almost all aggregation should use this rather than the existing two options as it's much less error prone and easier to maintain due to not having to always add in "job" plus whatever other common job-level labels you have like "region".	9 years ago
Tobias Schmidt	1a91cd6e09	Rename matrix to range selector in external error messages The documentation speaks about range vectors and range vector selectors. This change does not fix all issues, we might still expose the term "Matrix" in error messages using %T.	9 years ago
Tobias Schmidt	411ca4dba1	Consolidate offset modifier parsing Remove duplicated offset modifier parsing and ensure offset can only appear at the end of a selector statement.	9 years ago
Fabian Reinartz	6b4a6962d2	Support old alerting rule syntax	9 years ago
Fabian Reinartz	4209ec6864	Change WITH keyword to LABELS	9 years ago
Fabian Reinartz	af3a6661ed	Implement new alerting rule syntax	9 years ago
Brian Brazil	c36961130b	promql: Remove scalar/scalar comparisons. This change is breaking, use the 'bool' modifier for such comprisons. After this change all comparisons without 'bool' will filter, and all comparisons with 'bool' will return 0/1. This makes the language more consistent and orthogonal, and ultimately easier to learn and use. If we ever figure out sane semantics for filtering scalar/scalar comparisons we can add them in, which will most likely come out of how the new vector() function is used.	9 years ago
Julius Volz	46c5260761	Support escape sequences in strings and add raw strings. This adapts some functionality from the Go standard library for string literal lexing and unquoting/unescaping. The following string types are now supported: Double- or single-quoted strings: These support all escape sequences that Go supports in double-quoted string literals. The difference is that Prometheus also has single-quoted strings (instead of single-quoted runes in Go). Raw newlines are not allowed. Backtick-quoted raw strings: Strings quoted in backticks are treated as raw strings just like in Go and may contain raw newlines and other special characters directly. Fixes https://github.com/prometheus/prometheus/issues/1122 Fixes https://github.com/prometheus/prometheus/issues/1121	9 years ago
Fabian Reinartz	171f50706a	Fix unkeyed field errors.	9 years ago
Brian Brazil	9ec11b1847	Merge pull request #1049 from prometheus/bool-nofilter promql: Add 'bool' modifier to comparison functions	9 years ago
Brian Brazil	29e8dc2c49	promql: Add 'bool' modifier to comparison functions When doing comparison operations on vectors, filtering sometimes gets in the way and you have to go to a fair bit of effort to workaround it in order to always return a result. The 'bool' modifier instead of filtering returns 0/1 depending on the result of the compairson. This is also a prerequisite to removing plain scalar/scalar comparisons, as it maintains the current behaviour under a new syntax.	9 years ago
Julius Volz	963ad82dcb	Fix "go vet" errors. I ignored all errors of the type "composite literal uses unkeyed fields". Most of them are wrong because of https://github.com/golang/go/issues/9171.	9 years ago
Fabian Reinartz	438e232c9b	Fix grouping of import blocks	9 years ago
Fabian Reinartz	306e8468a0	Switch from client_golang/model to common/model	9 years ago
Brian Brazil	296f551418	Merge pull request #1014 from prometheus/scalar-rules rules: Allow recorded rules expressions to be scalars.	9 years ago
Brian Brazil	e6a67476c2	rules: Allow recorded rules expressions to be scalars. This is useful if you want to build up a constant metric, such as a set of alert thresholds that vary by label value.	9 years ago
Laurie Malau	cdf38ab93a	Log runtime errors during query evaluation instead of panicking.	9 years ago
Fabian Reinartz	579fdf65e2	Implement unary expression for vector types. Closes #956	9 years ago
Fabian Reinartz	adf109795c	forbid unexpected (runtime) errors in parse tests	9 years ago
Fabian Reinartz	c20e25f718	Add missing check for nil expression	9 years ago
Fabian Reinartz	5279d50d92	Handle parser runtime panics gracefully	9 years ago
Fabian Reinartz	749ae450c5	promql: add runbook to alert statement. This commit adds the RUNBOOK keyword to alert statements. The field is optional and expected to be a link.	10 years ago
Fabian Reinartz	94cd321be1	promql: error if all label matchers are empty.	10 years ago
Julius Volz	5e2d1c1464	Deprecate `keeping_extra`, rename it to `keep_common`. `keep_common` is more in line with the function name `drop_common_labels()` terminology-wise, and also more in line with `group_left`/`group_right` (no `...ing` verb suffix). We could also go the full way and call it `keep_common_labels`. That would have the benefit of being even more consistent with the function `drop_common_labels()` and would be more explanatory, but it also seems quite long.	10 years ago
Fabian Reinartz	319068a7a6	promql: fix parsing ambiguity for serial repetitions	10 years ago
Fabian Reinartz	03094eff04	Migrate parsing error tests. The promql_test checks failure of various bad syntaxed queries. Those are moved into the parser tests as the new testing language only deals with valid queries.	10 years ago
Fabian Reinartz	6321964738	Add parsing and execution of new test format. This commit adds a new test structure that parses and executes the new testing language.	10 years ago
Fabian Reinartz	a236c01457	Add time series description parsing. This commit adds parsing of time series description to the exisiting query language parser. Time series descriptions are defined by a metric followed by a sequence of values.	10 years ago
Fabian Reinartz	8707c54508	Fix single quote parsing, add tests	10 years ago
Fabian Reinartz	279831cdf1	Fix and improve parsing error output.	10 years ago
Fabian Reinartz	25cdff3527	Remove `name` arg from `Parse*` functions, enhance parsing errors.	10 years ago
Fabian Reinartz	5602328c7c	Refactor query evaluation. This copies the evaluation logic from the current rules/ package. The new engine handles the execution process from query string to final result. It provides query timeout and cancellation and general flexibility for future changes. functions.go: Add evaluation implementation. Slight changes to in/out data but not to the processing logic. quantile.go: No changes. analyzer.go: No changes. engine.go: Actually new part. Mainly consists of evaluation methods which were not changed. setup_test.go: Copy of rules/helpers_test.go to setup test storage. promql_test.go: Copy of rules/rules_test.go.	10 years ago
Fabian Reinartz	32b7595c47	Create promql package with lexer/parser. This commit creates a (so far unused) package. It contains the a custom lexer/parser for the query language. ast.go: New AST that interacts well with the parser. lex.go: Custom lexer (new). lex_test.go: Lexer tests (new). parse.go: Custom parser (new). parse_test.go: Parser tests (new). functions.go: Changed function type, dummies for parser testing (barely changed/dummies). printer.go: Adapted from rules/ and adjusted to new AST (mostly unchanged, few additions).	10 years ago

50 Commits (4fc619b605a24c72b8b3423e62a660e54d2513e8)