prometheus

Commit Graph

Author	SHA1	Message	Date
Fabian Reinartz	83cd270ea4	*: adapt to storage interface changes	7 years ago
David Kaltschmidt	87c46ea6c3	Renamed TotalEvalTime to EvalTotalTime * TotalFoo suggested a comprehensive timing, but TotalEvalTime was part of the Exec timings, together with Queue timings * The other option was to rename ExecTotalTime to TotalExecTime, but there was already ExecQueueTime, suggesting Exec to be some sort of group	7 years ago
David Kaltschmidt	c93e54d240	Adds execution timer stats to the range query API consumers should be able to get insight into the query run times. The UI currently measures total roundtrip times. This PR allows for more fine grained metrics to be exposed. * adds new timer for total execution time (queue + eval) * expose new timer, queue timer, and eval timer in stats field of the range query response: ```json { "status": "success", "data": { "resultType": "matrix", "result": [], "stats": { "execQueueTimeNs": 4683, "execTotalTimeNs": 2086587, "totalEvalTimeNs": 2077851 } } } ``` * stats field is optional, only set when query parameter `stats` is not empty Try it via ```sh curl 'http://localhost:9090/api/v1/query_range?query=up&start=1486480279&end=1486483879&step=14000&stats=true' ``` Review feedback * moved query stats json generation to query_stats.go * use seconds for all query timers * expose all timers available * Changed ExecTotalTime string representation from Exec queue total time to Exec total time	7 years ago
Brian Brazil	99905f82a6	Remove keep_common modifier. See #3060	7 years ago
Brian Brazil	67274f0794	Remove 4 interval staleness heuristic. (#3244 ) This means that if there is no stale marker, only the usual staleness delta (5m) applies. It has occured to me that there is an oddity in the heurestic. It works fine as long as you have 2 points within the last 5m, but breaks down when the time window advances to the point where you have just 1 point. Consider you had points at t=0 and t=10. With the heurestic it goes stale at t=51, up until t=300. However from t=301 until t=310 we only see the t=10 point and the series comes back to life. That is not desirable. I don't see a way to keep this form of heurestic working given this issue, so thus I'm removing it.	7 years ago
Julius Volz	f7e8348a88	Re-add contexts to storage.Storage.Querier() (#3230 ) * Re-add contexts to storage.Storage.Querier() These are needed when replacing the storage by a multi-tenant implementation where the tenant is stored in the context. The 1.x query interfaces already had contexts, but they got lost in 2.x. * Convert promql.Engine to use native contexts	7 years ago
Fabian Reinartz	d21f149745	*: migrate to go-kit/log	7 years ago
Fabian Reinartz	25f3e1c424	Merge branch 'master' into mergemaster	7 years ago
Alexey Palazhchenko	695ec0b981	Fix few typos. (#2962 )	7 years ago
Goutham Veeramachaneni	4194d2ac79	Call At() only if Next() is true Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	7 years ago
Goutham Veeramachaneni	d407bd150c	Consolidate the duration params in CLI * All CLI params moved to model.Duration Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	507790a357	Rework logging to use explicitly passed logger Mostly cleaned up the global logger use. Still some uses in discovery package. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Brian Brazil	220e78b9c3	Consider a series stale after 4.1 intervals with no data. To cover the cases where stale markers may not be available, we need to infer the interval and mark series stale based on that. As we're lacking stale markers this is less accurate, however it should be good enough for these cases. We need 4 intervals as if say we had data at t=0 and t=10, coming via federation. The next data point should be at t=20 however it could take up to t=30 for it actually to be ingested, t=40 for it to be scraped via federation and t=50 for it to be ingested. We then add 10% on to that for slack, as we do elsewhere.	8 years ago
Brian Brazil	c02c25d5ba	Allow peeking back further in buffer.	8 years ago
Brian Brazil	a5cf25743c	Move stalness check into a function	8 years ago
Brian Brazil	80b40e6d91	Add initial staleness handing to promql. For instant vectors, if "stale" is the newest sample ignore the timeseries. For range vectors, filter out "stale" samples. Make it possible to inject "stale" samples in promql tests.	8 years ago
Fabian Reinartz	6e804b3497	Merge branch 'master' into dev-2.0	8 years ago
Brian Brazil	fcc88f0e1e	query/query_range should return eval timestamp Query and query_range should return the timestamp at which an evaluation is performed, not the timestamp of the data. This is as that's what query range asked for, and we need to keep query consistent with that. Query for a matrix remains unchanged, returning the literal matrix.	8 years ago
Brian Brazil	517b81f927	Add timestamp() function. Make the timestamp of instant vectors be the timestamp of the sample rather than the evaluation. We were not using this anywhere, so this is safe. Add a function to return the timestamp of samples in an instant vector. Fixes #1557	8 years ago
Tom Wilkie	4d9b917d11	Instrument Prometheus with OpenTracing (#2554 ) * Use request.Context() instead of a global map of contexts. * Add some basic opentracing instrumentation on the query path. * Remove tracehandler endpoint.	8 years ago
Fabian Reinartz	73b8ff0ddc	Merge branch 'master' into dev-2.0	8 years ago
Tom Wilkie	f0e8a5f37c	Add promql.ErrStorage, which is interpreted by the API as a 500.	8 years ago
Fabian Reinartz	c389193b37	Merge branch 'master' into dev-2.0	8 years ago
Fabian Reinartz	0ecd205794	promql: Use buffer pool for matrix allocations	8 years ago
Fabian Reinartz	b09b90a940	Correctly close querier on error, revendor tsdb	8 years ago
Goutham Veeramachaneni	6634984a38	Comments and Typo Fixes	8 years ago
Fabian Reinartz	9304179ef7	Merge branch 'master' into dev-2.0	8 years ago
Alex Somesan	18cd7246b5	Instrument query engine timings (#2418 ) * Instrument query engine statistics	8 years ago
Fabian Reinartz	1d3cdd0d67	Merge branch 'master' into dev-2.0-rebase	8 years ago
André Carvalho	c43dfaba1c	Add max concurrent and current queries engine metrics (#2326 ) * Add max concurrent and current queries engine metrics This commit adds two metrics to the promql/engine: the number of max concurrent queries, as configured by the flag, and the number of current queries being served+blocked in the engine.	8 years ago
Fabian Reinartz	bc20d93f0a	storage: rename iterator value getters to At()	8 years ago
Fabian Reinartz	28f547bcc7	api/v1: fix tests, restore series queries	8 years ago
Fabian Reinartz	f8fc1f5bb2	*: migrate ingestion to new batch Appender	8 years ago
Fabian Reinartz	71fe0c58a8	promql: misc fixes	8 years ago
Fabian Reinartz	fecf9532b9	*: fix misc compile errors	8 years ago
Fabian Reinartz	0492ddbd4d	*: fully decouple tsdb, add new storage interfaces	8 years ago
Fabian Reinartz	9ea10d5265	promql: use labels.Builder to modify labels	8 years ago
Fabian Reinartz	c6cd998905	promql: use local labels, add conversion	8 years ago
Fabian Reinartz	ff504af2aa	promql: undo accidental exports	8 years ago
Fabian Reinartz	6dedf89cc3	promql: rename SampleStream to Series	8 years ago
Fabian Reinartz	c5f225b920	promql: export Sample	8 years ago
Fabian Reinartz	65581a3d46	promql: export SmapleStream	8 years ago
Fabian Reinartz	6315d00942	promql: export String value	8 years ago
Fabian Reinartz	ac5d3bc05e	promql: scalar T/V and Point	8 years ago
Fabian Reinartz	09666e2e2a	promql: make scalar public	8 years ago
Fabian Reinartz	b3f71df350	promql: make matrix exported	8 years ago
Fabian Reinartz	a62df87022	promql: rename vector	8 years ago
Fabian Reinartz	15a931dbdb	promql: migrate model types, use tsdb interfaces	8 years ago
Tristan Colgate	68fc15fe4e	Report type names in the form used in documentation	8 years ago
beorn7	4e3abc6cbf	Simply use `math.Mod(float64, float64)` after all This circumvents all the problems with int overflow, plus it is what was originally intended.	8 years ago
beorn7	5cf5bb427a	Check for int64 overflow when converting from float64	8 years ago
beorn7	92c0ef1a92	Merge branch 'release-1.2' into beorn7/release	8 years ago
beorn7	07f1bdfe94	Fix MOD binop for scalars and vectors Previously, a floating point number that would round down to 0 would cause a "division by zero" panic.	8 years ago
Fabian Reinartz	8fa18d564a	storage: enhance Querier interface usage This extracts Querier as an instantiateable and closeable object rather than just defining extending methods of the storage interface. This improves composability and allows abstracting query transactions, which can be useful for transaction-level caches, consistent data views, and encapsulating teardown.	8 years ago
Julius Volz	c187308366	storage: Contextify storage interfaces. This is based on https://github.com/prometheus/prometheus/pull/1997. This adds contexts to the relevant Storage methods and already passes PromQL's new per-query context into the storage's query methods. The immediate motivation supporting multi-tenancy in Frankenstein, but this could also be used by Prometheus's normal local storage to support cancellations and timeouts at some point.	8 years ago
Julius Volz	ed5a0f0abe	promql: Allow per-query contexts. For Weaveworks' Frankenstein, we need to support multitenancy. In Frankenstein, we initially solved this without modifying the promql package at all: we constructed a new promql.Engine for every query and injected a storage implementation into that engine which would be primed to only collect data for a given user. This is problematic to upstream, however. Prometheus assumes that there is only one engine: the query concurrency gate is part of the engine, and the engine contains one central cancellable context to shut down all queries. Also, creating a new engine for every query seems like overkill. Thus, we want to be able to pass per-query contexts into a single engine. This change gets rid of the promql.Engine's built-in base context and allows passing in a per-query context instead. Central cancellation of all queries is still possible by deriving all passed-in contexts from one central one, but this is now the responsibility of the caller. The central query context is now created in main() and passed into the relevant components (web handler / API, rule manager). In a next step, the per-query context would have to be passed to the storage implementation, so that the storage can implement multi-tenancy or other features based on the contextual information.	8 years ago
beorn7	71571a8ec4	promql: Fix (and simplify) populating iterators This was only relevant so far for the benchmark suite as it would recycle Expr for repetitions. However, the append is unnecessary as each node is only inspected once when populating iterators, and population must always start from scratch. This also introduces error checking during benchmarks and fixes the so far undetected test errors during benchmarking. Also, remove a style nit (two golint warnings less…).	8 years ago
Julius Volz	3bfec97d46	Make the storage interface higher-level. See discussion in https://groups.google.com/forum/#!topic/prometheus-developers/bkuGbVlvQ9g The main idea is that the user of a storage shouldn't have to deal with fingerprints anymore, and should not need to do an individual preload call for each metric. The storage interface needs to be made more high-level to not expose these details. This also makes it easier to reuse the same storage interface for remote storages later, as fewer roundtrips are required and the fingerprint concept doesn't work well across the network. NOTE: this deliberately gets rid of a small optimization in the old query Analyzer, where we dedupe instants and ranges for the same series. This should have a minor impact, as most queries do not have multiple selectors loading the same series (and at the same offset).	8 years ago
Brian Brazil	0303ccc6a7	Add quantile aggregator.	8 years ago
Brian Brazil	16690736ab	Add count_values() aggregator. This is useful for counting how many instances of a job are running a particular version/build. Fixes #622	9 years ago
Brian Brazil	3e5136e36d	Make topk/bottomk aggregators.	9 years ago
Brian Brazil	3b89616d82	Allow on, ignoring, by and without wit empty laberls. This offers new semantics in allowing on() for matching two single-element vectors with no known common labels. Previosuly this was often done using on(dummy). This also allows making it explict that you meant to do an aggregation without labels via by(). Fixes #1597.	9 years ago
Brian Brazil	246a817300	Flip vector matching to be ignoring by default. This is a noop semantically.	9 years ago
Julius Volz	b7b6717438	Separate query interface out of local.Storage. PromQL only requires a much narrower interface than local.Storage in order to run queries. Narrower interfaces are easier to replace and test, too. We could also change the web interface to use local.Querier, except that we'll probably use appending functions from there in the future.	9 years ago
royels	2fdc5717a3	promql: add power binary operation	9 years ago
Ali Reza	e7eba75690	remove keeping_extra because it's replaced with keep_common change all keepExtra label into keepCommon, and move action into removed list change incorrect token list	9 years ago
Brian Brazil	7201c010c4	Rename On to MatchingLabels	9 years ago
Brian Brazil	d991f0cf47	For many-to-one matches, always copy label from one side. This is a breaking change for everyone using the machine roles labeling approach.	9 years ago
Brian Brazil	768d09fd2a	Change on+group_* to take copy from the one side. If the label doesn't exist on the one side, it's not copied. All labels on the many inside are included, this is a breaking change but likely low impact.	9 years ago
Brian Brazil	d1edfb25b3	Add support for OneToMany with IGNORING. The labels listed in the group_ modifier will be copied from the one side to the many side. It will be valid to specify no labels. This is intended to replace the existing ON/GROUP_* support.,	9 years ago
Brian Brazil	1d08c4fef0	Add 'ignoring' as modifier for binops. Where 'on' uses the given labels to match, 'ignoring' uses all other labels to match. group_left/right is not supported yet.	9 years ago
Tobias Schmidt	8cc86f25c0	Implement relative complement set operator "unless" The `unless` set operator can be used to return all vector elements from the LHS which do not match the elements on the RHS. A use case is to return all metrics for nodes which do not have a specific role: node_load1 unless on(instance) chef_role{role="app"}	9 years ago
beorn7	c740789ce3	Improve predict_linear Fixes https://github.com/prometheus/prometheus/issues/1401 This remove the last (and in fact bogus) use of BoundaryValues. Thus, a whole lot of unused (and arguably sub-optimal / ugly) code can be removed here, too.	9 years ago
beorn7	0e202dacb4	Streamline series iterator creation This will fix issue #1035 and will also help to make issue #1264 less bad. The fundamental problem in the current code: In the preload phase, we quite accurately determine which chunks will be used for the query being executed. However, in the subsequent step of creating series iterators, the created iterators are referencing _all_ in-memory chunks in their series, even the un-pinned ones. In iterator creation, we copy a pointer to each in-memory chunk of a series into the iterator. While this creates a certain amount of allocation churn, the worst thing about it is that copying the chunk pointer out of the chunkDesc requires a mutex acquisition. (Remember that the iterator will also reference un-pinned chunks, so we need to acquire the mutex to protect against concurrent eviction.) The worst case happens if a series doesn't even contain any relevant samples for the query time range. We notice that during preloading but then we will still create a series iterator for it. But even for series that do contain relevant samples, the overhead is quite bad for instant queries that retrieve a single sample from each series, but still go through all the effort of series iterator creation. All of that is particularly bad if a series has many in-memory chunks. This commit addresses the problem from two sides: First, it merges preloading and iterator creation into one step, i.e. the preload call returns an iterator for exactly the preloaded chunks. Second, the required mutex acquisition in chunkDesc has been greatly reduced. That was enabled by a side effect of the first step, which is that the iterator is only referencing pinned chunks, so there is no risk of concurrent eviction anymore, and chunks can be accessed without mutex acquisition. To simplify the code changes for the above, the long-planned change of ValueAtTime to ValueAtOrBefore time was performed at the same time. (It should have been done first, but it kind of accidentally happened while I was in the middle of writing the series iterator changes. Sorry for that.) So far, we actively filtered the up to two values that were returned by ValueAtTime, i.e. we invested work to retrieve up to two values, and then we invested more work to throw one of them away. The SeriesIterator.BoundaryValues method can be removed once #1401 is fixed. But I really didn't want to load even more changes into this PR. Benchmarks: The BenchmarkFuzz.* benchmarks run 83% faster (i.e. about six times faster) and allocate 95% fewer bytes. The reason for that is that the benchmark reads one sample after another from the time series and creates a new series iterator for each sample read. To find out how much these improvements matter in practice, I have mirrored a beefy Prometheus server at SoundCloud that suffers from both issues #1035 and #1264. To reach steady state that would be comparable, the server needs to run for 15d. So far, it has run for 1d. The test server currently has only half as many memory time series and 60% of the memory chunks the main server has. The 90th percentile rule evaluation cycle time is ~11s on the main server and only ~3s on the test server. However, these numbers might get much closer over time. In addition to performance improvements, this commit removes about 150 LOC.	9 years ago
Julius Volz	9b6d69610a	Fix various typos in comments. Helpfully reported by https://goreportcard.com/report/github.com/prometheus/prometheus :)	9 years ago
Brian Brazil	9d0112d7cf	Add without aggregator modifier. This has the advantage that the user doesn't need to list all labels they want to keep (as with "by") but without having to worry about inconsistent labels as when there's only one time series (as with "keeping_common"). Almost all aggregation should use this rather than the existing two options as it's much less error prone and easier to maintain due to not having to always add in "job" plus whatever other common job-level labels you have like "region".	9 years ago
Brian Brazil	89760dd77d	Handle NaN for min/max. Similar to topk and sort, prefer not returning NaN where possible.	9 years ago
Fabian Reinartz	e3b6ec9784	Switch to common/log	9 years ago
Brian Brazil	29e8dc2c49	promql: Add 'bool' modifier to comparison functions When doing comparison operations on vectors, filtering sometimes gets in the way and you have to go to a fair bit of effort to workaround it in order to always return a result. The 'bool' modifier instead of filtering returns 0/1 depending on the result of the compairson. This is also a prerequisite to removing plain scalar/scalar comparisons, as it maintains the current behaviour under a new syntax.	9 years ago
Julius Volz	077a753e6b	Merge pull request #1006 from prometheus/true-values promql: Remove interpolation of vector values.	9 years ago
Fabian Reinartz	d6b8da8d43	Switch promql types to common/model	9 years ago
Brian Brazil	fb585e4591	promql: Remove interpolation of vector values. The current behaviour produces values that are not from rules or scrapes. So if for example I have a boolean 0/1 it can be returned as 0.2344589. This prevents a number of advanced use cases, introduces race conditions and can produce misleading graphs.	9 years ago
Fabian Reinartz	1535ef1457	Replace metric.SamplePair with model.SamplePair	9 years ago
Fabian Reinartz	438e232c9b	Fix grouping of import blocks	9 years ago
Fabian Reinartz	306e8468a0	Switch from client_golang/model to common/model	9 years ago
Laurie Malau	cdf38ab93a	Log runtime errors during query evaluation instead of panicking.	9 years ago
Julius Volz	27ed874358	Implement label_replace() Implements part of https://github.com/prometheus/prometheus/issues/959.	9 years ago
Fabian Reinartz	690b5f1575	Remove multi-statement queries This commit removes the possibility to have multi-statement queries which had no full support anyway. This makes the caller responsible for multi-statement semantics. Multiple tests are no longer timing-dependent.	9 years ago
Fabian Reinartz	579fdf65e2	Implement unary expression for vector types. Closes #956	9 years ago
Fabian Reinartz	3d67d75935	promql: implement JSON array format for scalar and string	10 years ago
Fabian Reinartz	77e8983221	promql: add MarshalJSON method for SamplePair	10 years ago
Fabian Reinartz	70d7a987a7	promql: add json tags, fix query constructor.	10 years ago
Fabian Reinartz	fe301d7946	promql: remove global flags	10 years ago
Fabian Reinartz	c32ae22119	promql: fix missing metric in range results.	10 years ago
Fabian Reinartz	cb10ceac18	promql: allow scalar expressions in range queries, improve errors. These changes allow to do range queries over scalar expressions. Errors on bad types for range queries are now raised on query creation rather than evaluation.	10 years ago
Fabian Reinartz	0de6edbdfc	Move pkg/ to util/	10 years ago
Fabian Reinartz	ccf51b132e	Move stats package to pkg/stats	10 years ago
beorn7	3b9c421a69	Weed out all the [Gg]et* method names. The only exception is getNumChunksToPersist to avoid naming the struct member numChunksToPersist in a weird way.	10 years ago
Fabian Reinartz	ac4d63b833	Merge pull request #689 from prometheus/fabxc/qltest Add basic testing language, migrate tests	10 years ago
Fabian Reinartz	6321964738	Add parsing and execution of new test format. This commit adds a new test structure that parses and executes the new testing language.	10 years ago

1 2 3 4

158 Commits (160186da18d8daa39b2318f9bb7263fc106847b9)