prometheus

Commit Graph

Author	SHA1	Message	Date
Vishnunarayan K I	fd3ef6ba34	Add metric rule_group_rules_loaded to get the number of rules loaded (#5090 ) Signed-off-by: Vishnunarayan K I <appukuttancr@gmail.com>	6 years ago
Tom Wilkie	121603c417	Expose rules.NewGroupMetrics and rules.Metrics. (#5059 ) Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Bartek Płotka	de213d4a5e	rule manager: Moved metric registration to custom registerer which is already available. (#4961 ) Signed-off-by: Bartek Plotka <bwplotka@gmail.com>	6 years ago
mknapphrt	f0e9196dca	Return warnings on a remote read fail (#4832 ) Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>	6 years ago
Krasi Georgiev	0754e5334b	querier for RestoreForState not closed. (#4922 ) Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Wei Guo	e329cbf673	Add metric prometheus_rule_group_last_evaluation for recording and alerting (#4852 ) * add metric prometheus_rule_group_last_evaluation for recording and alerting Signed-off-by: Wei Guo <me@imkira.com> * fix issues from comments Signed-off-by: Wei Guo <me@imkira.com>	6 years ago
Will Hegedus	193ebe7e34	Updates to /targets and /rules (scrape duration, last evaluation time) (#4722 ) * Add evaluationTimestamp (Last Evaluation) column to display on /rules Signed-off-by: Will Hegedus <wbhegedus@liberty.edu> * Add lastScrapeDuration ("Scrape Duration") to display on /targets Signed-off-by: Will Hegedus <wbhegedus@liberty.edu> * Updates based on Julius' feedback Signed-off-by: Will Hegedus <wbhegedus@liberty.edu> * Update to set timestamp to when eval started (after eval completes) Signed-off-by: Will Hegedus <wbhegedus@liberty.edu> * Update /rules to display time since last evaluation Signed-off-by: Will Hegedus <wbhegedus@liberty.edu> * Re-order Last Eval/Eval Time to be consistent with targets page Signed-off-by: Will Hegedus <wbhegedus@liberty.edu>	6 years ago
Ganesh Vernekar	5790d23fd8	Unit testing for rules (#4350 ) * Unit testing for rules * Specifying order of group evaluation in unit tests Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Chris Marchbanks	63ed9d1b70	Send EndsAt along with alerts (#4550 ) Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	6 years ago
Chris Marchbanks	87f1dad16d	throttle resends of alerts to 1 minute by default (#4538 ) Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	6 years ago
Julius Volz	8fbe1b5133	Handle a bunch of unchecked errors (#4461 ) There are many more (mostly finalizers like Close/Stop/etc.), but most of the others seemed like one couldn't do much about them anyway. Signed-off-by: Julius Volz <julius.volz@gmail.com>	6 years ago
Julien Pivotto	0b4d22b245	rules/manager: remove a no-longer-relevant comment (#4503 ) Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	6 years ago
Benji Visser	8bb6e0dd6e	Show rule evaluation errors on rules page (#4457 ) * adding information about the health and errors for Rules adding Health() and LastError() to the Rule interface. This will allow us to easily surface information about rules. Signed-off-by: noqcks <benny@noqcks.io> * updating rules.html with fields for Rule errors and health state Signed-off-by: noqcks <benny@noqcks.io> * fix code comment grammar & access Rule health/error info using a mutex Signed-off-by: noqcks <benny@noqcks.io> * s/Errors/Error/ in rules.html to remain consistent with targets.html Signed-off-by: noqcks <benny@noqcks.io> * adding periods to code comments in reporting/alerting Signed-off-by: noqcks <benny@noqcks.io> * putting health/error below mutex in struct field Signed-off-by: noqcks <benny@noqcks.io>	6 years ago
Julius Volz	90521a65f8	Remove error return value from NotifyFunc() (#4459 ) It's always nil and we also forgot to check it. Signed-off-by: Julius Volz <julius.volz@gmail.com>	6 years ago
Ganesh Vernekar	f1db699dff	Persist alert 'for' state across restarts (#4061 ) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Max Leonard Inden	71fafad099	api/v1: Coninue work exposing rules and alerts Signed-off-by: Max Leonard Inden <IndenML@gmail.com>	6 years ago
Bryan Boreham	afdb66dfac	Expose Group.CopyState() (#4304 ) This makes the `rules` package more useful to projects that use Prometheus as a library. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	6 years ago
Julius Volz	9e3171f6e3	rules: Minor naming/comment cleanups (#4328 ) Signed-off-by: Julius Volz <julius.volz@gmail.com>	6 years ago
Alin Sinpalean	9dc763cc03	Run rule evaluation with timestamps precisely evaluation_interval apart (#4201 ) * Run rule evaluation with timestamps precisely evaluation_interval apart from one another. Signed-off-by: Alin Sinpalean <alin.sinpalean@gmail.com>	7 years ago
Mario Trangoni	464e747f1e	fix some comments typos (#4059 )	7 years ago
Bryan Boreham	93494d8b7e	Add an OpenTracing span for each rule (#4027 ) * Add an OpenTracing span for each rule So that tags and child spans can be traced back to the rule that they refer to.	7 years ago
ferhat elmas	ffa673f7d8	General simplifications (#3887 ) Another try as in #1516	7 years ago
Fabian Reinartz	7ccd4b39b8	*: implement query params This adds a parameter to the storage selection interface which allows query engine(s) to pass information about the operations surrounding a data selection. This can for example be used by remote storage backends to infer the correct downsampling aggregates that need to be provided.	7 years ago
Brian Brazil	30b4439bbd	Remove rule_type label from rule metrics. This is not really needed now that we have rule groups to distinguish rules.	7 years ago
Brian Brazil	b97f4cf48c	Add metrics for rule group interval and last duration.	7 years ago
Brian Brazil	0a42a9fc8f	Copy over rule group duration on reload. This is currently getting lost, this will soon be in a metric and we don't want it dropping to 0 on every reload.	7 years ago
Brian Brazil	aa370fa568	Clarify metric names around rule groups. Make it clear they're about overall rule groups.	7 years ago
Fabian Reinartz	62461379b7	rules: decouple notifier packages The dependency on the notifier packages caused a transitive dependency on discovery and with that all client libraries our service discovery uses.	7 years ago
Fabian Reinartz	4d964a0a0d	rules: make glob expansion a concern of main	7 years ago
Fabian Reinartz	bd9f7460eb	rules: remove config package dependency	7 years ago
Fabian Reinartz	2d0e3746ac	rules: remove dependency on promql.Engine	7 years ago
Goutham Veeramachaneni	a880c86375	Fix unexported method on exported interface. Also move to model.Duration Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	7 years ago
conorbroderick	55aaece116	Add rule evaluation time	7 years ago
Goutham Veeramachaneni	e1117715fe	rules: remove skipped iterations cuz no throttling Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	7 years ago
Krasi Georgiev	e86d82ad2d	Fix regression of alert rules state loss on config reload. (#3382 ) * incorrect map name for the group prevented copying state from existing alert rules on config reload * applyConfig test * few nits * nits 2	7 years ago
Julius Volz	099df0c5f0	Migrate "golang.org/x/net/context" -> "context" (#3333 ) In some places, where ctxhttp or gRPC are concerned, we still need to use the old contexts.	7 years ago
Brian Brazil	ee88f0d222	Ensure all values are used or _	7 years ago
Fabian Reinartz	2d0b8e8b94	Merge branch 'master' into dev-2.0	7 years ago
beorn7	c2e9a151ab	Make all rule links link to the "Console" tab rather than "Graph" Clicking on a rule, either the name or the expression, opens the rule result (or the corresponding expression, repsectively) in the expression browser. This should by default happen in the console tab, as, more often than not, displaying it in the graph tab runs into a timeout.	7 years ago
Fabian Reinartz	d21f149745	*: migrate to go-kit/log	7 years ago
Goutham Veeramachaneni	37e7b69f56	Merge remote-tracking branch 'upstream/dev-2.0' into rulegroups Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	c472316fb3	Check done before every rule evaluation. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	6b70a4d850	Incorporate PR feedback * Move fingerprint to Hash() * Move away from tsdb.MultiError * 0777 -> 0666 for files * checkOverflow of extra fields Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	507790a357	Rework logging to use explicitly passed logger Mostly cleaned up the global logger use. Still some uses in discovery package. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	dc69645e92	Move back to go-yaml Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	5ff283a7b7	Reflect the grouping in the UI Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	8cca666cf2	Add file name to group. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	e893c89333	Validate labels and annotations Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	a48a018368	Make sure groups are unique in a single file Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	cea1e99f78	Add update-rules command to promtool Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	e8f55669ea	Move rules to new format Signed-off-by: Goutham Veeramachaneni <goutham@boomerangcommerce.com>	8 years ago
Brian Brazil	cc867dae60	Copy previous series and alert state more intelligently. Usually rules don't more around, and if they do it's likely that rules/alerts with the same name stay in the same order. If rules/alerts with the same name are added/removed this could cause a blip for one cycle, but this is unavoidable without requiring rule and alert names to be unique - which we don't want to do.	8 years ago
Brian Brazil	9bc68db7e6	Track staleness per rule rather than per group.	8 years ago
Brian Brazil	0451d6d31b	Add unittest for rule staleness, and rules generally.	8 years ago
Brian Brazil	0400f3cfd2	Very basic staleness handling for rules.	8 years ago
Fabian Reinartz	06c2b76cd4	Merge branch 'master' into uptsdb	8 years ago
Julius Volz	ac203ef0ee	Add externalURL template function (#2716 ) This allows users to e.g. add links back to the generating Prometheus right in their alert templates.	8 years ago
Fabian Reinartz	8ffc851147	Merge branch 'master' into dev-2.0	8 years ago
Tobias Schmidt	eaf33759fb	Register forgotten prometheus_evaluator_iterations_total metric	8 years ago
Tobias Schmidt	aaaba57184	Export number of missed rule evaluations In case the execution of all rules takes longer than the configured rule evaluation interval, one or more iterations will be skipped. This needs to be visible to the opterator.	8 years ago
Fabian Reinartz	5772f1a7ba	retrieval/storage: adapt to new interface This simplifies the interface to two add methods for appends with labels or faster reference numbers.	8 years ago
Fabian Reinartz	ad9bc62e4c	storage: extend appender and adapt it	8 years ago
Fabian Reinartz	e94b0899ee	rules: fix tests, remove model types	8 years ago
Fabian Reinartz	f8fc1f5bb2	*: migrate ingestion to new batch Appender	8 years ago
Fabian Reinartz	5817cb5bde	: migrate from model. to promql.* types	8 years ago
Jonathan Lange	d78dd3593d	Set evaluation interval on Group construction Prevents having object in invalid state, and allows users of public API to construct valid Groups.	8 years ago
Jonathan Lange	31fc357cd8	Make NewGroup and Group.Eval public Allows callers to execute evaluate lists of rules without first writing them to disk.	8 years ago
Jonathan Lange	2a2da40223	Make rule evaluation publicly available Means that a third-party can parse rules and run them with their own execution model.	8 years ago
Matt Bostock	926a5ab3dd	rules/manager.go: Fix race between reload and stop On one relatively large Prometheus instance (1.7M series), I noticed that upgrades were frequently resulting in Prometheus undergoing crash recovery on start-up. On closer examination, I found that Prometheus was panicking on shutdown. It seems that our configuration management (or misconfiguration thereof) is reloading Prometheus then immediately restarting it, which I suspect is causing this race: Sep 21 15:12:42 host systemd[1]: Reloading prometheus monitoring system. Sep 21 15:12:42 host prometheus[18734]: time="2016-09-21T15:12:42Z" level=info msg="Loading configuration file /etc/prometheus/config.yaml" source="main.go:221" Sep 21 15:12:42 host systemd[1]: Reloaded prometheus monitoring system. Sep 21 15:12:44 host systemd[1]: Stopping prometheus monitoring system... Sep 21 15:12:44 host prometheus[18734]: time="2016-09-21T15:12:44Z" level=warning msg="Received SIGTERM, exiting gracefully..." source="main.go:203" Sep 21 15:12:44 host prometheus[18734]: time="2016-09-21T15:12:44Z" level=info msg="See you next time!" source="main.go:210" Sep 21 15:12:44 host prometheus[18734]: time="2016-09-21T15:12:44Z" level=info msg="Stopping target manager..." source="targetmanager.go:90" Sep 21 15:12:52 host prometheus[18734]: time="2016-09-21T15:12:52Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:548" Sep 21 15:12:56 host prometheus[18734]: time="2016-09-21T15:12:56Z" level=warning msg="Error on ingesting out-of-order samples" numDropped=1 source="scrape.go:467" Sep 21 15:12:56 host prometheus[18734]: time="2016-09-21T15:12:56Z" level=error msg="Error adding file watch for \"/etc/prometheus/targets\": no such file or directory" source="file.go:84" Sep 21 15:12:56 host prometheus[18734]: time="2016-09-21T15:12:56Z" level=error msg="Error adding file watch for \"/etc/prometheus/targets\": no such file or directory" source="file.go:84" Sep 21 15:13:01 host prometheus[18734]: time="2016-09-21T15:13:01Z" level=info msg="Stopping rule manager..." source="manager.go:366" Sep 21 15:13:01 host prometheus[18734]: time="2016-09-21T15:13:01Z" level=info msg="Rule manager stopped." source="manager.go:372" Sep 21 15:13:01 host prometheus[18734]: time="2016-09-21T15:13:01Z" level=info msg="Stopping notification handler..." source="notifier.go:325" Sep 21 15:13:01 host prometheus[18734]: time="2016-09-21T15:13:01Z" level=info msg="Stopping local storage..." source="storage.go:381" Sep 21 15:13:01 host prometheus[18734]: time="2016-09-21T15:13:01Z" level=info msg="Stopping maintenance loop..." source="storage.go:383" Sep 21 15:13:01 host prometheus[18734]: panic: close of closed channel Sep 21 15:13:01 host prometheus[18734]: goroutine 7686074 [running]: Sep 21 15:13:01 host prometheus[18734]: panic(0xba57a0, 0xc60c42b500) Sep 21 15:13:01 host prometheus[18734]: /usr/local/go/src/runtime/panic.go:500 +0x1a1 Sep 21 15:13:01 host prometheus[18734]: github.com/prometheus/prometheus/rules.(Manager).ApplyConfig.func1(0xc6645a9901, 0xc420271ef0, 0xc420338ed0, 0xc60c42b4f0, 0xc6645a9900) Sep 21 15:13:01 host prometheus[18734]: /home/build/packages/prometheus/tmp/build/gopath/src/github.com/prometheus/prometheus/rules/manager.go:412 +0x3c Sep 21 15:13:01 host prometheus[18734]: created by github.com/prometheus/prometheus/rules.(Manager).ApplyConfig Sep 21 15:13:01 host prometheus[18734]: /home/build/packages/prometheus/tmp/build/gopath/src/github.com/prometheus/prometheus/rules/manager.go:423 +0x56b Sep 21 15:13:03 host systemd[1]: prometheus.service: main process exited, code=exited, status=2/INVALIDARGUMENT	8 years ago
Julius Volz	c187308366	storage: Contextify storage interfaces. This is based on https://github.com/prometheus/prometheus/pull/1997. This adds contexts to the relevant Storage methods and already passes PromQL's new per-query context into the storage's query methods. The immediate motivation supporting multi-tenancy in Frankenstein, but this could also be used by Prometheus's normal local storage to support cancellations and timeouts at some point.	8 years ago
Julius Volz	ed5a0f0abe	promql: Allow per-query contexts. For Weaveworks' Frankenstein, we need to support multitenancy. In Frankenstein, we initially solved this without modifying the promql package at all: we constructed a new promql.Engine for every query and injected a storage implementation into that engine which would be primed to only collect data for a given user. This is problematic to upstream, however. Prometheus assumes that there is only one engine: the query concurrency gate is part of the engine, and the engine contains one central cancellable context to shut down all queries. Also, creating a new engine for every query seems like overkill. Thus, we want to be able to pass per-query contexts into a single engine. This change gets rid of the promql.Engine's built-in base context and allows passing in a per-query context instead. Central cancellation of all queries is still possible by deriving all passed-in contexts from one central one, but this is now the responsibility of the caller. The central query context is now created in main() and passed into the relevant components (web handler / API, rule manager). In a next step, the per-query context would have to be passed to the storage implementation, so that the storage can implement multi-tenancy or other features based on the contextual information.	8 years ago
Dmitry Vorobev	273e457da4	web: return status code and error message for config resource	8 years ago
Brian Brazil	0509b0f2db	Expand alert templates at eval time. Fixes #1678 #1677	8 years ago
beorn7	064b57858e	Consistently use the `Seconds()` method for conversion of durations This also fixes one remaining case of recording integral numbers of seconds only for a metric, i.e. this will probably fix #1796.	9 years ago
beorn7	45e5775f9b	Add missing logging of out-of-order samples So far, out-of-order samples during rule evaluation were not logged, and neither scrape health samples. The latter are unlikely to cause any errors. That's why I'm logging them always now. (It's alway highly irregular should it happen.) For rules, I have used the same plumbing as for samples, just with a different wording in the message to mark them as a result of rule evaluation.	9 years ago
Fabian Reinartz	d89c254849	Make copying alerting state safer. This considers static labels in the equality of alerts to avoid falsely copying state from a different alert definition with the same name across reloads. To be safe, it also copies the state map rather than just its pointer so that remaining collisions disappear after one evaluation interval.	9 years ago
Fabian Reinartz	bfa8aaa017	Rename notification to notifier	9 years ago
beorn7	663a1550d0	Fix the instrumentation fixes	9 years ago
beorn7	ec08c9a391	Rework the way to communicate backpressure (AKA suspended ingestion) This gives up on the idea to communicate throuh the Append() call (by either not returning as it is now or returning an error as suggested/explored elsewhere). Here I have added a Throttled() call, which has the advantage that it can be called before a whole _batch_ of Append()'s. Scrapes will happen completely or not at all. Same for rule group evaluations. That's a highly desired behavior (as discussed elsewhere). The code is even simpler now as the whole ingestion buffer could be removed. Logging of throttled mode has been streamlined and will create at most one message per minute.	9 years ago
Fabian Reinartz	b0adfea8d5	Fix swapped constants, improve instrumentation	9 years ago
Fabian Reinartz	a8c38c3ac5	Don't log rule evaluation failure on shutdown	9 years ago
Fabian Reinartz	6eee86dce8	Terminate rule groups during initial sleep When an evaluation group runs initially, it waits a deterministic amount of time. During that time it also has to accept a termination singnal so shutdown doesn't hang during the first evaluation iteration after a configuration reload. Fixes #1307	9 years ago
Fabian Reinartz	37d80c4b25	Fix premature rule evaluation This commit prevents rule evaluation from starting until after the storage is ready.	9 years ago
Fabian Reinartz	0cf3c6a9ef	Add comments, rename a method	9 years ago
Fabian Reinartz	bf6abac8f4	Send resolved notifications	9 years ago
Fabian Reinartz	f69e668fc4	Improve rules/ instrumentation This commit adds a counter for the total number of rule evaluations and standardizes the units to seconds.	9 years ago
Fabian Reinartz	52e5224f5a	Refactor rules/ package	9 years ago
Fabian Reinartz	e4fabe135a	Set StartsAt to time of first firing state	9 years ago
Fabian Reinartz	7c90db22ed	Use annotation based alerts in rules/ This commit breaks the previously used alert format.	9 years ago
Fabian Reinartz	e114ce0ff7	Refactor notification handler	9 years ago
Fabian Reinartz	e3b6ec9784	Switch to common/log	9 years ago
Julius Volz	995d3b831d	Fix most golint warnings. This is with `golint -min_confidence=0.5`. I left several lint warnings untouched because they were either incorrect or I felt it was better not to change them at the moment.	9 years ago
Fabian Reinartz	d6b8da8d43	Switch promql types to common/model	9 years ago
Brian Brazil	fdf0d0642e	Cast value to float, as that's what the console templates expect.	9 years ago
Fabian Reinartz	438e232c9b	Fix grouping of import blocks	9 years ago
Fabian Reinartz	306e8468a0	Switch from client_golang/model to common/model	9 years ago
Fabian Reinartz	7a67472fc1	Resolve relative paths on configuration loading This moves the concern of resolving the files relative to the config file into the configuration loading itself. It also fixes #921 which did not load the cert and token files relatively.	9 years ago
Fabian Reinartz	feb8a03503	rules: load rule files relative to a base dir	10 years ago
Julius Volz	fcff35b43e	Consolidate external reachability flags into one. Besides fixing https://github.com/prometheus/prometheus/issues/805 by making the entire externally reachable server URL configurable, this adds tests for the "globalURL" template function and makes it easier to test other such functions in the future. This breaks the `web.Hostname` flag (and introduces `web.external-url`). This flag is likely only used by few users, so I hope that's justifiable. Fixes https://github.com/prometheus/prometheus/issues/805	10 years ago
Fabian Reinartz	f06cf664e1	rules: cleanup alerting test	10 years ago

1 2 3 4

192 Commits (b524ca8447c9aef9c44e3db5b08f99c550e1d18c)