217 Commits (291ab4d0bc6ea9f0e61073f303bd3982e9727b8d)

Author	SHA1	Message	Date
Kristiyan Nikolov	be85ba3842	discovery/ec2: Support filtering instances in discovery (#4011)	7 years ago
Corentin Chary	60dafd425c	consul: improve consul service discovery (#3814) ... * consul: improve consul service discovery Related to #3711 - Add the ability to filter by tag and node-meta in an efficient way (`/catalog/services` allow filtering by node-meta, and returns a `map[string]string` or `service`->`tags`). Tags and nore-meta are also used in `/catalog/service` requests. - Do not require a call to the catalog if services are specified by name. This is important because on large cluster `/catalog/services` changes all the time. - Add `allow_stale` configuration option to do stale reads. Non-stale reads can be costly, even more when you are doing them to a remote datacenter with 10k+ targets over WAN (which is common for federation). - Add `refresh_interval` to minimize the strain on the catalog and on the service endpoint. This is needed because of that kind of behavior from consul: https://github.com/hashicorp/consul/issues/3712 and because a catalog on a large cluster would basically change *all* the time. No need to discover targets in 1sec if we scrape them every minute. - Added plenty of unit tests. Benchmarks ---------- ```yaml scrape_configs: - job_name: prometheus scrape_interval: 60s static_configs: - targets: ["127.0.0.1:9090"] - job_name: "observability-by-tag" scrape_interval: "60s" metrics_path: "/metrics" consul_sd_configs: - server: consul.service.par.consul.prod.crto.in:8500 tag: marathon-user-observability # Used in After refresh_interval: 30s # Used in After+delay relabel_configs: - source_labels: [__meta_consul_tags] regex: ^(.*,)?marathon-user-observability(,.*)?$ action: keep - job_name: "observability-by-name" scrape_interval: "60s" metrics_path: "/metrics" consul_sd_configs: - server: consul.service.par.consul.prod.crto.in:8500 services: - observability-cerebro - observability-portal-web - job_name: "fake-fake-fake" scrape_interval: "15s" metrics_path: "/metrics" consul_sd_configs: - server: consul.service.par.consul.prod.crto.in:8500 services: - fake-fake-fake ``` Note: tested with ~1200 services, ~5000 nodes. | Resource | Empty | Before | After | After + delay | | -------- |:-----:|:------:|:-----:|:-------------:| |/service-discovery size|5K|85MiB|27k|27k|27k| |`go_memstats_heap_objects`|100k|1M|120k|110k| |`go_memstats_heap_alloc_bytes`|24MB|150MB|28MB|27MB| |`rate(go_memstats_alloc_bytes_total[5m])`|0.2MB/s|28MB/s|2MB/s|0.3MB/s| |`rate(process_cpu_seconds_total[5m])`|0.1%|15%|2%|0.01%| |`process_open_fds`|16|*1236*|22|22| |`rate(prometheus_sd_consul_rpc_duration_seconds_count{call="services"}[5m])`|~0|1|1|*0.03*| |`rate(prometheus_sd_consul_rpc_duration_seconds_count{call="service"}[5m])`|0.1|*80*|0.5|0.5| |`prometheus_target_sync_length_seconds{quantile="0.9",scrape_job="observability-by-tag"}`|N/A|200ms|0.2ms|0.2ms| |Network bandwidth|~10kbps|~2.8Mbps|~1.6Mbps|~10kbps| Filtering by tag using relabel_configs uses **100kiB and 23kiB/s per service per job** and quite a lot of CPU. Also sends and additional *1Mbps* of traffic to consul. Being a little bit smarter about this reduces the overhead quite a lot. Limiting the number of `/catalog/services` queries per second almost removes the overhead of service discovery. * consul: tweak `refresh_interval` behavior `refresh_interval` now does what is advertised in the documentation, there won't be more that one update per `refresh_interval`. It now defaults to 30s (which was also the current waitTime in the consul query). This also make sure we don't wait another 30s if we already waited 29s in the blocking call by substracting the number of elapsed seconds. Hopefully this will do what people expect it does and will be safer for existing consul infrastructures.	7 years ago
pasquier-s	fc8cf08f42	Prevent invalid label names with labelmap (#3868) ... This change ensures that the relabeling configurations using labelmap can't generate invalid label names.	7 years ago
Shubheksha Jalan	0471e64ad1	Use shared types from the `common` repo (#3674) ... * refactor: use shared types from common repo, remove util/config * vendor: add common/config * fix nit	7 years ago
Shubheksha Jalan	ec94df49d4	Refactor SD configuration to remove `config` dependency (#3629) ... * refactor: move targetGroup struct and CheckOverflow() to their own package * refactor: move auth and security related structs to a utility package, fix import error in utility package * refactor: Azure SD, remove SD struct from config * refactor: DNS SD, remove SD struct from config into dns package * refactor: ec2 SD, move SD struct from config into the ec2 package * refactor: file SD, move SD struct from config to file discovery package * refactor: gce, move SD struct from config to gce discovery package * refactor: move HTTPClientConfig and URL into util/config, fix import error in httputil * refactor: consul, move SD struct from config into consul discovery package * refactor: marathon, move SD struct from config into marathon discovery package * refactor: triton, move SD struct from config to triton discovery package, fix test * refactor: zookeeper, move SD structs from config to zookeeper discovery package * refactor: openstack, remove SD struct from config, move into openstack discovery package * refactor: kubernetes, move SD struct from config into kubernetes discovery package * refactor: notifier, use targetgroup package instead of config * refactor: tests for file, marathon, triton SD - use targetgroup package instead of config.TargetGroup * refactor: retrieval, use targetgroup package instead of config.TargetGroup * refactor: storage, use config util package * refactor: discovery manager, use targetgroup package instead of config.TargetGroup * refactor: use HTTPClient and TLS config from configUtil instead of config * refactor: tests, use targetgroup package instead of config.TargetGroup * refactor: fix tagetgroup.Group pointers that were removed by mistake * refactor: openstack, kubernetes: drop prefixes * refactor: remove import aliases forced due to vscode bug * refactor: move main SD struct out of config into discovery/config * refactor: rename configUtil to config_util * refactor: rename yamlUtil to yaml_config * refactor: kubernetes, remove prefixes * refactor: move the TargetGroup package to discovery/ * refactor: fix order of imports	7 years ago
Alberto Cortés	29da2fb9cd	testutil: update to go1.9 testing.Helper	7 years ago
Alberto Cortés	8f6a9f7833	config: simplify tests by using testutil.NotOk (#3289) ... Also include filename in all LoadFile errors Also add mesage to testuitl.NotOk so we can identify failing tests when using table driven tests.	7 years ago
Tobias Schmidt	7098c56474	Add remote read filter option ... For special remote read endpoints which have only data for specific queries, it is desired to limit the number of queries sent to the configured remote read endpoint to reduce latency and performance overhead.	7 years ago
Thibault Chataigner	bf4a279a91	Remote storage reads based on oldest timestamp in primary storage (#3129) ... Currently all read queries are simply pushed to remote read clients. This is fine, except for remote storage for wich it unefficient and make query slower even if remote read is unnecessary. So we need instead to compare the oldest timestamp in primary/local storage with the query range lower boundary. If the oldest timestamp is older than the mint parameter, then there is no need for remote read. This is an optionnal behavior per remote read client. Signed-off-by: Thibault Chataigner <t.chataigner@criteo.com>	7 years ago
Alberto Cortés	6c67296423	config: fix error message for unexpected result of yaml marshal	7 years ago
Alberto Cortés	0f3d8ea075	config: use testutil package	7 years ago
Fabian Reinartz	87918f3097	Merge branch 'master' into dev-2.0	7 years ago
Max Leonard Inden	1c96fbb992	Expose current Prometheus config via /status/config ... This PR adds the `/status/config` endpoint which exposes the currently loaded Prometheus config. This is the same config that is displayed on `/config` in the UI in YAML format. The response payload looks like such: ``` { "status": "success", "data": { "yaml": <CONFIG> } } ```	7 years ago
Fabian Reinartz	25f3e1c424	Merge branch 'master' into mergemaster	7 years ago
Yuki Ito	1bf3b91ae0	Make sure that url for remote_read/write is not nil (#3024)	7 years ago
Tom Wilkie	5169f990f9	Review feedback: add yaml struct tags, don't embed queue config. ... Also, rename QueueManageConfig to QueueConfig, for consistency with tags.	7 years ago
Tom Wilkie	454b661145	Make queue manager configurable.	7 years ago
Fabian Reinartz	dba7586671	Merge branch 'master' into dev-2.0	7 years ago
Fuente, Pablo Andres	902fafb8e7	Fixing tests for Windows ... Fixing the config/config_test, the discovery/file/file_test and the promql/promql_test tests for Windows. For most of the tests, the fix involved correct handling of path separators. In the case of the promql tests, the issue was related to the removal of the temporal directories used by the storage. The issue is that the RemoveAll() call returns an error when it tries to remove a directory which is not empty, which seems to be true due to some kind of process that is still running after closing the storage. To fix it I added some retries to the remove of the temporal directories. Adding tags file from Universal Ctags to .gitignore	7 years ago
Fabian Reinartz	65b087bcc1	config: resolve file SD paths relative to config	8 years ago
Roman Vynar	dbe2eb2afc	Hide consul token on UI. (#2797)	8 years ago
Julius Volz	240bb671e2	config: Fix overflow checking in global config (#2783)	8 years ago
Conor Broderick	6766123f93	Replace regex with Secret type and remarshal config to hide secrets (#2775)	8 years ago
Brian Akins	27d66628a1	Allow limiting Kubernetes service discover to certain namespaces ... Allow namespace discovery to be more easily extended in the future by using a struct rather than just a list. Rename fields for kubernetes namespace discovery	8 years ago
Julius Volz	525da88c35	Merge pull request #2479 from YKlausz/consul-tls ... Adding consul capability to connect via tls	8 years ago
Goutham Veeramachaneni	5c89cec65c	Stricter Relabel Config Checking for Labeldrop/keep (#2510) ... * Minor code cleanup * Labeldrop/Labelkeep Now *Only* Support Regex Ref promtheus/prometheus#2368	8 years ago
yklausz	75880b594f	Adding consul capability to connect via tls	8 years ago
Julius Volz	e9476b35d5	Re-add multiple remote writers ... Each remote write endpoint gets its own set of relabeling rules. This is based on the (yet-to-be-merged) https://github.com/prometheus/prometheus/pull/2419, which removes legacy remote write implementations.	8 years ago
Fabian Reinartz	7eb849e6a8	Merge pull request #2307 from joyent/triton_discovery ... Add Joyent Triton discovery	8 years ago
Richard Kiene	f3d9692d09	Add Joyent Triton discovery	8 years ago
Björn Rabenstein	ad40d0abbc	Merge pull request #2288 from prometheus/limit-scrape ... Add ability to limit scrape samples, and related metrics	8 years ago
Brian Brazil	30448286c7	Add sample_limit to scrape config. ... This imposes a hard limit on the number of samples ingested from the target. This is counted after metric relabelling, to allow dropping of problemtic metrics. This is intended as a very blunt tool to prevent overload due to misbehaving targets that suddenly jump in sample count (e.g. adding a label containing email addresses). Add metric to track how often this happens. Fixes #2137	8 years ago
Tristan Colgate-McFarlane	4d9134e6d8	Add labeldrop and labelkeep actions. (#2279) ... Introduce two new relabel actions. labeldrop, and labelkeep. These can be used to filter the set of labels by matching regex - labeldrop: drops all labels that match the regex - labelkeep: drops all labels that do not match the regex	8 years ago
Fabian Reinartz	3fb4d1191b	config: rename AlertingConfig, resolve file paths	8 years ago
Fabian Reinartz	183c5749b9	config: add Alertmanager configuration	8 years ago
Fabian Reinartz	200bbe1bad	config: extract SD and HTTPClient configurations	8 years ago
Fabian Reinartz	ec66082749	Merge branch 'ec2_sd_profile_support' of https://github.com/Ticketmaster/prometheus into Ticketmaster-ec2_sd_profile_support	8 years ago
Kraig Amador	bec6870ed4	ec2_sd_configs: Support profiles for configuring the ec2 service	8 years ago
beorn7	b2f28a9e82	Merge branch 'release-1.2'	8 years ago
Brian Brazil	d1ece12c70	Handle null Regex in the config as the empty regex. (#2150)	8 years ago
bekbulatov	2bc12fa2fb	Set timeout for marathon_sd	8 years ago
bekbulatov	c689b35858	Merge branch 'master' into marathon_tls	8 years ago
Matti Savolainen	ec6524ce74	test the labelTarget regex to make sure it properly validates pre-interpolated label names.	8 years ago
Matti Savolainen	56907ba6e3	Add interpolation to good test config. Fix regex	8 years ago
Björn Rabenstein	d93f73874f	Merge pull request #2093 from dominikschulz/spelling ... Trivial spelling corrections	8 years ago
Dominik Schulz	182e17958a	Trivial spelling corrections and a small comment.	8 years ago
bekbulatov	ac702f66eb	Resolve merge conflicts	8 years ago
Frederic Branczyk	2e18c81a00	config: adapt unit tests	8 years ago
bekbulatov	01b53c1180	Add tls support	8 years ago
Brian Brazil	77605649a9	Add support for remote write relabelling. ... Switch back to a single remote writer, as we were only ever meant to have one and the relabel semantics are clearer that way.	8 years ago
Donatas Abraitis	1aa8898b66	Allow number to be the first letter as well for `job_name`	8 years ago
Alexey Miroshkin	e29d9394e5	Forbid invalid relabel configurations ... This fix adds check if target_label value is set in case if action is replace or hashmod Issue [#1900]	8 years ago
Fabian Reinartz	7a0b3af0b7	config: validate Kubernetes role correctly.	8 years ago
Fabian Reinartz	7221228843	discovery/kubernetes: select between discovery role ... This adds `role` field to the Kubernetes SD config, which indicates which type of Kubernetes SD should be run. This no longer allows discovering pods and nodes with the same SD configuration for example.	9 years ago
Fabian Reinartz	0f21bd31ca	config: deprecate `target_groups` for `static_configs` ... This change deprecates the `target_groups` option in favor of `static_configs`. The old configuration is still accepted but prints a warning. Configuration loading errors if both options are set.	9 years ago
Jimmi Dyson	206bcfcdaa	Kubernetes SD: Remove kubeletPort config option	9 years ago
Ali Reza	c81b4e8a87	change config names to files for consistency	9 years ago
Seth Miller	0988e3b937	Add support for Azure discovery ... This change adds the ability to do target discovery with Microsoft's Azure platform.	9 years ago
Julius Volz	829a029dda	Update two more __meta_dns_srv_name references. ... Although they are only in examples/tests and don't affect anything, they could be confusing (the label has been renamed in the rest of the code a while ago).	9 years ago
Fabian Reinartz	e26e4b6e89	Restrict scrape timeout to interval length	9 years ago
beorn7	a7408bfb47	Unify duration parsing ... It's actually happening in several places (and for flags, we use the standard Go time.Duration...). This at least reduces all our home-grown parsing to one place (in model).	9 years ago
Julien Dehee	061fe2f364	Support AirBnB's Smartstack Nerve client for SD ... nerve's registration format differs from serverset. With this commit there is now a dedicated treecache file in util, and two separate files for serverset and nerve. Reference: https://github.com/airbnb/nerve	9 years ago
Fabian Reinartz	4d1c9296d5	Add new defaults for relabel configurations	9 years ago
Brian Brazil	fd2bd81cd8	Allow all instance labels in target groups ... With the blackbox exporter, the instance label will commonly be used for things other than hostnames so remove this restriction. https://example.com or https://example.com/probe/me are some examples. To prevent user error, check that urls aren't provided as targets when there's no relabelling that could potentically fix them.	9 years ago
Jimmi Dyson	87940ec213	Kubernetes SD: Rename `masters` to `api_servers` in config	9 years ago
Jimmi Dyson	7ff5cc66ea	Kubernetes SD authentication options cleanup	9 years ago
Brian Brazil	1ddf75240d	config: Don't hide username, it's not secret. ... Usernames are not generally considered to be secrets, and treating them as secrets may lead to confusion as to how secure they are. Obscuring them also makes debugging harder.	9 years ago
Matt Jibson	dcb4856d72	Add SD for Amazon EC2 instances	9 years ago
Julius Volz	dac26cef71	Rename global "labels" config option to "external_labels".	9 years ago
Matt Jibson	0e99fa6c46	Allow labelmap action	9 years ago
Jimmi Dyson	a1574aa2b3	Move TLS options to scrape config ... Fixes #1013, fixes #989	9 years ago
Fabian Reinartz	1ef5ed0cf2	Merge pull request #1048 from xperimental/fix/marathon-config ... Fix parsing Marathon SD config	9 years ago
Robert Jacob	eb7416e71f	Fix missing unmarshal for Marathon SD config.	9 years ago
Jimmi Dyson	d7a7fd4589	Kubernetes SD improvements ... * Support multiple masters with retries against each master as required. * Scrape masters' metrics. * Add role meta label for node/service/master to make it easier for relabeling.	9 years ago
Julius Volz	f63a899744	Change config regexes to full-string matches. ... This anchors all regular expressions entered via the config to match a full string vs. a substring. THIS IS A BREAKING CHANGE! Fixes part of https://github.com/prometheus/prometheus/issues/996	9 years ago
Fabian Reinartz	438e232c9b	Fix grouping of import blocks	9 years ago
Fabian Reinartz	306e8468a0	Switch from client_golang/model to common/model	9 years ago
Fabian Reinartz	ac0be60bb9	Add license headers	9 years ago
Fabian Reinartz	139f27bf8a	Increase default retry interval for file SD ... The automatic refresh is a safety mechanism in case file watches fail. As they seem to be working well the interval can be increased.	9 years ago
Fabian Reinartz	b964da4b75	Merge pull request #905 from fabric8io/kubernetes-discovery ... Kubernetes discovery	9 years ago
Brian Brazil	43449b0581	config: Update tests/examples to use __tmp_	9 years ago
Jimmi Dyson	923f8111d4	Initial Kubernetes discovery ... Fixes #904	9 years ago
Fabian Reinartz	73f1cc807d	Check token and cert file existence in promtool	9 years ago
Fabian Reinartz	7a67472fc1	Resolve relative paths on configuration loading ... This moves the concern of resolving the files relative to the config file into the configuration loading itself. It also fixes #921 which did not load the cert and token files relatively.	9 years ago
Jimmi Dyson	52cf6b3e6e	Configuration options for bearer tokens, client certs & CA certs ... Fixes #918, fixes #917	9 years ago
Johannes 'fish' Ziemke	9ab340e95e	Add support for A record based DNS SD ... If using A records, the user needs to specify "port" and set "type" to "A".	9 years ago
Fabian Reinartz	187fe4e3d3	Fix missing defaults for empty global config blocks	9 years ago
Fabian Reinartz	2a53b107c1	Fix missing defaults in empty configurations	9 years ago
Fabian Reinartz	02e06839f2	config: hide authentication credentials in String() output	10 years ago
Brian Brazil	682f949ab1	Add 'hashmod' relabel action. ... This takes the modulus of a hash of some labels. Combined with a keep relabel action, this allows for sharding of targets across multiple prometheus servers.	10 years ago
Fabian Reinartz	dc7d27ab9a	retrieval: add honor label handling and parametrized querying. ... This commit adds the honor_labels and params arguments to the scrape config. This allows to specify query parameters used by the scrapers and handling scraped labels with precedence.	10 years ago
Brian Brazil	0dbae36d36	Allow ingested metrics to be relabeled. ... The main purpose of this is to allow for blacklisting of expensive metrics as a tactical option. It could also find uses for renaming and removing labels from federation.	10 years ago
Brian Brazil	58ceae82bc	Revert "Allow ingested metrics to be relabeled." ... This reverts commit f2f26ca08f. Was accidentally pushed to master instead of a branch for PR.	10 years ago
Brian Brazil	f2f26ca08f	Allow ingested metrics to be relabeled. ... The main purpose of this is to allow for blacklisting of expensive metrics as a tactical option. It could also find uses for renaming and removing labels from federation.	10 years ago
Fabian Reinartz	116e6df096	config: raise error on unknown config parameters. ... The YAML parser ignores additional parameters on unmarshaling. This causes frequent confusion with bad configs that pass parsing. These changes raise errors on additional parameters.	10 years ago
Fabian Reinartz	3a24a7779d	config: extend and format config example/test.	10 years ago
Fabian Reinartz	458550560c	config: error on missing regex in relabel config. ... Fixes issue #787.	10 years ago
Fabian Reinartz	b5fe2e9afe	Merge pull request #773 from prometheus/fabxc/simple-cfg ... config: simplify default config handling.	10 years ago
Fabian Reinartz	f6c33a2347	config: prevent overwrite of DefaultGlobalConfig	10 years ago
Fabian Reinartz	db3367e83f	config: ensure correct labelname in JSON target group.	10 years ago
Fabian Reinartz	0af1cff8af	config: simplify default config handling.	10 years ago
Fabian Reinartz	c44ac7bc26	Load rule files from entire directories	10 years ago
Fabian Reinartz	93548a8882	Add initial file based service discovery. ... This commits adds file based service discovery which reads target groups from specified files. It detects changes based on file watches and regular refreshes.	10 years ago
Fabian Reinartz	3b21c7037a	Add file SD to configuration.	10 years ago
Fabian Reinartz	d5aa012fd0	Make HTTP basic auth configurable for scrape targets.	10 years ago
Fabian Reinartz	86087120dd	Replace example config with new YAML format.	10 years ago
Fabian Reinartz	5fbde88919	Switch config to YAML format.	10 years ago
Fabian Reinartz	b5a8f7b8fa	Cleanup, test, and document config.	10 years ago
Fabian Reinartz	0b619b46d6	Change JobConfig to ScrapeConfig. ... This commit changes the configuration interface from job configs to scrape configs. This includes allowing multiple ways of target definition at once and moving DNS SD to its own config message. DNS SD can now contain multiple DNS names per configured discovery.	10 years ago
Bjoern Rabenstein	5859b74f1b	Clean up license issues. ... - Move CONTRIBUTORS.md to the more common AUTHORS. - Added the required NOTICE file. - Changed "Prometheus Team" to "The Prometheus Authors". - Reverted the erroneous changes to the Apache License.	10 years ago
Bjoern Rabenstein	b1e4956142	Apply a giant code cleanup. ... Essentially: - Remove unused code. - Make it 'go vet' clean. The only remaining warnings are in generated code. - Make it 'golint' clean. The only remaining warnings are in gerenated code. - Smoothed out same minor things. Change-Id: I3fe5c1fbead27b0e7a9c247fee2f5a45bc2d42c6	10 years ago
Julius Volz	78ebc1a61f	Ensure that job names are unique in parsed configs. ... Change-Id: I6bd89e6401bd924315981db797af21bdf0b81252	11 years ago
Julius Volz	d9b4f98b44	Integrate DNS-SD support for discovering job targets.	12 years ago
Julius Volz	9cea5d9df8	Convert the Prometheus configuration to protocol buffers.	12 years ago
Julius Volz	c0601abf46	Implement initial no-op alert parsing and rule parsing tests.	12 years ago
Julius Volz	39411b516d	Add missing config test fixtures.	12 years ago
Julius Volz	d137362257	Config tests and config cleanups+fixes.	12 years ago