221 Commits (9fe8bcf4be00a7ebd73932df0cf39cc837cbb993)

Author	SHA1	Message	Date
Paul Gier	d24d2acd11	config: set target group source index during unmarshalling (#4245) ... * config: set target group source index during unmarshalling Fixes issue #4214 where the scrape pool is unnecessarily reloaded for a config reload where the config hasn't changed. Previously, the discovery manager changed the static config after loading which caused the in-memory config to differ from a freshly reloaded config. Signed-off-by: Paul Gier <pgier@redhat.com> * [issue #4214] Test that static targets are not modified by discovery manager Signed-off-by: Paul Gier <pgier@redhat.com>	7 years ago
Philippe Laflamme	2aba238f31	Use common HTTPClientConfig for marathon_sd configuration (#4009) ... This adds support for basic authentication which closes #3090 The support for specifying the client timeout was removed as discussed in https://github.com/prometheus/common/pull/123. Marathon was the only sd mechanism doing this and configuring the timeout is done through `Context`. DC/OS uses a custom `Authorization` header for authenticating. This adds 2 new configuration properties to reflect this. Existing configuration files that use the bearer token will no longer work. More work is required to make this backwards compatible.	7 years ago
Manos Fokas	25f929b772	Yaml UnmarshalStrict implementation. (#4033) ... * Updated yaml vendor package. * remove checkOverflow duplicate in rulefmt * remove duplicated HTTPClientConfig.Validate() * Added yaml static check.	7 years ago
Kristiyan Nikolov	be85ba3842	discovery/ec2: Support filtering instances in discovery (#4011)	7 years ago
Corentin Chary	60dafd425c	consul: improve consul service discovery (#3814) ... * consul: improve consul service discovery Related to #3711 - Add the ability to filter by tag and node-meta in an efficient way (`/catalog/services` allow filtering by node-meta, and returns a `map[string]string` or `service`->`tags`). Tags and nore-meta are also used in `/catalog/service` requests. - Do not require a call to the catalog if services are specified by name. This is important because on large cluster `/catalog/services` changes all the time. - Add `allow_stale` configuration option to do stale reads. Non-stale reads can be costly, even more when you are doing them to a remote datacenter with 10k+ targets over WAN (which is common for federation). - Add `refresh_interval` to minimize the strain on the catalog and on the service endpoint. This is needed because of that kind of behavior from consul: https://github.com/hashicorp/consul/issues/3712 and because a catalog on a large cluster would basically change *all* the time. No need to discover targets in 1sec if we scrape them every minute. - Added plenty of unit tests. Benchmarks ---------- ```yaml scrape_configs: - job_name: prometheus scrape_interval: 60s static_configs: - targets: ["127.0.0.1:9090"] - job_name: "observability-by-tag" scrape_interval: "60s" metrics_path: "/metrics" consul_sd_configs: - server: consul.service.par.consul.prod.crto.in:8500 tag: marathon-user-observability # Used in After refresh_interval: 30s # Used in After+delay relabel_configs: - source_labels: [__meta_consul_tags] regex: ^(.*,)?marathon-user-observability(,.*)?$ action: keep - job_name: "observability-by-name" scrape_interval: "60s" metrics_path: "/metrics" consul_sd_configs: - server: consul.service.par.consul.prod.crto.in:8500 services: - observability-cerebro - observability-portal-web - job_name: "fake-fake-fake" scrape_interval: "15s" metrics_path: "/metrics" consul_sd_configs: - server: consul.service.par.consul.prod.crto.in:8500 services: - fake-fake-fake ``` Note: tested with ~1200 services, ~5000 nodes. | Resource | Empty | Before | After | After + delay | | -------- |:-----:|:------:|:-----:|:-------------:| |/service-discovery size|5K|85MiB|27k|27k|27k| |`go_memstats_heap_objects`|100k|1M|120k|110k| |`go_memstats_heap_alloc_bytes`|24MB|150MB|28MB|27MB| |`rate(go_memstats_alloc_bytes_total[5m])`|0.2MB/s|28MB/s|2MB/s|0.3MB/s| |`rate(process_cpu_seconds_total[5m])`|0.1%|15%|2%|0.01%| |`process_open_fds`|16|*1236*|22|22| |`rate(prometheus_sd_consul_rpc_duration_seconds_count{call="services"}[5m])`|~0|1|1|*0.03*| |`rate(prometheus_sd_consul_rpc_duration_seconds_count{call="service"}[5m])`|0.1|*80*|0.5|0.5| |`prometheus_target_sync_length_seconds{quantile="0.9",scrape_job="observability-by-tag"}`|N/A|200ms|0.2ms|0.2ms| |Network bandwidth|~10kbps|~2.8Mbps|~1.6Mbps|~10kbps| Filtering by tag using relabel_configs uses **100kiB and 23kiB/s per service per job** and quite a lot of CPU. Also sends and additional *1Mbps* of traffic to consul. Being a little bit smarter about this reduces the overhead quite a lot. Limiting the number of `/catalog/services` queries per second almost removes the overhead of service discovery. * consul: tweak `refresh_interval` behavior `refresh_interval` now does what is advertised in the documentation, there won't be more that one update per `refresh_interval`. It now defaults to 30s (which was also the current waitTime in the consul query). This also make sure we don't wait another 30s if we already waited 29s in the blocking call by substracting the number of elapsed seconds. Hopefully this will do what people expect it does and will be safer for existing consul infrastructures.	7 years ago
pasquier-s	fc8cf08f42	Prevent invalid label names with labelmap (#3868) ... This change ensures that the relabeling configurations using labelmap can't generate invalid label names.	7 years ago
Shubheksha Jalan	0471e64ad1	Use shared types from the `common` repo (#3674) ... * refactor: use shared types from common repo, remove util/config * vendor: add common/config * fix nit	7 years ago
Shubheksha Jalan	ec94df49d4	Refactor SD configuration to remove `config` dependency (#3629) ... * refactor: move targetGroup struct and CheckOverflow() to their own package * refactor: move auth and security related structs to a utility package, fix import error in utility package * refactor: Azure SD, remove SD struct from config * refactor: DNS SD, remove SD struct from config into dns package * refactor: ec2 SD, move SD struct from config into the ec2 package * refactor: file SD, move SD struct from config to file discovery package * refactor: gce, move SD struct from config to gce discovery package * refactor: move HTTPClientConfig and URL into util/config, fix import error in httputil * refactor: consul, move SD struct from config into consul discovery package * refactor: marathon, move SD struct from config into marathon discovery package * refactor: triton, move SD struct from config to triton discovery package, fix test * refactor: zookeeper, move SD structs from config to zookeeper discovery package * refactor: openstack, remove SD struct from config, move into openstack discovery package * refactor: kubernetes, move SD struct from config into kubernetes discovery package * refactor: notifier, use targetgroup package instead of config * refactor: tests for file, marathon, triton SD - use targetgroup package instead of config.TargetGroup * refactor: retrieval, use targetgroup package instead of config.TargetGroup * refactor: storage, use config util package * refactor: discovery manager, use targetgroup package instead of config.TargetGroup * refactor: use HTTPClient and TLS config from configUtil instead of config * refactor: tests, use targetgroup package instead of config.TargetGroup * refactor: fix tagetgroup.Group pointers that were removed by mistake * refactor: openstack, kubernetes: drop prefixes * refactor: remove import aliases forced due to vscode bug * refactor: move main SD struct out of config into discovery/config * refactor: rename configUtil to config_util * refactor: rename yamlUtil to yaml_config * refactor: kubernetes, remove prefixes * refactor: move the TargetGroup package to discovery/ * refactor: fix order of imports	7 years ago
Brian Brazil	fba80da635	Fix default of read_recent to be false. (#3617) ... This is what is documented in the migration guide, and the default settings should make sense for a true long term storage. Document the setting.	7 years ago
Krasi Georgiev	e405e2f1ea	refactored discovery	7 years ago
Alberto Cortés	29da2fb9cd	testutil: update to go1.9 testing.Helper	7 years ago
Alberto Cortés	8f6a9f7833	config: simplify tests by using testutil.NotOk (#3289) ... Also include filename in all LoadFile errors Also add mesage to testuitl.NotOk so we can identify failing tests when using table driven tests.	7 years ago
Tobias Schmidt	7098c56474	Add remote read filter option ... For special remote read endpoints which have only data for specific queries, it is desired to limit the number of queries sent to the configured remote read endpoint to reduce latency and performance overhead.	7 years ago
Krasi Georgiev	e86d82ad2d	Fix regression of alert rules state loss on config reload. (#3382) ... * incorrect map name for the group prevented copying state from existing alert rules on config reload * applyConfig test * few nits * nits 2	7 years ago
Thibault Chataigner	bf4a279a91	Remote storage reads based on oldest timestamp in primary storage (#3129) ... Currently all read queries are simply pushed to remote read clients. This is fine, except for remote storage for wich it unefficient and make query slower even if remote read is unnecessary. So we need instead to compare the oldest timestamp in primary/local storage with the query range lower boundary. If the oldest timestamp is older than the mint parameter, then there is no need for remote read. This is an optionnal behavior per remote read client. Signed-off-by: Thibault Chataigner <t.chataigner@criteo.com>	7 years ago
Alberto Cortés	6c67296423	config: fix error message for unexpected result of yaml marshal	7 years ago
Alberto Cortés	0f3d8ea075	config: use testutil package	7 years ago
Fabian Reinartz	2d0b8e8b94	Merge branch 'master' into dev-2.0	7 years ago
Alberto Cortés	bb3dad9cba	config: simplify some returns	7 years ago
Bryan Boreham	9d6b945e41	Default HTTP keep-alive ON for remote read/write	7 years ago
Bryan Boreham	e0a4d18301	Allow http keep-alive setting to be overridden in config	7 years ago
Fabian Reinartz	e746282772	Merge branch 'master' into dev-2.0	7 years ago
Jamie Moore	7a135e0a1b	Add the ability to assume a role for ec2 discovery	7 years ago
Fabian Reinartz	87918f3097	Merge branch 'master' into dev-2.0	7 years ago
Johannes 'fish' Ziemke	70f3d1e9f9	k8s: Support discovery of ingresses (#3111) ... * k8s: Support discovery of ingresses * Move additional labels below allocation This makes it more obvious why the additional elements are allocated. Also fix allocation for node where we only set a single label. * k8s: Remove port from ingress discovery * k8s: Add comment to ingress discovery example	7 years ago
CuiHaozhi	b1c18bf29b	discovery openstack: support discovery hosts, add rule option. ... Signed-off-by: CuiHaozhi <cuihz@wise2c.com>	7 years ago
Max Leonard Inden	1c96fbb992	Expose current Prometheus config via /status/config ... This PR adds the `/status/config` endpoint which exposes the currently loaded Prometheus config. This is the same config that is displayed on `/config` in the UI in YAML format. The response payload looks like such: ``` { "status": "success", "data": { "yaml": <CONFIG> } } ```	7 years ago
Fabian Reinartz	25f3e1c424	Merge branch 'master' into mergemaster	7 years ago
Yuki Ito	1bf3b91ae0	Make sure that url for remote_read/write is not nil (#3024)	7 years ago
Tom Wilkie	5169f990f9	Review feedback: add yaml struct tags, don't embed queue config. ... Also, rename QueueManageConfig to QueueConfig, for consistency with tags.	7 years ago
Tom Wilkie	454b661145	Make queue manager configurable.	7 years ago
Fabian Reinartz	dba7586671	Merge branch 'master' into dev-2.0	7 years ago
Fuente, Pablo Andres	9eb8c6e1d2	Renaming the config_notwin test to config_default	7 years ago
Fuente, Pablo Andres	fe73de9452	Renaming config test file to fix build tags ... Renaming the name of a file of the config tests, in order to properly use the Go build tags feature.	7 years ago
Fuente, Pablo Andres	193dc47230	Fixing code style to adhere gofmt	7 years ago
Fuente, Pablo Andres	902fafb8e7	Fixing tests for Windows ... Fixing the config/config_test, the discovery/file/file_test and the promql/promql_test tests for Windows. For most of the tests, the fix involved correct handling of path separators. In the case of the promql tests, the issue was related to the removal of the temporal directories used by the storage. The issue is that the RemoveAll() call returns an error when it tries to remove a directory which is not empty, which seems to be true due to some kind of process that is still running after closing the storage. To fix it I added some retries to the remove of the temporal directories. Adding tags file from Universal Ctags to .gitignore	7 years ago
Goutham Veeramachaneni	98d20d5880	Make sure rendering config produces valid config ... Fixes #2899 Signed-off-by: Goutham Veeramachaneni <goutham@boomerangcommerce.com>	7 years ago
Fabian Reinartz	65b087bcc1	config: resolve file SD paths relative to config	8 years ago
Fabian Reinartz	9ba61df45a	Merge pull request #2789 from mtanda/aws_default_region ... config: set default region for EC2 SD	8 years ago
Mitsuhiro Tanda	64cef5cd05	handle NewSession() error	8 years ago
Christian Groschupp	8f781e411c	Openstack Service Discovery (#2701) ... * Add openstack service discovery. * Add gophercloud code for openstack service discovery. * first changes for juliusv comments. * add gophercloud code for floatingip. * Add tests to openstack sd. * Add testify suite vendor files. * add copyright and make changes for code climate. * Fixed typos in provider openstack. * Renamed tenant to project in openstack sd. * Change type of password to Secret in openstack sd.	8 years ago
Roman Vynar	dbe2eb2afc	Hide consul token on UI. (#2797)	8 years ago
Mitsuhiro Tanda	a1ddab463e	config: set default region for EC2 SD	8 years ago
Tobias Schmidt	287ec6e6cc	Fix outdated target_group naming in error message ... The target_groups config has been renamed to static_configs, the error message for overflow attributes should reflect that.	8 years ago
Julius Volz	240bb671e2	config: Fix overflow checking in global config (#2783)	8 years ago
Conor Broderick	6766123f93	Replace regex with Secret type and remarshal config to hide secrets (#2775)	8 years ago
Brian Akins	27d66628a1	Allow limiting Kubernetes service discover to certain namespaces ... Allow namespace discovery to be more easily extended in the future by using a struct rather than just a list. Rename fields for kubernetes namespace discovery	8 years ago
Julius Volz	eb14678a25	Make remote read/write use config.HTTPClientConfig	8 years ago
Julius Volz	02395a224d	[WIP] Remote Read	8 years ago
Julius Volz	525da88c35	Merge pull request #2479 from YKlausz/consul-tls ... Adding consul capability to connect via tls	8 years ago