prometheus

Commit Graph

Author	SHA1	Message	Date
Julien Pivotto	8920024323	Add PuppetDB service discovery We have been Puppet user for 10 years and we are users of https://github.com/camptocamp/prometheus-puppetdb-sd However, that file_sd implementation contains business logic and assumptions around e.g. the modules which you are using. This pull request adds a simple PuppetDB service discovery, which will enable more use cases than the upstream sd. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	3 years ago
DrAuYueng	e8be1d0a5c	Check relabel action at yaml unmarshal stage (#9224 ) Signed-off-by: DrAuYueng <ouyang1204@gmail.com>	3 years ago
austin ce	bbc951f50b	Add config tests for kuma SD Signed-off-by: austin ce <austin.cawley@gmail.com>	3 years ago
Julien Pivotto	17700e5600	Fix yaml indent to make CI happy Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	3 years ago
Ben Kochie	7cb55d5732	Merge pull request #8802 from mwasilew2/yaml-linting Adds yamllinting to Makefile.common	3 years ago
3Xpl0it3r	a0bac4b488	add kubeconfig support in discovery module (#8811 ) Signed-off-by: 3Xpl0it3r <shouc.wang@hotmail.com>	3 years ago
Michal Wasilewski	3f686cad8b	fixes yamllint errors Signed-off-by: Michal Wasilewski <mwasilewski@gmx.com>	4 years ago
Julien Pivotto	9444698ae2	http_sd (#8839 ) Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Julien Pivotto	20c6739adc	Merge pull request #8833 from hanjm/feature/add-scape-read-body-limit Add body_size_limit to prevent bad targets response large body cause Prometheus server OOM (#8827)	4 years ago
TJ Hoplock	dc22c65349	Add Linode Service Discovery (#8846 ) * Add Linode Service Discovery Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	4 years ago
hanjm	1df05bfd49	Add body_size_limit to prevent bad targets response large body cause Prometheus server OOM (#8827 ) Signed-off-by: hanjm <hanjinming@outlook.com>	4 years ago
Levi Harrison	fa184a5fc3	Add OAuth 2.0 Config (#8761 ) * Introduced oauth2 config into the codebase Signed-off-by: Levi Harrison <git@leviharrison.dev>	4 years ago
n888	7c028d59c2	Add lightsail service discovery (#8693 ) Signed-off-by: N888 <drifto@gmail.com>	4 years ago
Julien Pivotto	e635ca834b	Add environment variable expansion in external label values Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Robert Jacob	b253056163	Implement Docker discovery (#8629 ) * Implement Docker discovery Signed-off-by: Robert Jacob <xperimental@solidproject.de>	4 years ago
Julien Pivotto	5a6d244b00	Scaleway SD: Add the ability to read token from file Prometheus adds the ability to read secrets from files. This add this feature for the scaleway service discovery. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Rémy Léone	f690b811c5	add support for scaleway service discovery (#8555 ) Co-authored-by: Patrik <patrik@ptrk.io> Co-authored-by: Julien Pivotto <roidelapluie@inuits.eu> Signed-off-by: Rémy Léone <rleone@scaleway.com>	4 years ago
Harkishen-Singh	79ba53a6c4	Custom headers on remote-read and refactor implementation to roundtripper. Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>	4 years ago
Julien Pivotto	8787f0aed7	Update common to support credentials type Most of the backwards compat tests is done in common. Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Harkishen-Singh	77c20fd2f8	Adds support to configure retry on Rate-Limiting from remote-write config. Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>	4 years ago
Nándor István Krácser	509000269a	remote_write: allow passing along custom HTTP headers (#8416 ) * remote_write: allow passing along custom HTTP headers Signed-off-by: Nandor Kracser <bonifaido@gmail.com> * add warning Signed-off-by: Nandor Kracser <bonifaido@gmail.com> * remote_write: add header valadtion Signed-off-by: Nandor Kracser <bonifaido@gmail.com> * extend tests for bad remote write headers Signed-off-by: Nandor Kracser <bonifaido@gmail.com> * remote_write: add note about the authorization header Signed-off-by: Nandor Kracser <bonifaido@gmail.com>	4 years ago
Alexey Shumkin	73ddf603af	discovery/kubernetes: Fix valid label selector causing config error Label selector can be "set-based"(https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#set-based-requirement) but such a selector causes Prometheus start failure with the "unexpected error: parsing YAML file ...: invalid selector: 'foo in (bar,baz)'; can't understand 'baz)'"-like error. This is caused by the `fields.ParseSelector(string)` function that simply splits an expression as a CSV-list, so a comma confuses such a parsing method and lead to the error. Use `labels.Parse(string)` to use a valid lexer to parse a selector expression. Closes #8284. Signed-off-by: Alexey Shumkin <Alex.Crezoff@gmail.com>	4 years ago
kangwoo	7c0d5ae4e7	Add Eureka Service Discovery (#3369 ) Signed-off-by: kangwoo <kangwoo@gmail.com>	4 years ago
Lukas Kämmerling	b6955bf1ca	Add hetzner service discovery (#7822 ) Signed-off-by: Lukas Kämmerling <lukas.kaemmerling@hetzner-cloud.de>	4 years ago
Andy Bursavich	4e6a94a27d	Invert service discovery dependencies (#7701 ) This also fixes a bug in query_log_file, which now is relative to the config file like all other paths. Signed-off-by: Andy Bursavich <abursavich@gmail.com>	4 years ago
Julien Pivotto	f8ec72d730	Add digitalocean test Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Julien Pivotto	a197508d09	Add docker swarm test Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	4 years ago
Steffen Neubauer	9c9b872087	OpenStack SD: Add availability config option, to choose endpoint type (#7494 ) * OpenStack SD: Add availability config option, to choose endpoint type In some environments Prometheus must query OpenStack via an alternative endpoint type (gophercloud calls this `availability`. This commit implements this option. Co-Authored-By: Dennis Kuhn <d.kuhn@syseleven.de> Signed-off-by: Steffen Neubauer <s.neubauer@syseleven.de>	4 years ago
Aleksandra Gacek	8e53c19f9c	discovery/kubernetes: expose label_selector and field_selector Close #6807 Co-authored-by @shuttie Signed-off-by: Aleksandra Gacek <algacek@google.com>	5 years ago
Grebennikov Roman	b4445ff03f	discovery/kubernetes: expose label_selector and field_selector Closes #6096 Signed-off-by: Grebennikov Roman <grv@dfdx.me>	5 years ago
Callum Styan	67838643ee	Add config option for remote job name (#6043 ) * Track remote write queues via a map so we don't care about index. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Support a job name for remote write/read so we can differentiate between them using the name. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Remote write/read has Name to not confuse the meaning of the field with scrape job names. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Split queue/client label into remote_name and url labels. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Don't allow for duplicate remote write/read configs. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Ensure we restart remote write queues if the hash of their config has not changed, but the remote name has changed. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Include name in remote read/write config hashes, simplify duplicates check, update test accordingly. Signed-off-by: Callum Styan <callumstyan@gmail.com>	5 years ago
johncming	8d3083e256	config: add test case for scrape interval larger than timeout. (#6037 ) Signed-off-by: johncming <johncming@yahoo.com>	5 years ago
Callum Styan	5603b857a9	Check if label value is valid when unmarhsaling external labels from YAML, add a test to config_tests for valid/invalid external label value. Signed-off-by: Callum Styan <callumstyan@gmail.com>	6 years ago
Julien Pivotto	4397916cb2	Add honor_timestamps (#5304 ) Fixes #5302 Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>	6 years ago
Callum Styan	83c46fd549	update Consul vendor code so that catalog.ServiceMultipleTags can be (#5151 ) Signed-off-by: Callum Styan <callumstyan@gmail.com>	6 years ago
Simon Pasquier	027d2ece14	config: resolve more file paths (#5284 ) Signed-off-by: Simon Pasquier <spasquie@redhat.com>	6 years ago
Simon Pasquier	e72c875e63	config: fix Kubernetes config with empty API server (#5256 ) Signed-off-by: Simon Pasquier <spasquie@redhat.com>	6 years ago
Simon Pasquier	c8a1a5a93c	discovery/kubernetes: fix support for password_file and bearer_token_file (#5211 ) * discovery/kubernetes: fix support for password_file Signed-off-by: Simon Pasquier <spasquie@redhat.com> * Create and pass custom RoundTripper to Kubernetes client Signed-off-by: Simon Pasquier <spasquie@redhat.com> * Use inline HTTPClientConfig Signed-off-by: Simon Pasquier <spasquie@redhat.com>	6 years ago
Marcel D. Juhnke	c7d83b2b6a	discovery: add support for Managed Identity authentication in Azure SD (#4590 ) Signed-off-by: Marcel Juhnke <marrat@marrat.de>	6 years ago
Julius Volz	d28246e337	Fix config loading panics on nil pointer slice elements (#4942 ) Fixes https://github.com/prometheus/prometheus/issues/4902 Fixes https://github.com/prometheus/prometheus/issues/4889 Signed-off-by: Julius Volz <julius.volz@gmail.com>	6 years ago
mengnan	a5d39361ab	discovery/azure: Fail hard when Azure authentication parameters are missing (#4907 ) * discovery/azure: fail hard when client_id/client_secret is empty Signed-off-by: mengnan <supernan1994@gmail.com> * discovery/azure: fail hard when authentication parameters are missing Signed-off-by: mengnan <supernan1994@gmail.com> * add unit test Signed-off-by: mengnan <supernan1994@gmail.com> * add unit test Signed-off-by: mengnan <supernan1994@gmail.com> * format code Signed-off-by: mengnan <supernan1994@gmail.com>	6 years ago
Simon Pasquier	ff08c40091	discovery/openstack: support tls_config Signed-off-by: Simon Pasquier <spasquie@redhat.com>	6 years ago
Simon Pasquier	128ff546b8	config: add test for OpenStack SD (#4594 ) Signed-off-by: Simon Pasquier <spasquie@redhat.com>	6 years ago
Tariq Ibrahim	f708fd5c99	Adding support for multiple azure environments (#4569 ) Signed-off-by: Tariq Ibrahim <tariq.ibrahim@microsoft.com>	6 years ago
Philippe Laflamme	2aba238f31	Use common HTTPClientConfig for marathon_sd configuration (#4009 ) This adds support for basic authentication which closes #3090 The support for specifying the client timeout was removed as discussed in https://github.com/prometheus/common/pull/123. Marathon was the only sd mechanism doing this and configuring the timeout is done through `Context`. DC/OS uses a custom `Authorization` header for authenticating. This adds 2 new configuration properties to reflect this. Existing configuration files that use the bearer token will no longer work. More work is required to make this backwards compatible.	7 years ago
Manos Fokas	25f929b772	Yaml UnmarshalStrict implementation. (#4033 ) * Updated yaml vendor package. * remove checkOverflow duplicate in rulefmt * remove duplicated HTTPClientConfig.Validate() * Added yaml static check.	7 years ago
Kristiyan Nikolov	be85ba3842	discovery/ec2: Support filtering instances in discovery (#4011 )	7 years ago
Corentin Chary	60dafd425c	consul: improve consul service discovery (#3814 ) * consul: improve consul service discovery Related to #3711 - Add the ability to filter by tag and node-meta in an efficient way (`/catalog/services` allow filtering by node-meta, and returns a `map[string]string` or `service`->`tags`). Tags and nore-meta are also used in `/catalog/service` requests. - Do not require a call to the catalog if services are specified by name. This is important because on large cluster `/catalog/services` changes all the time. - Add `allow_stale` configuration option to do stale reads. Non-stale reads can be costly, even more when you are doing them to a remote datacenter with 10k+ targets over WAN (which is common for federation). - Add `refresh_interval` to minimize the strain on the catalog and on the service endpoint. This is needed because of that kind of behavior from consul: https://github.com/hashicorp/consul/issues/3712 and because a catalog on a large cluster would basically change all the time. No need to discover targets in 1sec if we scrape them every minute. - Added plenty of unit tests. Benchmarks ---------- ```yaml scrape_configs: - job_name: prometheus scrape_interval: 60s static_configs: - targets: ["127.0.0.1:9090"] - job_name: "observability-by-tag" scrape_interval: "60s" metrics_path: "/metrics" consul_sd_configs: - server: consul.service.par.consul.prod.crto.in:8500 tag: marathon-user-observability # Used in After refresh_interval: 30s # Used in After+delay relabel_configs: - source_labels: [__meta_consul_tags] regex: ^(.,)?marathon-user-observability(,.)?$ action: keep - job_name: "observability-by-name" scrape_interval: "60s" metrics_path: "/metrics" consul_sd_configs: - server: consul.service.par.consul.prod.crto.in:8500 services: - observability-cerebro - observability-portal-web - job_name: "fake-fake-fake" scrape_interval: "15s" metrics_path: "/metrics" consul_sd_configs: - server: consul.service.par.consul.prod.crto.in:8500 services: - fake-fake-fake ``` Note: tested with ~1200 services, ~5000 nodes. \| Resource \| Empty \| Before \| After \| After + delay \| \| -------- \|:-----:\|:------:\|:-----:\|:-------------:\| \|/service-discovery size\|5K\|85MiB\|27k\|27k\|27k\| \|`go_memstats_heap_objects`\|100k\|1M\|120k\|110k\| \|`go_memstats_heap_alloc_bytes`\|24MB\|150MB\|28MB\|27MB\| \|`rate(go_memstats_alloc_bytes_total[5m])`\|0.2MB/s\|28MB/s\|2MB/s\|0.3MB/s\| \|`rate(process_cpu_seconds_total[5m])`\|0.1%\|15%\|2%\|0.01%\| \|`process_open_fds`\|16\|1236\|22\|22\| \|`rate(prometheus_sd_consul_rpc_duration_seconds_count{call="services"}[5m])`\|~0\|1\|1\|0.03\| \|`rate(prometheus_sd_consul_rpc_duration_seconds_count{call="service"}[5m])`\|0.1\|80\|0.5\|0.5\| \|`prometheus_target_sync_length_seconds{quantile="0.9",scrape_job="observability-by-tag"}`\|N/A\|200ms\|0.2ms\|0.2ms\| \|Network bandwidth\|~10kbps\|~2.8Mbps\|~1.6Mbps\|~10kbps\| Filtering by tag using relabel_configs uses 100kiB and 23kiB/s per service per job and quite a lot of CPU. Also sends and additional 1Mbps of traffic to consul. Being a little bit smarter about this reduces the overhead quite a lot. Limiting the number of `/catalog/services` queries per second almost removes the overhead of service discovery. * consul: tweak `refresh_interval` behavior `refresh_interval` now does what is advertised in the documentation, there won't be more that one update per `refresh_interval`. It now defaults to 30s (which was also the current waitTime in the consul query). This also make sure we don't wait another 30s if we already waited 29s in the blocking call by substracting the number of elapsed seconds. Hopefully this will do what people expect it does and will be safer for existing consul infrastructures.	7 years ago
pasquier-s	fc8cf08f42	Prevent invalid label names with labelmap (#3868 ) This change ensures that the relabeling configurations using labelmap can't generate invalid label names.	7 years ago
Tobias Schmidt	7098c56474	Add remote read filter option For special remote read endpoints which have only data for specific queries, it is desired to limit the number of queries sent to the configured remote read endpoint to reduce latency and performance overhead.	7 years ago
Krasi Georgiev	e86d82ad2d	Fix regression of alert rules state loss on config reload. (#3382 ) * incorrect map name for the group prevented copying state from existing alert rules on config reload * applyConfig test * few nits * nits 2	7 years ago
Thibault Chataigner	bf4a279a91	Remote storage reads based on oldest timestamp in primary storage (#3129 ) Currently all read queries are simply pushed to remote read clients. This is fine, except for remote storage for wich it unefficient and make query slower even if remote read is unnecessary. So we need instead to compare the oldest timestamp in primary/local storage with the query range lower boundary. If the oldest timestamp is older than the mint parameter, then there is no need for remote read. This is an optionnal behavior per remote read client. Signed-off-by: Thibault Chataigner <t.chataigner@criteo.com>	7 years ago
Yuki Ito	1bf3b91ae0	Make sure that url for remote_read/write is not nil (#3024 )	7 years ago
Fuente, Pablo Andres	902fafb8e7	Fixing tests for Windows Fixing the config/config_test, the discovery/file/file_test and the promql/promql_test tests for Windows. For most of the tests, the fix involved correct handling of path separators. In the case of the promql tests, the issue was related to the removal of the temporal directories used by the storage. The issue is that the RemoveAll() call returns an error when it tries to remove a directory which is not empty, which seems to be true due to some kind of process that is still running after closing the storage. To fix it I added some retries to the remove of the temporal directories. Adding tags file from Universal Ctags to .gitignore	7 years ago
Roman Vynar	dbe2eb2afc	Hide consul token on UI. (#2797 )	8 years ago
Julius Volz	240bb671e2	config: Fix overflow checking in global config (#2783 )	8 years ago
Conor Broderick	6766123f93	Replace regex with Secret type and remarshal config to hide secrets (#2775 )	8 years ago
Brian Akins	27d66628a1	Allow limiting Kubernetes service discover to certain namespaces Allow namespace discovery to be more easily extended in the future by using a struct rather than just a list. Rename fields for kubernetes namespace discovery	8 years ago
Julius Volz	525da88c35	Merge pull request #2479 from YKlausz/consul-tls Adding consul capability to connect via tls	8 years ago
Goutham Veeramachaneni	5c89cec65c	Stricter Relabel Config Checking for Labeldrop/keep (#2510 ) * Minor code cleanup * Labeldrop/Labelkeep Now Only Support Regex Ref promtheus/prometheus#2368	8 years ago
yklausz	75880b594f	Adding consul capability to connect via tls	8 years ago
Julius Volz	e9476b35d5	Re-add multiple remote writers Each remote write endpoint gets its own set of relabeling rules. This is based on the (yet-to-be-merged) https://github.com/prometheus/prometheus/pull/2419, which removes legacy remote write implementations.	8 years ago
Fabian Reinartz	7eb849e6a8	Merge pull request #2307 from joyent/triton_discovery Add Joyent Triton discovery	8 years ago
Richard Kiene	f3d9692d09	Add Joyent Triton discovery	8 years ago
Björn Rabenstein	ad40d0abbc	Merge pull request #2288 from prometheus/limit-scrape Add ability to limit scrape samples, and related metrics	8 years ago
Brian Brazil	30448286c7	Add sample_limit to scrape config. This imposes a hard limit on the number of samples ingested from the target. This is counted after metric relabelling, to allow dropping of problemtic metrics. This is intended as a very blunt tool to prevent overload due to misbehaving targets that suddenly jump in sample count (e.g. adding a label containing email addresses). Add metric to track how often this happens. Fixes #2137	8 years ago
Tristan Colgate-McFarlane	4d9134e6d8	Add labeldrop and labelkeep actions. (#2279 ) Introduce two new relabel actions. labeldrop, and labelkeep. These can be used to filter the set of labels by matching regex - labeldrop: drops all labels that match the regex - labelkeep: drops all labels that do not match the regex	8 years ago
Fabian Reinartz	183c5749b9	config: add Alertmanager configuration	8 years ago
Fabian Reinartz	ec66082749	Merge branch 'ec2_sd_profile_support' of https://github.com/Ticketmaster/prometheus into Ticketmaster-ec2_sd_profile_support	8 years ago
Kraig Amador	bec6870ed4	ec2_sd_configs: Support profiles for configuring the ec2 service	8 years ago
beorn7	b2f28a9e82	Merge branch 'release-1.2'	8 years ago
Brian Brazil	d1ece12c70	Handle null Regex in the config as the empty regex. (#2150 )	8 years ago
bekbulatov	c689b35858	Merge branch 'master' into marathon_tls	8 years ago
Matti Savolainen	f867c1fd58	formating and text fixes, adjust regexp	8 years ago
Matti Savolainen	56907ba6e3	Add interpolation to good test config. Fix regex	8 years ago
Matti Savolainen	5a1e909b5d	Make TargetLabel in RelabelConfig a string	8 years ago
bekbulatov	ac702f66eb	Resolve merge conflicts	8 years ago
Fabian Reinartz	1b6dfa32a9	config: rename role 'endpoint' to 'endpoints'	8 years ago
Frederic Branczyk	2e18c81a00	config: adapt unit tests	8 years ago
bekbulatov	01b53c1180	Add tls support	8 years ago
Brian Brazil	77605649a9	Add support for remote write relabelling. Switch back to a single remote writer, as we were only ever meant to have one and the relabel semantics are clearer that way.	8 years ago
Donatas Abraitis	1aa8898b66	Allow number to be the first letter as well for `job_name`	8 years ago
Alexey Miroshkin	e29d9394e5	Forbid invalid relabel configurations This fix adds check if target_label value is set in case if action is replace or hashmod Issue [#1900]	8 years ago
Fabian Reinartz	7a0b3af0b7	config: validate Kubernetes role correctly.	8 years ago
Fabian Reinartz	7221228843	discovery/kubernetes: select between discovery role This adds `role` field to the Kubernetes SD config, which indicates which type of Kubernetes SD should be run. This no longer allows discovering pods and nodes with the same SD configuration for example.	9 years ago
Fabian Reinartz	0f21bd31ca	config: deprecate `target_groups` for `static_configs` This change deprecates the `target_groups` option in favor of `static_configs`. The old configuration is still accepted but prints a warning. Configuration loading errors if both options are set.	9 years ago
Ali Reza	c81b4e8a87	change config names to files for consistency	9 years ago
Seth Miller	0988e3b937	Add support for Azure discovery This change adds the ability to do target discovery with Microsoft's Azure platform.	9 years ago
Fabian Reinartz	37c709f917	Fix global config YAML issues	9 years ago
Julius Volz	829a029dda	Update two more __meta_dns_srv_name references. Although they are only in examples/tests and don't affect anything, they could be confusing (the label has been renamed in the rest of the code a while ago).	9 years ago
Fabian Reinartz	e26e4b6e89	Restrict scrape timeout to interval length	9 years ago
Julien Dehee	061fe2f364	Support AirBnB's Smartstack Nerve client for SD nerve's registration format differs from serverset. With this commit there is now a dedicated treecache file in util, and two separate files for serverset and nerve. Reference: https://github.com/airbnb/nerve	9 years ago
Fabian Reinartz	4d1c9296d5	Add new defaults for relabel configurations	9 years ago
Brian Brazil	fd2bd81cd8	Allow all instance labels in target groups With the blackbox exporter, the instance label will commonly be used for things other than hostnames so remove this restriction. https://example.com or https://example.com/probe/me are some examples. To prevent user error, check that urls aren't provided as targets when there's no relabelling that could potentically fix them.	9 years ago
Jimmi Dyson	87940ec213	Kubernetes SD: Rename `masters` to `api_servers` in config	9 years ago
Jimmi Dyson	7ff5cc66ea	Kubernetes SD authentication options cleanup	9 years ago
Matt Jibson	dcb4856d72	Add SD for Amazon EC2 instances	9 years ago
Julius Volz	dac26cef71	Rename global "labels" config option to "external_labels".	9 years ago
Matt Jibson	0e99fa6c46	Allow labelmap action	9 years ago
Jimmi Dyson	a1574aa2b3	Move TLS options to scrape config Fixes #1013, fixes #989	9 years ago

1 2 3 4

177 Commits (main)