756 Commits (main)

Author	SHA1	Message	Date
Krasi Georgiev	78ba5e62a6	few mote usefull comments	7 years ago
Krasi Georgiev	cabce21b70	delete empty targets sets to avoid memory leaks	7 years ago
Krasi Georgiev	abfd9f1920	nits	7 years ago
Shubheksha Jalan	0471e64ad1	Use shared types from the `common` repo (#3674) ... * refactor: use shared types from common repo, remove util/config * vendor: add common/config * fix nit	7 years ago
Krasi Georgiev	546c29af5b	return early for nil target groups	7 years ago
Callum Styan	97464236c7	comments with TargetProvider should read Discoverer instead (#3667)	7 years ago
Krasi Georgiev	77bf6bece0	discovery-manager comment update	7 years ago
Krasi Georgiev	135ea0f793	discovery manager - doesn't need sorting of the target groups so move it in the discovery manager tests as we only need it there. ... discovery manager - refactor the discovery tests. Signed-off-by: Krasi Georgiev <krasi.root@gmail.com>	7 years ago
Krasi Georgiev	638818a974	some Discoverers send nil targetgroup so need to check for it when updating a group	7 years ago
Krasi Georgiev	7e28397a2c	discovery - handle Discoverers that send only target Group updates rather than all Targets on every update. ... Signed-off-by: Krasi Georgiev <krasi.root@gmail.com>	7 years ago
Shubheksha Jalan	ec94df49d4	Refactor SD configuration to remove `config` dependency (#3629) ... * refactor: move targetGroup struct and CheckOverflow() to their own package * refactor: move auth and security related structs to a utility package, fix import error in utility package * refactor: Azure SD, remove SD struct from config * refactor: DNS SD, remove SD struct from config into dns package * refactor: ec2 SD, move SD struct from config into the ec2 package * refactor: file SD, move SD struct from config to file discovery package * refactor: gce, move SD struct from config to gce discovery package * refactor: move HTTPClientConfig and URL into util/config, fix import error in httputil * refactor: consul, move SD struct from config into consul discovery package * refactor: marathon, move SD struct from config into marathon discovery package * refactor: triton, move SD struct from config to triton discovery package, fix test * refactor: zookeeper, move SD structs from config to zookeeper discovery package * refactor: openstack, remove SD struct from config, move into openstack discovery package * refactor: kubernetes, move SD struct from config into kubernetes discovery package * refactor: notifier, use targetgroup package instead of config * refactor: tests for file, marathon, triton SD - use targetgroup package instead of config.TargetGroup * refactor: retrieval, use targetgroup package instead of config.TargetGroup * refactor: storage, use config util package * refactor: discovery manager, use targetgroup package instead of config.TargetGroup * refactor: use HTTPClient and TLS config from configUtil instead of config * refactor: tests, use targetgroup package instead of config.TargetGroup * refactor: fix tagetgroup.Group pointers that were removed by mistake * refactor: openstack, kubernetes: drop prefixes * refactor: remove import aliases forced due to vscode bug * refactor: move main SD struct out of config into discovery/config * refactor: rename configUtil to config_util * refactor: rename yamlUtil to yaml_config * refactor: kubernetes, remove prefixes * refactor: move the TargetGroup package to discovery/ * refactor: fix order of imports	7 years ago
Callum Styan	d76d5de66f	refactor to make timestamp collector work for multiple file_sd's	7 years ago
KalivarapuReshma	a00fc883c3	Add metric for timestamp of the files file_sd is using.	7 years ago
pasquier-s	78625f85a7	Fix race condition on file SD (#3468) ... The file discovery should only stop the watcher if it has been created otherwise it may trigger a segmentation fault.	7 years ago
Krasi Georgiev	587dec9eb9	rebased and resolved conflicts with the new Discovery GUI page ... Signed-off-by: Krasi Georgiev <krasi.root@gmail.com>	7 years ago
Krasi Georgiev	80182a5d82	use poolKey as the pool map key to avoid multi dimensional maps	7 years ago
Krasi Georgiev	1ec76d1950	rearange the contexts variables and logic ... split the groupsMerge function to set and get other small nits	7 years ago
Krasi Georgiev	f2df712166	updated README	7 years ago
Krasi Georgiev	aca8f85699	fixed the tests	7 years ago
Krasi Georgiev	fe6c544532	some renaming and comments fixes. ... remove some select state that is most likely obsoleete and hoepfully doesn't braje anything :) merge targets will sort by Discoverer name so we can have consistent tests for the maps.	7 years ago
Krasi Georgiev	f5c2c5ff8f	brake the start provider func so that can run unit tests against it.	7 years ago
Krasi Georgiev	c5cb0d2910	simplify naming and API.	7 years ago
Krasi Georgiev	9c61f0e8a0	scrape pool doesn't rely on context as Stop() needs to be blocking to prevent Scrape loops trying to write to a closed TSDB storage.	7 years ago
Krasi Georgiev	e405e2f1ea	refactored discovery	7 years ago
Brian Brazil	81db4716c1	Mention SD moratorium in README (#3573)	7 years ago
Will Howard	6a80fc24cf	Parse the normalized container.PortMappings presented by the Marathon 1.5.x API ... Fixes #3465	7 years ago
Brian Brazil	d7b3df5ae1	Fix staticcheck errors	7 years ago
Krasi Georgiev	29506e0bca	one meaningless write to the config file to trigger anothe fsnotify (#3492)	7 years ago
Tom Wilkie	099c50ce93	Avoid empty pod UID in test.	7 years ago
Tom Wilkie	9811e90d65	Fix tests.	7 years ago
Tom Wilkie	06dc1e8797	Include Pod UID in the discovery metadata.	7 years ago
Tobias Schmidt	91be55ebf0	Merge pull request #3458 from grandbora/test-race ... Fix race in test	7 years ago
Bora Tunca	493fd6bd1f	Fix race in test	7 years ago
Krasi Georgiev	1005ef0a70	Fix flaky file discovery tests - sync the channel draining goroutine	7 years ago
Bora Tunca	3cc01a3088	Add more discovery tests for updating target groups (#3426) ... * Adds a test covering the case where a target providers sends updated versions of the same target groups and the system should reconcile to the latest version of each of the target groups * Refactors how input data is represented in the tests. It used to be literal declarations of necessary structs, now it is parsing yaml. Yaml declarations are half as long as the former. And these can be put in a fixture file * Adds a tiny bit of refactoring on test timeouts	7 years ago
Krasi Georgiev	c8a735ceb6	Fix flaky file discovery tests (#3438) ... * flaky test caused by invalid fsnotify updates before the test files are written to disk causing the fd service to send empty `group[]` struct * `close(filesReady)` needs to be before the file closing so that fsnotify triggers a new loop of the discovery service. * nits * use filepath.Join for the path url to be cross platform * stupid mistake revert	7 years ago
Bora Tunca	e63219ae6a	Add discovery test (#3417)	7 years ago
Bora Tunca	09be10a553	Add test to prove redundant calls to identical target providers (#3404)	7 years ago
beorn7	348ea482ea	Merge branch 'beorn7/release'	7 years ago
Dominik Schulz	a731a43302	Guard against tags being nil in EC2 discovery ... Fixes #3001	7 years ago
Callum Styan	7776527390	bump consul HTTP client timeout by 5s so it doesn't match up exactly with the consul SD watch timeout	7 years ago
Jason Anderson	808f79f00a	Feature: Allow getting credentials via EC2 role (#3343) ... * Allow getting credentials via EC2 role This is subtly different than the existing `role_arn` solution, which allows Prometheus to assume an IAM role given some set of credentials already in-scope. With EC2 roles, one specifies the role at instance launch time (via an instance profile.) The instance then exposes temporary credentials via its metadata. The AWS Go SDK exposes a credential provider that polls the [instance metadata endpoint][1] already, so we can simply use that and it will take care of renewing the credentials when they expire. Without this, if this is being used inside EC2, it is difficult to cleanly allow the use of STS credentials. One has to set up a proxy role that can assume the role you really want, and launch the EC2 instance with the proxy role. This isn't very clean, and also doesn't seem to be [supported very well][2]. [1]: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html [2]: https://github.com/aws/aws-cli/issues/1390 * Automatically try to detect EC2 role credentials The `Available()` function exposed on ec2metadata returns a simple true/false if the ec2 metadata is available. This is the best way to know if we're actually running in EC2 (which is the only valid use-case for this credential provider.) This allows this to "just work" if you are using EC2 instance roles.	7 years ago
Julius Volz	099df0c5f0	Migrate "golang.org/x/net/context" -> "context" (#3333) ... In some places, where ctxhttp or gRPC are concerned, we still need to use the old contexts.	7 years ago
Julius Volz	c3d6abc8e6	Fix some lint errors (#3334) ... I left the promql ones and some others untouched as I remember that @fabxc prefers them that way.	7 years ago
Callum Styan	45f9f3c539	use a timeout in the HTTP client used for consul sd (#3303)	7 years ago
Alexander Kazarin	2c163f32a5	fix for issue 2976 (#3313) ... fix for null pointer exception in ZookeeperLogger	7 years ago
pasquier-s	88e4815bb7	Get OpenStack variables from env as fallback (#3293) ... This change enables the OpenStack service discovery to read the authentication parameters from the OS_* environment variables when the identity endpoint URL is not defined in the Prometheus configuration file.	7 years ago
Marc Sluiter	6a633eece1	Added go-conntrack for monitoring http connections (#3241) ... Added metrics for in- and outgoing traffic with go-conntrack.	7 years ago
Fabian Reinartz	2d0b8e8b94	Merge branch 'master' into dev-2.0	7 years ago
Goutham Veeramachaneni	3f0267c548	Merge branch 'dev-2.0' into go-kit/log ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	7 years ago
beorn7	84211bd2df	Foward-merge bug fixes and cherry-picks from 'release-1.7'	7 years ago
Matt Palmer	3369422327	Improve DNS response handling to prevent "stuck" records [Fixes #2799] (#3138) ... The problem reported in #2799 was that in the event that all records for a name were removed, the target group was never updated to be the "empty" set. Essentially, whatever Prometheus last saw as a non-empty list of targets would stay that way forever (or at least until Prometheus restarted...). This came about because of a fairly naive interpretation of what a valid-looking DNS response actually looked like -- essentially, the only valid DNS responses were ones that had a non-empty record list. That's fine as long as your config always lists only target names which have non-empty record sets; if your environment happens to legitimately have empty record sets sometimes, all hell breaks loose (otherwise-cleanly shutdown systems trigger up==0 alerts, for instance). This patch is a refactoring of the DNS lookup behaviour that maintains existing behaviour with regard to search paths, but correctly handles empty and non-existent record sets. RFC1034 s4.3.1 says there's three ways a recursive DNS server can respond: 1. Here is your answer (possibly an empty answer, because of the way DNS considers all records for a name, regardless of type, when deciding whether the name exists). 2. There is no spoon (the name you asked for definitely does not exist). 3. I am a teapot (something has gone terribly wrong). Situations 1 and 2 are fine and dandy; whatever the answer is (empty or otherwise) is the list of targets. If something has gone wrong, then we shouldn't go updating the target list because we don't really *know* what the target list should be. Multiple DNS servers to query is a straightforward augmentation; if you get an error, then try the next server in the list, until you get an answer or run out servers to ask. Only if *all* the servers return errors should you return an error to the calling code. Where things get complicated is the search path. In order to be able to confidently say, "this name does not exist anywhere, you can remove all the targets for this name because it's definitely GORN", at least one server for *all* the possible names need to return either successful-but-empty responses, or NXDOMAIN. If any name errors out, then -- since that one might have been the one where the records came from -- you need to say "maintain the status quo until we get a known-good response". It is possible, though unlikely, that a poorly-configured DNS setup (say, one which had a domain in its search path for which all configured recursive resolvers respond with REFUSED) could result in the same "stuck" records problem we're solving here, but the DNS configuration should be fixed in that case, and there's nothing we can do in Prometheus itself to fix the problem. I've tested this patch on a local scratch instance in all the various ways I can think of: 1. Adding records (targets get scraped) 2. Adding records of a different type 3. Remove records of the requested type, leaving other type records intact (targets don't get scraped) 4. Remove all records for the name (targets don't get scraped) 5. Shutdown the resolver (targets still get scraped) There's no automated test suite additions, because there isn't a test suite for DNS discovery, and I was stretching my Go skills to the limit to make this happen; mock objects are beyond me.	7 years ago
Goutham Veeramachaneni	f5aed810f9	logging: Port to common/promlog ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	7 years ago
Matt Bostock	e758260986	Marathon SD: Set port index label ... The changes [1][] to Marathon service discovery to support multiple ports mean that Prometheus now attempts to scrape all ports belonging to a Marathon service. You can use port definition or port mapping labels to filter out which ports to scrape but that requires service owners to update their Marathon configuration. To allow for a smoother migration path, add a `__meta_marathon_port_index` label, whose value is set to the port's sequential index integer. For example, PORT0 has the value `0`, PORT1 has the value `1`, and so on. This allows you to support scraping both the first available port (the previous behaviour) in addition to ports with a `metrics` label. For example, here's the relabel configuration we might use with this patch: - action: keep source_labels: ['__meta_marathon_port_definition_label_metrics', '__meta_marathon_port_mapping_label_metrics', '__meta_marathon_port_index'] # Keep if port mapping or definition has a 'metrics' label with any # non-empty value, or if no 'metrics' port label exists but this is the # service's first available port regex: ([^;]+;;[^;]+|;[^;]+;[^;]+|;;0) This assumes that the Marathon API returns the ports in sorted order (matching PORT0, PORT1, etc), which it appears that it does. [1]: https://github.com/prometheus/prometheus/pull/2506	7 years ago
Fabian Reinartz	e746282772	Merge branch 'master' into dev-2.0	7 years ago
Jamie Moore	7a135e0a1b	Add the ability to assume a role for ec2 discovery	7 years ago
Fabian Reinartz	d21f149745	*: migrate to go-kit/log	7 years ago
Johannes 'fish' Ziemke	75aec7d970	k8s: Use versioned struct for ingress discovery	7 years ago
Fabian Reinartz	87918f3097	Merge branch 'master' into dev-2.0	7 years ago
Johannes 'fish' Ziemke	70f3d1e9f9	k8s: Support discovery of ingresses (#3111) ... * k8s: Support discovery of ingresses * Move additional labels below allocation This makes it more obvious why the additional elements are allocated. Also fix allocation for node where we only set a single label. * k8s: Remove port from ingress discovery * k8s: Add comment to ingress discovery example	7 years ago
Tobias Schmidt	29fff1eca4	Merge pull request #2966 from alkalinecoffee/consul-node-metadata ... Add support for consul's node metadata	7 years ago
Tobias Schmidt	d0a02703a2	Merge pull request #3105 from sak0/dev ... discovery openstack: support discovery hypervisors, add rule option.	7 years ago
CuiHaozhi	b1c18bf29b	discovery openstack: support discovery hosts, add rule option. ... Signed-off-by: CuiHaozhi <cuihz@wise2c.com>	7 years ago
Colstuwjx	2b49df2c61	Fix target group foreach nil bug, directly return err.	7 years ago
CuiHaozhi	31b6f8b04c	discovery openstack: handle instances without ip ... Signed-off-by: CuiHaozhi <cuihz@wise2c.com>	7 years ago
Fabian Reinartz	25f3e1c424	Merge branch 'master' into mergemaster	7 years ago
Fabian Reinartz	ac511ecf30	Merge pull request #2970 from Gouthamve/docs/sd-interface ... Add docs about SD interface	7 years ago
Goutham Veeramachaneni	ab96e79bc8	Add docs about SD interface ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	7 years ago
Fabian Reinartz	40db026381	Merge pull request #2957 from prometheus/sd-doc ... Tweaks to SD README from review	7 years ago
Joe Martin	aba41c7d0f	add support for consul's node metadata	7 years ago
J. Taylor O'Connor	5a19ffb315	A few spelling corrections. (#2960)	7 years ago
Brian Brazil	84be97bd98	Tweaks to SD README from review	7 years ago
Brian Brazil	2a9ca394dd	Document how/when to write service discovery (#2943)	7 years ago
Fabian Reinartz	dba7586671	Merge branch 'master' into dev-2.0	7 years ago
Fuente, Pablo Andres	902fafb8e7	Fixing tests for Windows ... Fixing the config/config_test, the discovery/file/file_test and the promql/promql_test tests for Windows. For most of the tests, the fix involved correct handling of path separators. In the case of the promql tests, the issue was related to the removal of the temporal directories used by the storage. The issue is that the RemoveAll() call returns an error when it tries to remove a directory which is not empty, which seems to be true due to some kind of process that is still running after closing the storage. To fix it I added some retries to the remove of the temporal directories. Adding tags file from Universal Ctags to .gitignore	7 years ago
Matt Bostock	ab4d64959f	Marathon SD: Set port index label ... The changes [1][] to Marathon service discovery to support multiple ports mean that Prometheus now attempts to scrape all ports belonging to a Marathon service. You can use port definition or port mapping labels to filter out which ports to scrape but that requires service owners to update their Marathon configuration. To allow for a smoother migration path, add a `__meta_marathon_port_index` label, whose value is set to the port's sequential index integer. For example, PORT0 has the value `0`, PORT1 has the value `1`, and so on. This allows you to support scraping both the first available port (the previous behaviour) in addition to ports with a `metrics` label. For example, here's the relabel configuration we might use with this patch: - action: keep source_labels: ['__meta_marathon_port_definition_label_metrics', '__meta_marathon_port_mapping_label_metrics', '__meta_marathon_port_index'] # Keep if port mapping or definition has a 'metrics' label with any # non-empty value, or if no 'metrics' port label exists but this is the # service's first available port regex: ([^;]+;;[^;]+|;[^;]+;[^;]+|;;0) This assumes that the Marathon API returns the ports in sorted order (matching PORT0, PORT1, etc), which it appears that it does. [1]: https://github.com/prometheus/prometheus/pull/2506	8 years ago
Goutham Veeramachaneni	507790a357	Rework logging to use explicitly passed logger ... Mostly cleaned up the global logger use. Still some uses in discovery package. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Christian Groschupp	8f781e411c	Openstack Service Discovery (#2701) ... * Add openstack service discovery. * Add gophercloud code for openstack service discovery. * first changes for juliusv comments. * add gophercloud code for floatingip. * Add tests to openstack sd. * Add testify suite vendor files. * add copyright and make changes for code climate. * Fixed typos in provider openstack. * Renamed tenant to project in openstack sd. * Change type of password to Secret in openstack sd.	8 years ago
Roman Vynar	dbe2eb2afc	Hide consul token on UI. (#2797)	8 years ago
Chris Goller	42de0ae013	Use log.Logger interface for all discovery services	8 years ago
Tobias Schmidt	287ec6e6cc	Fix outdated target_group naming in error message ... The target_groups config has been renamed to static_configs, the error message for overflow attributes should reflect that.	8 years ago
Conor Broderick	6766123f93	Replace regex with Secret type and remarshal config to hide secrets (#2775)	8 years ago
Fabian Reinartz	11aa049b05	Merge branch 'release-1.6' into merge16	8 years ago
Fabian Reinartz	ddbbd2b712	Merge branch 'release-1.5' into cut162	8 years ago
Fabian Reinartz	2ff8855ae6	discovery/k8s: update client library	8 years ago
Fabian Reinartz	aaaec6431e	Merge pull request #2642 from bakins/kubernetes-namespaces ... Allow limiting Kubernetes service discover to certain namespaces	8 years ago
Stephan Erb	0b9fca983b	Fix reload of ZooKeeper service discovery config (#2669) ... Rational: * When the config is reloaded and the provider context is canceled, we need to exit the current ZK `TargetProvider.Run` method as a new provider will be instantiated. * In case `Stop` is called on the `ZookeeperTreeCache`, the update/events channel may not be closed as it is shared by multiple caches and would thus be double closed. * Stopping all `zookeeperTreeCacheNode`s on teardown ensures all associated watcher go-routines will be closed eagerly rather than implicityly on connection close events.	8 years ago
Brian Akins	27d66628a1	Allow limiting Kubernetes service discover to certain namespaces ... Allow namespace discovery to be more easily extended in the future by using a struct rather than just a list. Rename fields for kubernetes namespace discovery	8 years ago
Goutham Veeramachaneni	0f48d07f95	Fix Map Race by Moving Locking closer to the Write (#2476)	8 years ago
Richard Kiene	ec692f6161	Add triton zone brand metadata	8 years ago
Julius Volz	525da88c35	Merge pull request #2479 from YKlausz/consul-tls ... Adding consul capability to connect via tls	8 years ago
Robson Roberto Souza Peixoto	cc3e859d9e	Add support for multiple ports in Marathon (#2506) ... - create a target for every port - add meta labels for Marathon labels in portMappings and portDefinitions	8 years ago
yklausz	75880b594f	Adding consul capability to connect via tls	8 years ago
Tobias Schmidt	7bde44e98e	Remove testing.T usage in goroutines ... The staticcheck warns about testing.T usage in goroutines. Moving the t.Fatal* calls to the main thread showed immediately that this is a good practice, as one of the test setups didn't work.	8 years ago
Tobias Schmidt	58cd39aacd	Follow golang naming conventions in discovery packages	8 years ago
Robert Neumayer	feb7670929	Add tests for consul service discovery (#2490) ... * Add tests for consul service discovery * Add license header * Address comments * inline variables * check for extra error * Fix error formatting	8 years ago
Michael Kraus	690b49e503	Fix marathon tests	8 years ago
Michael Kraus	31252cc1b5	Clarify explicit use of authorization header	8 years ago
Michael Kraus	47bdcf0f67	Allow the use of bearer_token or bearer_token_file for MarathonSD	8 years ago
James Hartig	865f28bb15	discovery: Instead of looping over conf.Search, use NameList()	8 years ago
Alex Somesan	b22eb65d0f	Cleaner separation between ServiceAccount and custom authentication in K8S SD (#2348) ... * Canonical usage of cluster service-account in K8S SD * Early validation for opt-in custom auth in K8S SD * Fix typo in condition	8 years ago
Richard Kiene	f3d9692d09	Add Joyent Triton discovery	8 years ago
Fabian Reinartz	35da23fd82	consul: start service watch as goroutine	8 years ago
Fabian Reinartz	200bbe1bad	config: extract SD and HTTPClient configurations	8 years ago
Fabian Reinartz	d7f4f8b879	discovery: move TargetSet into discovery package	8 years ago
Fabian Reinartz	d19d1bcad3	discovery: move into top-level package	8 years ago