90 Commits (7a13850db23b2e64bead4e1c88d8e390c35c042a)

Author	SHA1	Message	Date
Sarah Christoff	5e26971864	Prune Unhealthy Agents (#6571) ... * Add -prune flag to ForceLeave	5 years ago
Freddy	fdd10dd8b8	Expose HTTP-based paths through Connect proxy (#6446) ... Fixes: #5396 This PR adds a proxy configuration stanza called expose. These flags register listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only listening on the loopback interface, while still accepting traffic from non Connect-enabled services. Under expose there is a boolean checks flag that would automatically expose all registered HTTP and gRPC check paths. This stanza also accepts a paths list to expose individual paths. The primary use case for this functionality would be to expose paths for third parties like Prometheus or the kubelet. Listeners for requests to exposed paths are be configured dynamically at run time. Any time a proxy, or check can be registered, a listener can also be created. In this initial implementation requests to these paths are not authenticated/encrypted.	5 years ago
Matt Keeler	6a1e0dfed8	Update the v1/agent/service/:service endpoint to output tagged… (#6304)	5 years ago
Mike Morris	65be58703c	connect: remove managed proxies (#6220) ... * connect: remove managed proxies implementation and all supporting config options and structs * connect: remove deprecated ProxyDestination * command: remove CONNECT_PROXY_TOKEN env var * agent: remove entire proxyprocess proxy manager * test: remove all managed proxy tests * test: remove irrelevant managed proxy note from TestService_ServerTLSConfig * test: update ContentHash to reflect managed proxy removal * test: remove deprecated ProxyDestination test * telemetry: remove managed proxy note * http: remove /v1/agent/connect/proxy endpoint * ci: remove deprecated test exclusion * website: update managed proxies deprecation page to note removal * website: remove managed proxy configuration API docs * website: remove managed proxy note from built-in proxy config * website: add note on removing proxy subdirectory of data_dir	5 years ago
Matt Keeler	8d953f5840	Implement Mesh Gateways ... This includes both ingress and egress functionality.	5 years ago
Matt Keeler	6cc1451895	Update some tests to fix ContentHash broken by the tagged service addresses (#5996)	6 years ago
Matt Keeler	f3d9b999ee	Add tagged addresses for services (#5965) ... This allows addresses to be tagged at the service level similar to what we allow for nodes already. The address translation that can be enabled with the `translate_wan_addrs` config was updated to take these new addresses into account as well.	6 years ago
Matt Keeler	afa1cc98d1	Implement data filtering of some endpoints (#5579) ... Fixes: #4222 # Data Filtering This PR will implement filtering for the following endpoints: ## Supported HTTP Endpoints - `/agent/checks` - `/agent/services` - `/catalog/nodes` - `/catalog/service/:service` - `/catalog/connect/:service` - `/catalog/node/:node` - `/health/node/:node` - `/health/checks/:service` - `/health/service/:service` - `/health/connect/:service` - `/health/state/:state` - `/internal/ui/nodes` - `/internal/ui/services` More can be added going forward and any endpoint which is used to list some data is a good candidate. ## Usage When using the HTTP API a `filter` query parameter can be used to pass a filter expression to Consul. Filter Expressions take the general form of: ``` <selector> == <value> <selector> != <value> <value> in <selector> <value> not in <selector> <selector> contains <value> <selector> not contains <value> <selector> is empty <selector> is not empty not <other expression> <expression 1> and <expression 2> <expression 1> or <expression 2> ``` Normal boolean logic and precedence is supported. All of the actual filtering and evaluation logic is coming from the [go-bexpr](https://github.com/hashicorp/go-bexpr) library ## Other changes Adding the `Internal.ServiceDump` RPC endpoint. This will allow the UI to filter services better.	6 years ago
Alvin Huang	f45e495e38	Merge pull request #5376 from hashicorp/fix-tests ... Fix tests in prep for CircleCI Migration	6 years ago
Jeff Mitchell	4243c3ae42	Move internal/ to sdk/ (#5568) ... * Move internal/ to sdk/ * Add a readme to the SDK folder	6 years ago
Jeff Mitchell	47c390025b	Convert to Go Modules (#5517) ... * First conversion * Use serf 0.8.2 tag and associated updated deps * * Move freeport and testutil into internal/ * Make internal/ its own module * Update imports * Add replace statements so API and normal Consul code are self-referencing for ease of development * Adapt to newer goe/values * Bump to new cleanhttp * Fix ban nonprintable chars test * Update lock bad args test The error message when the duration cannot be parsed changed in Go 1.12 (ae0c435877d3aacb9af5e706c40f9dddde5d3e67). This updates that test. * Update another test as well * Bump travis * Bump circleci * Bump go-discover and godo to get rid of launchpad dep * Bump dockerfile go version * fix tar command * Bump go-cleanhttp	6 years ago
R.B. Boyer	f8bcfcfa46	api: fix panic in 'consul acl set-agent-token' (#5533) ... api: fix panic in 'consul acl set-agent-token' Fixes #5531	6 years ago
Matt Keeler	118adbb123	ACL Token Persistence and Reloading (#5328) ... This PR adds two features which will be useful for operators when ACLs are in use. 1. Tokens set in configuration files are now reloadable. 2. If `acl.enable_token_persistence` is set to `true` in the configuration, tokens set via the `v1/agent/token` endpoint are now persisted to disk and loaded when the agent starts (or during configuration reload) Note that token persistence is opt-in so our users who do not want tokens on the local disk will see no change. Some other secondary changes: * Refactored a bunch of places where the replication token is retrieved from the token store. This token isn't just for replicating ACLs and now it is named accordingly. * Allowed better paths in the `v1/agent/token/` API. Instead of paths like: `v1/agent/token/acl_replication_token` the path can now be just `v1/agent/token/replication`. The old paths remain to be valid. * Added a couple new API functions to set tokens via the new paths. Deprecated the old ones and pointed to the new names. The names are also generally better and don't imply that what you are setting is for ACLs but rather are setting ACL tokens. There is a minor semantic difference there especially for the replication token as again, its no longer used only for ACL token/policy replication. The new functions will detect 404s and fallback to using the older token paths when talking to pre-1.4.3 agents. * Docs updated to reflect the API additions and to show using the new endpoints. * Updated the ACL CLI set-agent-tokens command to use the non-deprecated APIs.	6 years ago
Alvin Huang	aed9d123fa	add serf check to various api/agent tests	6 years ago
Alvin Huang	a5bd6d7c00	add serf check to TestAPI_AgentServices_ManagedConnectProxyDeprecatedUpstreams	6 years ago
Paul Banks	bb7145f27d	agent: add default weights to service in local state to prevent AE churn (#5126) ... * Add default weights when adding a service with no weights to local state to prevent constant AE re-sync. This fix was contributed by @42wim in https://github.com/hashicorp/consul/pull/5096 but was merged against the wrong base. This adds it to master and adds a test to cover the behaviour. * Fix tests that broke due to comparing internal state which now has default weights	6 years ago
Grégoire Seux	4f62a3b528	Implement /v1/agent/health/service/<service name> endpoint (#3551) ... This endpoint aggregates all checks related to <service id> on the agent and return an appropriate http code + the string describing the worst check. This allows to cleanly expose service status to other component, hiding complexity of multiple checks. This is especially useful to use consul to feed a load balancer which would delegate health checking to consul agent. Exposing this endpoint on the agent is necessary to avoid a hit on consul servers and avoid decreasing resiliency (this endpoint will work even if there is no consul leader in the cluster).	6 years ago
Jack Pearkes	8c684db488	New command: consul debug (#4754) ... * agent/debug: add package for debugging, host info * api: add v1/agent/host endpoint * agent: add v1/agent/host endpoint * command/debug: implementation of static capture * command/debug: tests and only configured targets * agent/debug: add basic test for host metrics * command/debug: add methods for dynamic data capture * api: add debug/pprof endpoints * command/debug: add pprof * command/debug: timing, wg, logs to disk * vendor: add gopsutil/disk * command/debug: add a usage section * website: add docs for consul debug * agent/host: require operator:read * api/host: improve docs and no retry timing * command/debug: fail on extra arguments * command/debug: fixup file permissions to 0644 * command/debug: remove server flags * command/debug: improve clarity of usage section * api/debug: add Trace for profiling, fix profile * command/debug: capture profile and trace at the same time * command/debug: add index document * command/debug: use "clusters" in place of members * command/debug: remove address in output * command/debug: improve comment on metrics sleep * command/debug: clarify usage * agent: always register pprof handlers and protect This will allow us to avoid a restart of a target agent for profiling by always registering the pprof handlers. Given this is a potentially sensitive path, it is protected with an operator:read ACL and enable debug being set to true on the target agent. enable_debug still requires a restart. If ACLs are disabled, enable_debug is sufficient. * command/debug: use trace.out instead of .prof More in line with golang docs. * agent: fix comment wording * agent: wrap table driven tests in t.run()	6 years ago
Pierre Souchay	251156eb68	Added SOA configuration for DNS settings. (#4714) ... This will allow to fine TUNE SOA settings sent by Consul in DNS responses, for instance to be able to control negative ttl. Will fix: https://github.com/hashicorp/consul/issues/4713 # Example Override all settings: * min_ttl: 0 => 60s * retry: 600 (10m) => 300s (5 minutes), * expire: 86400 (24h) => 43200 (12h) * refresh: 3600 (1h) => 1800 (30 minutes) ``` consul agent -dev -hcl 'dns_config={soa={min_ttl=60,retry=300,expire=43200,refresh=1800}}' ``` Result: ``` dig +multiline @localhost -p 8600 service.consul ; <<>> DiG 9.12.1 <<>> +multiline @localhost -p 8600 service.consul ; (2 servers found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 36557 ;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;service.consul. IN A ;; AUTHORITY SECTION: consul. 0 IN SOA ns.consul. hostmaster.consul. ( 1537959133 ; serial 1800 ; refresh (30 minutes) 300 ; retry (5 minutes) 43200 ; expire (12 hours) 60 ; minimum (1 minute) ) ;; Query time: 4 msec ;; SERVER: 127.0.0.1#8600(127.0.0.1) ;; WHEN: Wed Sep 26 12:52:13 CEST 2018 ;; MSG SIZE rcvd: 93 ```	6 years ago
Paul Banks	161482d2cd	Fix up tests broken by master merge; add proxy tests to services command (and fix it!); actually run the proxycfg.Manager	6 years ago
Paul Banks	e812f5516a	Add -sidecar-for and new /agent/service/:service_id endpoint (#4691) ... - A new endpoint `/v1/agent/service/:service_id` which is a generic way to look up the service for a single instance. The primary value here is that it: - **supports hash-based blocking** and so; - **replaces `/agent/connect/proxy/:proxy_id`** as the mechanism the built-in proxy uses to read its config. - It's not proxy specific and so works for any service. - It has a temporary shim to call through to the existing endpoint to preserve current managed proxy config defaulting behaviour until that is removed entirely (tested). - The built-in proxy now uses the new endpoint exclusively for it's config - The built-in proxy now has a `-sidecar-for` flag that allows the service ID of the _target_ service to be specified, on the condition that there is exactly one "sidecar" proxy (that is one that has `Proxy.DestinationServiceID` set) for the service registered. - Several fixes for edge cases for SidecarService - A fix for `Alias` checks - when running locally they didn't update their state until some external thing updated the target. If the target service has no checks registered as below, then the alias never made it past critical.	6 years ago
Paul Banks	1e7eace066	Add SidecarService Syntax sugar to Service Definition (#4686) ... * Added new Config for SidecarService in ServiceDefinitions. * WIP: all the code needed for SidecarService is written... none of it is tested other than config :). Need API updates too. * Test coverage for the new sidecarServiceFromNodeService method. * Test API registratrion with SidecarService * Recursive Key Translation 🤦 * Add tests for nested sidecar defintion arrays to ensure they are translated correctly * Use dedicated internal state rather than Service Meta for tracking sidecars for deregistration. Add tests for deregistration. * API struct for agent register. No other endpoint should be affected yet. * Additional test cases to cover updates to API registrations	6 years ago
Paul Banks	b83bbf248c	Add Proxy Upstreams to Service Definition (#4639) ... * Refactor Service Definition ProxyDestination. This includes: - Refactoring all internal structs used - Updated tests for both deprecated and new input for: - Agent Services endpoint response - Agent Service endpoint response - Agent Register endpoint - Unmanaged deprecated field - Unmanaged new fields - Managed deprecated upstreams - Managed new - Catalog Register - Unmanaged deprecated field - Unmanaged new fields - Managed deprecated upstreams - Managed new - Catalog Services endpoint response - Catalog Node endpoint response - Catalog Service endpoint response - Updated API tests for all of the above too (both deprecated and new forms of register) TODO: - config package changes for on-disk service definitions - proxy config endpoint - built-in proxy support for new fields * Agent proxy config endpoint updated with upstreams * Config file changes for upstreams. * Add upstream opaque config and update all tests to ensure it works everywhere. * Built in proxy working with new Upstreams config * Command fixes and deprecations * Fix key translation, upstream type defaults and a spate of other subtele bugs found with ned to end test scripts... TODO: tests still failing on one case that needs a fix. I think it's key translation for upstreams nested in Managed proxy struct. * Fix translated keys in API registration. ≈ * Fixes from docs - omit some empty undocumented fields in API - Bring back ServiceProxyDestination in Catalog responses to not break backwards compat - this was removed assuming it was only used internally. * Documentation updates for Upstreams in service definition * Fixes for tests broken by many refactors. * Enable travis on f-connect branch in this branch too. * Add consistent Deprecation comments to ProxyDestination uses * Update version number on deprecation notices, and correct upstream datacenter field with explanation in docs	6 years ago
Paul Banks	8cbeb29e73	Fixes #4421: General solution to stop blocking queries with index 0 (#4437) ... * Fix theoretical cache collision bug if/when we use more cache types with same result type * Generalized fix for blocking query handling when state store methods return zero index * Refactor test retry to only affect CI * Undo make file merge * Add hint to error message returned to end-user requests if Connect is not enabled when they try to request cert * Explicit error for Roots endpoint if connect is disabled * Fix tests that were asserting old behaviour	6 years ago
Paul Banks	b5f24a21cb	Fix test broken by final telemetry PR change!	7 years ago
Paul Banks	597e55e8e2	Misc test fixes	7 years ago
Paul Banks	96c416012e	Misc rebase and test fixes	7 years ago
Paul Banks	43b48bc06b	Get agent cache tests passing without global hit count (which is racy). ... Few other fixes in here just to get a clean run locally - they are all also fixed in other PRs but shouldn't conflict. This should be robust to timing between goroutines now.	7 years ago
Paul Banks	df2cb30b01	Make tests pass and clean proxy persistence. No detached child changes yet. ... This is a good state for persistence stuff to re-start the detached child work that got mixed up last time.	7 years ago
Paul Banks	cdc7cfaa36	Abandon daemonize for simpler solution (preserving history): ... Reverts: - bdb274852ae469c89092d6050697c0ff97178465 - 2c689179c4f61c11f0016214c0fc127a0b813bfe - d62e25c4a7ab753914b6baccd66f88ffd10949a3 - c727ffbcc98e3e0bf41e1a7bdd40169bd2d22191 - 31b4d18933fd0acbe157e28d03ad59c2abf9a1fb - 85c3f8df3eabc00f490cd392213c3b928a85aa44	7 years ago
Paul Banks	ba0fb58a72	Make daemoinze an option on test binary without hacks. Misc fixes for racey or broken tests. Still failing on several though.	7 years ago
Mitchell Hashimoto	a5ecdc5798	api: IntentionCheck	7 years ago
Paul Banks	7c7c858a9a	More test cleanup	7 years ago
Paul Banks	4aeab3897c	Fixed many tests after rebase. Some still failing and seem unrelated to any connect changes.	7 years ago
Paul Banks	e0e12e165b	TLS watching integrated into Service with some basic tests. ... There are also a lot of small bug fixes found when testing lots of things end-to-end for the first time and some cleanup now it's integrated with real CA code.	7 years ago
Paul Banks	cd88b2a351	Basic `watch` support for connect proxy config and certificate endpoints. ... - Includes some bug fixes for previous `api` work and `agent` that weren't tested - Needed somewhat pervasive changes to support hash based blocking - some TODOs left in our watch toolchain that will explicitly fail on hash-based watches. - Integration into `connect` is partially done here but still WIP	7 years ago
Paul Banks	36dbd878c9	Adds `api` client code and tests for new Proxy Config endpoint, registering with proxy and seeing proxy config in /agent/services list.	7 years ago
Paul Banks	88541bba17	Add tests all the way up through the endpoints to ensure duplicate src/destination is supported and so ultimately deny/allow nesting works. ... Also adds a sanity check test for `api.Agent().ConnectAuthorize()` and a fix for a trivial bug in it.	7 years ago
Mitchell Hashimoto	263e2c7cf7	api: endpoints for working with CA roots, agent authorize, etc.	7 years ago
Paul Banks	9309422fd9	Add Connect agent, catalog and health endpoints to api Client	7 years ago
Kyle Havlovitz	b73323aa42	Remove the script field from checks in favor of args	7 years ago
Pierre Souchay	c152cb7bdf	Added Missing Service Meta synchronization and field	7 years ago
James Hartig	ac225b2e7d	Added CheckID and Name to AgentServiceCheck struct	7 years ago
James Phillips	3177c2b450	Opens up the timing on the metrics test so it's less flaky.	7 years ago
James Phillips	172d5c0c40	Fetches metrics in the retry loop so there's some hope. ... Fixes #3656	7 years ago
James Phillips	21f38bff6f	Adds retry to API metrics test (flaky test).	7 years ago
James Phillips	53f67c3993	Fixes API client for ScriptArgs and updates documentation. (#3589) ... * Updates the API client to support the current `ScriptArgs` parameter for checks. * Updates docs for checks to explain the `ScriptArgs` parameter issue. * Adds mappings for "args" and "script-args" to give th API parity with config. * Adds checks on return codes. * Removes debug logging that shows empty when args are used.	7 years ago
Frank Schröder	ce887a0c45	Provide stable config for agent/self (#3532) ... * config: provide stable config for /v1/agent/self (#3530) This patch adds a stable subset of the previous Config struct to the agent/self response. The actual runtime configuration is moved into DebugConfig and will be documented to change. Fixes #3530 * config: fix tests * doc: update api documentation for /v1/agent/self	7 years ago
Frank Schröder	21118cafeb	Recursive sanitize (#3505) ... * vendor: add github.com/sergi/go-diff/diffmatchpatch for diff'ing test output * config: refactor Sanitize to recursively clean runtime config and format complex fields * Removes an extra int cast. * Adds a top-level check test case for sanitization.	7 years ago
Preetha Appan	00840929ea	Fix import order	7 years ago