61 Commits (7a418f70cf88aa915a1c956c299de689fbefa728)

Author	SHA1	Message	Date
Aestek	2aac4d5168	Register and deregisters services and their checks atomically in the local state (#5012) ... Prevent race between register and deregister requests by saving them together in the local state on registration. Also adds more cleaning in case of failure when registering services / checks.	6 years ago
Aestek	5960974db1	[Fix] Services sometimes not being synced with acl_enforce_version_8 = false (#4771) ... Fixes: https://github.com/hashicorp/consul/issues/3676 This fixes a bug were registering an agent with a non-existent ACL token can prevent other services registered with a good token from being synced to the server when using `acl_enforce_version_8 = false`. ## Background When `acl_enforce_version_8` is off the agent does not check the ACL token validity before storing the service in its state. When syncing a service registered with a missing ACL token we fall into the default error handling case (https://github.com/hashicorp/consul/blob/master/agent/local/state.go#L1255) and stop the sync (https://github.com/hashicorp/consul/blob/master/agent/local/state.go#L1082) without setting its Synced property to true like in the permission denied case. This means that the sync will always stop at the faulty service(s). The order in which the services are synced is random since we iterate on a map. So eventually all services with good ACL tokens will be synced, this can however take some time and is influenced by the cluster size, the bigger the slower because retries are less frequent. Having a service in this state also prevent all further sync of checks as they are done after the services. ## Changes This change modify the sync process to continue even if there is an error. This fixes the issue described above as well as making the sync more error tolerant: if the server repeatedly refuses a service (the ACL token could have been deleted by the time the service is synced, the servers were upgraded to a newer version that has more strict checks on the service definition...). Then all services and check that can be synced will, and those that don't will be marked as errors in the logs instead of blocking the whole process.	6 years ago
Paul Banks	0f27ffd163	Proxy Config Manager (#4729) ... * Proxy Config Manager This component watches for local state changes on the agent and ensures that each service registered locally with Kind == connect-proxy has it's state being actively populated in the cache. This serves two purposes: 1. For the built-in proxy, it ensures that the state needed to accept connections is available in RAM shortly after registration and likely before the proxy actually starts accepting traffic. 2. For (future - next PR) xDS server and other possible future proxies that require _push_ based config discovery, this provides a mechanism to subscribe and be notified about updates to a proxy instance's config including upstream service discovery results. * Address review comments * Better comments; Better delivery of latest snapshot for slow watchers; Embed Config * Comment typos * Add upstream Stringer for funsies	6 years ago
Paul Banks	e812f5516a	Add -sidecar-for and new /agent/service/:service_id endpoint (#4691) ... - A new endpoint `/v1/agent/service/:service_id` which is a generic way to look up the service for a single instance. The primary value here is that it: - **supports hash-based blocking** and so; - **replaces `/agent/connect/proxy/:proxy_id`** as the mechanism the built-in proxy uses to read its config. - It's not proxy specific and so works for any service. - It has a temporary shim to call through to the existing endpoint to preserve current managed proxy config defaulting behaviour until that is removed entirely (tested). - The built-in proxy now uses the new endpoint exclusively for it's config - The built-in proxy now has a `-sidecar-for` flag that allows the service ID of the _target_ service to be specified, on the condition that there is exactly one "sidecar" proxy (that is one that has `Proxy.DestinationServiceID` set) for the service registered. - Several fixes for edge cases for SidecarService - A fix for `Alias` checks - when running locally they didn't update their state until some external thing updated the target. If the target service has no checks registered as below, then the alias never made it past critical.	6 years ago
Paul Banks	b83bbf248c	Add Proxy Upstreams to Service Definition (#4639) ... * Refactor Service Definition ProxyDestination. This includes: - Refactoring all internal structs used - Updated tests for both deprecated and new input for: - Agent Services endpoint response - Agent Service endpoint response - Agent Register endpoint - Unmanaged deprecated field - Unmanaged new fields - Managed deprecated upstreams - Managed new - Catalog Register - Unmanaged deprecated field - Unmanaged new fields - Managed deprecated upstreams - Managed new - Catalog Services endpoint response - Catalog Node endpoint response - Catalog Service endpoint response - Updated API tests for all of the above too (both deprecated and new forms of register) TODO: - config package changes for on-disk service definitions - proxy config endpoint - built-in proxy support for new fields * Agent proxy config endpoint updated with upstreams * Config file changes for upstreams. * Add upstream opaque config and update all tests to ensure it works everywhere. * Built in proxy working with new Upstreams config * Command fixes and deprecations * Fix key translation, upstream type defaults and a spate of other subtele bugs found with ned to end test scripts... TODO: tests still failing on one case that needs a fix. I think it's key translation for upstreams nested in Managed proxy struct. * Fix translated keys in API registration. ≈ * Fixes from docs - omit some empty undocumented fields in API - Bring back ServiceProxyDestination in Catalog responses to not break backwards compat - this was removed assuming it was only used internally. * Documentation updates for Upstreams in service definition * Fixes for tests broken by many refactors. * Enable travis on f-connect branch in this branch too. * Add consistent Deprecation comments to ProxyDestination uses * Update version number on deprecation notices, and correct upstream datacenter field with explanation in docs	6 years ago
Martin	feb3ce4ee0	Use target service name instead of ID as connect proxy service name (#4620)	6 years ago
Mitchell Hashimoto	f97bfd5be8	agent: address some basic feedback	6 years ago
Mitchell Hashimoto	7543d270e2	agent/local: support local alias checks	6 years ago
Pierre Souchay	ff53648df2	Merge remote-tracking branch 'origin/master' into ACL_additional_info	6 years ago
Paul Banks	e21723a891	Persist proxy state through agent restart	7 years ago
Mitchell Hashimoto	d9bd4ffebd	agent/local: clarify the non-risk of a full buffer	7 years ago
Mitchell Hashimoto	437689e83c	agent/local: remove outdated comment	7 years ago
Mitchell Hashimoto	2bd39a84a6	agent/local: add Notify mechanism for proxy changes	7 years ago
Mitchell Hashimoto	476ea7b04a	agent: start/stop proxies	7 years ago
Mitchell Hashimoto	7355a614fe	agent/local: store proxy on local state, wip, not working yet	7 years ago
Paul Banks	e0e12e165b	TLS watching integrated into Service with some basic tests. ... There are also a lot of small bug fixes found when testing lots of things end-to-end for the first time and some cleanup now it's integrated with real CA code.	7 years ago
Paul Banks	2a69663448	Agent Connect Proxy config endpoint with hash-based blocking	7 years ago
Paul Banks	e6071051cf	Added connect proxy config and local agent state setup on boot.	7 years ago
Pierre Souchay	c83124a94c	Removed labels from new ACL denied metrics	7 years ago
Pierre Souchay	064f8ad170	Removed consul prefix from metrics as requested by @kyhavlov	7 years ago
Pierre Souchay	65d3a2b26e	Fixed import	7 years ago
Pierre Souchay	f13aa5ba9b	Added labels to improve new metric	7 years ago
Pierre Souchay	d9a23bb2fa	Track calls blocked by ACLs using metrics	7 years ago
Guido Iaquinti	8cd11d5888	Add package name to log output	7 years ago
Josh Soref	94835a2715	Spelling (#3958) ... * spelling: another * spelling: autopilot * spelling: beginning * spelling: circonus * spelling: default * spelling: definition * spelling: distance * spelling: encountered * spelling: enterprise * spelling: expands * spelling: exits * spelling: formatting * spelling: health * spelling: hierarchy * spelling: imposed * spelling: independence * spelling: inspect * spelling: last * spelling: latest * spelling: client * spelling: message * spelling: minimum * spelling: notify * spelling: nonexistent * spelling: operator * spelling: payload * spelling: preceded * spelling: prepared * spelling: programmatically * spelling: required * spelling: reconcile * spelling: responses * spelling: request * spelling: response * spelling: results * spelling: retrieve * spelling: service * spelling: significantly * spelling: specifies * spelling: supported * spelling: synchronization * spelling: synchronous * spelling: themselves * spelling: unexpected * spelling: validations * spelling: value	7 years ago
Frank Schroeder	52e73301f6	local state: clone check to avoid side effect	7 years ago
Frank Schroeder	6bc9d66192	local state: use synchronized access to internal maps	7 years ago
Frank Schroeder	58d52ac580	local state: rename Add{Check,Service}State to Set{Check,Service}State	7 years ago
Frank Schroeder	e144f51b29	local state: move Metadata methods together	7 years ago
Frank Schroeder	4f9e05f634	local state: update documentation of updateSyncState	7 years ago
Frank Schroeder	41c7b0927e	local state: update comments	7 years ago
Frank Schroeder	de57b16d99	local state: address review comments ... * move non-blocking notification mechanism into ae.Trigger * move Pause/Resume into separate type	7 years ago
Frank Schroeder	71c74e62c7	local state: fix anti-entropy state tests ... The anti-entropy tests relied on the side-effect of the StartSync() method to perform a full sync instead of a partial sync. This lead to multiple anti-entropy go routines being started unnecessary retry loops. This change changes the behavior to perform synchronous full syncs when necessary removing the need for all of the time.Sleep and most of the retry loops.	7 years ago
Frank Schroeder	37b95ef98e	local state: fix test with updated error message	7 years ago
Frank Schroeder	b803bf3091	local state: tests compile	7 years ago
Frank Schroeder	0a9ac9749e	local state: replace multi-map state with structs ... The state of the service and health check records was spread out over multiple maps guarded by a single lock. Access to the maps has to happen in a coordinated effort and the tests often violated this which made them brittle and racy. This patch replaces the multiple maps with a single one for both checks and services to make the code less fragile. This is also necessary since moving the local state into its own package creates circular dependencies for the tests. To avoid this the tests can no longer access internal data structures which they should not be doing in the first place. The tests still don't compile but this is a ncessary step in that direction.	7 years ago
Frank Schroeder	6027a9e2a5	local state: move to separate package ... This patch moves the local state to a separate package to further decouple it from the agent code. The code compiles but the tests do not yet.	7 years ago
Frank Schroeder	67a0689f71	Revert "local state: move to separate package" ... This reverts commit d447e823c6.	7 years ago
Frank Schroeder	623e07760a	Revert "local state: replace multi-map state with structs" ... This reverts commit ccbae7da5b.	7 years ago
Frank Schroeder	9ed4b2d631	Revert "local state: tests compile" ... This reverts commit 1af52bf7be.	7 years ago
Frank Schroeder	5ae4c52ca6	Revert "local state: fix test with updated error message" ... This reverts commit e9149f64d9.	7 years ago
Frank Schroeder	67cdfc038e	Revert "local state: fix anti-entropy state tests" ... This reverts commit f8e20cd996.	7 years ago
Frank Schroeder	46641e44d9	Revert "local state: address review comments" ... This reverts commit 1d315075b1.	7 years ago
Frank Schroeder	abd83f2d28	Revert "local state: update comments" ... This reverts commit 42188164f8.	7 years ago
Frank Schroeder	c6e441dd60	Revert "local state: update documentation of updateSyncState" ... This reverts commit e86521e637.	7 years ago
Frank Schroeder	648b4da4f8	Revert "local state: move Metadata methods together" ... This reverts commit 9bc8127728.	7 years ago
Frank Schroeder	fa45a6a547	Revert "local state: rename Add{Check,Service}State to Set{Check,Service}State" ... This reverts commit 9280841a80.	7 years ago
Frank Schroeder	d6f52d9ed1	Revert "local state: use synchronized access to internal maps" ... This reverts commit 39a2d8d25e.	7 years ago
Frank Schroeder	4e862d126d	Revert "local state: clone check to avoid side effect" ... This reverts commit af1243c725.	7 years ago
Frank Schroeder	af1243c725	local state: clone check to avoid side effect	7 years ago