142 Commits (3c58d29fad40917b124e66c161d834fdbc793f95)

Author	SHA1	Message	Date
Matt Keeler	973341a592	ACL Authorizer overhaul (#6620) ... * ACL Authorizer overhaul To account for upcoming features every Authorization function can now take an extra *acl.EnterpriseAuthorizerContext. These are unused in OSS and will always be nil. Additionally the acl package has received some thorough refactoring to enable all of the extra Consul Enterprise specific authorizations including moving sentinel enforcement into the stubbed structs. The Authorizer funcs now return an acl.EnforcementDecision instead of a boolean. This improves the overall interface as it makes multiple Authorizers easily chainable as they now indicate whether they had an authoritative decision or should use some other defaults. A ChainedAuthorizer was added to handle this Authorizer enforcement chain and will never itself return a non-authoritative decision. * Include stub for extra enterprise rules in the global management policy * Allow for an upgrade of the global-management policy	5 years ago
Sarah Christoff	5e26971864	Prune Unhealthy Agents (#6571) ... * Add -prune flag to ForceLeave	5 years ago
R.B. Boyer	06ca8cd2d7	agent: updates to the agent token trigger anti-entropy full syncs (#6577)	5 years ago
Freddy	fdd10dd8b8	Expose HTTP-based paths through Connect proxy (#6446) ... Fixes: #5396 This PR adds a proxy configuration stanza called expose. These flags register listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only listening on the loopback interface, while still accepting traffic from non Connect-enabled services. Under expose there is a boolean checks flag that would automatically expose all registered HTTP and gRPC check paths. This stanza also accepts a paths list to expose individual paths. The primary use case for this functionality would be to expose paths for third parties like Prometheus or the kubelet. Listeners for requests to exposed paths are be configured dynamically at run time. Any time a proxy, or check can be registered, a listener can also be created. In this initial implementation requests to these paths are not authenticated/encrypted.	5 years ago
R.B. Boyer	5882e21b2b	agent: tolerate more failure scenarios during service registration with central config enabled (#6472) ... Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.	5 years ago
Aestek	7c7b7f24fd	Add option to register services and their checks idempotently (#4905)	5 years ago
Alvin Huang	c516fabfac	revert commits on master (#6413)	5 years ago
tradel	672e181399	add serviceID to leaf cert request	5 years ago
Matt Keeler	6a1e0dfed8	Update the v1/agent/service/:service endpoint to output tagged… (#6304)	5 years ago
Mike Morris	65be58703c	connect: remove managed proxies (#6220) ... * connect: remove managed proxies implementation and all supporting config options and structs * connect: remove deprecated ProxyDestination * command: remove CONNECT_PROXY_TOKEN env var * agent: remove entire proxyprocess proxy manager * test: remove all managed proxy tests * test: remove irrelevant managed proxy note from TestService_ServerTLSConfig * test: update ContentHash to reflect managed proxy removal * test: remove deprecated ProxyDestination test * telemetry: remove managed proxy note * http: remove /v1/agent/connect/proxy endpoint * ci: remove deprecated test exclusion * website: update managed proxies deprecation page to note removal * website: remove managed proxy configuration API docs * website: remove managed proxy note from built-in proxy config * website: add note on removing proxy subdirectory of data_dir	5 years ago
Alvin Huang	ef6b80bab2	resolve circleci config conflicts	5 years ago
Christian Muehlhaeuser	7753b97cc7	Simplified code in various places (#6176) ... All these changes should have no side-effects or change behavior: - Use bytes.Buffer's String() instead of a conversion - Use time.Since and time.Until where fitting - Drop unnecessary returns and assignment	5 years ago
Matt Keeler	8d953f5840	Implement Mesh Gateways ... This includes both ingress and egress functionality.	5 years ago
hashicorp-ci	f4304e2e5b	Merge Consul OSS branch 'master' at commit 4eb73973b6	6 years ago
Pierre Souchay	0e907f5aa8	Support for maximum size for Output of checks (#5233) ... * Support for maximum size for Output of checks This PR allows users to limit the size of output produced by checks at the agent and check level. When set at the agent level, it will limit the output for all checks monitored by the agent. When set at the check level, it can override the agent max for a specific check but only if it is lower than the agent max. Default value is 4k, and input must be at least 1.	6 years ago
Matt Keeler	f3d9b999ee	Add tagged addresses for services (#5965) ... This allows addresses to be tagged at the service level similar to what we allow for nodes already. The address translation that can be enabled with the `translate_wan_addrs` config was updated to take these new addresses into account as well.	6 years ago
Matt Keeler	f665695b6b	Ensure ServiceName is populated correctly for agent service checks ... Also update some snapshot agent docs * Enforce correct permissions when registering a check Previously we had attempted to enforce service:write for a check associated with a service instead of node:write on the agent but due to how we decoded the health check from the request it would never do it properly. This commit fixes that. * Update website/source/docs/commands/snapshot/agent.html.markdown.erb Co-Authored-By: mkeeler <mkeeler@users.noreply.github.com>	6 years ago
Matt Keeler	d0f410cd84	Make a few config entry endpoints return 404s and allow for snake_case and lowercase key names. (#5748)	6 years ago
Matt Keeler	afa1cc98d1	Implement data filtering of some endpoints (#5579) ... Fixes: #4222 # Data Filtering This PR will implement filtering for the following endpoints: ## Supported HTTP Endpoints - `/agent/checks` - `/agent/services` - `/catalog/nodes` - `/catalog/service/:service` - `/catalog/connect/:service` - `/catalog/node/:node` - `/health/node/:node` - `/health/checks/:service` - `/health/service/:service` - `/health/connect/:service` - `/health/state/:state` - `/internal/ui/nodes` - `/internal/ui/services` More can be added going forward and any endpoint which is used to list some data is a good candidate. ## Usage When using the HTTP API a `filter` query parameter can be used to pass a filter expression to Consul. Filter Expressions take the general form of: ``` <selector> == <value> <selector> != <value> <value> in <selector> <value> not in <selector> <selector> contains <value> <selector> not contains <value> <selector> is empty <selector> is not empty not <other expression> <expression 1> and <expression 2> <expression 1> or <expression 2> ``` Normal boolean logic and precedence is supported. All of the actual filtering and evaluation logic is coming from the [go-bexpr](https://github.com/hashicorp/go-bexpr) library ## Other changes Adding the `Internal.ServiceDump` RPC endpoint. This will allow the UI to filter services better.	6 years ago
R.B. Boyer	f4a3b9d518	fix typos reported by golangci-lint:misspell (#5434)	6 years ago
Matt Keeler	118adbb123	ACL Token Persistence and Reloading (#5328) ... This PR adds two features which will be useful for operators when ACLs are in use. 1. Tokens set in configuration files are now reloadable. 2. If `acl.enable_token_persistence` is set to `true` in the configuration, tokens set via the `v1/agent/token` endpoint are now persisted to disk and loaded when the agent starts (or during configuration reload) Note that token persistence is opt-in so our users who do not want tokens on the local disk will see no change. Some other secondary changes: * Refactored a bunch of places where the replication token is retrieved from the token store. This token isn't just for replicating ACLs and now it is named accordingly. * Allowed better paths in the `v1/agent/token/` API. Instead of paths like: `v1/agent/token/acl_replication_token` the path can now be just `v1/agent/token/replication`. The old paths remain to be valid. * Added a couple new API functions to set tokens via the new paths. Deprecated the old ones and pointed to the new names. The names are also generally better and don't imply that what you are setting is for ACLs but rather are setting ACL tokens. There is a minor semantic difference there especially for the replication token as again, its no longer used only for ACL token/policy replication. The new functions will detect 404s and fallback to using the older token paths when talking to pre-1.4.3 agents. * Docs updated to reflect the API additions and to show using the new endpoints. * Updated the ACL CLI set-agent-tokens command to use the non-deprecated APIs.	6 years ago
Matt Keeler	baa8946ea6	cache: Pass through wait query param to the cache.Get (#5203) ... This adds a MaxQueryTime field to the connect ca leaf cache request type and populates it via the wait query param. The cache will then do the right thing and timeout the operation as expected if no new leaf cert is available within that time. Fixes #4462 The reproduction scenario in the original issue now times out appropriately.	6 years ago
Grégoire Seux	4f62a3b528	Implement /v1/agent/health/service/<service name> endpoint (#3551) ... This endpoint aggregates all checks related to <service id> on the agent and return an appropriate http code + the string describing the worst check. This allows to cleanly expose service status to other component, hiding complexity of multiple checks. This is especially useful to use consul to feed a load balancer which would delegate health checking to consul agent. Exposing this endpoint on the agent is necessary to avoid a hit on consul servers and avoid decreasing resiliency (this endpoint will work even if there is no consul leader in the cluster).	6 years ago
Jack Pearkes	a90c29e60d	Doc changes for 1.4 Final (#4870) ... * website: add multi-dc enterprise landing page * website: switch all 1.4.0 alerts/RC warnings * website: connect product wording Co-Authored-By: pearkes <jackpearkes@gmail.com> * website: remove RC notification * commmand/acl: fix usage docs for ACL tokens * agent: remove comment, OperatorRead * website: improve multi-dc docs Still not happy with this but tried to make it slightly more informative. * website: put back acl guide warning for 1.4.0 * website: simplify multi-dc page and respond to feedback * Fix Multi-DC typos on connect index page. * Improve Multi-DC overview. A full guide is a WIP and will be added post-release. * Fixes typo avaiable > available	6 years ago
Jack Pearkes	8c684db488	New command: consul debug (#4754) ... * agent/debug: add package for debugging, host info * api: add v1/agent/host endpoint * agent: add v1/agent/host endpoint * command/debug: implementation of static capture * command/debug: tests and only configured targets * agent/debug: add basic test for host metrics * command/debug: add methods for dynamic data capture * api: add debug/pprof endpoints * command/debug: add pprof * command/debug: timing, wg, logs to disk * vendor: add gopsutil/disk * command/debug: add a usage section * website: add docs for consul debug * agent/host: require operator:read * api/host: improve docs and no retry timing * command/debug: fail on extra arguments * command/debug: fixup file permissions to 0644 * command/debug: remove server flags * command/debug: improve clarity of usage section * api/debug: add Trace for profiling, fix profile * command/debug: capture profile and trace at the same time * command/debug: add index document * command/debug: use "clusters" in place of members * command/debug: remove address in output * command/debug: improve comment on metrics sleep * command/debug: clarify usage * agent: always register pprof handlers and protect This will allow us to avoid a restart of a target agent for profiling by always registering the pprof handlers. Given this is a potentially sensitive path, it is protected with an operator:read ACL and enable debug being set to true on the target agent. enable_debug still requires a restart. If ACLs are disabled, enable_debug is sufficient. * command/debug: use trace.out instead of .prof More in line with golang docs. * agent: fix comment wording * agent: wrap table driven tests in t.run()	6 years ago
Kyle Havlovitz	c617326470	re-add Connect multi-dc config changes ... This reverts commit 8bcfbaffb6.	6 years ago
Jack Pearkes	8bcfbaffb6	Revert "Connect multi-dc config" (#4784)	6 years ago
Aestek	25f04fbd21	[Security] Add finer control over script checks (#4715) ... * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording	6 years ago
Kyle Havlovitz	56dc426227	agent: add primary_datacenter and connect replication config options	6 years ago
Paul Banks	c9217c958e	merge feedback: fix typos; actually use deliverLatest added previously but not plumbed in	6 years ago
Paul Banks	161482d2cd	Fix up tests broken by master merge; add proxy tests to services command (and fix it!); actually run the proxycfg.Manager	6 years ago
Paul Banks	dca1303d05	Connect Envoy Command (#4735) ... * Plumb xDS server and proxyxfg into the agent startup * Add `consul connect envoy` command to allow running Envoy as a connect sidecar. * Add test for help tabs; typos and style fixups from review	6 years ago
Paul Banks	e812f5516a	Add -sidecar-for and new /agent/service/:service_id endpoint (#4691) ... - A new endpoint `/v1/agent/service/:service_id` which is a generic way to look up the service for a single instance. The primary value here is that it: - **supports hash-based blocking** and so; - **replaces `/agent/connect/proxy/:proxy_id`** as the mechanism the built-in proxy uses to read its config. - It's not proxy specific and so works for any service. - It has a temporary shim to call through to the existing endpoint to preserve current managed proxy config defaulting behaviour until that is removed entirely (tested). - The built-in proxy now uses the new endpoint exclusively for it's config - The built-in proxy now has a `-sidecar-for` flag that allows the service ID of the _target_ service to be specified, on the condition that there is exactly one "sidecar" proxy (that is one that has `Proxy.DestinationServiceID` set) for the service registered. - Several fixes for edge cases for SidecarService - A fix for `Alias` checks - when running locally they didn't update their state until some external thing updated the target. If the target service has no checks registered as below, then the alias never made it past critical.	6 years ago
Paul Banks	1e7eace066	Add SidecarService Syntax sugar to Service Definition (#4686) ... * Added new Config for SidecarService in ServiceDefinitions. * WIP: all the code needed for SidecarService is written... none of it is tested other than config :). Need API updates too. * Test coverage for the new sidecarServiceFromNodeService method. * Test API registratrion with SidecarService * Recursive Key Translation 🤦 * Add tests for nested sidecar defintion arrays to ensure they are translated correctly * Use dedicated internal state rather than Service Meta for tracking sidecars for deregistration. Add tests for deregistration. * API struct for agent register. No other endpoint should be affected yet. * Additional test cases to cover updates to API registrations	6 years ago
Paul Banks	b83bbf248c	Add Proxy Upstreams to Service Definition (#4639) ... * Refactor Service Definition ProxyDestination. This includes: - Refactoring all internal structs used - Updated tests for both deprecated and new input for: - Agent Services endpoint response - Agent Service endpoint response - Agent Register endpoint - Unmanaged deprecated field - Unmanaged new fields - Managed deprecated upstreams - Managed new - Catalog Register - Unmanaged deprecated field - Unmanaged new fields - Managed deprecated upstreams - Managed new - Catalog Services endpoint response - Catalog Node endpoint response - Catalog Service endpoint response - Updated API tests for all of the above too (both deprecated and new forms of register) TODO: - config package changes for on-disk service definitions - proxy config endpoint - built-in proxy support for new fields * Agent proxy config endpoint updated with upstreams * Config file changes for upstreams. * Add upstream opaque config and update all tests to ensure it works everywhere. * Built in proxy working with new Upstreams config * Command fixes and deprecations * Fix key translation, upstream type defaults and a spate of other subtele bugs found with ned to end test scripts... TODO: tests still failing on one case that needs a fix. I think it's key translation for upstreams nested in Managed proxy struct. * Fix translated keys in API registration. ≈ * Fixes from docs - omit some empty undocumented fields in API - Bring back ServiceProxyDestination in Catalog responses to not break backwards compat - this was removed assuming it was only used internally. * Documentation updates for Upstreams in service definition * Fixes for tests broken by many refactors. * Enable travis on f-connect branch in this branch too. * Add consistent Deprecation comments to ProxyDestination uses * Update version number on deprecation notices, and correct upstream datacenter field with explanation in docs	6 years ago
Paul O'Connor	6b7f03911e	Fix prometheus error message (#4745)	6 years ago
Pierre Souchay	2fe728c7bd	Ensure that Proxies ARE always cleaned up, event with DeregisterCriticalServiceAfter (#4649) ... This fixes https://github.com/hashicorp/consul/issues/4648	6 years ago
Pierre Souchay	eddcf228ea	Implementation of Weights Data structures (#4468) ... * Implementation of Weights Data structures Adding this datastructure will allow us to resolve the issues #1088 and #4198 This new structure defaults to values: ``` { Passing: 1, Warning: 0 } ``` Which means, use weight of 0 for a Service in Warning State while use Weight 1 for a Healthy Service. Thus it remains compatible with previous Consul versions. * Implemented weights for DNS SRV Records * DNS properly support agents with weight support while server does not (backwards compatibility) * Use Warning value of Weights of 1 by default When using DNS interface with only_passing = false, all nodes with non-Critical healthcheck used to have a weight value of 1. While having weight.Warning = 0 as default value, this is probably a bad idea as it breaks ascending compatibility. Thus, we put a default value of 1 to be consistent with existing behaviour. * Added documentation for new weight field in service description * Better documentation about weights as suggested by @banks * Return weight = 1 for unknown Check states as suggested by @banks * Fixed typo (of -> or) in error message as requested by @mkeeler * Fixed unstable unit test TestRetryJoin * Fixed unstable tests * Fixed wrong Fatalf format in `testrpc/wait.go` * Added notes regarding DNS SRV lookup limitations regarding number of instances * Documentation fixes and clarification regarding SRV records with weights as requested by @banks * Rephrase docs	6 years ago
Matt Keeler	870a6ad6a8	Handle resolving proxy tokens when parsing HTTP requests (#4453) ... Fixes: #4441 This fixes the issue with Connect Managed Proxies + ACLs being broken. The underlying problem was that the token parsed for most http endpoints was sent untouched to the servers via the RPC request. These changes make it so that at the HTTP endpoint when parsing the token we additionally attempt to convert potential proxy tokens into regular tokens before sending to the RPC endpoint. Proxy tokens are only valid on the agent with the managed proxy so the resolution has to happen before it gets forwarded anywhere.	6 years ago
Matt Keeler	3fe5f566f2	Persist proxies from config files ... Also change how loadProxies works. Now it will load all persisted proxies into a map, then when loading config file proxies will look up the previous proxy token in that map.	6 years ago
Paul Banks	17789d4fe3	register TCP check for managed proxies	7 years ago
Paul Banks	420ae3df69	Limit proxy telemetry config to only be visible with authenticated with a proxy token	7 years ago
Paul Banks	c6ef6a61c9	Refactor to use embedded struct.	7 years ago
Paul Banks	38405bd4a9	Allow user override of proxy telemetry config	7 years ago
Paul Banks	7649d630c6	Basic proxy telemetry working; not sure if it's too ugly; need to instrument things we care about	7 years ago
Paul Banks	d83f2e8e21	Expose telemetry config from RuntimeConfig to proxy config endpoint	7 years ago
Paul Banks	43b48bc06b	Get agent cache tests passing without global hit count (which is racy). ... Few other fixes in here just to get a clean run locally - they are all also fixed in other PRs but shouldn't conflict. This should be robust to timing between goroutines now.	7 years ago
Mitchell Hashimoto	e9e6514c9b	agent: disallow deregistering a managed proxy directly	7 years ago
Mitchell Hashimoto	66a573e496	agent: deregister service deregisters the proxy along with it	7 years ago
Mitchell Hashimoto	0d6dcbd2f1	agent: disallow API registration with managed proxy if not enabled	7 years ago