106 Commits (6c39a9825580e3e4375acafed557d73146b1443f)

Author	SHA1	Message	Date
Iryna Shustava	d7d44f6ae7	Save exposed ports in agent's store and expose them via API (#10173) ... * Save exposed HTTP or GRPC ports to the agent's store * Add those the health checks API so we can retrieve them from the API * Change redirect-traffic command to also exclude those ports from inbound traffic redirection when expose.checks is set to true.	4 years ago
Mark Anderson	4d0346bc46	Fixup filtering tests ... Missed annotations for SocketPath, LocalSocketPath, LocalServiceSocketPath Signed-off-by: Mark Anderson <manderson@hashicorp.com>	4 years ago
Mark Anderson	8040f91a43	Add support for downstreams ... Enhance config by adding SocketPath and LocalSocketPath config values Supports syntax of the form: ``` services { name = "sock_forwarder" id = "sock_forwarder.1" socket_path = "/tmp/downstream_3.sock" connect { sidecar_service { proxy { local_service_socket_path = "/tmp/downstream.sock" } } } } ``` Signed-off-by: Mark Anderson <manderson@hashicorp.com>	4 years ago
Mark Anderson	583ae65d5b	Convert mode to string representation ... Signed-off-by: Mark Anderson <manderson@hashicorp.com>	4 years ago
Mark Anderson	8b1217417a	First changes for unix domain sockets upstreams ... Start making structure changes to support unix domain socket address for upstreams upstreams = [ { destination_name = "echo-service" local_bind_socket_path = "/tmp/upstream.sock" config { passive_health_check { interval = "10s" max_failures = 42 } } } Signed-off-by: Mark Anderson <manderson@hashicorp.com>	4 years ago
freddygv	7cb3f32672	Convert new tproxy structs in api module into ptrs ... This way we avoid serializing these when empty. Otherwise users of the latest version of the api submodule cannot interact with older versions of Consul, because a new api client would send keys that the older Consul doesn't recognize yet.	4 years ago
freddygv	7bd51ff536	Replace TransparentProxy bool with ProxyMode ... This PR replaces the original boolean used to configure transparent proxy mode. It was replaced with a string mode that can be set to: - "": Empty string is the default for when the setting should be defaulted from other configuration like config entries. - "direct": Direct mode is how applications originally opted into the mesh. Proxy listeners need to be dialed directly. - "transparent": Transparent mode enables configuring Envoy as a transparent proxy. Traffic must be captured and redirected to the inbound and outbound listeners. This PR also adds a struct for transparent proxy specific configuration. Initially this is not stored as a pointer. Will revisit that decision before GA.	4 years ago
Seth Hoenig	cd1cd4febd	api: enable query options on agent endpoints ... This PR adds support for setting QueryOptions on a few agent API endpoints. Nomad needs to be able to set the Namespace field on these endpoints to: - query for services / checks in a namespace - deregister services / checks in a namespace - update TTL status on checks in a namespace	4 years ago
Freddy	8207b832df	Add TransparentProxy option to proxy definitions	4 years ago
Christopher Broglie	f0307c73e5	Add support for configuring TLS ServerName for health checks ... Some TLS servers require SNI, but the Golang HTTP client doesn't include it in the ClientHello when connecting to an IP address. This change adds a new TLSServerName field to health check definitions to optionally set it. This fixes #9473.	4 years ago
freddygv	6090cfcf68	PR comments	4 years ago
freddygv	23ffa3d3f3	And another test fix	4 years ago
freddygv	6fd30d0384	Add TransparentProxy opt to proxy definition	4 years ago
Daniel Nephin	5a7f4c0dea	Merge pull request #8609 from hashicorp/dnephin/add-query-options-to-ServiceRegister ... api: Add a context to ServiceRegisterOpts	4 years ago
R.B. Boyer	7c7a3e5165	command: when generating envoy bootstrap configs use the datacenter returned from the agent services endpoint (#9229) ... Fixes #9215	4 years ago
Matt Keeler	66fd23d67f	Refactor to call non-voting servers read replicas (#9191) ... Co-authored-by: Kit Patella <kit@jepsen.io>	4 years ago
Matt Keeler	7f87adbcf4	Remove this constant as it is soon to be changing and we want to prevent backwards compat issues (#9193)	4 years ago
Daniel Nephin	d083d4717b	api: Add a context to ServiceRegisterOpts ... So that operations can timeout instead of hanging forever	4 years ago
Matt Keeler	7c3914d89e	Add helpers to the API client to help with getting information from `AgentMember` tags (#8575) ... Lots of constants were added for various tags that would concern users and are not already parsed out. Additionally two methods on the AgentMember type were added to ask a member what its ACL Mode is and whether its a server or not.	4 years ago
Seth Hoenig	95f46eb3ed	api/agent: enable setting SuccessBeforePassing and FailuresBeforeCritical in API (#7949) ... Fixes #7764 Until now these two fields could only be set through on-disk agent configuration. This change adds the fields to the agent API struct definition so that they can be set using the agent HTTP API.	4 years ago
s-christoff	818d00fda3	Add AgentMemberStatus const (#8110) ... * Add AgentMemberStatus const	4 years ago
Chris Piraino	3d2de925d8	Add support for ingress-gateway in CLI command (#7618) ... * Add support for ingress-gateway in CLI command - Supports -register command - Creates a static Envoy listener that exposes only the /ready API so that we can register a TCP healthcheck against the ingress gateway itself - Updates ServiceAddressValue.String() to be more in line with Value()	5 years ago
Freddy	18d356899c	Enable CLI to register terminating gateways (#7500) ... * Enable CLI to register terminating gateways * Centralize gateway proxy configuration	5 years ago
Akshay Ganeshen	8beb716414	feat: support sending body in HTTP checks (#6602)	5 years ago
Fouad Zaryouh	ef6399928d	api: add replace-existing-checks param to the api package (#7136)	5 years ago
Chris Piraino	401221de58	Allow users to configure either unstructured or JSON logging (#7130) ... * hclog Allow users to choose between unstructured and JSON logging	5 years ago
neillyt	a2ab9b4aee	Fix Namespace Typo (#7030)	5 years ago
Matt Keeler	5934f803bf	Sync of OSS changes to support namespaces (#6909)	5 years ago
Freddy	60f6ec0c2f	Store check type in catalog (#6561)	5 years ago
Sarah Christoff	5e26971864	Prune Unhealthy Agents (#6571) ... * Add -prune flag to ForceLeave	5 years ago
Freddy	fdd10dd8b8	Expose HTTP-based paths through Connect proxy (#6446) ... Fixes: #5396 This PR adds a proxy configuration stanza called expose. These flags register listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only listening on the loopback interface, while still accepting traffic from non Connect-enabled services. Under expose there is a boolean checks flag that would automatically expose all registered HTTP and gRPC check paths. This stanza also accepts a paths list to expose individual paths. The primary use case for this functionality would be to expose paths for third parties like Prometheus or the kubelet. Listeners for requests to exposed paths are be configured dynamically at run time. Any time a proxy, or check can be registered, a listener can also be created. In this initial implementation requests to these paths are not authenticated/encrypted.	5 years ago
Mike Morris	65be58703c	connect: remove managed proxies (#6220) ... * connect: remove managed proxies implementation and all supporting config options and structs * connect: remove deprecated ProxyDestination * command: remove CONNECT_PROXY_TOKEN env var * agent: remove entire proxyprocess proxy manager * test: remove all managed proxy tests * test: remove irrelevant managed proxy note from TestService_ServerTLSConfig * test: update ContentHash to reflect managed proxy removal * test: remove deprecated ProxyDestination test * telemetry: remove managed proxy note * http: remove /v1/agent/connect/proxy endpoint * ci: remove deprecated test exclusion * website: update managed proxies deprecation page to note removal * website: remove managed proxy configuration API docs * website: remove managed proxy note from built-in proxy config * website: add note on removing proxy subdirectory of data_dir	5 years ago
Matt Keeler	bdebe62fd0	Fix some tests that I broke when refactoring the ConfigSnapshot (#6051) ... * Fix some tests that I broke when refactoring the ConfigSnapshot * Make sure the MeshGateway config is added to all the right api structs * Fix some more tests	5 years ago
Matt Keeler	8d953f5840	Implement Mesh Gateways ... This includes both ingress and egress functionality.	5 years ago
Matt Keeler	f3d9b999ee	Add tagged addresses for services (#5965) ... This allows addresses to be tagged at the service level similar to what we allow for nodes already. The address translation that can be enabled with the `translate_wan_addrs` config was updated to take these new addresses into account as well.	6 years ago
Matt Keeler	afa1cc98d1	Implement data filtering of some endpoints (#5579) ... Fixes: #4222 # Data Filtering This PR will implement filtering for the following endpoints: ## Supported HTTP Endpoints - `/agent/checks` - `/agent/services` - `/catalog/nodes` - `/catalog/service/:service` - `/catalog/connect/:service` - `/catalog/node/:node` - `/health/node/:node` - `/health/checks/:service` - `/health/service/:service` - `/health/connect/:service` - `/health/state/:state` - `/internal/ui/nodes` - `/internal/ui/services` More can be added going forward and any endpoint which is used to list some data is a good candidate. ## Usage When using the HTTP API a `filter` query parameter can be used to pass a filter expression to Consul. Filter Expressions take the general form of: ``` <selector> == <value> <selector> != <value> <value> in <selector> <value> not in <selector> <selector> contains <value> <selector> not contains <value> <selector> is empty <selector> is not empty not <other expression> <expression 1> and <expression 2> <expression 1> or <expression 2> ``` Normal boolean logic and precedence is supported. All of the actual filtering and evaluation logic is coming from the [go-bexpr](https://github.com/hashicorp/go-bexpr) library ## Other changes Adding the `Internal.ServiceDump` RPC endpoint. This will allow the UI to filter services better.	6 years ago
R.B. Boyer	f8bcfcfa46	api: fix panic in 'consul acl set-agent-token' (#5533) ... api: fix panic in 'consul acl set-agent-token' Fixes #5531	6 years ago
Matt Keeler	118adbb123	ACL Token Persistence and Reloading (#5328) ... This PR adds two features which will be useful for operators when ACLs are in use. 1. Tokens set in configuration files are now reloadable. 2. If `acl.enable_token_persistence` is set to `true` in the configuration, tokens set via the `v1/agent/token` endpoint are now persisted to disk and loaded when the agent starts (or during configuration reload) Note that token persistence is opt-in so our users who do not want tokens on the local disk will see no change. Some other secondary changes: * Refactored a bunch of places where the replication token is retrieved from the token store. This token isn't just for replicating ACLs and now it is named accordingly. * Allowed better paths in the `v1/agent/token/` API. Instead of paths like: `v1/agent/token/acl_replication_token` the path can now be just `v1/agent/token/replication`. The old paths remain to be valid. * Added a couple new API functions to set tokens via the new paths. Deprecated the old ones and pointed to the new names. The names are also generally better and don't imply that what you are setting is for ACLs but rather are setting ACL tokens. There is a minor semantic difference there especially for the replication token as again, its no longer used only for ACL token/policy replication. The new functions will detect 404s and fallback to using the older token paths when talking to pre-1.4.3 agents. * Docs updated to reflect the API additions and to show using the new endpoints. * Updated the ACL CLI set-agent-tokens command to use the non-deprecated APIs.	6 years ago
Grégoire Seux	4f62a3b528	Implement /v1/agent/health/service/<service name> endpoint (#3551) ... This endpoint aggregates all checks related to <service id> on the agent and return an appropriate http code + the string describing the worst check. This allows to cleanly expose service status to other component, hiding complexity of multiple checks. This is especially useful to use consul to feed a load balancer which would delegate health checking to consul agent. Exposing this endpoint on the agent is necessary to avoid a hit on consul servers and avoid decreasing resiliency (this endpoint will work even if there is no consul leader in the cluster).	6 years ago
Jack Pearkes	8c684db488	New command: consul debug (#4754) ... * agent/debug: add package for debugging, host info * api: add v1/agent/host endpoint * agent: add v1/agent/host endpoint * command/debug: implementation of static capture * command/debug: tests and only configured targets * agent/debug: add basic test for host metrics * command/debug: add methods for dynamic data capture * api: add debug/pprof endpoints * command/debug: add pprof * command/debug: timing, wg, logs to disk * vendor: add gopsutil/disk * command/debug: add a usage section * website: add docs for consul debug * agent/host: require operator:read * api/host: improve docs and no retry timing * command/debug: fail on extra arguments * command/debug: fixup file permissions to 0644 * command/debug: remove server flags * command/debug: improve clarity of usage section * api/debug: add Trace for profiling, fix profile * command/debug: capture profile and trace at the same time * command/debug: add index document * command/debug: use "clusters" in place of members * command/debug: remove address in output * command/debug: improve comment on metrics sleep * command/debug: clarify usage * agent: always register pprof handlers and protect This will allow us to avoid a restart of a target agent for profiling by always registering the pprof handlers. Given this is a potentially sensitive path, it is protected with an operator:read ACL and enable debug being set to true on the target agent. enable_debug still requires a restart. If ACLs are disabled, enable_debug is sufficient. * command/debug: use trace.out instead of .prof More in line with golang docs. * agent: fix comment wording * agent: wrap table driven tests in t.run()	6 years ago
R.B. Boyer	c310451b2b	cli: avoid passing envoy bootstrap configuration as arguments (#4747) ... Play a trick with CLOEXEC to pass the envoy bootstrap configuration as an open file descriptor to the exec'd envoy process. The file only briefly touches disk before being unlinked. We convince envoy to read from this open file descriptor by using the /dev/fd/$FDNUMBER mechanism to read the open file descriptor as a file. Because the filename no longer has an extension envoy's sniffing logic falls back on JSON instead of YAML, so the bootstrap configuration must be generated as JSON instead.	6 years ago
Paul Banks	e812f5516a	Add -sidecar-for and new /agent/service/:service_id endpoint (#4691) ... - A new endpoint `/v1/agent/service/:service_id` which is a generic way to look up the service for a single instance. The primary value here is that it: - **supports hash-based blocking** and so; - **replaces `/agent/connect/proxy/:proxy_id`** as the mechanism the built-in proxy uses to read its config. - It's not proxy specific and so works for any service. - It has a temporary shim to call through to the existing endpoint to preserve current managed proxy config defaulting behaviour until that is removed entirely (tested). - The built-in proxy now uses the new endpoint exclusively for it's config - The built-in proxy now has a `-sidecar-for` flag that allows the service ID of the _target_ service to be specified, on the condition that there is exactly one "sidecar" proxy (that is one that has `Proxy.DestinationServiceID` set) for the service registered. - Several fixes for edge cases for SidecarService - A fix for `Alias` checks - when running locally they didn't update their state until some external thing updated the target. If the target service has no checks registered as below, then the alias never made it past critical.	6 years ago
Paul Banks	1e7eace066	Add SidecarService Syntax sugar to Service Definition (#4686) ... * Added new Config for SidecarService in ServiceDefinitions. * WIP: all the code needed for SidecarService is written... none of it is tested other than config :). Need API updates too. * Test coverage for the new sidecarServiceFromNodeService method. * Test API registratrion with SidecarService * Recursive Key Translation 🤦 * Add tests for nested sidecar defintion arrays to ensure they are translated correctly * Use dedicated internal state rather than Service Meta for tracking sidecars for deregistration. Add tests for deregistration. * API struct for agent register. No other endpoint should be affected yet. * Additional test cases to cover updates to API registrations	6 years ago
Paul Banks	b83bbf248c	Add Proxy Upstreams to Service Definition (#4639) ... * Refactor Service Definition ProxyDestination. This includes: - Refactoring all internal structs used - Updated tests for both deprecated and new input for: - Agent Services endpoint response - Agent Service endpoint response - Agent Register endpoint - Unmanaged deprecated field - Unmanaged new fields - Managed deprecated upstreams - Managed new - Catalog Register - Unmanaged deprecated field - Unmanaged new fields - Managed deprecated upstreams - Managed new - Catalog Services endpoint response - Catalog Node endpoint response - Catalog Service endpoint response - Updated API tests for all of the above too (both deprecated and new forms of register) TODO: - config package changes for on-disk service definitions - proxy config endpoint - built-in proxy support for new fields * Agent proxy config endpoint updated with upstreams * Config file changes for upstreams. * Add upstream opaque config and update all tests to ensure it works everywhere. * Built in proxy working with new Upstreams config * Command fixes and deprecations * Fix key translation, upstream type defaults and a spate of other subtele bugs found with ned to end test scripts... TODO: tests still failing on one case that needs a fix. I think it's key translation for upstreams nested in Managed proxy struct. * Fix translated keys in API registration. ≈ * Fixes from docs - omit some empty undocumented fields in API - Bring back ServiceProxyDestination in Catalog responses to not break backwards compat - this was removed assuming it was only used internally. * Documentation updates for Upstreams in service definition * Fixes for tests broken by many refactors. * Enable travis on f-connect branch in this branch too. * Add consistent Deprecation comments to ProxyDestination uses * Update version number on deprecation notices, and correct upstream datacenter field with explanation in docs	6 years ago
Pierre Souchay	eddcf228ea	Implementation of Weights Data structures (#4468) ... * Implementation of Weights Data structures Adding this datastructure will allow us to resolve the issues #1088 and #4198 This new structure defaults to values: ``` { Passing: 1, Warning: 0 } ``` Which means, use weight of 0 for a Service in Warning State while use Weight 1 for a Healthy Service. Thus it remains compatible with previous Consul versions. * Implemented weights for DNS SRV Records * DNS properly support agents with weight support while server does not (backwards compatibility) * Use Warning value of Weights of 1 by default When using DNS interface with only_passing = false, all nodes with non-Critical healthcheck used to have a weight value of 1. While having weight.Warning = 0 as default value, this is probably a bad idea as it breaks ascending compatibility. Thus, we put a default value of 1 to be consistent with existing behaviour. * Added documentation for new weight field in service description * Better documentation about weights as suggested by @banks * Return weight = 1 for unknown Check states as suggested by @banks * Fixed typo (of -> or) in error message as requested by @mkeeler * Fixed unstable unit test TestRetryJoin * Fixed unstable tests * Fixed wrong Fatalf format in `testrpc/wait.go` * Added notes regarding DNS SRV lookup limitations regarding number of instances * Documentation fixes and clarification regarding SRV records with weights as requested by @banks * Rephrase docs	6 years ago
Mitchell Hashimoto	297e4f272e	api: support native connect	7 years ago
Paul Banks	7c7c858a9a	More test cleanup	7 years ago
Mitchell Hashimoto	aaa2431350	agent: change connect command paths to be slices, not strings ... This matches other executable configuration and allows us to cleanly separate executable from arguments without trying to emulate shell parsing.	7 years ago
Paul Banks	e0e12e165b	TLS watching integrated into Service with some basic tests. ... There are also a lot of small bug fixes found when testing lots of things end-to-end for the first time and some cleanup now it's integrated with real CA code.	7 years ago
Paul Banks	36dbd878c9	Adds `api` client code and tests for new Proxy Config endpoint, registering with proxy and seeing proxy config in /agent/services list.	7 years ago