73 Commits (add-downstream-service-meta)

Author	SHA1	Message	Date
Paul Glass	bcda205f88	Add consul.xds.server.streamStart metric (#14957) ... This adds a new consul.xds.server.streamStart metric to measure the time taken to first generate xDS resources after an xDS stream is opened.	2 years ago
malizz	84b0f408fa	Support Stale Queries for Trust Bundle Lookups (#14724) ... * initial commit * add tags, add conversations * add test for query options utility functions * update previous tests * fix test * don't error out on empty context * add changelog * update decode config	2 years ago
Dan Upton	1c2c975b0b	xDS Load Balancing (#14397) ... Prior to #13244, connect proxies and gateways could only be configured by an xDS session served by the local client agent. In an upcoming release, it will be possible to deploy a Consul service mesh without client agents. In this model, xDS sessions will be handled by the servers themselves, which necessitates load-balancing to prevent a single server from receiving a disproportionate amount of load and becoming overwhelmed. This introduces a simple form of load-balancing where Consul will attempt to achieve an even spread of load (xDS sessions) between all healthy servers. It does so by implementing a concurrent session limiter (limiter.SessionLimiter) and adjusting the limit according to autopilot state and proxy service registrations in the catalog. If a server is already over capacity (i.e. the session limit is lowered), Consul will begin draining sessions to rebalance the load. This will result in the client receiving a `RESOURCE_EXHAUSTED` status code. It is the client's responsibility to observe this response and reconnect to a different server. Users of the gRPC client connection brokered by the consul-server-connection-manager library will get this for free. The rate at which Consul will drain sessions to rebalance load is scaled dynamically based on the number of proxies in the catalog.	2 years ago
Daniel Upton	13c04a13af	proxycfg: terminate stream on irrecoverable errors ... This is the OSS portion of enterprise PR 2339. It improves our handling of "irrecoverable" errors in proxycfg data sources. The canonical example of this is what happens when the ACL token presented by Envoy is deleted/revoked. Previously, the stream would get "stuck" until the xDS server re-checked the token (after 5 minutes) and terminated the stream. Materializers would also sit burning resources retrying something that could never succeed. Now, it is possible for data sources to mark errors as "terminal" which causes the xDS stream to be closed immediately. Similarly, the submatview.Store will evict materializers when it observes they have encountered such an error.	2 years ago
Dan Upton	b9e525d689	grpc: rename public/private directories to external/internal (#13721) ... Previously, public referred to gRPC services that are both exposed on the dedicated gRPC port and have their definitions in the proto-public directory (so were considered usable by 3rd parties). Whereas private referred to services on the multiplexed server port that are only usable by agents and other servers. Now, we're splitting these definitions, such that external/internal refers to the port and public/private refers to whether they can be used by 3rd parties. This is necessary because the peering replication API needs to be exposed on the dedicated port, but is not (yet) suitable for use by 3rd parties.	2 years ago
Dan Upton	b168424398	xds: remove HTTPCheckFetcher dependency (#13366) ... This is the OSS portion of enterprise PR 1994 Rather than directly interrogating the agent-local state for HTTP checks using the `HTTPCheckFetcher` interface, we now rely on the config snapshot containing the checks. This reduces the number of changes required to support server xDS sessions. It's not clear why the fetching approach was introduced in 931d167ebb.	3 years ago
Dan Upton	2427e38839	Enable servers to configure arbitrary proxies from the catalog (#13244) ... OSS port of enterprise PR 1822 Includes the necessary changes to the `proxycfg` and `xds` packages to enable Consul servers to configure arbitrary proxies using catalog data. Broadly, `proxycfg.Manager` now has public methods for registering, deregistering, and listing registered proxies — the existing local agent state-sync behavior has been moved into a separate component that makes use of these methods. When an xDS session is started for a proxy service in the catalog, a goroutine will be spawned to watch the service in the server's state store and re-register it with the `proxycfg.Manager` whenever it is updated (and clean it up when the client goes away).	3 years ago
Dan Upton	a70e1886c9	WatchRoots gRPC endpoint (#12678) ... Adds a new gRPC streaming endpoint (WatchRoots) that dataplane clients will use to fetch the current list of active Connect CA roots and receive new lists whenever the roots are rotated.	3 years ago
Dan Upton	7298967070	Restructure gRPC server setup (#12586) ... OSS sync of enterprise changes at 0b44395e	3 years ago
Dan Upton	b36d4e16b6	Support per-listener TLS configuration ⚙️ (#12504) ... Introduces the capability to configure TLS differently for Consul's listeners/ports (i.e. HTTPS, gRPC, and the internal multiplexed RPC port) which is useful in scenarios where you may want the HTTPS or gRPC interfaces to present a certificate signed by a well-known/public CA, rather than the certificate used for internal communication which must have a SAN in the form `server.<dc>.consul`.	3 years ago
Eric	cf3e517d0e	Create and wire up the serverless patcher	3 years ago
Eric Haberkorn	44609c0ca5	Merge pull request #12539 from hashicorp/make-xds-lib ... Make the xdscommon package	3 years ago
Mark Anderson	aaefe15613	Bulk acl message fixup oss (#12470) ... * First pass for helper for bulk changes Signed-off-by: Mark Anderson <manderson@hashicorp.com> * Convert ACLRead and ACLWrite to new form Signed-off-by: Mark Anderson <manderson@hashicorp.com> * AgentRead and AgentWRite Signed-off-by: Mark Anderson <manderson@hashicorp.com> * Fix EventWrite Signed-off-by: Mark Anderson <manderson@hashicorp.com> * KeyRead, KeyWrite, KeyList Signed-off-by: Mark Anderson <manderson@hashicorp.com> * KeyRing Signed-off-by: Mark Anderson <manderson@hashicorp.com> * NodeRead NodeWrite Signed-off-by: Mark Anderson <manderson@hashicorp.com> * OperatorRead and OperatorWrite Signed-off-by: Mark Anderson <manderson@hashicorp.com> * PreparedQuery Signed-off-by: Mark Anderson <manderson@hashicorp.com> * Intention partial Signed-off-by: Mark Anderson <manderson@hashicorp.com> * Fix ServiceRead, Write ,etc Signed-off-by: Mark Anderson <manderson@hashicorp.com> * Error check ServiceRead? Signed-off-by: Mark Anderson <manderson@hashicorp.com> * Fix Sessionread/Write Signed-off-by: Mark Anderson <manderson@hashicorp.com> * Fixup snapshot ACL Signed-off-by: Mark Anderson <manderson@hashicorp.com> * Error fixups for txn Signed-off-by: Mark Anderson <manderson@hashicorp.com> * Add changelog Signed-off-by: Mark Anderson <manderson@hashicorp.com> * Fixup review comments Signed-off-by: Mark Anderson <manderson@hashicorp.com>	3 years ago
Eric	f5c9fa6fa6	Make an xdscommon package that will be shared between Consul and Envoy plugins	3 years ago
Giulio Micheloni	af7b7b5693	Merge branch 'main' into serve-panic-recovery	3 years ago
Evan Culver	61be9371f5	connect: Remove support for Envoy 1.16 (#11354)	3 years ago
Giulio Micheloni	fecce25658	Separete test file and no stack trace in ret error	3 years ago
Giulio Micheloni	0c78ddacde	Merge branch 'main' of https://github.com/hashicorp/consul into hashicorp-main	3 years ago
Daniel Nephin	cc310224aa	command/envoy: stop using the DebugConfig from Self endpoint ... The DebugConfig in the self endpoint can change at any time. It's not a stable API. This commit adds the XDSPort to a stable part of the XDS api, and changes the envoy command to read this new field. It includes support for the old API as well, in case a newer CLI is used with an older API, and adds a test for both cases.	3 years ago
Daniel Nephin	1502547e38	Revert "Merge pull request #10588 from hashicorp/dnephin/config-fix-ports-grpc" ... This reverts commit 74fb650b6b, reversing changes made to 58bd817336.	3 years ago
R.B. Boyer	b2d17ac448	xds: fix representation of incremental xDS subscriptions (#10987) ... Fixes #10563 The `resourceVersion` map was doing two jobs prior to this PR. The first job was to track what version of every resource we know envoy currently has. The second was to track subscriptions to those resources (by way of the empty string for a version). This mostly works out fine, but occasionally leads to consul removing a resource and accidentally (effectively) unsubscribing at the same time. The fix separates these two jobs. When all of the resources for a subscription are removed we continue to track the subscription until envoy explicitly unsubscribes	3 years ago
R.B. Boyer	5fe613dd05	xds: ensure the active streams counters are 64 bit aligned on 32 bit systems (#11085)	3 years ago
Giulio Micheloni	7fa01105cc	Fix merge conflicts	3 years ago
Giulio Micheloni	655da1fc42	Merge branch 'main' into serve-panic-recovery	3 years ago
Giulio Micheloni	4b0eaa4bff	grpc, xds: recovery middleware to return and log error in case of panic ... 1) xds and grpc servers: 1.1) to use recovery middleware with callback that prints stack trace to log 1.2) callback turn the panic into a core.Internal error 2) added unit test for grpc server	3 years ago
Daniel Nephin	8252a2691c	xds: document how authorization works	3 years ago
Daniel Nephin	e637cd71f3	acl: use authz consistently as the variable name for an acl.Authorizer ... Follow up to https://github.com/hashicorp/consul/pull/10737#discussion_r682147950 Renames all variables for acl.Authorizer to use `authz`. Previously some places used `rule` which I believe was an old name carried over from the legacy ACL system. A couple places also used authorizer. This commit also removes another couple of authorizer nil checks that are no longer necessary.	3 years ago
Giulio Micheloni	2b14a9b59a	grpc Server: turn panic into error through middleware	3 years ago
Daniel Nephin	84fac3ce0e	acl: use acl.ManangeAll when ACLs are disabled ... Instead of returning nil and checking for nilness Removes a bunch of nil checks, and fixes one test failures.	3 years ago
Daniel Nephin	7d73fd7ae5	rename GRPC->XDS where appropriate	3 years ago
Freddy	429f9d8bb8	Add flag for transparent proxies to dial individual instances (#10329)	4 years ago
R.B. Boyer	ede14b7c54	xds: emit a labeled gauge of connected xDS streams by version (#10243) ... Fixes #10099	4 years ago
R.B. Boyer	71d45a3460	Support Incremental xDS mode (#9855) ... This adds support for the Incremental xDS protocol when using xDS v3. This is best reviewed commit-by-commit and will not be squashed when merged. Union of all commit messages follows to give an overarching summary: xds: exclusively support incremental xDS when using xDS v3 Attempts to use SoTW via v3 will fail, much like attempts to use incremental via v2 will fail. Work around a strange older envoy behavior involving empty CDS responses over incremental xDS. xds: various cleanups and refactors that don't strictly concern the addition of incremental xDS support Dissolve the connectionInfo struct in favor of per-connection ResourceGenerators instead. Do a better job of ensuring the xds code uses a well configured logger that accurately describes the connected client. xds: pull out checkStreamACLs method in advance of a later commit xds: rewrite SoTW xDS protocol tests to use protobufs rather than hand-rolled json strings In the test we very lightly reuse some of the more boring protobuf construction helper code that is also technically under test. The important thing of the protocol tests is testing the protocol. The actual inputs and outputs are largely already handled by the xds golden output tests now so these protocol tests don't have to do double-duty. This also updates the SoTW protocol test to exclusively use xDS v2 which is the only variant of SoTW that will be supported in Consul 1.10. xds: default xds.Server.AuthCheckFrequency at use-time instead of construction-time	4 years ago
freddygv	7bd51ff536	Replace TransparentProxy bool with ProxyMode ... This PR replaces the original boolean used to configure transparent proxy mode. It was replaced with a string mode that can be set to: - "": Empty string is the default for when the setting should be defaulted from other configuration like config entries. - "direct": Direct mode is how applications originally opted into the mesh. Proxy listeners need to be dialed directly. - "transparent": Transparent mode enables configuring Envoy as a transparent proxy. Traffic must be captured and redirected to the inbound and outbound listeners. This PR also adds a struct for transparent proxy specific configuration. Initially this is not stored as a pointer. Will revisit that decision before GA.	4 years ago
freddygv	ce964f8ea5	Update xds for transparent proxy	4 years ago
R.B. Boyer	398b766532	xds: default to speaking xDS v3, but allow for v2 to be spoken upon request (#9658) ... - Also add support for envoy 1.17.0	4 years ago
R.B. Boyer	3b6ffc447b	xds: remove deprecated usages of xDS (#9602) ... Note that this does NOT upgrade to xDS v3. That will come in a future PR. Additionally: - Ignored staticcheck warnings about how github.com/golang/protobuf is deprecated. - Shuffled some agent/xds imports in advance of a later xDS v3 upgrade. - Remove support for envoy 1.13.x but don't add in 1.17.x yet. We have to wait until the xDS v3 support is added in a follow-up PR. Fixes #8425	4 years ago
freddygv	de0cb1af7f	Make xDS labeling consistent with proxycfg	4 years ago
freddygv	5ba14ad41d	Add trace logs to proxycfg state runner and xds srv	4 years ago
Daniel Nephin	4b8b2a4291	xds: remove Server.Initialize ... Requiring a call to initialize to set a single field is not really substantially different from having to set that field to a value.	4 years ago
Daniel Nephin	375aed5ed6	xds: Pass in logger ... small cleanup in tests	4 years ago
freddygv	959d9913b8	Add server receiver to routes and log tgw err	4 years ago
freddygv	2bbbd9e1da	Log error as error	4 years ago
R.B. Boyer	74d5df7c7a	xds: use envoy's rbac filter to handle intentions entirely within envoy (#8569)	4 years ago
R.B. Boyer	c599a2f5f4	xds: add support for envoy 1.15.0 and drop support for 1.11.x (#8424) ... Related changes: - hard-fail the xDS connection attempt if the envoy version is known to be too old to be supported - remove the RouterMatchSafeRegex proxy feature since all supported envoy versions have it - stop using --max-obj-name-len (due to: envoyproxy/envoy#11740)	4 years ago
R.B. Boyer	1eef096dfe	xds: version sniff envoy and switch regular expressions from 'regex' to 'safe_regex' on newer envoy versions (#8222) ... - cut down on extra node metadata transmission - split the golden file generation to compare all envoy version	4 years ago
R.B. Boyer	c63c994b04	connect: upgrade github.com/envoyproxy/go-control-plane to v0.9.5 (#8165)	4 years ago
Chris Piraino	f55e20a2f7	Allow ingress gateways to send empty clusters, routes, and listeners (#7795) ... This is useful when updating an config entry with no services, and the expected behavior is that envoy closes all listeners and clusters. We also allow empty routes because ingress gateways name route configurations based on the port of the listener, so it is important we remove any stale routes. Then, if a new listener with an old port is added, we will not have to deal with stale routes hanging around routing to the wrong place. Endpoints are associated with clusters, and thus by deleting the clusters we don't have to care about sending empty endpoint responses.	5 years ago
freddygv	4710410cb5	Remove fallthrough	5 years ago
freddygv	c0e1751878	Allow terminating-gateway to setup listener before servicegroups are known	5 years ago