112 Commits (aa4709ab74f8feda5363df7390deeb8cb77f7d0a)

Author	SHA1	Message	Date
Gabriel Santos	e53af28bd7	Middleware: `RequestRecorder` reports calls below 1ms as decimal value (#12905) ... * Typos * Test failing * Convert values <1ms to decimal * Fix test * Update docs and test error msg * Applied suggested changes to test case * Changelog file and suggested changes * Update .changelog/12905.txt Co-authored-by: Chris S. Kim <kisunji92@gmail.com> * suggested change - start duration with microseconds instead of nanoseconds * fix error * suggested change - floats Co-authored-by: alex <8968914+acpana@users.noreply.github.com> Co-authored-by: Chris S. Kim <kisunji92@gmail.com>	2 years ago
DanStough	485e1b5d4e	fix(peering): generate token metrics only for leader	2 years ago
DanStough	2a2debee64	feat(peering): validate server name conflicts on establish	2 years ago
Dan Upton	1c2c975b0b	xDS Load Balancing (#14397) ... Prior to #13244, connect proxies and gateways could only be configured by an xDS session served by the local client agent. In an upcoming release, it will be possible to deploy a Consul service mesh without client agents. In this model, xDS sessions will be handled by the servers themselves, which necessitates load-balancing to prevent a single server from receiving a disproportionate amount of load and becoming overwhelmed. This introduces a simple form of load-balancing where Consul will attempt to achieve an even spread of load (xDS sessions) between all healthy servers. It does so by implementing a concurrent session limiter (limiter.SessionLimiter) and adjusting the limit according to autopilot state and proxy service registrations in the catalog. If a server is already over capacity (i.e. the session limit is lowered), Consul will begin draining sessions to rebalance the load. This will result in the client receiving a `RESOURCE_EXHAUSTED` status code. It is the client's responsibility to observe this response and reconnect to a different server. Users of the gRPC client connection brokered by the consul-server-connection-manager library will get this for free. The rate at which Consul will drain sessions to rebalance load is scaled dynamically based on the number of proxies in the catalog.	2 years ago
freddygv	650e48624d	Allow terminated peerings to be deleted ... Peerings are terminated when a peer decides to delete the peering from their end. Deleting a peering sends a termination message to the peer and triggers them to mark the peering as terminated but does NOT delete the peering itself. This is to prevent peerings from disappearing from both sides just because one side deleted them. Previously the Delete endpoint was skipping the deletion if the peering was not marked as active. However, terminated peerings are also inactive. This PR makes some updates so that peerings marked as terminated can be deleted by users.	2 years ago
Chris S. Kim	6ddcc04613	Replace ring buffer with async version (#14314) ... We need to watch for changes to peerings and update the server addresses which get served by the ring buffer. Also, if there is an active connection for a peer, we are getting up-to-date server addresses from the replication stream and can safely ignore the token's addresses which may be stale.	2 years ago
freddygv	c04515a844	Use proto message for each secrets write op ... Previously there was a field indicating the operation that triggered a secrets write. Now there is a message for each operation and it contains the secret ID being persisted.	2 years ago
freddygv	60d6e28c97	Pass explicit signal with op for secrets write ... Previously the updates to the peering secrets UUID table relied on inferring what action triggered the update based on a reconciliation against the existing secrets. Instead we now explicitly require the operation to be given so that the inference isn't necessary. This makes the UUID table logic easier to reason about and fixes some related bugs. There is also an update so that the peering secrets get handled on snapshots/restores.	2 years ago
freddygv	9ca687bc7c	Avoid deleting peering secret UUIDs at dialers ... Dialers do not keep track of peering secret UUIDs, so they should not attempt to clean up data from that table when their peering is deleted. We also now keep peer server addresses when marking peerings for deletion. Peer server addresses are used by the ShouldDial() helper when determining whether the peering is for a dialer or an acceptor. We need to keep this data so that peering secrets can be cleaned up accordingly.	2 years ago
Luke Kysow	988e1fd35d	peering: default to false (#13963) ... * defaulting to false because peering will be released as beta * Ignore peering disabled error in bundles cachetype Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com> Co-authored-by: freddygv <freddy@hashicorp.com> Co-authored-by: Matt Keeler <mjkeeler7@gmail.com>	2 years ago
Matt Keeler	f74d0cef7a	Implement/Utilize secrets for Peering Replication Stream (#13977)	2 years ago
acpana	6033584349	use EqualPartitions ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
acpana	0351ca5136	better fix ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
acpana	8b2ef80336	sync w ent ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
alex	437a28d18a	peering: prevent peering in same partition (#13851) ... Co-authored-by: Chris S. Kim <ckim@hashicorp.com>	2 years ago
Nitya Dhanushkodi	27bd895ac8	peering: remove validation that forces peering token server addresses to be an IP, allow hostname based addresses (#13874)	2 years ago
Luke Kysow	3530d3782d	peering: read endpoints can now return failing status (#13849) ... Track streams that have been disconnected due to an error and set their statuses to failing.	2 years ago
Chris S. Kim	73a84f256f	Preserve PeeringState on upsert (#13666) ... Fixes a bug where if the generate token is called twice, the second call upserts the zero-value (undefined) of PeeringState.	2 years ago
freddygv	b544ce6485	Add ACL enforcement to peering endpoints	2 years ago
alex	279d458e6e	peering: use ShouldDial to validate peer role (#13823) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
Luke Kysow	a1e6d69454	peering: add config to enable/disable peering (#13867) ... * peering: add config to enable/disable peering Add config: ``` peering { enabled = true } ``` Defaults to true. When disabled: 1. All peering RPC endpoints will return an error 2. Leader won't start its peering establishment goroutines 3. Leader won't start its peering deletion goroutines	2 years ago
Nitya Dhanushkodi	f47319b7c6	update generate token endpoint to take external addresses (#13844) ... Update generate token endpoint (rpc, http, and api module) If ServerExternalAddresses are set, it will override any addresses gotten from the "consul" service, and be used in the token instead, and dialed by the dialer. This allows for setting up a load balancer for example, in front of the consul servers.	2 years ago
alex	de5a991d8c	peering: refactor reconcile, cleanup (#13795) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
alex	a9ae2ff4fa	peering: track exported services (#13784) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
R.B. Boyer	bb4d4040fb	server: ensure peer replication can successfully use TLS over external gRPC (#13733) ... Ensure that the peer stream replication rpc can successfully be used with TLS activated. Also: - If key material is configured for the gRPC port but HTTPS is not enabled now TLS will still be activated for the gRPC port. - peerstream replication stream opened by the establishing-side will now ignore grpc.WithBlock so that TLS errors will bubble up instead of being awkwardly delayed or suppressed	2 years ago
alex	adb5ffa1a6	peering: track imported services (#13718)	2 years ago
Dan Upton	b9e525d689	grpc: rename public/private directories to external/internal (#13721) ... Previously, public referred to gRPC services that are both exposed on the dedicated gRPC port and have their definitions in the proto-public directory (so were considered usable by 3rd parties). Whereas private referred to services on the multiplexed server port that are only usable by agents and other servers. Now, we're splitting these definitions, such that external/internal refers to the port and public/private refers to whether they can be used by 3rd parties. This is necessary because the peering replication API needs to be exposed on the dedicated port, but is not (yet) suitable for use by 3rd parties.	2 years ago
R.B. Boyer	af04851637	peering: move peer replication to the external gRPC port (#13698) ... Peer replication is intended to be between separate Consul installs and effectively should be considered "external". This PR moves the peer stream replication bidirectional RPC endpoint to the external gRPC server and ensures that things continue to function.	2 years ago
Chris S. Kim	f07132dacc	Revise possible states for a peering. (#13661) ... These changes are primarily for Consul's UI, where we want to be more specific about the state a peering is in. - The "initial" state was renamed to pending, and no longer applies to peerings being established from a peering token. - Upon request to establish a peering from a peering token, peerings will be set as "establishing". This will help distinguish between the two roles: the cluster that generates the peering token and the cluster that establishes the peering. - When marked for deletion, peering state will be set to "deleting". This way the UI determines the deletion via the state rather than the "DeletedAt" field. Co-authored-by: freddygv <freddy@hashicorp.com>	2 years ago
Daniel Upton	653b8c4f9d	proxycfg: server-local config entry data sources ... This is the OSS portion of enterprise PR 2056. This commit provides server-local implementations of the proxycfg.ConfigEntry and proxycfg.ConfigEntryList interfaces, that source data from streaming events. It makes use of the LocalMaterializer type introduced for peering replication, adding the necessary support for authorization. It also adds support for "wildcard" subscriptions (within a topic) to the event publisher, as this is needed to fetch service-resolvers for all services when configuring mesh gateways. Currently, events will be emitted for just the ingress-gateway, service-resolver, and mesh config entry types, as these are the only entries required by proxycfg — the events will be emitted on topics named IngressGateway, ServiceResolver, and MeshConfig topics respectively. Though these events will only be consumed "locally" for now, they can also be consumed via the gRPC endpoint (confirmed using grpcurl) so using them from client agents should be a case of swapping the LocalMaterializer for an RPCMaterializer.	2 years ago
alex	cd9ca4290a	peering: add imported/exported counts to peering (#13644) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com> Co-authored-by: Chris S. Kim <ckim@hashicorp.com>	2 years ago
alex	beb8b03e8a	peering: reconcile/ hint active state for list (#13619) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
R.B. Boyer	0fa828db76	peering: replicate all SpiffeID values necessary for the importing side to do SAN validation (#13612) ... When traversing an exported peered service, the discovery chain evaluation at the other side may re-route the request to a variety of endpoints. Furthermore we intend to terminate mTLS at the mesh gateway for arriving peered traffic that is http-like (L7), so the caller needs to know the mesh gateway's SpiffeID in that case as well. The following new SpiffeID values will be shipped back in the peerstream replication: - tcp: all possible SpiffeIDs resulting from the service-resolver component of the exported discovery chain - http-like: the SpiffeID of the mesh gateway	2 years ago
R.B. Boyer	e8ea3d7c3b	state: peering ID assignment cannot happen inside of the state store (#13525) ... Move peering ID assignment outisde of the FSM, so that the ID is written to the raft log and the same ID is used by all voters, and after restarts.	2 years ago
R.B. Boyer	201d1458c3	xds: mesh gateways now have their own leaf certificate when involved in a peering (#13460) ... This is only configured in xDS when a service with an L7 protocol is exported. They also load any relevant trust bundles for the peered services to eventually use for L7 SPIFFE validation during mTLS termination.	2 years ago
freddygv	f3843809da	Avoid deleting peerings marked as terminated. ... When our peer deletes the peering it is locally marked as terminated. This termination should kick off deleting all imported data, but should not delete the peering object itself. Keeping peerings marked as terminated acts as a signal that the action took place.	2 years ago
freddygv	6453375ab2	Add leader routine to clean up peerings ... Once a peering is marked for deletion a new leader routine will now clean up all imported resources and then the peering itself. A lot of the logic was grabbed from the namespace/partitions deferred deletions but with a handful of simplifications: - The rate limiting is not configurable. - Deleting imported nodes/services/checks is done by deleting nodes with the Txn API. The services and checks are deleted as a side-effect. - There is no "round rate limiter" like with namespaces and partitions. This is because peerings are purely local, and deleting a peering in the datacenter does not depend on deleting data from other DCs like with WAN-federated namespaces. All rate limiting is handled by the Raft rate limiter.	2 years ago
freddygv	6c8ab1bbac	Fixup stream tear-down steps. ... 1. Fix a bug where the peering leader routine would not track all active peerings in the "stored" reconciliation map. This could lead to tearing down streams where the token was generated, since the ConnectedStreams() method used for reconciliation returns all streams and not just the ones initiated by this leader routine. 2. Fix a race where stream contexts were being canceled before termination messages were being processed by a peer. Previously the leader routine would tear down streams by canceling their context right after the termination message was sent. This context cancelation could be propagated to the server side faster than the termination message. Now there is a change where the dialing peer uses CloseSend() to signal when no more messages will be sent. Eventually the server peer will read an EOF after receiving and processing the preceding termination message. Using CloseSend() is actually not enough to address the issue mentioned, since it doesn't wait for the server peer to finish processing messages. Because of this now the dialing peer also reads from the stream until an error signals that there are no more messages. Receiving an EOF from our peer indicates that they processed the termination message and have no additional work to do. Given that the stream is being closed, all the messages received by Recv are discarded. We only check for errors to avoid importing new data.	2 years ago
freddygv	cc921a9c78	Update peering state and RPC for deferred deletion ... When deleting a peering we do not want to delete the peering and all imported data in a single operation, since deleting a large amount of data at once could overload Consul. Instead we defer deletion of peerings so that: 1. When a peering deletion request is received via gRPC the peering is marked for deletion by setting the DeletedAt field. 2. A leader routine will monitor for peerings that are marked for deletion and kick off a throttled deletion of all imported resources before deleting the peering itself. This commit mostly addresses point #1 by modifying the peering service to mark peerings for deletion. Another key change is to add a PeeringListDeleted state store function which can return all peerings marked for deletion. This function is what will be watched by the deferred deletion leader routine.	2 years ago
Freddy	71b254522e	Clean up imported nodes/services/checks as needed (#13367) ... Previously, imported data would never be deleted. As nodes/services/checks were registered and deregistered, resources deleted from the exporting cluster would accumulate in the imported cluster. This commit makes updates to replication so that whenever an update is received for a service name we reconcile what was present in the catalog against what was received. This handleUpdateService method can handle both updates and deletions.	2 years ago
Chris S. Kim	a02e9abcc1	Update RBAC to handle imported services (#13404) ... When converting from Consul intentions to xds RBAC rules, services imported from other peers must encode additional data like partition (from the remote cluster) and trust domain. This PR updates the PeeringTrustBundle to hold the sending side's local partition as ExportedPartition. It also updates RBAC code to encode SpiffeIDs of imported services with the ExportedPartition and TrustDomain.	2 years ago
R.B. Boyer	7001e1151c	peering: rename initiate to establish in the context of the APIs (#13419)	3 years ago
R.B. Boyer	bba3eb8cdd	peering: mesh gateways are required for cross-peer service mesh communication (#13410) ... Require use of mesh gateways in order for service mesh data plane traffic to flow between peers. This also adds plumbing for envoy integration tests involving peers, and one starter peering test.	3 years ago
R.B. Boyer	7423886136	peering: allow protobuf requests to populate the default partition or namespace (#13398)	3 years ago
R.B. Boyer	ab758b7b32	peering: allow mesh gateways to proxy L4 peered traffic (#13339) ... Mesh gateways will now enable tcp connections with SNI names including peering information so that those connections may be proxied. Note: this does not change the callers to use these mesh gateways.	3 years ago
alex	bbbc50815a	peering: send leader addr (#13342) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	3 years ago
R.B. Boyer	019aeaa57d	peering: update how cross-peer upstreams and represented in proxycfg and rendered in xds (#13362) ... This removes unnecessary, vestigal remnants of discovery chains.	3 years ago
freddygv	647c57a416	Add agent cache-type for TrustBundleListByService ... There are a handful of changes in this commit: * When querying trust bundles for a service we need to be able to specify the namespace of the service. * The endpoint needs to track the index because the cache watches use it. * Extracted bulk of the endpoint's logic to a state store function so that index tracking could be tested more easily. * Removed check for service existence, deferring that sort of work to ACL authz * Added the cache type	3 years ago
freddygv	8b58fa8afe	Update assumptions around exported-service config ... Given that the exported-services config entry can use wildcards, the precedence for wildcards is handled as with intentions. The most exact match is the match that applies for any given service. We do not take the union of all that apply. Another update that was made was to reflect that only one exported-services config entry applies to any given service in a partition. This is a pre-existing constraint that gets enforced by the Normalize() method on that config entry type.	3 years ago
freddygv	870e7c72d7	Return SPIFFE ID for connect proxies in PeerMeta ... Proxies dialing exporting services need to know the SPIFFE ID of services dialed so that the upstream's SANs can be validated. This commit attaches the SPIFFE ID to all connect proxies exported over the peering stream so that they are available to importing clusters. The data in the SPIFFE ID cannot be re-constructed in peer clusters because the partition of exported services is overwritten on imports.	3 years ago