139 Commits (641347fe14a3629fc0d359ab4fa5c6079cefc186)

Author	SHA1	Message	Date
Dan Upton	641347fe14	proto: deep-copy PeeringTrustBundle using proto.Clone (#15004) ... Fixes a `go vet` warning caused by the pragma.DoNotCopy on the protobuf message type. Originally I'd hoped we wouldn't need any reflection in the proxycfg hot path, but it seems proto.Clone is the only supported way to copy a message.	2 years ago
Chris S. Kim	3d2dffff16	Merge pull request #13388 from deblasis/feature/health-checks_windows_service ... Feature: Health checks windows service	2 years ago
Dan Upton	e6b55d1d81	perf: remove expensive reflection from xDS hot path (#14934) ... Replaces the reflection-based implementation of proxycfg's ConfigSnapshot.Clone with code generated by deep-copy. While load testing server-based xDS (for consul-dataplane) we discovered this method is extremely expensive. The ConfigSnapshot struct, directly or indirectly, contains a copy of many of the structs in the agent/structs package, which creates a large graph for copystructure.Copy to traverse at runtime, on every proxy reconfiguration.	2 years ago
Derek Menteer	caa1396255	Add remote peer partition and datacenter info.	2 years ago
Chris S. Kim	b0a4c5c563	Include stream-related information in peering endpoints	2 years ago
DanStough	77ab28c5c7	feat: xDS updates for peerings control plane through mesh gw	2 years ago
Eric Haberkorn	80e51ff907	Add exported services event to cluster peering replication. (#14797)	2 years ago
cskh	69f40df548	feat(ingress gateway: support configuring limits in ingress-gateway c… (#14749) ... * feat(ingress gateway: support configuring limits in ingress-gateway config entry - a new Defaults field with max_connections, max_pending_connections, max_requests is added to ingress gateway config entry - new field max_connections, max_pending_connections, max_requests in individual services to overwrite the value in Default - added unit test and integration test - updated doc Co-authored-by: Chris S. Kim <ckim@hashicorp.com> Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> Co-authored-by: Dan Stough <dan.stough@hashicorp.com>	2 years ago
Alessandro De Blasis	461b42ed48	fix(check): added missing OSService props	2 years ago
DanStough	0150e88200	feat: add PeerThroughMeshGateways to mesh config	2 years ago
Chris S. Kim	1c4a6eef4f	Merge pull request #14285 from hashicorp/NET-638-push-server-address-updates-to-the-peer ... peering: Subscribe to server address changes and push updates to peers	2 years ago
Freddy	f4dfd42e0a	Add SpiffeID for Consul server agents (#14485) ... Co-authored-by: Eric Haberkorn <erichaberkorn@gmail.com> By adding a SpiffeID for server agents, servers can now request a leaf certificate from the Connect CA. This new Spiffe ID has a key property: servers are identified by their datacenter name and trust domain. All servers that share these attributes will share a ServerURI. The aim is to use these certificates to verify the server name of ANY server in a Consul datacenter.	2 years ago
Chris S. Kim	ecc43db774	Fix early return in prototest.AssertElementsMatch (#14467)	2 years ago
Alessandro De Blasis	60c7c831c6	Merge remote-tracking branch 'hashicorp/main' into feature/health-checks_windows_service	2 years ago
Chris S. Kim	560d410c6d	Merge branch 'main' into NET-638-push-server-address-updates-to-the-peer ... # Conflicts: # agent/grpc-external/services/peerstream/stream_test.go	2 years ago
Eric Haberkorn	1099665473	Update the structs and discovery chain for service resolver redirects to cluster peers. (#14366)	2 years ago
Alessandro De Blasis	f3437eaf05	Merge remote-tracking branch 'hashicorp/main' into feature/health-checks_windows_service ... Signed-off-by: Alessandro De Blasis <alex@deblasis.net>	2 years ago
Chris S. Kim	1c43a1a7b4	Merge branch 'main' into NET-638-push-server-address-updates-to-the-peer ... # Conflicts: # agent/grpc-external/services/peerstream/stream_test.go	2 years ago
Chris S. Kim	6ddcc04613	Replace ring buffer with async version (#14314) ... We need to watch for changes to peerings and update the server addresses which get served by the ring buffer. Also, if there is an active connection for a peer, we are getting up-to-date server addresses from the replication stream and can safely ignore the token's addresses which may be stale.	2 years ago
Chris S. Kim	584d3409c4	Send server addresses on update from server	2 years ago
Chris S. Kim	09c0fe22f2	Add PeeringServerAddresses proto	2 years ago
Eric Haberkorn	1a73b0ca20	Add `Targets` field to service resolver failovers. (#14162) ... This field will be used for cluster peering failover.	2 years ago
Alessandro De Blasis	5dee555888	Merge remote-tracking branch 'hashicorp/main' into feature/health-checks_windows_service ... Signed-off-by: Alessandro De Blasis <alex@deblasis.net>	2 years ago
Alessandro De Blasis	ab611eabc3	Merge remote-tracking branch 'hashicorp/main' into feature/health-checks_windows_service ... Signed-off-by: Alessandro De Blasis <alex@deblasis.net>	2 years ago
freddygv	1031ffc3c7	Re-validate existing secrets at state store ... Previously establishment and pending secrets were only checked at the RPC layer. However, given that these are Check-and-Set transactions we should ensure that the given secrets are still valid when persisting a secret exchange or promotion. Otherwise it would be possible for concurrent requests to overwrite each other.	2 years ago
freddygv	0ea4bfae94	Test fixes	2 years ago
freddygv	c04515a844	Use proto message for each secrets write op ... Previously there was a field indicating the operation that triggered a secrets write. Now there is a message for each operation and it contains the secret ID being persisted.	2 years ago
freddygv	60d6e28c97	Pass explicit signal with op for secrets write ... Previously the updates to the peering secrets UUID table relied on inferring what action triggered the update based on a reconciliation against the existing secrets. Instead we now explicitly require the operation to be given so that the inference isn't necessary. This makes the UUID table logic easier to reason about and fixes some related bugs. There is also an update so that the peering secrets get handled on snapshots/restores.	2 years ago
Matt Keeler	f74d0cef7a	Implement/Utilize secrets for Peering Replication Stream (#13977)	2 years ago
freddygv	b544ce6485	Add ACL enforcement to peering endpoints	2 years ago
Nitya Dhanushkodi	f47319b7c6	update generate token endpoint to take external addresses (#13844) ... Update generate token endpoint (rpc, http, and api module) If ServerExternalAddresses are set, it will override any addresses gotten from the "consul" service, and be used in the token instead, and dialed by the dialer. This allows for setting up a load balancer for example, in front of the consul servers.	2 years ago
Luke Kysow	741c906b0d	Add heartbeat proto to peer stream (#13804)	2 years ago
R.B. Boyer	cd513aeead	peerstream: require a resource subscription to receive updates of that type (#13767) ... This mimics xDS's discovery protocol where you must request a resource explicitly for the exporting side to send those events to you. As part of this I aligned the overall ResourceURL with the TypeURL that gets embedded into the encoded protobuf Any construct. The CheckServiceNodes is now wrapped in a better named "ExportedService" struct now.	2 years ago
alex	b7043f7150	peering: add warning about AllowStaleRead (#13768)	2 years ago
Matt Keeler	05b5e7e2ca	Enable partition support for peering establishment (#13772) ... Prior to this the dialing side of the peering would only ever work within the default partition. This commit allows properly parsing the partition field out of the API struct request body, query param and header.	2 years ago
Daniel Upton	3c533ceea8	proxycfg-glue: server-local implementation of `ServiceList` ... This is the OSS portion of enterprise PR 2242. This PR introduces a server-local implementation of the proxycfg.ServiceList interface, backed by streaming events and a local materializer.	2 years ago
R.B. Boyer	f1cc185335	proto: add package prefixes for all proto files where it is safe (#13735) ... We cannot do this for "subscribe" and "partition" this easily without breakage so those are omitted. Any protobuf message passed around via an Any construct will have the fully qualified package name embedded in the protobuf as a string. Also RPC method dispatch will include the package of the service during serialization. - We will be passing pbservice and pbpeering through an Any as part of peer stream replication. - We will be exposing two new gRPC services via pbpeering and pbpeerstream.	2 years ago
R.B. Boyer	30fffd0c90	peerstream: some cosmetic refactors to make this easier to follow (#13732) ... - Use some protobuf construction helper methods for brevity. - Rename a local variable to avoid later shadowing. - Rename the Nonce field to be more like xDS's naming. - Be more explicit about which PeerID fields are empty.	2 years ago
R.B. Boyer	9a56eed86c	proto: ensure buf formatter has been applied to protobufs (#13709)	2 years ago
R.B. Boyer	af04851637	peering: move peer replication to the external gRPC port (#13698) ... Peer replication is intended to be between separate Consul installs and effectively should be considered "external". This PR moves the peer stream replication bidirectional RPC endpoint to the external gRPC server and ensures that things continue to function.	2 years ago
Chris S. Kim	f07132dacc	Revise possible states for a peering. (#13661) ... These changes are primarily for Consul's UI, where we want to be more specific about the state a peering is in. - The "initial" state was renamed to pending, and no longer applies to peerings being established from a peering token. - Upon request to establish a peering from a peering token, peerings will be set as "establishing". This will help distinguish between the two roles: the cluster that generates the peering token and the cluster that establishes the peering. - When marked for deletion, peering state will be set to "deleting". This way the UI determines the deletion via the state rather than the "DeletedAt" field. Co-authored-by: freddygv <freddy@hashicorp.com>	2 years ago
Daniel Upton	37ccbd2826	proxycfg: server-local intentions data source ... This is the OSS portion of enterprise PR 2141. This commit provides a server-local implementation of the `proxycfg.Intentions` interface that sources data from streaming events. It adds events for the `service-intentions` config entry type, and then consumes event streams (via materialized views) for the service's explicit intentions and any applicable wildcard intentions, merging them into a single list of intentions. An alternative approach I considered was to consume _all_ intention events (via `SubjectWildcard`) and filter out the irrelevant ones. This would admittedly remove some complexity in the `agent/proxycfg-glue` package but at the expense of considerable overhead from waking potentially many thousands of connect proxies every time any intention is updated.	2 years ago
Daniel Upton	653b8c4f9d	proxycfg: server-local config entry data sources ... This is the OSS portion of enterprise PR 2056. This commit provides server-local implementations of the proxycfg.ConfigEntry and proxycfg.ConfigEntryList interfaces, that source data from streaming events. It makes use of the LocalMaterializer type introduced for peering replication, adding the necessary support for authorization. It also adds support for "wildcard" subscriptions (within a topic) to the event publisher, as this is needed to fetch service-resolvers for all services when configuring mesh gateways. Currently, events will be emitted for just the ingress-gateway, service-resolver, and mesh config entry types, as these are the only entries required by proxycfg — the events will be emitted on topics named IngressGateway, ServiceResolver, and MeshConfig topics respectively. Though these events will only be consumed "locally" for now, they can also be consumed via the gRPC endpoint (confirmed using grpcurl) so using them from client agents should be a case of swapping the LocalMaterializer for an RPCMaterializer.	2 years ago
alex	cd9ca4290a	peering: add imported/exported counts to peering (#13644) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com> Co-authored-by: Chris S. Kim <ckim@hashicorp.com>	2 years ago
R.B. Boyer	201d1458c3	xds: mesh gateways now have their own leaf certificate when involved in a peering (#13460) ... This is only configured in xDS when a service with an L7 protocol is exported. They also load any relevant trust bundles for the peered services to eventually use for L7 SPIFFE validation during mTLS termination.	2 years ago
freddygv	6c8ab1bbac	Fixup stream tear-down steps. ... 1. Fix a bug where the peering leader routine would not track all active peerings in the "stored" reconciliation map. This could lead to tearing down streams where the token was generated, since the ConnectedStreams() method used for reconciliation returns all streams and not just the ones initiated by this leader routine. 2. Fix a race where stream contexts were being canceled before termination messages were being processed by a peer. Previously the leader routine would tear down streams by canceling their context right after the termination message was sent. This context cancelation could be propagated to the server side faster than the termination message. Now there is a change where the dialing peer uses CloseSend() to signal when no more messages will be sent. Eventually the server peer will read an EOF after receiving and processing the preceding termination message. Using CloseSend() is actually not enough to address the issue mentioned, since it doesn't wait for the server peer to finish processing messages. Because of this now the dialing peer also reads from the stream until an error signals that there are no more messages. Receiving an EOF from our peer indicates that they processed the termination message and have no additional work to do. Given that the stream is being closed, all the messages received by Recv are discarded. We only check for errors to avoid importing new data.	2 years ago
freddygv	cc921a9c78	Update peering state and RPC for deferred deletion ... When deleting a peering we do not want to delete the peering and all imported data in a single operation, since deleting a large amount of data at once could overload Consul. Instead we defer deletion of peerings so that: 1. When a peering deletion request is received via gRPC the peering is marked for deletion by setting the DeletedAt field. 2. A leader routine will monitor for peerings that are marked for deletion and kick off a throttled deletion of all imported resources before deleting the peering itself. This commit mostly addresses point #1 by modifying the peering service to mark peerings for deletion. Another key change is to add a PeeringListDeleted state store function which can return all peerings marked for deletion. This function is what will be watched by the deferred deletion leader routine.	2 years ago
Freddy	71b254522e	Clean up imported nodes/services/checks as needed (#13367) ... Previously, imported data would never be deleted. As nodes/services/checks were registered and deregistered, resources deleted from the exporting cluster would accumulate in the imported cluster. This commit makes updates to replication so that whenever an update is received for a service name we reconcile what was present in the catalog against what was received. This handleUpdateService method can handle both updates and deletions.	2 years ago
Chris S. Kim	a02e9abcc1	Update RBAC to handle imported services (#13404) ... When converting from Consul intentions to xds RBAC rules, services imported from other peers must encode additional data like partition (from the remote cluster) and trust domain. This PR updates the PeeringTrustBundle to hold the sending side's local partition as ExportedPartition. It also updates RBAC code to encode SpiffeIDs of imported services with the ExportedPartition and TrustDomain.	2 years ago
R.B. Boyer	7001e1151c	peering: rename initiate to establish in the context of the APIs (#13419)	2 years ago