consul

Commit Graph

Author	SHA1	Message	Date
Dan Upton	641347fe14	proto: deep-copy PeeringTrustBundle using proto.Clone (#15004 ) Fixes a `go vet` warning caused by the pragma.DoNotCopy on the protobuf message type. Originally I'd hoped we wouldn't need any reflection in the proxycfg hot path, but it seems proto.Clone is the only supported way to copy a message.	2 years ago
Dan Upton	e6b55d1d81	perf: remove expensive reflection from xDS hot path (#14934 ) Replaces the reflection-based implementation of proxycfg's ConfigSnapshot.Clone with code generated by deep-copy. While load testing server-based xDS (for consul-dataplane) we discovered this method is extremely expensive. The ConfigSnapshot struct, directly or indirectly, contains a copy of many of the structs in the agent/structs package, which creates a large graph for copystructure.Copy to traverse at runtime, on every proxy reconfiguration.	2 years ago
Derek Menteer	caa1396255	Add remote peer partition and datacenter info.	2 years ago
boruszak	38b1a515f1	Mergimg	2 years ago
Chris S. Kim	b0a4c5c563	Include stream-related information in peering endpoints	2 years ago
DanStough	77ab28c5c7	feat: xDS updates for peerings control plane through mesh gw	2 years ago
Eric Haberkorn	80e51ff907	Add exported services event to cluster peering replication. (#14797 )	2 years ago
cskh	69f40df548	feat(ingress gateway: support configuring limits in ingress-gateway c… (#14749 ) * feat(ingress gateway: support configuring limits in ingress-gateway config entry - a new Defaults field with max_connections, max_pending_connections, max_requests is added to ingress gateway config entry - new field max_connections, max_pending_connections, max_requests in individual services to overwrite the value in Default - added unit test and integration test - updated doc Co-authored-by: Chris S. Kim <ckim@hashicorp.com> Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> Co-authored-by: Dan Stough <dan.stough@hashicorp.com>	2 years ago
Alessandro De Blasis	461b42ed48	fix(check): added missing OSService props	2 years ago
DanStough	0150e88200	feat: add PeerThroughMeshGateways to mesh config	2 years ago
Freddy	f4dfd42e0a	Add SpiffeID for Consul server agents (#14485 ) Co-authored-by: Eric Haberkorn <erichaberkorn@gmail.com> By adding a SpiffeID for server agents, servers can now request a leaf certificate from the Connect CA. This new Spiffe ID has a key property: servers are identified by their datacenter name and trust domain. All servers that share these attributes will share a ServerURI. The aim is to use these certificates to verify the server name of ANY server in a Consul datacenter.	2 years ago
Chris S. Kim	ecc43db774	Fix early return in prototest.AssertElementsMatch (#14467 )	2 years ago
Eric Haberkorn	1099665473	Update the structs and discovery chain for service resolver redirects to cluster peers. (#14366 )	2 years ago
Chris S. Kim	6ddcc04613	Replace ring buffer with async version (#14314 ) We need to watch for changes to peerings and update the server addresses which get served by the ring buffer. Also, if there is an active connection for a peer, we are getting up-to-date server addresses from the replication stream and can safely ignore the token's addresses which may be stale.	2 years ago
Chris S. Kim	584d3409c4	Send server addresses on update from server	2 years ago
Chris S. Kim	09c0fe22f2	Add PeeringServerAddresses proto	2 years ago
Eric Haberkorn	1a73b0ca20	Add `Targets` field to service resolver failovers. (#14162 ) This field will be used for cluster peering failover.	2 years ago
freddygv	1031ffc3c7	Re-validate existing secrets at state store Previously establishment and pending secrets were only checked at the RPC layer. However, given that these are Check-and-Set transactions we should ensure that the given secrets are still valid when persisting a secret exchange or promotion. Otherwise it would be possible for concurrent requests to overwrite each other.	2 years ago
freddygv	0ea4bfae94	Test fixes	2 years ago
freddygv	c04515a844	Use proto message for each secrets write op Previously there was a field indicating the operation that triggered a secrets write. Now there is a message for each operation and it contains the secret ID being persisted.	2 years ago
freddygv	60d6e28c97	Pass explicit signal with op for secrets write Previously the updates to the peering secrets UUID table relied on inferring what action triggered the update based on a reconciliation against the existing secrets. Instead we now explicitly require the operation to be given so that the inference isn't necessary. This makes the UUID table logic easier to reason about and fixes some related bugs. There is also an update so that the peering secrets get handled on snapshots/restores.	2 years ago
Matt Keeler	f74d0cef7a	Implement/Utilize secrets for Peering Replication Stream (#13977 )	2 years ago
freddygv	b544ce6485	Add ACL enforcement to peering endpoints	2 years ago
Nitya Dhanushkodi	f47319b7c6	update generate token endpoint to take external addresses (#13844 ) Update generate token endpoint (rpc, http, and api module) If ServerExternalAddresses are set, it will override any addresses gotten from the "consul" service, and be used in the token instead, and dialed by the dialer. This allows for setting up a load balancer for example, in front of the consul servers.	2 years ago
Luke Kysow	741c906b0d	Add heartbeat proto to peer stream (#13804 )	2 years ago
R.B. Boyer	cd513aeead	peerstream: require a resource subscription to receive updates of that type (#13767 ) This mimics xDS's discovery protocol where you must request a resource explicitly for the exporting side to send those events to you. As part of this I aligned the overall ResourceURL with the TypeURL that gets embedded into the encoded protobuf Any construct. The CheckServiceNodes is now wrapped in a better named "ExportedService" struct now.	2 years ago
alex	b7043f7150	peering: add warning about AllowStaleRead (#13768 )	2 years ago
Matt Keeler	05b5e7e2ca	Enable partition support for peering establishment (#13772 ) Prior to this the dialing side of the peering would only ever work within the default partition. This commit allows properly parsing the partition field out of the API struct request body, query param and header.	2 years ago
Daniel Upton	3c533ceea8	proxycfg-glue: server-local implementation of `ServiceList` This is the OSS portion of enterprise PR 2242. This PR introduces a server-local implementation of the proxycfg.ServiceList interface, backed by streaming events and a local materializer.	2 years ago
R.B. Boyer	f1cc185335	proto: add package prefixes for all proto files where it is safe (#13735 ) We cannot do this for "subscribe" and "partition" this easily without breakage so those are omitted. Any protobuf message passed around via an Any construct will have the fully qualified package name embedded in the protobuf as a string. Also RPC method dispatch will include the package of the service during serialization. - We will be passing pbservice and pbpeering through an Any as part of peer stream replication. - We will be exposing two new gRPC services via pbpeering and pbpeerstream.	2 years ago
R.B. Boyer	30fffd0c90	peerstream: some cosmetic refactors to make this easier to follow (#13732 ) - Use some protobuf construction helper methods for brevity. - Rename a local variable to avoid later shadowing. - Rename the Nonce field to be more like xDS's naming. - Be more explicit about which PeerID fields are empty.	2 years ago
R.B. Boyer	9a56eed86c	proto: ensure buf formatter has been applied to protobufs (#13709 )	2 years ago
R.B. Boyer	af04851637	peering: move peer replication to the external gRPC port (#13698 ) Peer replication is intended to be between separate Consul installs and effectively should be considered "external". This PR moves the peer stream replication bidirectional RPC endpoint to the external gRPC server and ensures that things continue to function.	2 years ago
Chris S. Kim	f07132dacc	Revise possible states for a peering. (#13661 ) These changes are primarily for Consul's UI, where we want to be more specific about the state a peering is in. - The "initial" state was renamed to pending, and no longer applies to peerings being established from a peering token. - Upon request to establish a peering from a peering token, peerings will be set as "establishing". This will help distinguish between the two roles: the cluster that generates the peering token and the cluster that establishes the peering. - When marked for deletion, peering state will be set to "deleting". This way the UI determines the deletion via the state rather than the "DeletedAt" field. Co-authored-by: freddygv <freddy@hashicorp.com>	2 years ago
Daniel Upton	37ccbd2826	proxycfg: server-local intentions data source This is the OSS portion of enterprise PR 2141. This commit provides a server-local implementation of the `proxycfg.Intentions` interface that sources data from streaming events. It adds events for the `service-intentions` config entry type, and then consumes event streams (via materialized views) for the service's explicit intentions and any applicable wildcard intentions, merging them into a single list of intentions. An alternative approach I considered was to consume _all_ intention events (via `SubjectWildcard`) and filter out the irrelevant ones. This would admittedly remove some complexity in the `agent/proxycfg-glue` package but at the expense of considerable overhead from waking potentially many thousands of connect proxies every time any intention is updated.	2 years ago
Daniel Upton	653b8c4f9d	proxycfg: server-local config entry data sources This is the OSS portion of enterprise PR 2056. This commit provides server-local implementations of the proxycfg.ConfigEntry and proxycfg.ConfigEntryList interfaces, that source data from streaming events. It makes use of the LocalMaterializer type introduced for peering replication, adding the necessary support for authorization. It also adds support for "wildcard" subscriptions (within a topic) to the event publisher, as this is needed to fetch service-resolvers for all services when configuring mesh gateways. Currently, events will be emitted for just the ingress-gateway, service-resolver, and mesh config entry types, as these are the only entries required by proxycfg — the events will be emitted on topics named IngressGateway, ServiceResolver, and MeshConfig topics respectively. Though these events will only be consumed "locally" for now, they can also be consumed via the gRPC endpoint (confirmed using grpcurl) so using them from client agents should be a case of swapping the LocalMaterializer for an RPCMaterializer.	2 years ago
alex	cd9ca4290a	peering: add imported/exported counts to peering (#13644 ) Signed-off-by: acpana <8968914+acpana@users.noreply.github.com> Co-authored-by: Chris S. Kim <ckim@hashicorp.com>	2 years ago
R.B. Boyer	201d1458c3	xds: mesh gateways now have their own leaf certificate when involved in a peering (#13460 ) This is only configured in xDS when a service with an L7 protocol is exported. They also load any relevant trust bundles for the peered services to eventually use for L7 SPIFFE validation during mTLS termination.	2 years ago
freddygv	6c8ab1bbac	Fixup stream tear-down steps. 1. Fix a bug where the peering leader routine would not track all active peerings in the "stored" reconciliation map. This could lead to tearing down streams where the token was generated, since the ConnectedStreams() method used for reconciliation returns all streams and not just the ones initiated by this leader routine. 2. Fix a race where stream contexts were being canceled before termination messages were being processed by a peer. Previously the leader routine would tear down streams by canceling their context right after the termination message was sent. This context cancelation could be propagated to the server side faster than the termination message. Now there is a change where the dialing peer uses CloseSend() to signal when no more messages will be sent. Eventually the server peer will read an EOF after receiving and processing the preceding termination message. Using CloseSend() is actually not enough to address the issue mentioned, since it doesn't wait for the server peer to finish processing messages. Because of this now the dialing peer also reads from the stream until an error signals that there are no more messages. Receiving an EOF from our peer indicates that they processed the termination message and have no additional work to do. Given that the stream is being closed, all the messages received by Recv are discarded. We only check for errors to avoid importing new data.	2 years ago
freddygv	cc921a9c78	Update peering state and RPC for deferred deletion When deleting a peering we do not want to delete the peering and all imported data in a single operation, since deleting a large amount of data at once could overload Consul. Instead we defer deletion of peerings so that: 1. When a peering deletion request is received via gRPC the peering is marked for deletion by setting the DeletedAt field. 2. A leader routine will monitor for peerings that are marked for deletion and kick off a throttled deletion of all imported resources before deleting the peering itself. This commit mostly addresses point #1 by modifying the peering service to mark peerings for deletion. Another key change is to add a PeeringListDeleted state store function which can return all peerings marked for deletion. This function is what will be watched by the deferred deletion leader routine.	2 years ago
Freddy	71b254522e	Clean up imported nodes/services/checks as needed (#13367 ) Previously, imported data would never be deleted. As nodes/services/checks were registered and deregistered, resources deleted from the exporting cluster would accumulate in the imported cluster. This commit makes updates to replication so that whenever an update is received for a service name we reconcile what was present in the catalog against what was received. This handleUpdateService method can handle both updates and deletions.	2 years ago
Chris S. Kim	a02e9abcc1	Update RBAC to handle imported services (#13404 ) When converting from Consul intentions to xds RBAC rules, services imported from other peers must encode additional data like partition (from the remote cluster) and trust domain. This PR updates the PeeringTrustBundle to hold the sending side's local partition as ExportedPartition. It also updates RBAC code to encode SpiffeIDs of imported services with the ExportedPartition and TrustDomain.	2 years ago
R.B. Boyer	7001e1151c	peering: rename initiate to establish in the context of the APIs (#13419 )	2 years ago
R.B. Boyer	edb2e55335	peering: avoid a race between peering establishment and termination (#13389 )	2 years ago
Alessandro De Blasis	b59c19bb06	feat: windows service health checks	2 years ago
Fulvio	d457d8b6ce	UDP check for service stanza #12221 (#12722 ) * UDP check for service stanza #12221 * add pass status on timeout condition * delete useless files * Update check_test.go improve comment in test * fix test * fix requested changes and update TestRuntimeConfig_Sanitize.golden * add freeport to TestCheckUDPCritical * improve comment for CheckUDP struct * fix requested changes * fix requested changes * fix requested changes * add UDP to proto * add UDP to proto and add a changelog * add requested test on agent_endpoint_test.go * add test for given endpoints * fix failing tests * add documentation for udp healthcheck * regenerate proto using buf * Update website/content/api-docs/agent/check.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * Update website/content/api-docs/agent/check.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * Update website/content/docs/discovery/checks.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * Update website/content/docs/ecs/configuration-reference.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * Update website/content/docs/ecs/configuration-reference.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * add debug echo * add debug circle-ci * add debug circle-ci bash * use echo instead of status_stage * remove debug and status from devtools script and use echo instead * Update website/content/api-docs/agent/check.mdx Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com> * fix test * replace status_stage with status * replace functions with echo Co-authored-by: Dhia Ayachi <dhia@hashicorp.com> Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com>	3 years ago
alex	bbbc50815a	peering: send leader addr (#13342 ) Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	3 years ago
Freddy	a09c776645	Update public listener with SPIFFE Validator Envoy's SPIFFE certificate validation extension allows for us to validate against different root certificates depending on the trust domain of the dialing proxy. If there are any trust bundles from peers in the config snapshot then we use the SPIFFE validator as the validation context, rather than the usual TrustedCA. The injected validation config includes the local root certificates as well.	3 years ago
freddygv	647c57a416	Add agent cache-type for TrustBundleListByService There are a handful of changes in this commit: * When querying trust bundles for a service we need to be able to specify the namespace of the service. * The endpoint needs to track the index because the cache watches use it. * Extracted bulk of the endpoint's logic to a state store function so that index tracking could be tested more easily. * Removed check for service existence, deferring that sort of work to ACL authz * Added the cache type	3 years ago
Freddy	74ca6406ea	Configure upstream TLS context with peer root certs (#13321 ) For mTLS to work between two proxies in peered clusters with different root CAs, proxies need to configure their outbound listener to use different root certificates for validation. Up until peering was introduced proxies would only ever use one set of root certificates to validate all mesh traffic, both inbound and outbound. Now an upstream proxy may have a leaf certificate signed by a CA that's different from the dialing proxy's. This PR makes changes to proxycfg and xds so that the upstream TLS validation uses different root certificates depending on which cluster is being dialed.	3 years ago

1 2 3 4

189 Commits (8617f8af16184e1de42f22aca7448e1d8e86fab6)