4536 Commits (70bb6a2abdbc5ed4a6e728e8da243c5394a631d1)

Author	SHA1	Message	Date
Daniel Upton	a8df87f574	proxycfg-glue: server-local implementation of `ExportedPeeredServices` ... This is the OSS portion of enterprise PR 2377. Adds a server-local implementation of the proxycfg.ExportedPeeredServices interface that sources data from a blocking query against the server's state store.	2 years ago
Eric Haberkorn	501089292e	Add Cluster Peering Failover Support to Prepared Queries (#13835) ... Add peering failover support to prepared queries	2 years ago
Nitya Dhanushkodi	f47319b7c6	update generate token endpoint to take external addresses (#13844) ... Update generate token endpoint (rpc, http, and api module) If ServerExternalAddresses are set, it will override any addresses gotten from the "consul" service, and be used in the token instead, and dialed by the dialer. This allows for setting up a load balancer for example, in front of the consul servers.	2 years ago
acpana	12b773ab02	Rename peering internal to ~ ... sync ENT to 5679392c81 Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
Luke Kysow	0c87be0845	peering: Add heartbeating to peering streams (#13806) ... * Add heartbeating to peering streams	2 years ago
Daniel Upton	3655802fdc	proxycfg-glue: server-local implementation of `PeeredUpstreams` ... This is the OSS portion of enterprise PR 2352. It adds a server-local implementation of the proxycfg.PeeredUpstreams interface based on a blocking query against the server's state store. It also fixes an omission in the Virtual IP freeing logic where we were never updating the max index (and therefore blocking queries against VirtualIPsForAllImportedServices would not return on service deletion).	2 years ago
Luke Kysow	c411e6b326	Add send mutex to protect against concurrent sends (#13805)	2 years ago
Kyle Havlovitz	0be7d923dc	Cancel upstream watches when the discovery chain has been removed	2 years ago
Kyle Havlovitz	31318d7049	Fix duplicate Notify calls for discovery chains in ingress gateways	2 years ago
Evan Culver	4116537b83	connect: Add support for Envoy 1.23, remove 1.19 (#13807)	2 years ago
Paul Glass	77afe0e76e	Extract AWS auth implementation out of Consul (#13760)	2 years ago
Chris S. Kim	495936300e	Make envoy resources for inferred peered upstreams (#13758) ... Peered upstreams has a separate loop in xds from discovery chain upstreams. This PR adds similar but slightly modified code to add filters for peered upstream listeners, clusters, and endpoints in the case of transparent proxy.	2 years ago
alex	de5a991d8c	peering: refactor reconcile, cleanup (#13795) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
Luke Kysow	e8d965e56f	peerstream: set keepalive enforcement to 15s (#13796) ... The client is set to send keepalive pings every 30s. The server keepalive enforcement must be set to a number less than that, otherwise it will disconnect clients for sending pings too often. MinTime governs the minimum amount of time between pings.	2 years ago
alex	a9ae2ff4fa	peering: track exported services (#13784) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
R.B. Boyer	cd513aeead	peerstream: require a resource subscription to receive updates of that type (#13767) ... This mimics xDS's discovery protocol where you must request a resource explicitly for the exporting side to send those events to you. As part of this I aligned the overall ResourceURL with the TypeURL that gets embedded into the encoded protobuf Any construct. The CheckServiceNodes is now wrapped in a better named "ExportedService" struct now.	2 years ago
R.B. Boyer	c737301093	peerstream: fix test assertions (#13780)	2 years ago
Luke Kysow	46381b1a7f	Add docs for peerStreamServer vs peeringServer. (#13781)	2 years ago
Luke Kysow	ca3d7c964c	peerstream: dialer should reconnect when stream closes (#13745) ... * peerstream: dialer should reconnect when stream closes If the stream is closed unexpectedly (i.e. when we haven't received a terminated message), the dialer should attempt to re-establish the stream. Previously, the `HandleStream` would return `nil` when the stream was closed. The caller then assumed the stream was terminated on purpose and so didn't reconnect when instead it was stopped unexpectedly and the dialer should have attempted to reconnect.	2 years ago
R.B. Boyer	bb4d4040fb	server: ensure peer replication can successfully use TLS over external gRPC (#13733) ... Ensure that the peer stream replication rpc can successfully be used with TLS activated. Also: - If key material is configured for the gRPC port but HTTPS is not enabled now TLS will still be activated for the gRPC port. - peerstream replication stream opened by the establishing-side will now ignore grpc.WithBlock so that TLS errors will bubble up instead of being awkwardly delayed or suppressed	2 years ago
alex	adb5ffa1a6	peering: track imported services (#13718)	2 years ago
Matt Keeler	257f88d4df	Use Node Name for peering healthSnapshot instead of ID (#13773) ... A Node ID is not a required field with Consul’s data model. Therefore we cannot reliably expect all uses to have it. However the node name is required and must be unique so its equally as good of a key for the internal healthSnapshot node tracking.	2 years ago
Matt Keeler	05b5e7e2ca	Enable partition support for peering establishment (#13772) ... Prior to this the dialing side of the peering would only ever work within the default partition. This commit allows properly parsing the partition field out of the API struct request body, query param and header.	2 years ago
Dan Stough	49f3dadb8f	feat: connect proxy xDS for destinations ... Signed-off-by: Dhia Ayachi <dhia@hashicorp.com>	2 years ago
Daniel Upton	3d74efa8ad	proxycfg-glue: server-local implementation of `FederationStateListMeshGateways` ... This is the OSS portion of enterprise PR 2265. This PR provides a server-local implementation of the proxycfg.FederationStateListMeshGateways interface based on blocking queries.	2 years ago
Daniel Upton	ccc672013e	proxycfg-glue: server-local implementation of `GatewayServices` ... This is the OSS portion of enterprise PR 2259. This PR provides a server-local implementation of the proxycfg.GatewayServices interface based on blocking queries.	2 years ago
Daniel Upton	15a319dbfe	proxycfg-glue: server-local implementation of `TrustBundle` and `TrustBundleList` ... This is the OSS portion of enterprise PR 2250. This PR provides server-local implementations of the proxycfg.TrustBundle and proxycfg.TrustBundleList interfaces, based on local blocking queries.	2 years ago
Daniel Upton	673d02d30f	proxycfg-glue: server-local implementation of the `Health` interface ... This is the OSS portion of enterprise PR 2249. This PR introduces an implementation of the proxycfg.Health interface based on a local materialized view of the health events. It reuses the view and request machinery from agent/rpcclient/health, which made it super straightforward.	2 years ago
Daniel Upton	3c533ceea8	proxycfg-glue: server-local implementation of `ServiceList` ... This is the OSS portion of enterprise PR 2242. This PR introduces a server-local implementation of the proxycfg.ServiceList interface, backed by streaming events and a local materializer.	2 years ago
Daniel Upton	fbf88d3b19	proxycfg-glue: server-local compiled discovery chain data source ... This is the OSS portion of enterprise PR 2236. Adds a local blocking query-based implementation of the proxycfg.CompiledDiscoveryChain interface.	2 years ago
Chris S. Kim	f56810132f	Check if an upstream is implicit from either intentions or peered services	2 years ago
Chris S. Kim	02cff2394d	Use new maps for proxycfg peered data	2 years ago
Chris S. Kim	7f32cba735	Add new watch.Map type to refactor proxycfg	2 years ago
Chris S. Kim	b4ffa9ae0c	Scrub VirtualIPs before exporting	2 years ago
Kyle Havlovitz	9097e2b0f0	Merge pull request #13699 from hashicorp/tgate-http2-upstream ... Respect http2 protocol for upstreams of terminating gateways	2 years ago
Dan Upton	b9e525d689	grpc: rename public/private directories to external/internal (#13721) ... Previously, public referred to gRPC services that are both exposed on the dedicated gRPC port and have their definitions in the proto-public directory (so were considered usable by 3rd parties). Whereas private referred to services on the multiplexed server port that are only usable by agents and other servers. Now, we're splitting these definitions, such that external/internal refers to the port and public/private refers to whether they can be used by 3rd parties. This is necessary because the peering replication API needs to be exposed on the dedicated port, but is not (yet) suitable for use by 3rd parties.	2 years ago
R.B. Boyer	30fffd0c90	peerstream: some cosmetic refactors to make this easier to follow (#13732) ... - Use some protobuf construction helper methods for brevity. - Rename a local variable to avoid later shadowing. - Rename the Nonce field to be more like xDS's naming. - Be more explicit about which PeerID fields are empty.	2 years ago
Kyle Havlovitz	7d0c692374	Use protocol from resolved config entry, not gateway service	2 years ago
Kyle Havlovitz	7162e3bde2	Enable http2 options for grpc protocol	2 years ago
R.B. Boyer	c5c216008d	peering: always send the mesh gateway SpiffeID even for tcp services (#13728) ... If someone were to switch a peer-exported service from L4 to L7 there would be a brief SAN validation hiccup as traffic shifted to the mesh gateway for termination. This PR sends the mesh gateway SpiffeID down all the time so the clients always expect a switch.	2 years ago
R.B. Boyer	f0e6e4e697	state: prohibit changing an exported tcp discovery chain in a way that would break SAN validation (#13727) ... For L4/tcp exported services the mesh gateways will not be terminating TLS. A caller in one peer will be directly establishing TLS connections to the ultimate exported service in the other peer. The caller will be doing SAN validation using the replicated SpiffeID values shipped from the exporting side. There are a class of discovery chain edits that could be done on the exporting side that would cause the introduction of a new SpiffeID value. In between the time of the config entry update on the exporting side and the importing side getting updated peer stream data requests to the exported service would fail due to SAN validation errors. This is unacceptable so instead prohibit the exporting peer from making changes that would break peering in this way.	2 years ago
R.B. Boyer	2317f37b4d	state: prohibit exported discovery chains to have cross-datacenter or cross-partition references (#13726) ... Because peerings are pairwise, between two tuples of (datacenter, partition) having any exported reference via a discovery chain that crosses out of the peered datacenter or partition will ultimately not be able to work for various reasons. The biggest one is that there is no way in the ultimate destination to configure an intention that can allow an external SpiffeID to access a service. This PR ensures that a user simply cannot do this, so they won't run into weird situations like this.	2 years ago
Chris S. Kim	a6634db4a5	Return error if ServerAddresses is empty (#13714)	2 years ago
Kyle Havlovitz	439eccdd80	Respect http2 protocol for upstreams of terminating gateways	2 years ago
R.B. Boyer	af04851637	peering: move peer replication to the external gRPC port (#13698) ... Peer replication is intended to be between separate Consul installs and effectively should be considered "external". This PR moves the peer stream replication bidirectional RPC endpoint to the external gRPC server and ensures that things continue to function.	2 years ago
R.B. Boyer	ea58f235f5	server: broadcast the public grpc port using lan serf and update the consul service in the catalog with the same data (#13687) ... Currently servers exchange information about their WAN serf port and RPC port with serf tags, so that they all learn of each other's addressing information. We intend to make larger use of the new public-facing gRPC port exposed on all of the servers, so this PR addresses that by passing around the gRPC port via serf tags and then ensuring the generated consul service in the catalog has metadata about that new port as well for ease of non-serf-based lookup.	2 years ago
Freddy	3542138e4d	Parse peer name for virtual IP DNS queries (#13602) ... This commit updates the DNS query locality parsing so that the virtual IP for an imported service can be queried. Note that: - Support for parsing a peer in other service discovery queries was not added. - Querying another datacenter for a virtual IP is not supported. This was technically allowed in 1.11 but is being rolled back for 1.13 because it is not a use-case we intended to support. Virtual IPs in different datacenters are going to collide because they are allocated sequentially.	2 years ago
R.B. Boyer	2a945facec	test: update mockery use to put mocks into test files (#13656) ... --testonly doesn't do anything anymore so switch to --filename instead	2 years ago
Chris S. Kim	f07132dacc	Revise possible states for a peering. (#13661) ... These changes are primarily for Consul's UI, where we want to be more specific about the state a peering is in. - The "initial" state was renamed to pending, and no longer applies to peerings being established from a peering token. - Upon request to establish a peering from a peering token, peerings will be set as "establishing". This will help distinguish between the two roles: the cluster that generates the peering token and the cluster that establishes the peering. - When marked for deletion, peering state will be set to "deleting". This way the UI determines the deletion via the state rather than the "DeletedAt" field. Co-authored-by: freddygv <freddy@hashicorp.com>	2 years ago
Daniel Upton	45886848b4	proxycfg: server-local intention upstreams data source ... This is the OSS portion of enterprise PR 2157. It builds on the local blocking query work in #13438 to implement the proxycfg.IntentionUpstreams interface using server-local data. Also moves the ACL filtering logic from agent/consul into the acl/filter package so that it can be reused here.	2 years ago
Daniel Upton	37ccbd2826	proxycfg: server-local intentions data source ... This is the OSS portion of enterprise PR 2141. This commit provides a server-local implementation of the `proxycfg.Intentions` interface that sources data from streaming events. It adds events for the `service-intentions` config entry type, and then consumes event streams (via materialized views) for the service's explicit intentions and any applicable wildcard intentions, merging them into a single list of intentions. An alternative approach I considered was to consume _all_ intention events (via `SubjectWildcard`) and filter out the irrelevant ones. This would admittedly remove some complexity in the `agent/proxycfg-glue` package but at the expense of considerable overhead from waking potentially many thousands of connect proxies every time any intention is updated.	2 years ago
Daniel Upton	653b8c4f9d	proxycfg: server-local config entry data sources ... This is the OSS portion of enterprise PR 2056. This commit provides server-local implementations of the proxycfg.ConfigEntry and proxycfg.ConfigEntryList interfaces, that source data from streaming events. It makes use of the LocalMaterializer type introduced for peering replication, adding the necessary support for authorization. It also adds support for "wildcard" subscriptions (within a topic) to the event publisher, as this is needed to fetch service-resolvers for all services when configuring mesh gateways. Currently, events will be emitted for just the ingress-gateway, service-resolver, and mesh config entry types, as these are the only entries required by proxycfg — the events will be emitted on topics named IngressGateway, ServiceResolver, and MeshConfig topics respectively. Though these events will only be consumed "locally" for now, they can also be consumed via the gRPC endpoint (confirmed using grpcurl) so using them from client agents should be a case of swapping the LocalMaterializer for an RPCMaterializer.	2 years ago
alex	cd9ca4290a	peering: add imported/exported counts to peering (#13644) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com> Co-authored-by: Chris S. Kim <ckim@hashicorp.com>	2 years ago
Chris S. Kim	b186731a2e	Fix ENT drift in files (#13647)	2 years ago
Chris S. Kim	d8b7940e40	Add internal endpoint to fetch peered upstream candidates from VirtualIP table (#13642) ... For initial cluster peering TProxy support we consider all imported services of a partition to be potential upstreams. We leverage the VirtualIP table because it stores plain service names (e.g. "api", not "api-sidecar-proxy").	2 years ago
Eric Haberkorn	653cb42944	Fix spelling mistake in serverless patcher (#13607) ... passhthrough -> passthrough	2 years ago
alex	07bc22e405	no 1.9 style metrics (#13532) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
alex	beb8b03e8a	peering: reconcile/ hint active state for list (#13619) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
R.B. Boyer	31b95c747b	xds: modify rbac rules to use the XFCC header for peered L7 enforcement (#13629) ... When the protocol is http-like, and an intention has a peered source then the normal RBAC mTLS SAN field check is replaces with a joint combo of: mTLS SAN field must be the service's local mesh gateway leaf cert AND the first XFCC header (from the MGW) must have a URI field that matches the original intention source Also: - Update the regex program limit to be much higher than the teeny defaults, since the RBAC regex constructions are more complicated now. - Fix a few stray panics in xds generation.	2 years ago
R.B. Boyer	de0f9ac519	xds: have mesh gateways forward peered SpiffeIDs using the XFCC header (#13625)	2 years ago
R.B. Boyer	1a9c86ea8f	xds: mesh gateways now correctly load up peer-exported discovery chains using L7 protocols (#13624) ... A mesh gateway will now configure the filter chains for L7 exported services using the correct discovery chain information.	2 years ago
R.B. Boyer	0fa828db76	peering: replicate all SpiffeID values necessary for the importing side to do SAN validation (#13612) ... When traversing an exported peered service, the discovery chain evaluation at the other side may re-route the request to a variety of endpoints. Furthermore we intend to terminate mTLS at the mesh gateway for arriving peered traffic that is http-like (L7), so the caller needs to know the mesh gateway's SpiffeID in that case as well. The following new SpiffeID values will be shipped back in the peerstream replication: - tcp: all possible SpiffeIDs resulting from the service-resolver component of the exported discovery chain - http-like: the SpiffeID of the mesh gateway	2 years ago
Max Bowsher	ef4b9e541f	Merge branch 'main' into fix-kv_entries-metric	2 years ago
alex	53f0cf5835	peering, internal: support UIServices, UINodes, UINodeInfo (#13577)	2 years ago
Chris S. Kim	2e4cb6f77d	Add new index for PeeredServiceName and ServiceVirtualIP (#13582) ... For TProxy we will be leveraging the VirtualIP table, which needs to become peer-aware	2 years ago
alex	20ecf0febd	Merge pull request #13570 from hashicorp/acpance/peering-oss-intentions ... oss: peering, http: get peer service intentions (#2098)	2 years ago
Will Jordan	34ecbc1d71	Add per-node max indexes (#12399) ... Adds fine-grained node.[node] entries to the index table, allowing blocking queries to return fine-grained indexes that prevent them from returning immediately when unrelated nodes/services are updated. Co-authored-by: kisunji <ckim@hashicorp.com>	2 years ago
Chris S. Kim	ba89a7d9b0	Make memdb indexers generic (#13558) ... We have many indexer functions in Consul which take interface{} and type assert before building the index. We can use generics to get rid of the initial plumbing and pass around functions with better defined signatures. This has two benefits: 1) Less verbosity; 2) Developers can parse the argument types to memdb schemas without having to introspect the function for the type assertion.	2 years ago
Matt Keeler	7a4d13b0b2	Port over the index 0 -> 1 code that lived in the old rpc setQueryMeta function. (#13561)	2 years ago
acpana	99c2e11328	oss: peering, http: get peer service intentions (#2098) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
R.B. Boyer	e8ea3d7c3b	state: peering ID assignment cannot happen inside of the state store (#13525) ... Move peering ID assignment outisde of the FSM, so that the ID is written to the raft log and the same ID is used by all voters, and after restarts.	2 years ago
Matt Keeler	cb01702cd2	Add server local blocking queries and watches (#13438) ... Co-authored-by: Dan Upton <daniel@floppy.co>	2 years ago
Chris S. Kim	fb5eb20563	Pass trust domain to RBAC to validate and fix use of wrong peer trust bundles (#13508)	2 years ago
Max Bowsher	7b97b8abd2	Delete definition of metric `consul.acl.blocked.node.registration` ... Although the metric is defined, there is no code which ever sets its value - the code in question is genuinely asymmetric - there are 3 types of object for which registration can be tracked, but only 2 for which deregistration can be tracked.	2 years ago
Max Bowsher	7c19c701e1	Fix incorrect name and doc for kv_entries metric ... The name of the metric as registered with the metrics library to provide the help string, was incorrect compared with the actual code that sets the metric value - bring them into sync. Also, the help message was incorrect. Rather than copy the help message from telemetry.mdx, which was correct, but felt a bit unnatural in the way it was worded, update both of them to a new wording.	2 years ago
Dan Upton	e00e3a0bc3	Move ACLResolveResult into acl/resolver package (#13467) ... Having this type live in the agent/consul package makes it difficult to put anything that relies on token resolution (e.g. the new gRPC services) in separate packages without introducing import cycles. For example, if package foo imports agent/consul for the ACLResolveResult type it means that agent/consul cannot import foo to register its service. We've previously worked around this by wrapping the ACLResolver to "downgrade" its return type to an acl.Authorizer - aside from the added complexity, this also loses the resolved identity information. In the future, we may want to move the whole ACLResolver into the acl/resolver package. For now, putting the result type there at least, fixes the immediate import cycle issues.	2 years ago
DanStough	4b402e3119	feat: tgtwy xDS generation for destinations ... Signed-off-by: Dhia Ayachi <dhia@hashicorp.com>	2 years ago
alex	bd4ddb3720	peering: block Intention.Apply ops (#13451) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
alex	b3e99784a6	peering, state: account for peer intentions (#13443) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
R.B. Boyer	da8cea58c9	xds: begin refactor to always pass test snapshots through all xDS types (#13461)	2 years ago
R.B. Boyer	201d1458c3	xds: mesh gateways now have their own leaf certificate when involved in a peering (#13460) ... This is only configured in xDS when a service with an L7 protocol is exported. They also load any relevant trust bundles for the peered services to eventually use for L7 SPIFFE validation during mTLS termination.	2 years ago
Riddhi Shah	411edc876b	[OSS] Support merge-central-config option in node services list API (#13450) ... Adds the merge-central-config query param option to the /catalog/node-services/:node-name API, to get a service definition in the response that is merged with central defaults (proxy-defaults/service-defaults). Updated the consul connect envoy command to use this option when retrieving the proxy service details so as to render the bootstrap configuration correctly.	2 years ago
Evan Culver	7f8c650d61	connect: Use Envoy 1.22.2 instead of 1.22.1 (#13444)	2 years ago
freddygv	f3843809da	Avoid deleting peerings marked as terminated. ... When our peer deletes the peering it is locally marked as terminated. This termination should kick off deleting all imported data, but should not delete the peering object itself. Keeping peerings marked as terminated acts as a signal that the action took place.	2 years ago
freddygv	6453375ab2	Add leader routine to clean up peerings ... Once a peering is marked for deletion a new leader routine will now clean up all imported resources and then the peering itself. A lot of the logic was grabbed from the namespace/partitions deferred deletions but with a handful of simplifications: - The rate limiting is not configurable. - Deleting imported nodes/services/checks is done by deleting nodes with the Txn API. The services and checks are deleted as a side-effect. - There is no "round rate limiter" like with namespaces and partitions. This is because peerings are purely local, and deleting a peering in the datacenter does not depend on deleting data from other DCs like with WAN-federated namespaces. All rate limiting is handled by the Raft rate limiter.	2 years ago
Evan Culver	ba6136eb42	connect: Update Envoy support matrix to latest patch releases (#13431)	2 years ago
alex	a0a49ce2a6	peering: intentions list test (#13435)	2 years ago
freddygv	6c8ab1bbac	Fixup stream tear-down steps. ... 1. Fix a bug where the peering leader routine would not track all active peerings in the "stored" reconciliation map. This could lead to tearing down streams where the token was generated, since the ConnectedStreams() method used for reconciliation returns all streams and not just the ones initiated by this leader routine. 2. Fix a race where stream contexts were being canceled before termination messages were being processed by a peer. Previously the leader routine would tear down streams by canceling their context right after the termination message was sent. This context cancelation could be propagated to the server side faster than the termination message. Now there is a change where the dialing peer uses CloseSend() to signal when no more messages will be sent. Eventually the server peer will read an EOF after receiving and processing the preceding termination message. Using CloseSend() is actually not enough to address the issue mentioned, since it doesn't wait for the server peer to finish processing messages. Because of this now the dialing peer also reads from the stream until an error signals that there are no more messages. Receiving an EOF from our peer indicates that they processed the termination message and have no additional work to do. Given that the stream is being closed, all the messages received by Recv are discarded. We only check for errors to avoid importing new data.	2 years ago
freddygv	cc921a9c78	Update peering state and RPC for deferred deletion ... When deleting a peering we do not want to delete the peering and all imported data in a single operation, since deleting a large amount of data at once could overload Consul. Instead we defer deletion of peerings so that: 1. When a peering deletion request is received via gRPC the peering is marked for deletion by setting the DeletedAt field. 2. A leader routine will monitor for peerings that are marked for deletion and kick off a throttled deletion of all imported resources before deleting the peering itself. This commit mostly addresses point #1 by modifying the peering service to mark peerings for deletion. Another key change is to add a PeeringListDeleted state store function which can return all peerings marked for deletion. This function is what will be watched by the deferred deletion leader routine.	2 years ago
Freddy	71b254522e	Clean up imported nodes/services/checks as needed (#13367) ... Previously, imported data would never be deleted. As nodes/services/checks were registered and deregistered, resources deleted from the exporting cluster would accumulate in the imported cluster. This commit makes updates to replication so that whenever an update is received for a service name we reconcile what was present in the catalog against what was received. This handleUpdateService method can handle both updates and deletions.	2 years ago
Mark Anderson	edbf19f4e8	Merge pull request #13357 from hashicorp/ma/add-build-date-oss ... Add build date (oss)	2 years ago
Chris S. Kim	a02e9abcc1	Update RBAC to handle imported services (#13404) ... When converting from Consul intentions to xds RBAC rules, services imported from other peers must encode additional data like partition (from the remote cluster) and trust domain. This PR updates the PeeringTrustBundle to hold the sending side's local partition as ExportedPartition. It also updates RBAC code to encode SpiffeIDs of imported services with the ExportedPartition and TrustDomain.	3 years ago
R.B. Boyer	f557509e58	xds: allow for peered upstreams to use tagged addresses that are hostnames (#13422) ... Mesh gateways can use hostnames in their tagged addresses (#7999). This is useful if you were to expose a mesh gateway using a cloud networking load balancer appliance that gives you a DNS name but no reliable static IPs. Envoy cannot accept hostnames via EDS and those must be configured using CDS. There was already logic when configuring gateways in other locations in the code, but given the illusions in play for peering the downstream of a peered service wasn't aware that it should be doing that. Also: - ensuring that we always try to use wan-like addresses to cross peer boundaries.	3 years ago
Kyle Havlovitz	7f62571419	Add dns node lookup support in partitions	3 years ago
R.B. Boyer	7001e1151c	peering: rename initiate to establish in the context of the APIs (#13419)	3 years ago
Mark Anderson	dd22ceccd1	Change default dates ... Signed-off-by: Mark Anderson <manderson@hashicorp.com>	3 years ago
Mark Anderson	f65093f1c6	Fixup some more tests ... Signed-off-by: Mark Anderson <manderson@hashicorp.com>	3 years ago
Mark Anderson	19c87be3a6	Add build date to self endpoint ... Signed-off-by: Mark Anderson <manderson@hashicorp.com>	3 years ago
Mark Anderson	ec060e5e37	Build date in config file ... Signed-off-by: Mark Anderson <manderson@hashicorp.com>	3 years ago
R.B. Boyer	bba3eb8cdd	peering: mesh gateways are required for cross-peer service mesh communication (#13410) ... Require use of mesh gateways in order for service mesh data plane traffic to flow between peers. This also adds plumbing for envoy integration tests involving peers, and one starter peering test.	3 years ago
kisunji	196a1c468a	Add missing index for read	3 years ago
kisunji	d026d84880	Add IntentionMatch tests for source peers	3 years ago
kisunji	bb0b42da12	Update ServiceIntentionSourceIndex to handle peer	3 years ago
Chris S. Kim	bb832e2bba	Add SourcePeer fields to relevant Intentions types (#13390)	3 years ago
R.B. Boyer	7423886136	peering: allow protobuf requests to populate the default partition or namespace (#13398)	3 years ago
Dhia Ayachi	ec0d267a35	Fix intentions wildcard dest (#13397) ... * when enterprise meta are wildcard assume it's a service intention * fix partition and namespace * move kind outside the loops * get the kind check outside the loop and add a comment Co-authored-by: github-team-consul-core <github-team-consul-core@hashicorp.com>	3 years ago
R.B. Boyer	edb2e55335	peering: avoid a race between peering establishment and termination (#13389)	3 years ago
Dhia Ayachi	7393374fc0	Egress gtw/intention rpc endpoint (#13354) ... * update gateway-services table with endpoints * fix failing test * remove unneeded config in test * rename "endpoint" to "destination" * more endpoint renaming to destination in tests * update isDestination based on service-defaults config entry creation * use a 3 state kind to be able to set the kind to unknown (when neither a service or a destination exist) * set unknown state to empty to avoid modifying alot of tests * fix logic to set the kind correctly on CRUD * fix failing tests * add missing tests and fix service delete * fix failing test * Apply suggestions from code review Co-authored-by: Dan Stough <dan.stough@hashicorp.com> * fix a bug with kind and add relevant test * fix compile error * fix failing tests * add kind to clone * fix failing tests * fix failing tests in catalog endpoint * fix service dump test * Apply suggestions from code review Co-authored-by: Dan Stough <dan.stough@hashicorp.com> * remove duplicate tests * first draft of destinations intention in connect proxy * remove ServiceDestinationList * fix failing tests * fix agent/consul failing tests * change to filter intentions in the state store instead of adding a field. * fix failing tests * fix comment * fix comments * store service kind destination and add relevant tests * changes based on review * filter on destinations when querying source match * change state store API to get an IntentionTarget parameter * add intentions tests * add destination upstream endpoint * fix failing test * fix failing test and a bug with wildcard intentions * fix failing test * Apply suggestions from code review Co-authored-by: alex <8968914+acpana@users.noreply.github.com> * add missing test and clarify doc * fix style * gofmt intention.go * fix merge introduced issue Co-authored-by: Dan Stough <dan.stough@hashicorp.com> Co-authored-by: alex <8968914+acpana@users.noreply.github.com> Co-authored-by: github-team-consul-core <github-team-consul-core@hashicorp.com>	3 years ago
Dhia Ayachi	5ec3274ae5	Egress gtw/connect destination intentions (#13341) ... * update gateway-services table with endpoints * fix failing test * remove unneeded config in test * rename "endpoint" to "destination" * more endpoint renaming to destination in tests * update isDestination based on service-defaults config entry creation * use a 3 state kind to be able to set the kind to unknown (when neither a service or a destination exist) * set unknown state to empty to avoid modifying alot of tests * fix logic to set the kind correctly on CRUD * fix failing tests * add missing tests and fix service delete * fix failing test * Apply suggestions from code review Co-authored-by: Dan Stough <dan.stough@hashicorp.com> * fix a bug with kind and add relevant test * fix compile error * fix failing tests * add kind to clone * fix failing tests * fix failing tests in catalog endpoint * fix service dump test * Apply suggestions from code review Co-authored-by: Dan Stough <dan.stough@hashicorp.com> * remove duplicate tests * first draft of destinations intention in connect proxy * remove ServiceDestinationList * fix failing tests * fix agent/consul failing tests * change to filter intentions in the state store instead of adding a field. * fix failing tests * fix comment * fix comments * store service kind destination and add relevant tests * changes based on review * filter on destinations when querying source match * Apply suggestions from code review Co-authored-by: alex <8968914+acpana@users.noreply.github.com> * fix style * Apply suggestions from code review Co-authored-by: Dan Stough <dan.stough@hashicorp.com> * rename destinationType to targetType. Co-authored-by: Dan Stough <dan.stough@hashicorp.com> Co-authored-by: alex <8968914+acpana@users.noreply.github.com> Co-authored-by: github-team-consul-core <github-team-consul-core@hashicorp.com>	3 years ago
R.B. Boyer	ab758b7b32	peering: allow mesh gateways to proxy L4 peered traffic (#13339) ... Mesh gateways will now enable tcp connections with SNI names including peering information so that those connections may be proxied. Note: this does not change the callers to use these mesh gateways.	3 years ago
Fulvio	d457d8b6ce	UDP check for service stanza #12221 (#12722) ... * UDP check for service stanza #12221 * add pass status on timeout condition * delete useless files * Update check_test.go improve comment in test * fix test * fix requested changes and update TestRuntimeConfig_Sanitize.golden * add freeport to TestCheckUDPCritical * improve comment for CheckUDP struct * fix requested changes * fix requested changes * fix requested changes * add UDP to proto * add UDP to proto and add a changelog * add requested test on agent_endpoint_test.go * add test for given endpoints * fix failing tests * add documentation for udp healthcheck * regenerate proto using buf * Update website/content/api-docs/agent/check.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * Update website/content/api-docs/agent/check.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * Update website/content/docs/discovery/checks.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * Update website/content/docs/ecs/configuration-reference.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * Update website/content/docs/ecs/configuration-reference.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * add debug echo * add debug circle-ci * add debug circle-ci bash * use echo instead of status_stage * remove debug and status from devtools script and use echo instead * Update website/content/api-docs/agent/check.mdx Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com> * fix test * replace status_stage with status * replace functions with echo Co-authored-by: Dhia Ayachi <dhia@hashicorp.com> Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com>	3 years ago
alex	bbbc50815a	peering: send leader addr (#13342) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	3 years ago
Dan Upton	b168424398	xds: remove HTTPCheckFetcher dependency (#13366) ... This is the OSS portion of enterprise PR 1994 Rather than directly interrogating the agent-local state for HTTP checks using the `HTTPCheckFetcher` interface, we now rely on the config snapshot containing the checks. This reduces the number of changes required to support server xDS sessions. It's not clear why the fetching approach was introduced in 931d167ebb.	3 years ago
R.B. Boyer	019aeaa57d	peering: update how cross-peer upstreams and represented in proxycfg and rendered in xds (#13362) ... This removes unnecessary, vestigal remnants of discovery chains.	3 years ago
cskh	74158a8aa2	Add isLeader metric to track if a server is a leader (#13304) ... CTIA-21: sdd is_leader metric to track if a server is a leader Co-authored-by: alex <8968914+acpana@users.noreply.github.com>	3 years ago
Freddy	32f125cc0f	Merge pull request #13340 from hashicorp/peering/public-listener	3 years ago
Chris S. Kim	73af9e9737	Fix KVSGet method to handle QueryOptions properly (#13344)	3 years ago
Freddy	a09c776645	Update public listener with SPIFFE Validator ... Envoy's SPIFFE certificate validation extension allows for us to validate against different root certificates depending on the trust domain of the dialing proxy. If there are any trust bundles from peers in the config snapshot then we use the SPIFFE validator as the validation context, rather than the usual TrustedCA. The injected validation config includes the local root certificates as well.	3 years ago
freddygv	647c57a416	Add agent cache-type for TrustBundleListByService ... There are a handful of changes in this commit: * When querying trust bundles for a service we need to be able to specify the namespace of the service. * The endpoint needs to track the index because the cache watches use it. * Extracted bulk of the endpoint's logic to a state store function so that index tracking could be tested more easily. * Removed check for service existence, deferring that sort of work to ACL authz * Added the cache type	3 years ago
freddygv	8b58fa8afe	Update assumptions around exported-service config ... Given that the exported-services config entry can use wildcards, the precedence for wildcards is handled as with intentions. The most exact match is the match that applies for any given service. We do not take the union of all that apply. Another update that was made was to reflect that only one exported-services config entry applies to any given service in a partition. This is a pre-existing constraint that gets enforced by the Normalize() method on that config entry type.	3 years ago
Freddy	74ca6406ea	Configure upstream TLS context with peer root certs (#13321) ... For mTLS to work between two proxies in peered clusters with different root CAs, proxies need to configure their outbound listener to use different root certificates for validation. Up until peering was introduced proxies would only ever use one set of root certificates to validate all mesh traffic, both inbound and outbound. Now an upstream proxy may have a leaf certificate signed by a CA that's different from the dialing proxy's. This PR makes changes to proxycfg and xds so that the upstream TLS validation uses different root certificates depending on which cluster is being dialed.	3 years ago
R.B. Boyer	8e530701ce	test: regenerate golden files (#13336) ... make envoy-regen go test ./agent/config -update	3 years ago
Chris S. Kim	fcdd031911	Revert getPathSuffixUnescaped (#13256)	3 years ago
Dan Upton	adeabed126	proxycfg: replace direct agent cache usage with interfaces (#13320) ... This is the OSS portion of enterprise PRs 1904, 1905, 1906, 1907, 1949, and 1971. It replaces the proxycfg manager's direct dependency on the agent cache with interfaces that will be implemented differently when serving xDS sessions from a Consul server.	3 years ago
Chris S. Kim	67860bd248	Reimplement fs.FileInfo interface (#13315) ... Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>	3 years ago
Dhia Ayachi	1b779240ae	update gateway-services table with endpoints (#13217) ... * update gateway-services table with endpoints * fix failing test * remove unneeded config in test * rename "endpoint" to "destination" * more endpoint renaming to destination in tests * update isDestination based on service-defaults config entry creation * use a 3 state kind to be able to set the kind to unknown (when neither a service or a destination exist) * set unknown state to empty to avoid modifying alot of tests * fix logic to set the kind correctly on CRUD * fix failing tests * add missing tests and fix service delete * fix failing test * Apply suggestions from code review Co-authored-by: Dan Stough <dan.stough@hashicorp.com> * fix a bug with kind and add relevant test * fix compile error * fix failing tests * add kind to clone * fix failing tests * fix failing tests in catalog endpoint * fix service dump test * Apply suggestions from code review Co-authored-by: Dan Stough <dan.stough@hashicorp.com> * remove duplicate tests * rename consts and fix kind when no destination is defined in the service-defaults. * rename Kind to ServiceKind and change switch to use .(type) Co-authored-by: Dan Stough <dan.stough@hashicorp.com>	3 years ago
Chris S. Kim	f0a9b30174	Update repo to use go:embed (#10996) ... Replace bindata packages with stdlib go:embed. Modernize some uiserver code with newer interfaces introduced in go 1.16 (mainly working with fs.File instead of http.File. Remove steps that are no longer used from our build files. Add Github Action to detect differences in agent/uiserver/dist and verify that the files are correct (by compiling UI assets and comparing contents).	3 years ago
Riddhi Shah	1a901953e2	[OSS] Fix merge central config tests (#13309) ... Setting the right enterprise meta to fix the merge central config tests. Re-added the tests that were failing on the OSS to ENT merge.	3 years ago
freddygv	364758ef2f	Use embedded SpiffeID for peered upstreams	3 years ago
freddygv	c8edec0ab6	Remove intermediate representation of SPIFFE IDs ... xDS only ever uses the string representation, so we can avoid passing around connect.SpiffeIDService objects around.	3 years ago
freddygv	870e7c72d7	Return SPIFFE ID for connect proxies in PeerMeta ... Proxies dialing exporting services need to know the SPIFFE ID of services dialed so that the upstream's SANs can be validated. This commit attaches the SPIFFE ID to all connect proxies exported over the peering stream so that they are available to importing clusters. The data in the SPIFFE ID cannot be re-constructed in peer clusters because the partition of exported services is overwritten on imports.	3 years ago
Freddy	9427700270	[OSS] Add grpc endpoint to fetch a specific trust bundle (#13292) ... Co-authored-by: R.B. Boyer <rb@hashicorp.com>	3 years ago
Matt Keeler	3795769729	Fix a flaky test (#13282) ... At the end of this test we were trying to ensure that updating a service in the local state causes it to re-register the service with the config manager. The config manager in the same method will also call RegisteredProxies to determine if any need to be removed. This portion of the test is not attempting to verify that behavior. Because the test is only blocked waiting for the Register event before it can end and assert all the mock expectations were met, we may not see the call to RegisteredProxies. This is especially apparent when tests are run with the race detector. As we don’t actually care if that method is executed before the end of the test we can simply transition from expecting it to be called exactly once to a 0 or 1 times assertion.	3 years ago
Dan Upton	2427e38839	Enable servers to configure arbitrary proxies from the catalog (#13244) ... OSS port of enterprise PR 1822 Includes the necessary changes to the `proxycfg` and `xds` packages to enable Consul servers to configure arbitrary proxies using catalog data. Broadly, `proxycfg.Manager` now has public methods for registering, deregistering, and listing registered proxies — the existing local agent state-sync behavior has been moved into a separate component that makes use of these methods. When an xDS session is started for a proxy service in the catalog, a goroutine will be spawned to watch the service in the server's state store and re-register it with the `proxycfg.Manager` whenever it is updated (and clean it up when the client goes away).	3 years ago
alex	fd7a403e11	monitor leadership in peering service (#13257) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com> Co-authored-by: Chris S. Kim <ckim@hashicorp.com> Co-authored-by: Freddy <freddygv@users.noreply.github.com>	3 years ago
Riddhi Shah	b6a4271c02	Termporarily disable validation of merge central config response (#13266) ... Temporarily disabling the validation of merge central config response since it is breaking OSS to ENT merging. A follow up PR will patch the fixes.	3 years ago
Chris S. Kim	6d3bea7129	Add support for streaming CA roots to peers (#13260) ... Sender watches for changes to CA roots and sends them through the replication stream. Receiver saves CA roots to tablePeeringTrustBundle	3 years ago
Riddhi Shah	c78ee7d48f	Remove tests failing on ent (#13255) ... Will follow up with the fixed version of these tests that passes in ent.	3 years ago
John Cowen	09c5bac102	Export top-level HCP Enabled go-template variable for UI (#13165) ... * Update ui template data to export HCPEnabled at the top level	3 years ago
DanStough	2e2c71d2f2	fix: multiple grpc/http2 services for ingress listeners	3 years ago
Riddhi Shah	d8d8c8603e	Add support for merge-central-config query param (#13001) ... Adds a new query param merge-central-config for use with the below endpoints: /catalog/service/:service /catalog/connect/:service /health/service/:service /health/connect/:service If set on the request, the response will include a fully resolved service definition which is merged with the proxy-defaults/global and service-defaults/:service config entries (on-demand style). This is useful to view the full service definition for a mesh service (connect-proxy kind or gateway kind) which might not be merged before being written into the catalog (example: in case of services in the agentless model).	3 years ago
R.B. Boyer	31526139fd	remove a source of test panics (#13227)	3 years ago
R.B. Boyer	a85b8a4705	api: ensure peering API endpoints do not use protobufs (#13204) ... I noticed that the JSON api endpoints for peerings json encodes protobufs directly, rather than converting them into their `api` package equivalents before marshal/unmarshaling them. I updated this and used `mog` to do the annoying part in the middle. Other changes: - the status enum was converted into the friendlier string form of the enum for readability with tools like `curl` - some of the `api` library functions were slightly modified to match other similar endpoints in UX (cc: @ndhanushkodi ) - peeringRead returns `nil` if not found - partitions are NOT inferred from the agent's partition (matching 1.11-style logic)	3 years ago
R.B. Boyer	1a8834e1c8	peering: replicate expected SNI, SPIFFE, and service protocol to peers (#13218) ... The importing peer will need to know what SNI and SPIFFE name corresponds to each exported service. Additionally it will need to know at a high level the protocol in use (L4/L7) to generate the appropriate connection pool and local metrics. For replicated connect synthetic entities we edit the `Connect{}` part of a `NodeService` to have a new section: { "PeerMeta": { "SNI": [ "web.default.default.owt.external.183150d5-1033-3672-c426-c29205a576b8.consul" ], "SpiffeID": [ "spiffe://183150d5-1033-3672-c426-c29205a576b8.consul/ns/default/dc/dc1/svc/web" ], "Protocol": "tcp" } } This data is then replicated and saved as-is at the importing side. Both SNI and SpiffeID are slices for now until I can be sure we don't need them for how mesh gateways will ultimately work.	3 years ago
R.B. Boyer	be631ebdce	peering: disable requirement for mesh gateways initially (#13213)	3 years ago
Kyle Havlovitz	0ed9ff8ef7	Merge pull request #13143 from hashicorp/envoy-connection-limit ... Add connection limit setting to service defaults	3 years ago
Kyle Havlovitz	f2fbe8aec9	Fix proto lint errors after version bump	3 years ago
Kyle Havlovitz	dbed8ae10b	Specify go_package explicitly	3 years ago
cskh	8712a088b1	fix: non-leader agents return 404 on Get Intention exact api (#13179) ... * fix: non-leader agents return 404 on Get Intention exact api - rpc call method appends extra error message, so change == to "Strings.Contains" Co-authored-by: Chris S. Kim <ckim@hashicorp.com>	3 years ago
Kyle Havlovitz	4bc6c23357	Add connection limit setting to service defaults	3 years ago