4370 Commits (2e4cb6f77d2be36b02e9be0b289b24e5b0afb794)

Author	SHA1	Message	Date
Chris S. Kim	2e4cb6f77d	Add new index for PeeredServiceName and ServiceVirtualIP (#13582) ... For TProxy we will be leveraging the VirtualIP table, which needs to become peer-aware	2 years ago
alex	20ecf0febd	Merge pull request #13570 from hashicorp/acpance/peering-oss-intentions ... oss: peering, http: get peer service intentions (#2098)	2 years ago
Will Jordan	34ecbc1d71	Add per-node max indexes (#12399) ... Adds fine-grained node.[node] entries to the index table, allowing blocking queries to return fine-grained indexes that prevent them from returning immediately when unrelated nodes/services are updated. Co-authored-by: kisunji <ckim@hashicorp.com>	2 years ago
Chris S. Kim	ba89a7d9b0	Make memdb indexers generic (#13558) ... We have many indexer functions in Consul which take interface{} and type assert before building the index. We can use generics to get rid of the initial plumbing and pass around functions with better defined signatures. This has two benefits: 1) Less verbosity; 2) Developers can parse the argument types to memdb schemas without having to introspect the function for the type assertion.	2 years ago
Matt Keeler	7a4d13b0b2	Port over the index 0 -> 1 code that lived in the old rpc setQueryMeta function. (#13561)	2 years ago
acpana	99c2e11328	oss: peering, http: get peer service intentions (#2098) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
R.B. Boyer	e8ea3d7c3b	state: peering ID assignment cannot happen inside of the state store (#13525) ... Move peering ID assignment outisde of the FSM, so that the ID is written to the raft log and the same ID is used by all voters, and after restarts.	2 years ago
Matt Keeler	cb01702cd2	Add server local blocking queries and watches (#13438) ... Co-authored-by: Dan Upton <daniel@floppy.co>	2 years ago
Chris S. Kim	fb5eb20563	Pass trust domain to RBAC to validate and fix use of wrong peer trust bundles (#13508)	2 years ago
Dan Upton	e00e3a0bc3	Move ACLResolveResult into acl/resolver package (#13467) ... Having this type live in the agent/consul package makes it difficult to put anything that relies on token resolution (e.g. the new gRPC services) in separate packages without introducing import cycles. For example, if package foo imports agent/consul for the ACLResolveResult type it means that agent/consul cannot import foo to register its service. We've previously worked around this by wrapping the ACLResolver to "downgrade" its return type to an acl.Authorizer - aside from the added complexity, this also loses the resolved identity information. In the future, we may want to move the whole ACLResolver into the acl/resolver package. For now, putting the result type there at least, fixes the immediate import cycle issues.	2 years ago
DanStough	4b402e3119	feat: tgtwy xDS generation for destinations ... Signed-off-by: Dhia Ayachi <dhia@hashicorp.com>	2 years ago
alex	bd4ddb3720	peering: block Intention.Apply ops (#13451) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
alex	b3e99784a6	peering, state: account for peer intentions (#13443) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
R.B. Boyer	da8cea58c9	xds: begin refactor to always pass test snapshots through all xDS types (#13461)	2 years ago
R.B. Boyer	201d1458c3	xds: mesh gateways now have their own leaf certificate when involved in a peering (#13460) ... This is only configured in xDS when a service with an L7 protocol is exported. They also load any relevant trust bundles for the peered services to eventually use for L7 SPIFFE validation during mTLS termination.	2 years ago
Riddhi Shah	411edc876b	[OSS] Support merge-central-config option in node services list API (#13450) ... Adds the merge-central-config query param option to the /catalog/node-services/:node-name API, to get a service definition in the response that is merged with central defaults (proxy-defaults/service-defaults). Updated the consul connect envoy command to use this option when retrieving the proxy service details so as to render the bootstrap configuration correctly.	2 years ago
Evan Culver	7f8c650d61	connect: Use Envoy 1.22.2 instead of 1.22.1 (#13444)	2 years ago
freddygv	f3843809da	Avoid deleting peerings marked as terminated. ... When our peer deletes the peering it is locally marked as terminated. This termination should kick off deleting all imported data, but should not delete the peering object itself. Keeping peerings marked as terminated acts as a signal that the action took place.	2 years ago
freddygv	6453375ab2	Add leader routine to clean up peerings ... Once a peering is marked for deletion a new leader routine will now clean up all imported resources and then the peering itself. A lot of the logic was grabbed from the namespace/partitions deferred deletions but with a handful of simplifications: - The rate limiting is not configurable. - Deleting imported nodes/services/checks is done by deleting nodes with the Txn API. The services and checks are deleted as a side-effect. - There is no "round rate limiter" like with namespaces and partitions. This is because peerings are purely local, and deleting a peering in the datacenter does not depend on deleting data from other DCs like with WAN-federated namespaces. All rate limiting is handled by the Raft rate limiter.	2 years ago
Evan Culver	ba6136eb42	connect: Update Envoy support matrix to latest patch releases (#13431)	2 years ago
alex	a0a49ce2a6	peering: intentions list test (#13435)	2 years ago
freddygv	6c8ab1bbac	Fixup stream tear-down steps. ... 1. Fix a bug where the peering leader routine would not track all active peerings in the "stored" reconciliation map. This could lead to tearing down streams where the token was generated, since the ConnectedStreams() method used for reconciliation returns all streams and not just the ones initiated by this leader routine. 2. Fix a race where stream contexts were being canceled before termination messages were being processed by a peer. Previously the leader routine would tear down streams by canceling their context right after the termination message was sent. This context cancelation could be propagated to the server side faster than the termination message. Now there is a change where the dialing peer uses CloseSend() to signal when no more messages will be sent. Eventually the server peer will read an EOF after receiving and processing the preceding termination message. Using CloseSend() is actually not enough to address the issue mentioned, since it doesn't wait for the server peer to finish processing messages. Because of this now the dialing peer also reads from the stream until an error signals that there are no more messages. Receiving an EOF from our peer indicates that they processed the termination message and have no additional work to do. Given that the stream is being closed, all the messages received by Recv are discarded. We only check for errors to avoid importing new data.	2 years ago
freddygv	cc921a9c78	Update peering state and RPC for deferred deletion ... When deleting a peering we do not want to delete the peering and all imported data in a single operation, since deleting a large amount of data at once could overload Consul. Instead we defer deletion of peerings so that: 1. When a peering deletion request is received via gRPC the peering is marked for deletion by setting the DeletedAt field. 2. A leader routine will monitor for peerings that are marked for deletion and kick off a throttled deletion of all imported resources before deleting the peering itself. This commit mostly addresses point #1 by modifying the peering service to mark peerings for deletion. Another key change is to add a PeeringListDeleted state store function which can return all peerings marked for deletion. This function is what will be watched by the deferred deletion leader routine.	2 years ago
Freddy	71b254522e	Clean up imported nodes/services/checks as needed (#13367) ... Previously, imported data would never be deleted. As nodes/services/checks were registered and deregistered, resources deleted from the exporting cluster would accumulate in the imported cluster. This commit makes updates to replication so that whenever an update is received for a service name we reconcile what was present in the catalog against what was received. This handleUpdateService method can handle both updates and deletions.	2 years ago
Mark Anderson	edbf19f4e8	Merge pull request #13357 from hashicorp/ma/add-build-date-oss ... Add build date (oss)	2 years ago
Chris S. Kim	a02e9abcc1	Update RBAC to handle imported services (#13404) ... When converting from Consul intentions to xds RBAC rules, services imported from other peers must encode additional data like partition (from the remote cluster) and trust domain. This PR updates the PeeringTrustBundle to hold the sending side's local partition as ExportedPartition. It also updates RBAC code to encode SpiffeIDs of imported services with the ExportedPartition and TrustDomain.	2 years ago
R.B. Boyer	f557509e58	xds: allow for peered upstreams to use tagged addresses that are hostnames (#13422) ... Mesh gateways can use hostnames in their tagged addresses (#7999). This is useful if you were to expose a mesh gateway using a cloud networking load balancer appliance that gives you a DNS name but no reliable static IPs. Envoy cannot accept hostnames via EDS and those must be configured using CDS. There was already logic when configuring gateways in other locations in the code, but given the illusions in play for peering the downstream of a peered service wasn't aware that it should be doing that. Also: - ensuring that we always try to use wan-like addresses to cross peer boundaries.	2 years ago
Kyle Havlovitz	7f62571419	Add dns node lookup support in partitions	2 years ago
R.B. Boyer	7001e1151c	peering: rename initiate to establish in the context of the APIs (#13419)	2 years ago
Mark Anderson	dd22ceccd1	Change default dates ... Signed-off-by: Mark Anderson <manderson@hashicorp.com>	3 years ago
Mark Anderson	f65093f1c6	Fixup some more tests ... Signed-off-by: Mark Anderson <manderson@hashicorp.com>	3 years ago
Mark Anderson	19c87be3a6	Add build date to self endpoint ... Signed-off-by: Mark Anderson <manderson@hashicorp.com>	3 years ago
Mark Anderson	ec060e5e37	Build date in config file ... Signed-off-by: Mark Anderson <manderson@hashicorp.com>	3 years ago
R.B. Boyer	bba3eb8cdd	peering: mesh gateways are required for cross-peer service mesh communication (#13410) ... Require use of mesh gateways in order for service mesh data plane traffic to flow between peers. This also adds plumbing for envoy integration tests involving peers, and one starter peering test.	3 years ago
kisunji	196a1c468a	Add missing index for read	3 years ago
kisunji	d026d84880	Add IntentionMatch tests for source peers	3 years ago
kisunji	bb0b42da12	Update ServiceIntentionSourceIndex to handle peer	3 years ago
Chris S. Kim	bb832e2bba	Add SourcePeer fields to relevant Intentions types (#13390)	3 years ago
R.B. Boyer	7423886136	peering: allow protobuf requests to populate the default partition or namespace (#13398)	3 years ago
Dhia Ayachi	ec0d267a35	Fix intentions wildcard dest (#13397) ... * when enterprise meta are wildcard assume it's a service intention * fix partition and namespace * move kind outside the loops * get the kind check outside the loop and add a comment Co-authored-by: github-team-consul-core <github-team-consul-core@hashicorp.com>	3 years ago
R.B. Boyer	edb2e55335	peering: avoid a race between peering establishment and termination (#13389)	3 years ago
Dhia Ayachi	7393374fc0	Egress gtw/intention rpc endpoint (#13354) ... * update gateway-services table with endpoints * fix failing test * remove unneeded config in test * rename "endpoint" to "destination" * more endpoint renaming to destination in tests * update isDestination based on service-defaults config entry creation * use a 3 state kind to be able to set the kind to unknown (when neither a service or a destination exist) * set unknown state to empty to avoid modifying alot of tests * fix logic to set the kind correctly on CRUD * fix failing tests * add missing tests and fix service delete * fix failing test * Apply suggestions from code review Co-authored-by: Dan Stough <dan.stough@hashicorp.com> * fix a bug with kind and add relevant test * fix compile error * fix failing tests * add kind to clone * fix failing tests * fix failing tests in catalog endpoint * fix service dump test * Apply suggestions from code review Co-authored-by: Dan Stough <dan.stough@hashicorp.com> * remove duplicate tests * first draft of destinations intention in connect proxy * remove ServiceDestinationList * fix failing tests * fix agent/consul failing tests * change to filter intentions in the state store instead of adding a field. * fix failing tests * fix comment * fix comments * store service kind destination and add relevant tests * changes based on review * filter on destinations when querying source match * change state store API to get an IntentionTarget parameter * add intentions tests * add destination upstream endpoint * fix failing test * fix failing test and a bug with wildcard intentions * fix failing test * Apply suggestions from code review Co-authored-by: alex <8968914+acpana@users.noreply.github.com> * add missing test and clarify doc * fix style * gofmt intention.go * fix merge introduced issue Co-authored-by: Dan Stough <dan.stough@hashicorp.com> Co-authored-by: alex <8968914+acpana@users.noreply.github.com> Co-authored-by: github-team-consul-core <github-team-consul-core@hashicorp.com>	3 years ago
Dhia Ayachi	5ec3274ae5	Egress gtw/connect destination intentions (#13341) ... * update gateway-services table with endpoints * fix failing test * remove unneeded config in test * rename "endpoint" to "destination" * more endpoint renaming to destination in tests * update isDestination based on service-defaults config entry creation * use a 3 state kind to be able to set the kind to unknown (when neither a service or a destination exist) * set unknown state to empty to avoid modifying alot of tests * fix logic to set the kind correctly on CRUD * fix failing tests * add missing tests and fix service delete * fix failing test * Apply suggestions from code review Co-authored-by: Dan Stough <dan.stough@hashicorp.com> * fix a bug with kind and add relevant test * fix compile error * fix failing tests * add kind to clone * fix failing tests * fix failing tests in catalog endpoint * fix service dump test * Apply suggestions from code review Co-authored-by: Dan Stough <dan.stough@hashicorp.com> * remove duplicate tests * first draft of destinations intention in connect proxy * remove ServiceDestinationList * fix failing tests * fix agent/consul failing tests * change to filter intentions in the state store instead of adding a field. * fix failing tests * fix comment * fix comments * store service kind destination and add relevant tests * changes based on review * filter on destinations when querying source match * Apply suggestions from code review Co-authored-by: alex <8968914+acpana@users.noreply.github.com> * fix style * Apply suggestions from code review Co-authored-by: Dan Stough <dan.stough@hashicorp.com> * rename destinationType to targetType. Co-authored-by: Dan Stough <dan.stough@hashicorp.com> Co-authored-by: alex <8968914+acpana@users.noreply.github.com> Co-authored-by: github-team-consul-core <github-team-consul-core@hashicorp.com>	3 years ago
R.B. Boyer	ab758b7b32	peering: allow mesh gateways to proxy L4 peered traffic (#13339) ... Mesh gateways will now enable tcp connections with SNI names including peering information so that those connections may be proxied. Note: this does not change the callers to use these mesh gateways.	3 years ago
Fulvio	d457d8b6ce	UDP check for service stanza #12221 (#12722) ... * UDP check for service stanza #12221 * add pass status on timeout condition * delete useless files * Update check_test.go improve comment in test * fix test * fix requested changes and update TestRuntimeConfig_Sanitize.golden * add freeport to TestCheckUDPCritical * improve comment for CheckUDP struct * fix requested changes * fix requested changes * fix requested changes * add UDP to proto * add UDP to proto and add a changelog * add requested test on agent_endpoint_test.go * add test for given endpoints * fix failing tests * add documentation for udp healthcheck * regenerate proto using buf * Update website/content/api-docs/agent/check.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * Update website/content/api-docs/agent/check.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * Update website/content/docs/discovery/checks.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * Update website/content/docs/ecs/configuration-reference.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * Update website/content/docs/ecs/configuration-reference.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * add debug echo * add debug circle-ci * add debug circle-ci bash * use echo instead of status_stage * remove debug and status from devtools script and use echo instead * Update website/content/api-docs/agent/check.mdx Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com> * fix test * replace status_stage with status * replace functions with echo Co-authored-by: Dhia Ayachi <dhia@hashicorp.com> Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com>	3 years ago
alex	bbbc50815a	peering: send leader addr (#13342) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	3 years ago
Dan Upton	b168424398	xds: remove HTTPCheckFetcher dependency (#13366) ... This is the OSS portion of enterprise PR 1994 Rather than directly interrogating the agent-local state for HTTP checks using the `HTTPCheckFetcher` interface, we now rely on the config snapshot containing the checks. This reduces the number of changes required to support server xDS sessions. It's not clear why the fetching approach was introduced in 931d167ebb.	3 years ago
R.B. Boyer	019aeaa57d	peering: update how cross-peer upstreams and represented in proxycfg and rendered in xds (#13362) ... This removes unnecessary, vestigal remnants of discovery chains.	3 years ago
cskh	74158a8aa2	Add isLeader metric to track if a server is a leader (#13304) ... CTIA-21: sdd is_leader metric to track if a server is a leader Co-authored-by: alex <8968914+acpana@users.noreply.github.com>	3 years ago
Freddy	32f125cc0f	Merge pull request #13340 from hashicorp/peering/public-listener	3 years ago