4703 Commits (62688107affc75861a8d2f89caca9a982523beca)

Author	SHA1	Message	Date
Alessandro De Blasis	c0d647d11e	fix(agent): removed redundant check on prev. running check	2 years ago
Chris S. Kim	def529edd3	Rename test	2 years ago
Chris S. Kim	93271f649c	Fix test	2 years ago
Eric Haberkorn	1099665473	Update the structs and discovery chain for service resolver redirects to cluster peers. (#14366)	2 years ago
Alessandro De Blasis	f3437eaf05	Merge remote-tracking branch 'hashicorp/main' into feature/health-checks_windows_service ... Signed-off-by: Alessandro De Blasis <alex@deblasis.net>	2 years ago
Alessandro De Blasis	f634e36811	fix(OSServiceCheck): fixes following code-review	2 years ago
Chris S. Kim	4d97e2f936	Adjust metrics reporting for peering tracker	2 years ago
freddygv	650e48624d	Allow terminated peerings to be deleted ... Peerings are terminated when a peer decides to delete the peering from their end. Deleting a peering sends a termination message to the peer and triggers them to mark the peering as terminated but does NOT delete the peering itself. This is to prevent peerings from disappearing from both sides just because one side deleted them. Previously the Delete endpoint was skipping the deletion if the peering was not marked as active. However, terminated peerings are also inactive. This PR makes some updates so that peerings marked as terminated can be deleted by users.	2 years ago
Chris S. Kim	937a8ec742	Fix casing	2 years ago
Chris S. Kim	87962b9713	Merge branch 'main' into catalog-service-list-filter	2 years ago
Chris S. Kim	e2fe8b8d65	Fix tests for enterprise	2 years ago
Chris S. Kim	1c43a1a7b4	Merge branch 'main' into NET-638-push-server-address-updates-to-the-peer ... # Conflicts: # agent/grpc-external/services/peerstream/stream_test.go	2 years ago
Chris S. Kim	6ddcc04613	Replace ring buffer with async version (#14314) ... We need to watch for changes to peerings and update the server addresses which get served by the ring buffer. Also, if there is an active connection for a peer, we are getting up-to-date server addresses from the replication stream and can safely ignore the token's addresses which may be stale.	2 years ago
alex	30ff2e9a35	peering: add peer health metric (#14004) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
Chris S. Kim	181063cd23	Exit loop when context is cancelled	2 years ago
cskh	41aea65214	Fix: the inboundconnection limit filter should be placed in front of http co… (#14325) ... * fix: the inboundconnection limit should be placed in front of http connection manager Co-authored-by: Freddy <freddygv@users.noreply.github.com>	2 years ago
Chris S. Kim	8c94d1a80c	Update test comment	2 years ago
Chris S. Kim	5f2959329f	Add check for zero-length server addresses	2 years ago
skpratt	919da33331	no-op: refactor usagemetrics tests for clarity and DRY cases (#14313)	2 years ago
Pablo Ruiz García	1f293e5244	Added new auto_encrypt.grpc_server_tls config option to control AutoTLS enabling of GRPC Server's TLS usage ... Fix for #14253 Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>	2 years ago
Dan Upton	3b993f2da7	dataplane: update envoy bootstrap params for consul-dataplane (#14017) ... Contains 2 changes to the GetEnvoyBootstrapParams response to support consul-dataplane. Exposing node_name and node_id: consul-dataplane will support providing either the node_id or node_name in its configuration. Unfortunately, supporting both in the xDS meta adds a fair amount of complexity (partly because most tables are currently indexed on node_name) so for now we're going to return them both from the bootstrap params endpoint, allowing consul-dataplane to exchange a node_id for a node_name (which it will supply in the xDS meta). Properly setting service for gateways: To avoid the need to special case gateways in consul-dataplane, service will now either be the destination service name for connect proxies, or the gateway service name. This means it can be used as-is in Envoy configuration (i.e. as a cluster name or in metric tags).	2 years ago
Daniel Upton	13c04a13af	proxycfg: terminate stream on irrecoverable errors ... This is the OSS portion of enterprise PR 2339. It improves our handling of "irrecoverable" errors in proxycfg data sources. The canonical example of this is what happens when the ACL token presented by Envoy is deleted/revoked. Previously, the stream would get "stuck" until the xDS server re-checked the token (after 5 minutes) and terminated the stream. Materializers would also sit burning resources retrying something that could never succeed. Now, it is possible for data sources to mark errors as "terminal" which causes the xDS stream to be closed immediately. Similarly, the submatview.Store will evict materializers when it observes they have encountered such an error.	2 years ago
Chris S. Kim	81e965479b	PR feedback to specify Node name in test mock	2 years ago
Eric Haberkorn	58901ad7df	Cluster peering failover disco chain changes (#14296)	2 years ago
Chris S. Kim	cdc8b0634d	Fix flakes	2 years ago
Chris S. Kim	03e92826aa	Increase heartbeat rate to reduce test flakes	2 years ago
Chris S. Kim	06ba9775ee	Remove check for ResponseNonce	2 years ago
Chris S. Kim	547fb9570e	Add missing mock assertions	2 years ago
Chris S. Kim	adff2eef16	Fix data race ... newMockSnapshotHandler has an assertion on t.Cleanup which gets called before the event publisher is cancelled. This commit reorders the context.WithCancel so it properly gets cancelled before the assertion is made.	2 years ago
cskh	060531a29a	Fix: add missing ent meta for test (#14289)	2 years ago
Chris S. Kim	4e40e1d222	Handle server addresses update as client	2 years ago
Chris S. Kim	584d3409c4	Send server addresses on update from server	2 years ago
Chris S. Kim	c9d8ad3939	Add new subscription for server addresses	2 years ago
Chris S. Kim	028b87d51f	Cleanup unused logger	2 years ago
Chris S. Kim	df951bd601	Expose external gRPC port in autopilot ... The grpc_port was added to a NodeService's meta in ea58f235f5	2 years ago
cskh	527ebd068a	fix: missing MaxInboundConnections field in service-defaults config entry (#14072) ... * fix: missing max_inbound_connections field in merge config	2 years ago
cskh	e84e4b8868	Fix: upgrade pkg imdario/merg to prevent merge config panic (#14237) ... * upgrade imdario/merg to prevent merge config panic * test: service definition takes precedence over service-defaults in merged results	2 years ago
James Hartig	f92883bbce	Use the maximum jitter when calculating the timeout ... The timeout should include the maximum possible jitter since the server will randomly add to it's timeout a jitter. If the server's timeout is less than the client's timeout then the client will return an i/o deadline reached error. Before: ``` time curl 'http://localhost:8500/v1/catalog/service/service?dc=other-dc&stale=&wait=600s&index=15820644' rpc error making call: i/o deadline reached real 10m11.469s user 0m0.018s sys 0m0.023s ``` After: ``` time curl 'http://localhost:8500/v1/catalog/service/service?dc=other-dc&stale=&wait=600s&index=15820644' [...] real 10m35.835s user 0m0.021s sys 0m0.021s ```	2 years ago
Eric Haberkorn	1a73b0ca20	Add `Targets` field to service resolver failovers. (#14162) ... This field will be used for cluster peering failover.	2 years ago
Alessandro De Blasis	5dee555888	Merge remote-tracking branch 'hashicorp/main' into feature/health-checks_windows_service ... Signed-off-by: Alessandro De Blasis <alex@deblasis.net>	2 years ago
Alessandro De Blasis	ab611eabc3	Merge remote-tracking branch 'hashicorp/main' into feature/health-checks_windows_service ... Signed-off-by: Alessandro De Blasis <alex@deblasis.net>	2 years ago
cskh	d46b515b64	fix: missing segment and partition (#14194)	2 years ago
Eric Haberkorn	ebd5513d4b	Refactor failover code to use Envoy's aggregate clusters (#14178)	2 years ago
cskh	81931e52c3	feat(telemetry): add labels to serf and memberlist metrics (#14161) ... * feat(telemetry): add labels to serf and memberlist metrics * changelog * doc update Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>	2 years ago
Chris S. Kim	4c928cb2f7	Handle breaking change for ServiceVirtualIP restore (#14149) ... Consul 1.13.0 changed ServiceVirtualIP to use PeeredServiceName instead of ServiceName which was a breaking change for those using service mesh and wanted to restore their snapshot after upgrading to 1.13.0. This commit handles existing data with older ServiceName and converts it during restore so that there are no issues when restoring from older snapshots.	2 years ago
Chris S. Kim	3926009405	Add test to verify forwarding	2 years ago
Chris S. Kim	1ef22360c3	Register peerStreamServer internally to enable RPC forwarding	2 years ago
Chris S. Kim	de73171202	Handle wrapped errors in isFailedPreconditionErr	2 years ago
Daniel Kimsey	3c4fa9b468	Add support for filtering the 'List Services' API ... 1. Create a bexpr filter for performing the filtering 2. Change the state store functions to return the raw (not aggregated) list of ServiceNodes. 3. Move the aggregate service tags by name logic out of the state store functions into a new function called from the RPC endpoint 4. Perform the filtering in the endpoint before aggregation.	2 years ago
cskh	11e7a0d547	fix: shadowed err in retryJoin() (#14112) ... - err value will be used later to surface the error message if r.join() returns any err.	2 years ago
skpratt	79c23a7cd2	Merge pull request #14056 from hashicorp/proxy-register-port-race ... Refactor sidecar_service method to separate port assignment	2 years ago
skpratt	aa77559819	Merge branch 'main' into proxy-register-port-race	2 years ago
Chris S. Kim	e3046120b3	Close active listeners on error ... If startListeners successfully created listeners for some of its input addresses but eventually failed, the function would return an error and existing listeners would not be cleaned up.	2 years ago
Chris S. Kim	6311c651de	Add retry in TestAgentConnectCALeafCert_good	2 years ago
Kyle Havlovitz	6938b8c755	Merge pull request #13958 from hashicorp/gateway-wildcard-fix ... Fix wildcard picking up services it shouldn't for ingress/terminating gateways	2 years ago
Kyle Havlovitz	fe1fcea34f	Add some extra handling for destination deletes	2 years ago
freddygv	d421e18172	Update snapshot test	2 years ago
freddygv	1031ffc3c7	Re-validate existing secrets at state store ... Previously establishment and pending secrets were only checked at the RPC layer. However, given that these are Check-and-Set transactions we should ensure that the given secrets are still valid when persisting a secret exchange or promotion. Otherwise it would be possible for concurrent requests to overwrite each other.	2 years ago
freddygv	0ea4bfae94	Test fixes	2 years ago
freddygv	c04515a844	Use proto message for each secrets write op ... Previously there was a field indicating the operation that triggered a secrets write. Now there is a message for each operation and it contains the secret ID being persisted.	2 years ago
Kyle Havlovitz	6580566c3b	Update ingress/terminating wildcard logic and handle destinations	2 years ago
freddygv	8067890787	Inherit active secret when exchanging	2 years ago
freddygv	60d6e28c97	Pass explicit signal with op for secrets write ... Previously the updates to the peering secrets UUID table relied on inferring what action triggered the update based on a reconciliation against the existing secrets. Instead we now explicitly require the operation to be given so that the inference isn't necessary. This makes the UUID table logic easier to reason about and fixes some related bugs. There is also an update so that the peering secrets get handled on snapshots/restores.	2 years ago
freddygv	9ca687bc7c	Avoid deleting peering secret UUIDs at dialers ... Dialers do not keep track of peering secret UUIDs, so they should not attempt to clean up data from that table when their peering is deleted. We also now keep peer server addresses when marking peerings for deletion. Peer server addresses are used by the ShouldDial() helper when determining whether the peering is for a dialer or an acceptor. We need to keep this data so that peering secrets can be cleaned up accordingly.	2 years ago
skpratt	58eed6b049	Merge pull request #13906 from skpratt/validate-port-agent-split ... Separate port and socket path validation for local agent	2 years ago
Dhia Ayachi	7154367892	add token to the request when creating a cacheIntentions query (#14005)	2 years ago
Kyle Havlovitz	499211f907	Fix wildcard picking up services it shouldn't for ingress/terminating gateways	2 years ago
Daniel Upton	6452118c15	proxycfg-sources: fix hot loop when service not found in catalog ... Fixes a bug where a service getting deleted from the catalog would cause the ConfigSource to spin in a hot loop attempting to look up the service. This is because we were returning a nil WatchSet which would always unblock the select. Kudos to @freddygv for discovering this!	2 years ago
Freddy	42996411cc	Various peering fixes (#13979) ... * Avoid logging StreamSecretID * Wrap additional errors in stream handler * Fix flakiness in leader test and rename servers for clarity. There was a race condition where the peering was being deleted in the test before the stream was active. Now the test waits for the stream to be connected on both sides before deleting the associated peering. * Run flaky test serially	2 years ago
DanStough	169ff71132	fix: ipv4 destination dns resolution	2 years ago
Luke Kysow	988e1fd35d	peering: default to false (#13963) ... * defaulting to false because peering will be released as beta * Ignore peering disabled error in bundles cachetype Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com> Co-authored-by: freddygv <freddy@hashicorp.com> Co-authored-by: Matt Keeler <mjkeeler7@gmail.com>	2 years ago
Freddy	dacf703d20	Merge branch 'main' into fix-kv_entries-metric	2 years ago
Freddy	72b6d69652	Merge pull request #13499 from maxb/delete-unused-metric ... Delete definition of metric `consul.acl.blocked.node.deregistration`	2 years ago
Dhia Ayachi	6fd65a4a45	Tgtwy egress HTTP support (#13953) ... * add golden files * add support to http in tgateway egress destination * fix slice sorting to include both address and port when using server_names * fix listener loop for http destination * fix routes to generate a route per port and a virtualhost per port-address combination * sort virtual hosts list to have a stable order * extract redundant serviceNode	2 years ago
Matt Keeler	f74d0cef7a	Implement/Utilize secrets for Peering Replication Stream (#13977)	2 years ago
alex	a45bb1f06b	block PeerName register requests (#13887) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
Luke Kysow	95096e2c03	peering: retry establishing connection more quickly on certain errors (#13938) ... When we receive a FailedPrecondition error, retry that more quickly because we expect it will resolve shortly. This is particularly important in the context of Consul servers behind a load balancer because when establishing a connection we have to retry until we randomly land on a leader node. The default retry backoff goes from 2s, 4s, 8s, etc. which can result in very long delays quite quickly. Instead, this backoff retries in 8ms five times, then goes exponentially from there: 16ms, 32ms, ... up to a max of 8152ms.	2 years ago
Sarah Pratt	10a4999a87	Separate port and socket path requirement in case of local agent assignment	2 years ago
alex	92c615c35f	Merge pull request #13952 from hashicorp/sync-more-acl ... sync more acl enforcement	2 years ago
Dhia Ayachi	256694b603	inject gateway addons to destination clusters (#13951)	2 years ago
acpana	eae4e71492	sync more acl enforcement ... sync w ent at 32756f7 Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
alex	41f3343eac	Merge pull request #13929 from hashicorp/fix-validation ... [sync] fix empty partitions matching	2 years ago
Sarah Pratt	a3ef6f016e	refactor sidecare_service method into parts	2 years ago
Ashwin Venkatesh	eef9edaed9	Add peer counts to emitted metrics. (#13930)	2 years ago
Luke Kysow	465a9801e1	Merge pull request #13924 from hashicorp/lkysow/util-metric-peering ... peering: don't track imported services/nodes in usage	2 years ago
acpana	6033584349	use EqualPartitions ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
acpana	0351ca5136	better fix ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
acpana	8b2ef80336	sync w ent ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
Chris S. Kim	0999e05a7d	Reduce arm64 flakes for TestConnectCA_ConfigurationSet_ChangeKeyConfig_Primary ... There were 16 combinations of tests but 4 of them were duplicates since the default key type and bits were "ec" and 256. That entry was commented out to reduce the subtest count to 12. testrpc.WaitForLeader was failing on arm64 environments; the cause is unknown but it might be due to the environment being flooded with parallel tests making RPC calls. The RPC polling+retry was replaced with a simpler check for leadership based on raft.	2 years ago
Chris S. Kim	8ead1caf53	Retry checks for virtual IP metadata	2 years ago
Chris S. Kim	62ed0250c3	Sort slice of ServiceNames deterministically	2 years ago
Sarah Pratt	f520f6dd0f	Separate port and socket path requirement in case of local agent assignment	2 years ago
Luke Kysow	740d54e730	peering: don't track imported services/nodes in usage ... Services/nodes that are imported from other peers are stored in state. We don't want to count those as part of our own cluster's usage.	2 years ago
cskh	4e292b7b72	chore: clarify the error message: service.service must not be empty (#13907) ... - when register service using catalog endpoint, the key of service name actually should be "service". Add this information to the error message will help user to quickly fix in the request.	2 years ago
cskh	59e81a728e	chore: removed unused method AddService (#13905) ... - This AddService is not used anywhere. AddServiceWithChecks is place of AddService - Test code is updated	2 years ago
Luke Kysow	021b00e321	Remove duplicate comment	2 years ago
alex	437a28d18a	peering: prevent peering in same partition (#13851) ... Co-authored-by: Chris S. Kim <ckim@hashicorp.com>	2 years ago
Nitya Dhanushkodi	27bd895ac8	peering: remove validation that forces peering token server addresses to be an IP, allow hostname based addresses (#13874)	2 years ago
Luke Kysow	8c5b70d227	Rename receive to recv in tracker (#13896) ... Because it's shorter	2 years ago
Luke Kysow	3530d3782d	peering: read endpoints can now return failing status (#13849) ... Track streams that have been disconnected due to an error and set their statuses to failing.	2 years ago
Kyle Havlovitz	93de25f87c	Merge pull request #13872 from hashicorp/remove-upstream-log ... Remove extra logging from ingress upstream watch shutdown	2 years ago
Chris S. Kim	73a84f256f	Preserve PeeringState on upsert (#13666) ... Fixes a bug where if the generate token is called twice, the second call upserts the zero-value (undefined) of PeeringState.	2 years ago
Chris S. Kim	8ed49ea4d0	Update envoy metrics label extraction for peered clusters and listeners (#13818) ... Now that peered upstreams can generate envoy resources (#13758), we need a way to disambiguate local from peered resources in our metrics. The key difference is that datacenter and partition will be replaced with peer, since in the context of peered resources partition is ambiguous (could refer to the partition in a remote cluster or one that exists locally). The partition and datacenter of the proxy will always be that of the source service. Regexes were updated to make emitting datacenter and partition labels mutually exclusive with peer labels. Listener filter names were updated to better match the existing regex. Cluster names assigned to peered upstreams were updated to be synthesized from local peer name (it previously used the externally provided primary SNI, which contained the peer name from the other side of the peering). Integration tests were updated to assert for the new peer labels.	2 years ago
DanStough	2da8949d78	feat: convert destination address to slice	2 years ago
Freddy	f03cca7576	[OSS] Add ACL enforcement to peering endpoints (#13878)	2 years ago
Matt Keeler	58e4d8235b	Enable/Disable Peering Support in the UI (#13816) ... We enabled/disable based on the config flag.	2 years ago
freddygv	b544ce6485	Add ACL enforcement to peering endpoints	2 years ago
Kyle Havlovitz	016f963e7e	Remove excess debug log from ingress upstream shutdown	2 years ago
alex	279d458e6e	peering: use ShouldDial to validate peer role (#13823) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
Luke Kysow	a1e6d69454	peering: add config to enable/disable peering (#13867) ... * peering: add config to enable/disable peering Add config: ``` peering { enabled = true } ``` Defaults to true. When disabled: 1. All peering RPC endpoints will return an error 2. Leader won't start its peering establishment goroutines 3. Leader won't start its peering deletion goroutines	2 years ago
Kyle Havlovitz	0786517b56	Merge pull request #13847 from hashicorp/gateway-goroutine-leak ... Fix goroutine leaks in proxycfg when using ingress gateway	2 years ago
Freddy	f99df57840	[OSS] Add new peering ACL rule (#13848) ... This commit adds a new ACL rule named "peering" to authorize actions taken against peering-related endpoints. The "peering" rule has several key properties: - It is scoped to a partition, and MUST be defined in the default namespace. - Its access level must be "read', "write", or "deny". - Granting an access level will apply to all peerings. This ACL rule cannot be used to selective grant access to some peerings but not others. - If the peering rule is not specified, we fall back to the "operator" rule and then the default ACL rule.	2 years ago
alex	927cee692b	peering: emit exported services count metric (#13811) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
Daniel Upton	a8df87f574	proxycfg-glue: server-local implementation of `ExportedPeeredServices` ... This is the OSS portion of enterprise PR 2377. Adds a server-local implementation of the proxycfg.ExportedPeeredServices interface that sources data from a blocking query against the server's state store.	2 years ago
Eric Haberkorn	501089292e	Add Cluster Peering Failover Support to Prepared Queries (#13835) ... Add peering failover support to prepared queries	2 years ago
Nitya Dhanushkodi	f47319b7c6	update generate token endpoint to take external addresses (#13844) ... Update generate token endpoint (rpc, http, and api module) If ServerExternalAddresses are set, it will override any addresses gotten from the "consul" service, and be used in the token instead, and dialed by the dialer. This allows for setting up a load balancer for example, in front of the consul servers.	2 years ago
acpana	12b773ab02	Rename peering internal to ~ ... sync ENT to 5679392c81 Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
Luke Kysow	0c87be0845	peering: Add heartbeating to peering streams (#13806) ... * Add heartbeating to peering streams	2 years ago
Daniel Upton	3655802fdc	proxycfg-glue: server-local implementation of `PeeredUpstreams` ... This is the OSS portion of enterprise PR 2352. It adds a server-local implementation of the proxycfg.PeeredUpstreams interface based on a blocking query against the server's state store. It also fixes an omission in the Virtual IP freeing logic where we were never updating the max index (and therefore blocking queries against VirtualIPsForAllImportedServices would not return on service deletion).	2 years ago
Luke Kysow	c411e6b326	Add send mutex to protect against concurrent sends (#13805)	2 years ago
Kyle Havlovitz	0be7d923dc	Cancel upstream watches when the discovery chain has been removed	2 years ago
Kyle Havlovitz	31318d7049	Fix duplicate Notify calls for discovery chains in ingress gateways	2 years ago
Evan Culver	4116537b83	connect: Add support for Envoy 1.23, remove 1.19 (#13807)	2 years ago
Paul Glass	77afe0e76e	Extract AWS auth implementation out of Consul (#13760)	2 years ago
Chris S. Kim	495936300e	Make envoy resources for inferred peered upstreams (#13758) ... Peered upstreams has a separate loop in xds from discovery chain upstreams. This PR adds similar but slightly modified code to add filters for peered upstream listeners, clusters, and endpoints in the case of transparent proxy.	2 years ago
alex	de5a991d8c	peering: refactor reconcile, cleanup (#13795) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
Luke Kysow	e8d965e56f	peerstream: set keepalive enforcement to 15s (#13796) ... The client is set to send keepalive pings every 30s. The server keepalive enforcement must be set to a number less than that, otherwise it will disconnect clients for sending pings too often. MinTime governs the minimum amount of time between pings.	2 years ago
alex	a9ae2ff4fa	peering: track exported services (#13784) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
R.B. Boyer	cd513aeead	peerstream: require a resource subscription to receive updates of that type (#13767) ... This mimics xDS's discovery protocol where you must request a resource explicitly for the exporting side to send those events to you. As part of this I aligned the overall ResourceURL with the TypeURL that gets embedded into the encoded protobuf Any construct. The CheckServiceNodes is now wrapped in a better named "ExportedService" struct now.	2 years ago
R.B. Boyer	c737301093	peerstream: fix test assertions (#13780)	2 years ago
Luke Kysow	46381b1a7f	Add docs for peerStreamServer vs peeringServer. (#13781)	2 years ago
Luke Kysow	ca3d7c964c	peerstream: dialer should reconnect when stream closes (#13745) ... * peerstream: dialer should reconnect when stream closes If the stream is closed unexpectedly (i.e. when we haven't received a terminated message), the dialer should attempt to re-establish the stream. Previously, the `HandleStream` would return `nil` when the stream was closed. The caller then assumed the stream was terminated on purpose and so didn't reconnect when instead it was stopped unexpectedly and the dialer should have attempted to reconnect.	2 years ago
R.B. Boyer	bb4d4040fb	server: ensure peer replication can successfully use TLS over external gRPC (#13733) ... Ensure that the peer stream replication rpc can successfully be used with TLS activated. Also: - If key material is configured for the gRPC port but HTTPS is not enabled now TLS will still be activated for the gRPC port. - peerstream replication stream opened by the establishing-side will now ignore grpc.WithBlock so that TLS errors will bubble up instead of being awkwardly delayed or suppressed	2 years ago
alex	adb5ffa1a6	peering: track imported services (#13718)	2 years ago
Matt Keeler	257f88d4df	Use Node Name for peering healthSnapshot instead of ID (#13773) ... A Node ID is not a required field with Consul’s data model. Therefore we cannot reliably expect all uses to have it. However the node name is required and must be unique so its equally as good of a key for the internal healthSnapshot node tracking.	2 years ago
Matt Keeler	05b5e7e2ca	Enable partition support for peering establishment (#13772) ... Prior to this the dialing side of the peering would only ever work within the default partition. This commit allows properly parsing the partition field out of the API struct request body, query param and header.	2 years ago
Dan Stough	49f3dadb8f	feat: connect proxy xDS for destinations ... Signed-off-by: Dhia Ayachi <dhia@hashicorp.com>	2 years ago
Daniel Upton	3d74efa8ad	proxycfg-glue: server-local implementation of `FederationStateListMeshGateways` ... This is the OSS portion of enterprise PR 2265. This PR provides a server-local implementation of the proxycfg.FederationStateListMeshGateways interface based on blocking queries.	2 years ago
Daniel Upton	ccc672013e	proxycfg-glue: server-local implementation of `GatewayServices` ... This is the OSS portion of enterprise PR 2259. This PR provides a server-local implementation of the proxycfg.GatewayServices interface based on blocking queries.	2 years ago
Daniel Upton	15a319dbfe	proxycfg-glue: server-local implementation of `TrustBundle` and `TrustBundleList` ... This is the OSS portion of enterprise PR 2250. This PR provides server-local implementations of the proxycfg.TrustBundle and proxycfg.TrustBundleList interfaces, based on local blocking queries.	2 years ago
Daniel Upton	673d02d30f	proxycfg-glue: server-local implementation of the `Health` interface ... This is the OSS portion of enterprise PR 2249. This PR introduces an implementation of the proxycfg.Health interface based on a local materialized view of the health events. It reuses the view and request machinery from agent/rpcclient/health, which made it super straightforward.	2 years ago
Daniel Upton	3c533ceea8	proxycfg-glue: server-local implementation of `ServiceList` ... This is the OSS portion of enterprise PR 2242. This PR introduces a server-local implementation of the proxycfg.ServiceList interface, backed by streaming events and a local materializer.	2 years ago
Daniel Upton	fbf88d3b19	proxycfg-glue: server-local compiled discovery chain data source ... This is the OSS portion of enterprise PR 2236. Adds a local blocking query-based implementation of the proxycfg.CompiledDiscoveryChain interface.	2 years ago
Chris S. Kim	f56810132f	Check if an upstream is implicit from either intentions or peered services	2 years ago
Chris S. Kim	02cff2394d	Use new maps for proxycfg peered data	2 years ago
Chris S. Kim	7f32cba735	Add new watch.Map type to refactor proxycfg	2 years ago
Chris S. Kim	b4ffa9ae0c	Scrub VirtualIPs before exporting	2 years ago
Kyle Havlovitz	9097e2b0f0	Merge pull request #13699 from hashicorp/tgate-http2-upstream ... Respect http2 protocol for upstreams of terminating gateways	2 years ago
Dan Upton	b9e525d689	grpc: rename public/private directories to external/internal (#13721) ... Previously, public referred to gRPC services that are both exposed on the dedicated gRPC port and have their definitions in the proto-public directory (so were considered usable by 3rd parties). Whereas private referred to services on the multiplexed server port that are only usable by agents and other servers. Now, we're splitting these definitions, such that external/internal refers to the port and public/private refers to whether they can be used by 3rd parties. This is necessary because the peering replication API needs to be exposed on the dedicated port, but is not (yet) suitable for use by 3rd parties.	2 years ago
R.B. Boyer	30fffd0c90	peerstream: some cosmetic refactors to make this easier to follow (#13732) ... - Use some protobuf construction helper methods for brevity. - Rename a local variable to avoid later shadowing. - Rename the Nonce field to be more like xDS's naming. - Be more explicit about which PeerID fields are empty.	2 years ago
Kyle Havlovitz	7d0c692374	Use protocol from resolved config entry, not gateway service	2 years ago
Kyle Havlovitz	7162e3bde2	Enable http2 options for grpc protocol	2 years ago
R.B. Boyer	c5c216008d	peering: always send the mesh gateway SpiffeID even for tcp services (#13728) ... If someone were to switch a peer-exported service from L4 to L7 there would be a brief SAN validation hiccup as traffic shifted to the mesh gateway for termination. This PR sends the mesh gateway SpiffeID down all the time so the clients always expect a switch.	2 years ago
R.B. Boyer	f0e6e4e697	state: prohibit changing an exported tcp discovery chain in a way that would break SAN validation (#13727) ... For L4/tcp exported services the mesh gateways will not be terminating TLS. A caller in one peer will be directly establishing TLS connections to the ultimate exported service in the other peer. The caller will be doing SAN validation using the replicated SpiffeID values shipped from the exporting side. There are a class of discovery chain edits that could be done on the exporting side that would cause the introduction of a new SpiffeID value. In between the time of the config entry update on the exporting side and the importing side getting updated peer stream data requests to the exported service would fail due to SAN validation errors. This is unacceptable so instead prohibit the exporting peer from making changes that would break peering in this way.	2 years ago
R.B. Boyer	2317f37b4d	state: prohibit exported discovery chains to have cross-datacenter or cross-partition references (#13726) ... Because peerings are pairwise, between two tuples of (datacenter, partition) having any exported reference via a discovery chain that crosses out of the peered datacenter or partition will ultimately not be able to work for various reasons. The biggest one is that there is no way in the ultimate destination to configure an intention that can allow an external SpiffeID to access a service. This PR ensures that a user simply cannot do this, so they won't run into weird situations like this.	2 years ago
Chris S. Kim	a6634db4a5	Return error if ServerAddresses is empty (#13714)	2 years ago
Kyle Havlovitz	439eccdd80	Respect http2 protocol for upstreams of terminating gateways	2 years ago
R.B. Boyer	af04851637	peering: move peer replication to the external gRPC port (#13698) ... Peer replication is intended to be between separate Consul installs and effectively should be considered "external". This PR moves the peer stream replication bidirectional RPC endpoint to the external gRPC server and ensures that things continue to function.	2 years ago
R.B. Boyer	ea58f235f5	server: broadcast the public grpc port using lan serf and update the consul service in the catalog with the same data (#13687) ... Currently servers exchange information about their WAN serf port and RPC port with serf tags, so that they all learn of each other's addressing information. We intend to make larger use of the new public-facing gRPC port exposed on all of the servers, so this PR addresses that by passing around the gRPC port via serf tags and then ensuring the generated consul service in the catalog has metadata about that new port as well for ease of non-serf-based lookup.	2 years ago
Freddy	3542138e4d	Parse peer name for virtual IP DNS queries (#13602) ... This commit updates the DNS query locality parsing so that the virtual IP for an imported service can be queried. Note that: - Support for parsing a peer in other service discovery queries was not added. - Querying another datacenter for a virtual IP is not supported. This was technically allowed in 1.11 but is being rolled back for 1.13 because it is not a use-case we intended to support. Virtual IPs in different datacenters are going to collide because they are allocated sequentially.	2 years ago
R.B. Boyer	2a945facec	test: update mockery use to put mocks into test files (#13656) ... --testonly doesn't do anything anymore so switch to --filename instead	2 years ago
Chris S. Kim	f07132dacc	Revise possible states for a peering. (#13661) ... These changes are primarily for Consul's UI, where we want to be more specific about the state a peering is in. - The "initial" state was renamed to pending, and no longer applies to peerings being established from a peering token. - Upon request to establish a peering from a peering token, peerings will be set as "establishing". This will help distinguish between the two roles: the cluster that generates the peering token and the cluster that establishes the peering. - When marked for deletion, peering state will be set to "deleting". This way the UI determines the deletion via the state rather than the "DeletedAt" field. Co-authored-by: freddygv <freddy@hashicorp.com>	2 years ago
Daniel Upton	45886848b4	proxycfg: server-local intention upstreams data source ... This is the OSS portion of enterprise PR 2157. It builds on the local blocking query work in #13438 to implement the proxycfg.IntentionUpstreams interface using server-local data. Also moves the ACL filtering logic from agent/consul into the acl/filter package so that it can be reused here.	2 years ago
Daniel Upton	37ccbd2826	proxycfg: server-local intentions data source ... This is the OSS portion of enterprise PR 2141. This commit provides a server-local implementation of the `proxycfg.Intentions` interface that sources data from streaming events. It adds events for the `service-intentions` config entry type, and then consumes event streams (via materialized views) for the service's explicit intentions and any applicable wildcard intentions, merging them into a single list of intentions. An alternative approach I considered was to consume _all_ intention events (via `SubjectWildcard`) and filter out the irrelevant ones. This would admittedly remove some complexity in the `agent/proxycfg-glue` package but at the expense of considerable overhead from waking potentially many thousands of connect proxies every time any intention is updated.	2 years ago
Daniel Upton	653b8c4f9d	proxycfg: server-local config entry data sources ... This is the OSS portion of enterprise PR 2056. This commit provides server-local implementations of the proxycfg.ConfigEntry and proxycfg.ConfigEntryList interfaces, that source data from streaming events. It makes use of the LocalMaterializer type introduced for peering replication, adding the necessary support for authorization. It also adds support for "wildcard" subscriptions (within a topic) to the event publisher, as this is needed to fetch service-resolvers for all services when configuring mesh gateways. Currently, events will be emitted for just the ingress-gateway, service-resolver, and mesh config entry types, as these are the only entries required by proxycfg — the events will be emitted on topics named IngressGateway, ServiceResolver, and MeshConfig topics respectively. Though these events will only be consumed "locally" for now, they can also be consumed via the gRPC endpoint (confirmed using grpcurl) so using them from client agents should be a case of swapping the LocalMaterializer for an RPCMaterializer.	2 years ago
alex	cd9ca4290a	peering: add imported/exported counts to peering (#13644) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com> Co-authored-by: Chris S. Kim <ckim@hashicorp.com>	2 years ago
Chris S. Kim	b186731a2e	Fix ENT drift in files (#13647)	2 years ago
Chris S. Kim	d8b7940e40	Add internal endpoint to fetch peered upstream candidates from VirtualIP table (#13642) ... For initial cluster peering TProxy support we consider all imported services of a partition to be potential upstreams. We leverage the VirtualIP table because it stores plain service names (e.g. "api", not "api-sidecar-proxy").	2 years ago
Eric Haberkorn	653cb42944	Fix spelling mistake in serverless patcher (#13607) ... passhthrough -> passthrough	2 years ago
alex	07bc22e405	no 1.9 style metrics (#13532) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
alex	beb8b03e8a	peering: reconcile/ hint active state for list (#13619) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
R.B. Boyer	31b95c747b	xds: modify rbac rules to use the XFCC header for peered L7 enforcement (#13629) ... When the protocol is http-like, and an intention has a peered source then the normal RBAC mTLS SAN field check is replaces with a joint combo of: mTLS SAN field must be the service's local mesh gateway leaf cert AND the first XFCC header (from the MGW) must have a URI field that matches the original intention source Also: - Update the regex program limit to be much higher than the teeny defaults, since the RBAC regex constructions are more complicated now. - Fix a few stray panics in xds generation.	2 years ago
R.B. Boyer	de0f9ac519	xds: have mesh gateways forward peered SpiffeIDs using the XFCC header (#13625)	2 years ago
R.B. Boyer	1a9c86ea8f	xds: mesh gateways now correctly load up peer-exported discovery chains using L7 protocols (#13624) ... A mesh gateway will now configure the filter chains for L7 exported services using the correct discovery chain information.	2 years ago
R.B. Boyer	0fa828db76	peering: replicate all SpiffeID values necessary for the importing side to do SAN validation (#13612) ... When traversing an exported peered service, the discovery chain evaluation at the other side may re-route the request to a variety of endpoints. Furthermore we intend to terminate mTLS at the mesh gateway for arriving peered traffic that is http-like (L7), so the caller needs to know the mesh gateway's SpiffeID in that case as well. The following new SpiffeID values will be shipped back in the peerstream replication: - tcp: all possible SpiffeIDs resulting from the service-resolver component of the exported discovery chain - http-like: the SpiffeID of the mesh gateway	2 years ago
Max Bowsher	ef4b9e541f	Merge branch 'main' into fix-kv_entries-metric	2 years ago
alex	53f0cf5835	peering, internal: support UIServices, UINodes, UINodeInfo (#13577)	2 years ago
Chris S. Kim	2e4cb6f77d	Add new index for PeeredServiceName and ServiceVirtualIP (#13582) ... For TProxy we will be leveraging the VirtualIP table, which needs to become peer-aware	2 years ago
alex	20ecf0febd	Merge pull request #13570 from hashicorp/acpance/peering-oss-intentions ... oss: peering, http: get peer service intentions (#2098)	2 years ago
Will Jordan	34ecbc1d71	Add per-node max indexes (#12399) ... Adds fine-grained node.[node] entries to the index table, allowing blocking queries to return fine-grained indexes that prevent them from returning immediately when unrelated nodes/services are updated. Co-authored-by: kisunji <ckim@hashicorp.com>	2 years ago
Chris S. Kim	ba89a7d9b0	Make memdb indexers generic (#13558) ... We have many indexer functions in Consul which take interface{} and type assert before building the index. We can use generics to get rid of the initial plumbing and pass around functions with better defined signatures. This has two benefits: 1) Less verbosity; 2) Developers can parse the argument types to memdb schemas without having to introspect the function for the type assertion.	2 years ago
Matt Keeler	7a4d13b0b2	Port over the index 0 -> 1 code that lived in the old rpc setQueryMeta function. (#13561)	2 years ago
acpana	99c2e11328	oss: peering, http: get peer service intentions (#2098) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
R.B. Boyer	e8ea3d7c3b	state: peering ID assignment cannot happen inside of the state store (#13525) ... Move peering ID assignment outisde of the FSM, so that the ID is written to the raft log and the same ID is used by all voters, and after restarts.	2 years ago
Matt Keeler	cb01702cd2	Add server local blocking queries and watches (#13438) ... Co-authored-by: Dan Upton <daniel@floppy.co>	2 years ago
Chris S. Kim	fb5eb20563	Pass trust domain to RBAC to validate and fix use of wrong peer trust bundles (#13508)	2 years ago
Max Bowsher	7b97b8abd2	Delete definition of metric `consul.acl.blocked.node.registration` ... Although the metric is defined, there is no code which ever sets its value - the code in question is genuinely asymmetric - there are 3 types of object for which registration can be tracked, but only 2 for which deregistration can be tracked.	2 years ago
Max Bowsher	7c19c701e1	Fix incorrect name and doc for kv_entries metric ... The name of the metric as registered with the metrics library to provide the help string, was incorrect compared with the actual code that sets the metric value - bring them into sync. Also, the help message was incorrect. Rather than copy the help message from telemetry.mdx, which was correct, but felt a bit unnatural in the way it was worded, update both of them to a new wording.	2 years ago
Dan Upton	e00e3a0bc3	Move ACLResolveResult into acl/resolver package (#13467) ... Having this type live in the agent/consul package makes it difficult to put anything that relies on token resolution (e.g. the new gRPC services) in separate packages without introducing import cycles. For example, if package foo imports agent/consul for the ACLResolveResult type it means that agent/consul cannot import foo to register its service. We've previously worked around this by wrapping the ACLResolver to "downgrade" its return type to an acl.Authorizer - aside from the added complexity, this also loses the resolved identity information. In the future, we may want to move the whole ACLResolver into the acl/resolver package. For now, putting the result type there at least, fixes the immediate import cycle issues.	2 years ago
DanStough	4b402e3119	feat: tgtwy xDS generation for destinations ... Signed-off-by: Dhia Ayachi <dhia@hashicorp.com>	2 years ago
alex	bd4ddb3720	peering: block Intention.Apply ops (#13451) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
alex	b3e99784a6	peering, state: account for peer intentions (#13443) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
R.B. Boyer	da8cea58c9	xds: begin refactor to always pass test snapshots through all xDS types (#13461)	2 years ago
R.B. Boyer	201d1458c3	xds: mesh gateways now have their own leaf certificate when involved in a peering (#13460) ... This is only configured in xDS when a service with an L7 protocol is exported. They also load any relevant trust bundles for the peered services to eventually use for L7 SPIFFE validation during mTLS termination.	2 years ago
Riddhi Shah	411edc876b	[OSS] Support merge-central-config option in node services list API (#13450) ... Adds the merge-central-config query param option to the /catalog/node-services/:node-name API, to get a service definition in the response that is merged with central defaults (proxy-defaults/service-defaults). Updated the consul connect envoy command to use this option when retrieving the proxy service details so as to render the bootstrap configuration correctly.	2 years ago
Evan Culver	7f8c650d61	connect: Use Envoy 1.22.2 instead of 1.22.1 (#13444)	2 years ago
freddygv	f3843809da	Avoid deleting peerings marked as terminated. ... When our peer deletes the peering it is locally marked as terminated. This termination should kick off deleting all imported data, but should not delete the peering object itself. Keeping peerings marked as terminated acts as a signal that the action took place.	2 years ago
freddygv	6453375ab2	Add leader routine to clean up peerings ... Once a peering is marked for deletion a new leader routine will now clean up all imported resources and then the peering itself. A lot of the logic was grabbed from the namespace/partitions deferred deletions but with a handful of simplifications: - The rate limiting is not configurable. - Deleting imported nodes/services/checks is done by deleting nodes with the Txn API. The services and checks are deleted as a side-effect. - There is no "round rate limiter" like with namespaces and partitions. This is because peerings are purely local, and deleting a peering in the datacenter does not depend on deleting data from other DCs like with WAN-federated namespaces. All rate limiting is handled by the Raft rate limiter.	2 years ago
Evan Culver	ba6136eb42	connect: Update Envoy support matrix to latest patch releases (#13431)	2 years ago
alex	a0a49ce2a6	peering: intentions list test (#13435)	2 years ago