250 Commits (acc5fd5c0ad01f5a54d079055e95c1ad355df51d)

Author	SHA1	Message	Date
freddygv	659ebc05a9	Ensure passthrough addresses get cleaned up ... Transparent proxies can set up filter chains that allow direct connections to upstream service instances. Services that can be dialed directly are stored in the PassthroughUpstreams map of the proxycfg snapshot. Previously these addresses were not being cleaned up based on new service health data. The list of addresses associated with an upstream service would only ever grow. As services scale up and down, eventually they will have instances assigned to an IP that was previously assigned to a different service. When IP addresses are duplicated across filter chain match rules the listener config will be rejected by Envoy. This commit updates the proxycfg snapshot management so that passthrough addresses can get cleaned up when no longer associated with a given upstream. There is still the possibility of a race condition here where due to timing an address is shared between multiple passthrough upstreams. That concern is mitigated by #12195, but will be further addressed in a follow-up.	3 years ago
freddygv	c31c1158a6	Add failing test ... The updated test fails because passthrough upstream addresses are not being cleaned up.	3 years ago
R.B. Boyer	b60d89e7ef	bulk rewrite using this script ... set -euo pipefail unset CDPATH cd "$(dirname "$0")" for f in $(git grep '\brequire := require\.New(' | cut -d':' -f1 | sort -u); do echo "=== require: $f ===" sed -i '/require := require.New(t)/d' $f # require.XXX(blah) but not require.XXX(tblah) or require.XXX(rblah) sed -i 's/\brequire\.\([a-zA-Z0-9_]*\)(\([^tr]\)/require.\1(t,\2/g' $f # require.XXX(tblah) but not require.XXX(t, blah) sed -i 's/\brequire\.\([a-zA-Z0-9_]*\)(\(t[^,]\)/require.\1(t,\2/g' $f # require.XXX(rblah) but not require.XXX(r, blah) sed -i 's/\brequire\.\([a-zA-Z0-9_]*\)(\(r[^,]\)/require.\1(t,\2/g' $f gofmt -s -w $f done for f in $(git grep '\bassert := assert\.New(' | cut -d':' -f1 | sort -u); do echo "=== assert: $f ===" sed -i '/assert := assert.New(t)/d' $f # assert.XXX(blah) but not assert.XXX(tblah) or assert.XXX(rblah) sed -i 's/\bassert\.\([a-zA-Z0-9_]*\)(\([^tr]\)/assert.\1(t,\2/g' $f # assert.XXX(tblah) but not assert.XXX(t, blah) sed -i 's/\bassert\.\([a-zA-Z0-9_]*\)(\(t[^,]\)/assert.\1(t,\2/g' $f # assert.XXX(rblah) but not assert.XXX(r, blah) sed -i 's/\bassert\.\([a-zA-Z0-9_]*\)(\(r[^,]\)/assert.\1(t,\2/g' $f gofmt -s -w $f done	3 years ago
R.B. Boyer	424f3cdd2c	proxycfg: introduce explicit UpstreamID in lieu of bare string (#12125) ... The gist here is that now we use a value-type struct proxycfg.UpstreamID as the map key in ConfigSnapshot maps where we used to use "upstream id-ish" strings. These are internal only and used just for bidirectional trips through the agent cache keyspace (like the discovery chain target struct). For the few places where the upstream id needs to be projected into xDS, that's what (proxycfg.UpstreamID).EnvoyID() is for. This lets us ALWAYS inject the partition and namespace into these things without making stuff like the golden testdata diverge.	3 years ago
Dhia Ayachi	e653f81919	reset `coalesceTimer` to nil as soon as the event is consumed (#11924) ... * reset `coalesceTimer` to nil as soon as the event is consumed * add change log * refactor to add relevant test. * fix linter * Apply suggestions from code review Co-authored-by: Freddy <freddygv@users.noreply.github.com> * remove non needed check Co-authored-by: Freddy <freddygv@users.noreply.github.com>	3 years ago
freddygv	21f2c2e68d	Purge chain if it shouldn't be there	3 years ago
freddygv	d26b4860fd	Account for new upstreams constraint in tests	3 years ago
freddygv	2fe27b748d	Check ingress upstreams when gating chain watches	3 years ago
freddygv	6af9a0d8cf	Avoid storing chain without an upstream	3 years ago
freddygv	ba12dc215b	Clean up chains separately from their watches	3 years ago
freddygv	70d6358426	Store intention upstreams in snapshot	3 years ago
R.B. Boyer	81ea8129d7	proxycfg: ensure all of the watches are canceled if they are cancelable (#11824)	3 years ago
R.B. Boyer	4aabbe529c	proxycfg: use external addresses in tproxy when crossing partition boundaries (#11823)	3 years ago
R.B. Boyer	631c649291	various partition related todos (#11822)	3 years ago
R.B. Boyer	1e02460bd1	re-run gofmt on 1.17 (#11579) ... This should let freshly recompiled golangci-lint binaries using Go 1.17 pass 'make lint'	3 years ago
freddygv	0e507492d0	Update proxycfg for ingress service partitions	3 years ago
Freddy	00b5b0a0a2	Update filter chain creation for sidecar/ingress listeners (#11245) ... The duo of `makeUpstreamFilterChainForDiscoveryChain` and `makeListenerForDiscoveryChain` were really hard to reason about, and led to concealing a bug in their branching logic. There were several issues here: - They tried to accomplish too much: determining filter name, cluster name, and whether RDS should be used. - They embedded logic to handle significantly different kinds of upstream listeners (passthrough, prepared query, typical services, and catch-all) - They needed to coalesce different data sources (Upstream and CompiledDiscoveryChain) Rather than handling all of those tasks inside of these functions, this PR pulls out the RDS/clusterName/filterName logic. This refactor also fixed a bug with the handling of [UpstreamDefaults](https://www.consul.io/docs/connect/config-entries/service-defaults#defaults). These defaults get stored as UpstreamConfig in the proxy snapshot with a DestinationName of "*", since they apply to all upstreams. However, this wildcard destination name must not be used when creating the name of the associated upstream cluster. The coalescing logic in the original functions here was in some situations creating clusters with a `*.` prefix, which is not a valid destination.	3 years ago
Daniel Upton	50a1f20ff9	xds: prefer fed state gateway definitions if they're fresher (#11522) ... Fixes an issue described in #10132, where if two DCs are WAN federated over mesh gateways, and the gateway in the non-primary DC is terminated and receives a new IP address (as is commonly the case when running them on ephemeral compute instances) the primary DC is unable to re-establish its connection until the agent running on its own gateway is restarted. This was happening because we always preferred gateways discovered by the `Internal.ServiceDump` RPC (which would fail because there's no way to dial the remote DC) over those discovered in the federation state, which is replicated as long as the primary DC's gateway is reachable.	3 years ago
freddygv	60066e5154	Exclude default partition from GatewayKey string ... This will behave the way we handle SNI and SPIFFE IDs, where the default partition is excluded. Excluding the default ensures that don't attempt to compare default.dc2 to dc2 in OSS.	3 years ago
freddygv	e3666b0bc4	Update GatewayKeys deduplication ... Federation states data is only keyed on datacenter, so it cannot be directly compared against keys for gateway groups.	3 years ago
freddygv	90ce897456	Store GatewayKey in proxycfg snapshot for re-use	3 years ago
freddygv	4d4ccedb3a	Update locality check in proxycfg	3 years ago
freddygv	3a2061544d	Fixup partitions assertion	3 years ago
freddygv	d28b9052b2	Move the exportingpartitions constant to enterprise	3 years ago
freddygv	448701dbd8	Replace default partition check	3 years ago
freddygv	12923f5ebc	PR comments	3 years ago
freddygv	a33b6923e0	Account for partitions in xds gen for mesh gw ... This commit avoids skipping gateways in remote partitions of the local DC when generating listeners/clusters/endpoints.	3 years ago
freddygv	110fae820a	Update xds pkg to account for GatewayKey	3 years ago
freddygv	7e65678c52	Update mesh gateway proxy watches for partitions ... This commit updates mesh gateway watches for cross-partitions communication. * Mesh gateways are keyed by partition and datacenter. * Mesh gateways will now watch gateways in partitions that export services to their partition. * Mesh gateways in non-default partitions will not have cross-datacenter watches. They are not involved in traditional WAN federation.	3 years ago
freddygv	37a16e9487	Replace Split with SplitN	3 years ago
freddygv	b9b6447977	Finish removing useInDatacenter	3 years ago
freddygv	62e0fc62c1	Configure sidecars to watch gateways in partitions ... Previously the datacenter of the gateway was the key identifier, now it is the datacenter and partition. When dialing services in other partitions or datacenters we now watch the appropriate partition.	3 years ago
Paul Banks	78a00f2e1c	Add support for enabling connect-based ingress TLS per listener.	3 years ago
Daniel Nephin	eb632c53a2	structs: rename the last helper method. ... This one gets used a bunch, but we can rename it to make the behaviour more obvious.	3 years ago
Daniel Nephin	6d72517682	structs: remove two methods that were only used once each. ... These methods only called a single function. Wrappers like this end up making code harder to read because it adds extra ways of doing things. We already have many helper functions for constructing these types, we don't need additional methods.	3 years ago
Paul Banks	136928a90f	Minor PR typo and cleanup fixes	3 years ago
Paul Banks	20d0bf81f7	Revert abandonned changes to proxycfg for Ent test consistency	3 years ago
Paul Banks	659321d008	Handle namespaces in route names correctly; add tests for enterprise	3 years ago
Paul Banks	ccbda0c285	Update proxycfg to hold more ingress config state	3 years ago
Paul Banks	4e39f03d5b	Add ingress-gateway config for SDS	3 years ago
freddygv	49248a0802	Fixup proxycfg tproxy case	3 years ago
freddygv	95a6db9cfa	Account for partitions in ixn match/decision	3 years ago
freddygv	3f3a61c6e1	Fixup manager tests	3 years ago
freddygv	77681b9f6c	Pass partition to intention match query	3 years ago
Paul Banks	e22cc9c53a	Header manip for split legs plumbing	3 years ago
Paul Banks	83fc8723a3	Header manip for service-router plumbed through	3 years ago
Paul Banks	f439dfc04f	Ingress gateway header manip plumbing	3 years ago
Dhia Ayachi	bc0e4f2f46	partition dicovery chains (#10983) ... * partition dicovery chains * fix default partition for OSS	3 years ago
Dhia Ayachi	09197c989c	add partition to SNI when partition is non default (#10917)	3 years ago
freddygv	f52bd80f6d	Update comment for test function	3 years ago
freddygv	af52d21884	Update prepared query cluster SAN validation ... Previously SAN validation for prepared queries was broken because we validated against the name, namespace, and datacenter for prepared queries. However, prepared queries can target: - Services with a name that isn't their own - Services in multiple datacenters This means that the SpiffeID to validate needs to be based on the prepared query endpoints, and not the prepared query's upstream definition. This commit updates prepared query clusters to account for that.	3 years ago
freddygv	85878685b7	Fixup proxy config test fixtures ... - The TestNodeService helper created services with the fixed name "web", and now that name is overridable. - The discovery chain snapshot didn't have prepared query endpoints so the endpoints tests were missing data for prepared queries	3 years ago
Dhia Ayachi	1950ebbe1f	oss portion of ent #1069 (#10883)	3 years ago
R.B. Boyer	097e1645e3	agent: ensure that most agent behavior correctly respects partition configuration (#10880)	3 years ago
Daniel Nephin	0575498d0d	proxycfg: Lookup the agent token as a default ... When no ACL token is provided with the service registration.	3 years ago
Daniel Nephin	b313f495b8	proxycfg: Add a test to show the bug ... When a token is not provided at registration, the agent token is not being used.	3 years ago
Freddy	19f6e1ca31	Log the correlation ID when blocking queries fire (#10689) ... Knowing that blocking queries are firing does not provide much information on its own. If we know the correlation IDs we can piece together which parts of the snapshot have been populated. Some of these responses might be empty from the blocking query timing out. But if they're returning quickly I think we can reasonably assume they contain data.	3 years ago
R.B. Boyer	188e8dc51f	agent/structs: add a bunch more EnterpriseMeta helper functions to help with partitioning (#10669)	3 years ago
freddygv	b4c5c58c9b	Add TODOs about partition handling	3 years ago
freddygv	47da00d3c7	Validate SANs for passthrough clusters and failovers	3 years ago
Daniel Nephin	10051cf6d3	proxycfg: remove unused method ... This method was accidentally re-introduced in an earlier rebase. It was removed in ed1082510d as part of the tproxy work.	3 years ago
Daniel Nephin	6bc5255028	proxycfg: move each handler into a seprate file ... There is no interaction between these handlers, so splitting them into separate files makes it easier to discover the full implementation of each kindHandler.	3 years ago
Daniel Nephin	19d3eeff3c	Merge pull request #9489 from hashicorp/dnephin/proxycfg-state-2 ... proxycfg: split state into a handler for each kind	4 years ago
Nitya Dhanushkodi	52043830b4	proxycfg: reference to entry in map should not panic	4 years ago
Daniel Nephin	e738fa3b80	Replace type conversion with embedded structs	4 years ago
Daniel Nephin	32c15d9a88	proxycfg: split state into kind-specific types ... This commit extracts all the kind-specific logic into handler types, and keeps the generic parts on the state struct. This change should make it easier to add new kinds, and see the implementation of each kind more clearly.	4 years ago
Daniel Nephin	cd05df7157	proxycfg: unmethod hostnameEndpoints ... the method receiver can be replaced by the first argument. This will allow us to extract more from the state struct in the future.	4 years ago
Daniel Nephin	97c6ee00d7	Remove duplicate import ... because two PRs crossed paths.	4 years ago
Daniel Nephin	0547d0c046	Merge pull request #9466 from hashicorp/dnephin/proxycfg-state ... proxycfg: prepare state for split by kind	4 years ago
Nitya Dhanushkodi	b8b44419a0	proxycfg: Ensure that endpoints for explicit upstreams in other datacenters are watched in transparent mode (#10391) ... Co-authored-by: Freddy Vallenilla <freddy@hashicorp.com>	4 years ago
Daniel Nephin	016c5611d1	proxycfg: extract two types from state struct ... These two new struct types will allow us to make polymorphic handler for each kind, instad of having all the logic for each proxy kind on the state struct.	4 years ago
Daniel Nephin	9c40aa729f	proxycfg: pass context around where it is needed ... context.Context should never be stored on a struct (as it says in the godoc) because it is easy to to end up with the wrong context when it is stored. Also see https://blog.golang.org/context-and-structs This change is also in preparation for splitting state into kind-specific handlers so that the implementation of each kind is grouped together.	4 years ago
Freddy	429f9d8bb8	Add flag for transparent proxies to dial individual instances (#10329)	4 years ago
freddygv	c73703c08b	Ensure entmeta is encoded in test correlationID	4 years ago
Daniel Nephin	347f3d2128	Merge pull request #10155 from hashicorp/dnephin/config-entry-remove-fields ... config-entry: remove Kind and Name field from Mesh config entry	4 years ago
Mark Anderson	6be9cebad0	Add tests for xds/listeners ... Signed-off-by: Mark Anderson <manderson@hashicorp.com>	4 years ago
Mark Anderson	06f0f79218	Continue working through proxy and agent ... Rework/listeners, rename makeListener Refactor, tests pass Signed-off-by: Mark Anderson <manderson@hashicorp.com>	4 years ago
Freddy	ed1082510d	Fixup discovery chain handling in transparent mode (#10168) ... Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com> Previously we would associate the address of a discovery chain target with the discovery chain's filter chain. This was broken for a few reasons: - If the upstream is a virtual service, the client proxy has no way of dialing it because virtual services are not targets of their discovery chains. The targets are distinct services. This is addressed by watching the endpoints of all upstream services, not just their discovery chain targets. - If multiple discovery chains resolve to the same target, that would lead to multiple filter chains attempting to match on the target's virtual IP. This is addressed by only matching on the upstream's virtual IP. NOTE: this implementation requires an intention to the redirecting virtual service and not just to the final destination. This is how we can know that the virtual service is an upstream to watch. A later PR will look into traversing discovery chains when computing upstreams so that intentions are only required to the discovery chain targets.	4 years ago
Daniel Nephin	62efaaab21	config-entry: remove Kind and Name field from Mesh config entry ... No config entry needs a Kind field. It is only used to determine the Go type to target. As we introduce new config entries (like this one) we can remove the kind field and have the GetKind method return the single supported value. In this case (similar to proxy-defaults) the Name field is also unnecessary. We always use the same value. So we can omit the name field entirely.	4 years ago
R.B. Boyer	71d45a3460	Support Incremental xDS mode (#9855) ... This adds support for the Incremental xDS protocol when using xDS v3. This is best reviewed commit-by-commit and will not be squashed when merged. Union of all commit messages follows to give an overarching summary: xds: exclusively support incremental xDS when using xDS v3 Attempts to use SoTW via v3 will fail, much like attempts to use incremental via v2 will fail. Work around a strange older envoy behavior involving empty CDS responses over incremental xDS. xds: various cleanups and refactors that don't strictly concern the addition of incremental xDS support Dissolve the connectionInfo struct in favor of per-connection ResourceGenerators instead. Do a better job of ensuring the xds code uses a well configured logger that accurately describes the connected client. xds: pull out checkStreamACLs method in advance of a later commit xds: rewrite SoTW xDS protocol tests to use protobufs rather than hand-rolled json strings In the test we very lightly reuse some of the more boring protobuf construction helper code that is also technically under test. The important thing of the protocol tests is testing the protocol. The actual inputs and outputs are largely already handled by the xds golden output tests now so these protocol tests don't have to do double-duty. This also updates the SoTW protocol test to exclusively use xDS v2 which is the only variant of SoTW that will be supported in Consul 1.10. xds: default xds.Server.AuthCheckFrequency at use-time instead of construction-time	4 years ago
Freddy	078c40425f	Rename "cluster" config entry to "mesh" (#10127) ... This config entry is being renamed primarily because in k8s the name cluster could be confusing given that the config entry applies across federated datacenters. Additionally, this config entry will only apply to Consul as a service mesh, so the more generic "cluster" name is not needed.	4 years ago
Daniel Nephin	2a26085b2c	connect: do not set QuerySource.Node ... Setting this field to a value is equivalent to using the 'near' query paramter. The intent is to sort the results by proximity to the node requesting them. However with connect we send the results to envoy, which doesn't care about the order, so setting this field is increasing the work performed for no gain. It is necessary to unset this field now because we would like connect to use streaming, but streaming does not support sorting by proximity.	4 years ago
Freddy	439a7fce2d	Split Upstream.Identifier() so non-empty namespace is always prepended in ent (#10031)	4 years ago
freddygv	8857195437	Fixup wildcard ent assertion	4 years ago
freddygv	7bd51ff536	Replace TransparentProxy bool with ProxyMode ... This PR replaces the original boolean used to configure transparent proxy mode. It was replaced with a string mode that can be set to: - "": Empty string is the default for when the setting should be defaulted from other configuration like config entries. - "direct": Direct mode is how applications originally opted into the mesh. Proxy listeners need to be dialed directly. - "transparent": Transparent mode enables configuring Envoy as a transparent proxy. Traffic must be captured and redirected to the inbound and outbound listeners. This PR also adds a struct for transparent proxy specific configuration. Initially this is not stored as a pointer. Will revisit that decision before GA.	4 years ago
freddygv	b21224a4c8	PR comments	4 years ago
freddygv	49a4a78fd5	Ensure mesh gateway mode override is set for upstreams for intentions	4 years ago
freddygv	5140c3e51f	Finish resolving upstream defaults in proxycfg	4 years ago
R.B. Boyer	499fee73b3	connect: add toggle to globally disable wildcard outbound network access when transparent proxy is enabled (#9973) ... This adds a new config entry kind "cluster" with a single special name "cluster" where this can be controlled.	4 years ago
freddygv	098b9af901	Fixup enterprise tests from tproxy changes	4 years ago
freddygv	eb1e0a1751	Cancel watch on all errors	4 years ago
freddygv	f4f45af6d0	Merge master and fix upstream config protocol defaulting	4 years ago
freddygv	0da8702f34	PR comments	4 years ago
freddygv	a54d6a9010	Update proxycfg for transparent proxy	4 years ago
Daniel Nephin	f40b76af2d	proxycfg: use rpcclient/health.Client instead of passing around cache name ... This should allow us to swap out the implementation with something other than `agent/cache` without making further code changes.	4 years ago
Daniel Nephin	906834ce8e	proxycfg: Use streaming in connect state	4 years ago
Freddy	82c269a7c5	Avoid potential proxycfg/xDS deadlock using non-blocking send	4 years ago
freddygv	ec5f75776b	Update comments on avoiding proxycfg deadlock	4 years ago
R.B. Boyer	43193a35c6	xds: prevent LDS flaps in mesh gateways due to unstable datacenter lists (#9651) ... Also fix a similar issue in Terminating Gateways that was masked by an overzealous test.	4 years ago
freddygv	6e443e5536	Retry send after timer fires, in case no updates occur	4 years ago