115 Commits (f01f00fc84892272d9a9dc37ee2b972e4fdc711a)

Author	SHA1	Message	Date
freddygv	cbea3d203c	Fix race of upstreams with same passthrough ip ... Due to timing, a transparent proxy could have two upstreams to dial directly with the same address. For example: - The orders service can dial upstreams shipping and payment directly. - An instance of shipping at address 10.0.0.1 is deregistered. - Payments is scaled up and scheduled to have address 10.0.0.1. - The orders service receives the event for the new payments instance before seeing the deregistration for the shipping instance. At this point two upstreams have the same passthrough address and Envoy will reject the listener configuration. To disambiguate this commit considers the Raft index when storing passthrough addresses. In the example above, 10.0.0.1 would only be associated with the newer payments service instance.	3 years ago
freddygv	659ebc05a9	Ensure passthrough addresses get cleaned up ... Transparent proxies can set up filter chains that allow direct connections to upstream service instances. Services that can be dialed directly are stored in the PassthroughUpstreams map of the proxycfg snapshot. Previously these addresses were not being cleaned up based on new service health data. The list of addresses associated with an upstream service would only ever grow. As services scale up and down, eventually they will have instances assigned to an IP that was previously assigned to a different service. When IP addresses are duplicated across filter chain match rules the listener config will be rejected by Envoy. This commit updates the proxycfg snapshot management so that passthrough addresses can get cleaned up when no longer associated with a given upstream. There is still the possibility of a race condition here where due to timing an address is shared between multiple passthrough upstreams. That concern is mitigated by #12195, but will be further addressed in a follow-up.	3 years ago
R.B. Boyer	424f3cdd2c	proxycfg: introduce explicit UpstreamID in lieu of bare string (#12125) ... The gist here is that now we use a value-type struct proxycfg.UpstreamID as the map key in ConfigSnapshot maps where we used to use "upstream id-ish" strings. These are internal only and used just for bidirectional trips through the agent cache keyspace (like the discovery chain target struct). For the few places where the upstream id needs to be projected into xDS, that's what (proxycfg.UpstreamID).EnvoyID() is for. This lets us ALWAYS inject the partition and namespace into these things without making stuff like the golden testdata diverge.	3 years ago
Kyle Havlovitz	0db874c38b	Add virtual IP generation for term gateway backed services	3 years ago
Mike Morris	1b1a97e8f9	ingress: allow setting TLS min version and cipher suites in ingress gateway config entries (#11576) ... * xds: refactor ingress listener SDS configuration * xds: update resolveListenerSDS call args in listeners_test * ingress: add TLS min, max and cipher suites to GatewayTLSConfig * xds: implement envoyTLSVersions and envoyTLSCipherSuites * xds: merge TLS config * xds: configure TLS parameters with ingress TLS context from leaf * xds: nil check in resolveListenerTLSConfig validation * xds: nil check in makeTLSParameters* functions * changelog: add entry for TLS params on ingress config entries * xds: remove indirection for TLS params in TLSConfig structs * xds: return tlsContext, nil instead of ambiguous err Co-authored-by: Chris S. Kim <ckim@hashicorp.com> * xds: switch zero checks to types.TLSVersionUnspecified * ingress: add validation for ingress config entry TLS params * ingress: validate listener TLS config * xds: add basic ingress with TLS params tests * xds: add ingress listeners mixed TLS min version defaults precedence test * xds: add more explicit tests for ingress listeners inheriting gateway defaults * xds: add test for single TLS listener on gateway without TLS defaults * xds: regen golden files for TLSVersionInvalid zero value, add TLSVersionAuto listener test * types/tls: change TLSVersion to string * types/tls: update TLSCipherSuite to string type * types/tls: implement validation functions for TLSVersion and TLSCipherSuites, make some maps private * api: add TLS params to GatewayTLSConfig, add tests * api: add TLSMinVersion to ingress gateway config entry test JSON * xds: switch to Envoy TLS cipher suite encoding from types package * xds: fixup validation for TLSv1_3 min version with cipher suites * add some kitchen sink tests and add a missing struct tag * xds: check if mergedCfg.TLSVersion is in TLSVersionsWithConfigurableCipherSuites * xds: update connectTLSEnabled comment * xds: remove unsued resolveGatewayServiceTLSConfig function * xds: add makeCommonTLSContextFromLeafWithoutParams * types/tls: add LessThan comparator function for concrete values * types/tls: change tlsVersions validation map from string to TLSVersion keys * types/tls: remove unused envoyTLSCipherSuites * types/tls: enable chacha20 cipher suites for Consul agent * types/tls: remove insecure cipher suites from allowed config TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 and TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 are both explicitly listed as insecure and disabled in the Go source. Refs https://cs.opensource.google/go/go/+/refs/tags/go1.17.3:src/crypto/tls/cipher_suites.go;l=329-330 * types/tls: add ValidateConsulAgentCipherSuites function, make direct lookup map private * types/tls: return all unmatched cipher suites in validation errors * xds: check that Envoy API value matching TLS version is found when building TlsParameters * types/tls: check that value is found in map before appending to slice in MarshalEnvoyTLSCipherSuiteStrings * types/tls: cast to string rather than fmt.Printf in TLSCihperSuite.String() * xds: add TLSVersionUnspecified to list of configurable cipher suites * structs: update note about config entry warning * xds: remove TLS min version cipher suite unconfigurable test placeholder * types/tls: update tests to remove assumption about private map values Co-authored-by: R.B. Boyer <rb@hashicorp.com>	3 years ago
freddygv	c5c290c503	Validate chains are associated with upstreams ... Previously we could get into a state where discovery chain entries were not cleaned up after the associated watch was cancelled. These changes add handling for that case where stray chain references are encountered.	3 years ago
freddygv	e7a7042c69	Update listener generation to account for consul VIP	3 years ago
Freddy	00b5b0a0a2	Update filter chain creation for sidecar/ingress listeners (#11245) ... The duo of `makeUpstreamFilterChainForDiscoveryChain` and `makeListenerForDiscoveryChain` were really hard to reason about, and led to concealing a bug in their branching logic. There were several issues here: - They tried to accomplish too much: determining filter name, cluster name, and whether RDS should be used. - They embedded logic to handle significantly different kinds of upstream listeners (passthrough, prepared query, typical services, and catch-all) - They needed to coalesce different data sources (Upstream and CompiledDiscoveryChain) Rather than handling all of those tasks inside of these functions, this PR pulls out the RDS/clusterName/filterName logic. This refactor also fixed a bug with the handling of [UpstreamDefaults](https://www.consul.io/docs/connect/config-entries/service-defaults#defaults). These defaults get stored as UpstreamConfig in the proxy snapshot with a DestinationName of "*", since they apply to all upstreams. However, this wildcard destination name must not be used when creating the name of the associated upstream cluster. The coalescing logic in the original functions here was in some situations creating clusters with a `*.` prefix, which is not a valid destination.	3 years ago
freddygv	e93c144d2f	Update comments	3 years ago
freddygv	448701dbd8	Replace default partition check	3 years ago
freddygv	12923f5ebc	PR comments	3 years ago
freddygv	a33b6923e0	Account for partitions in xds gen for mesh gw ... This commit avoids skipping gateways in remote partitions of the local DC when generating listeners/clusters/endpoints.	3 years ago
freddygv	935112a47a	Account for partition in SNI for gateways	3 years ago
freddygv	110fae820a	Update xds pkg to account for GatewayKey	3 years ago
Paul Banks	78a00f2e1c	Add support for enabling connect-based ingress TLS per listener.	3 years ago
Paul Banks	5cfd030d03	Refactor Ingress-specific lister code to separate file	3 years ago
Paul Banks	136928a90f	Minor PR typo and cleanup fixes	3 years ago
Paul Banks	9fa60c7472	Remove unused argument to fix lint error	3 years ago
Paul Banks	659321d008	Handle namespaces in route names correctly; add tests for enterprise	3 years ago
Paul Banks	16b3b1c737	Update xDS Listeners with SDS support	3 years ago
Chris S. Kim	f972048ebc	connect: Allow upstream listener escape hatch for prepared queries (#11109)	3 years ago
Dhia Ayachi	bc0e4f2f46	partition dicovery chains (#10983) ... * partition dicovery chains * fix default partition for OSS	3 years ago
Dhia Ayachi	09197c989c	add partition to SNI when partition is non default (#10917)	3 years ago
Dhia Ayachi	1950ebbe1f	oss portion of ent #1069 (#10883)	3 years ago
Dhia Ayachi	9b45107c1e	Format certificates properly (rfc7468) with a trailing new line (#10411) ... * trim carriage return from certificates when inserting rootCA in the inMemDB * format rootCA properly when returning the CA on the connect CA endpoint * Fix linter warnings * Fix providers to trim certs before returning it * trim newlines on write when possible * add changelog * make sure all provider return a trailing newline after the root and intermediate certs * Fix endpoint to return trailing new line * Fix failing test with vault provider * make test more robust * make sure all provider return a trailing newline after the leaf certs * Check for suffix before removing newline and use function * Add comment to consul provider * Update change log Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com> * fix typo * simplify code callflow Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com> * extract requireNewLine as shared func * remove dependency to testify in testing file * remove extra newline in vault provider * Add cert newline fix to envoy xds * remove new line from mock provider * Remove adding a new line from provider and fix it when the cert is read * Add a comment to explain the fix * Add missing for leaf certs * fix missing new line * fix missing new line in leaf certs * remove extra new line in test * updage changelog Co-authored-by: Daniel Nephin <dnephin@hashicorp.com> * fix in vault provider and when reading cache (RPC call) * fix AWS provider * fix failing test in the provider * remove comments and empty lines * add check for empty cert in test * fix linter warnings * add new line for leaf and private key * use string concat instead of Sprintf * fix new lines for leaf signing * preallocate slice and remove append * Add new line to `SignIntermediate` and `CrossSignCA` Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com> Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>	3 years ago
Freddy	0a38c8fe10	Update agent/xds/listeners.go ... Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>	4 years ago
freddygv	f3e4705923	Remove unused param	4 years ago
freddygv	0aec6761dc	Update ingress gateway stats labeling ... In the absence of stats_tags to handle this pattern, when we pass "ingress_upstream.$port" as the stat_prefix, Envoy splits up that prefix and makes the port a part of the metric name. For example: - stat_prefix: ingress_upstream.8080 This leads to metric names like envoy_http_8080_no_route. Changing the stat_prefix to ingress_upstream_80880 yields the expected metric names such as envoy_http_no_route. Note that we don't encode the destination's name/ns/dc in this stat_prefix because for HTTP services ingress gateways use a single filter chain. Only cluster metrics are available on a per-upstream basis.	4 years ago
freddygv	6f8c6043b6	Update terminating gateway stats labeling ... This change makes it so that the stat prefix for terminating gateways matches that of connect proxies. By using the structure of "upstream.svc.ns.dc" we can extract labels for the destination service, namespace, and datacenter.	4 years ago
R.B. Boyer	848ad8535b	xds: ensure that dependent xDS resources are reconfigured during primary type warming (#10381) ... Updates to a cluster will clear the associated endpoints, and updates to a listener will clear the associated routes. Update the incremental xDS logic to account for this implicit cleanup so that we can finish warming the clusters and listeners. Fixes #10379	4 years ago
Freddy	ffb13f35f1	Rename CatalogDestinationsOnly (#10397) ... CatalogDestinationsOnly is a passthrough that would enable dialing addresses outside of Consul's catalog. However, when this flag is set to true only _connect_ endpoints for services can be dialed. This flag is being renamed to signal that non-Connect endpoints can't be dialed by transparent proxies when the value is set to true.	4 years ago
Freddy	429f9d8bb8	Add flag for transparent proxies to dial individual instances (#10329)	4 years ago
Freddy	7577f0e991	Revert "Avoid adding original_dst filter when not needed" (#10365)	4 years ago
Freddy	19334e8abf	Avoid adding original_dst filter when not needed (#10302)	4 years ago
Mark Anderson	583ae65d5b	Convert mode to string representation ... Signed-off-by: Mark Anderson <manderson@hashicorp.com>	4 years ago
Mark Anderson	06f0f79218	Continue working through proxy and agent ... Rework/listeners, rename makeListener Refactor, tests pass Signed-off-by: Mark Anderson <manderson@hashicorp.com>	4 years ago
Freddy	ed1082510d	Fixup discovery chain handling in transparent mode (#10168) ... Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com> Previously we would associate the address of a discovery chain target with the discovery chain's filter chain. This was broken for a few reasons: - If the upstream is a virtual service, the client proxy has no way of dialing it because virtual services are not targets of their discovery chains. The targets are distinct services. This is addressed by watching the endpoints of all upstream services, not just their discovery chain targets. - If multiple discovery chains resolve to the same target, that would lead to multiple filter chains attempting to match on the target's virtual IP. This is addressed by only matching on the upstream's virtual IP. NOTE: this implementation requires an intention to the redirecting virtual service and not just to the final destination. This is how we can know that the virtual service is an upstream to watch. A later PR will look into traversing discovery chains when computing upstreams so that intentions are only required to the discovery chain targets.	4 years ago
Freddy	2ca3f481f8	Only consider virtual IPs for transparent proxies (#10162) ... Initially we were loading every potential upstream address into Envoy and then routing traffic to the logical upstream service. The downside of this behavior is that traffic meant to go to a specific instance would be load balanced across ALL instances. Traffic to specific instance IPs should be forwarded to the original destination and if it's a destination in the mesh then we should ensure the appropriate certificates are used. This PR makes transparent proxying a Kubernetes-only feature for now since support for other environments requires generating virtual IPs, and Consul does not do that at the moment.	4 years ago
R.B. Boyer	abc1dc0fe9	connect: update supported envoy versions to 1.18.2, 1.17.2, 1.16.3, and 1.15.4 (#10101) ... The only thing that needed fixing up pertained to this section of the 1.18.x release notes: > grpc_stats: the default value for stats_for_all_methods is switched from true to false, in order to avoid possible memory exhaustion due to an untrusted downstream sending a large number of unique method names. The previous default value was deprecated in version 1.14.0. This only changes the behavior when the value is not set. The previous behavior can be used by setting the value to true. This behavior change by be overridden by setting runtime feature envoy.deprecated_features.grpc_stats_filter_enable_stats_for_all_methods_by_default. For now to maintain status-quo I'm explicitly setting `stats_for_all_methods=true` in all versions to avoid relying upon the default. Additionally the naming of the emitted metrics for these gRPC requests changed slightly so the integration test assertions for `case-grpc` needed adjusting.	4 years ago
R.B. Boyer	71d45a3460	Support Incremental xDS mode (#9855) ... This adds support for the Incremental xDS protocol when using xDS v3. This is best reviewed commit-by-commit and will not be squashed when merged. Union of all commit messages follows to give an overarching summary: xds: exclusively support incremental xDS when using xDS v3 Attempts to use SoTW via v3 will fail, much like attempts to use incremental via v2 will fail. Work around a strange older envoy behavior involving empty CDS responses over incremental xDS. xds: various cleanups and refactors that don't strictly concern the addition of incremental xDS support Dissolve the connectionInfo struct in favor of per-connection ResourceGenerators instead. Do a better job of ensuring the xds code uses a well configured logger that accurately describes the connected client. xds: pull out checkStreamACLs method in advance of a later commit xds: rewrite SoTW xDS protocol tests to use protobufs rather than hand-rolled json strings In the test we very lightly reuse some of the more boring protobuf construction helper code that is also technically under test. The important thing of the protocol tests is testing the protocol. The actual inputs and outputs are largely already handled by the xds golden output tests now so these protocol tests don't have to do double-duty. This also updates the SoTW protocol test to exclusively use xDS v2 which is the only variant of SoTW that will be supported in Consul 1.10. xds: default xds.Server.AuthCheckFrequency at use-time instead of construction-time	4 years ago
Freddy	078c40425f	Rename "cluster" config entry to "mesh" (#10127) ... This config entry is being renamed primarily because in k8s the name cluster could be confusing given that the config entry applies across federated datacenters. Additionally, this config entry will only apply to Consul as a service mesh, so the more generic "cluster" name is not needed.	4 years ago
Freddy	55a3697b83	Merge pull request #9987 from hashicorp/remove-kube-dns-hack	4 years ago
freddygv	4e509aa768	Remove todo that was todone	4 years ago
freddygv	7bd51ff536	Replace TransparentProxy bool with ProxyMode ... This PR replaces the original boolean used to configure transparent proxy mode. It was replaced with a string mode that can be set to: - "": Empty string is the default for when the setting should be defaulted from other configuration like config entries. - "direct": Direct mode is how applications originally opted into the mesh. Proxy listeners need to be dialed directly. - "transparent": Transparent mode enables configuring Envoy as a transparent proxy. Traffic must be captured and redirected to the inbound and outbound listeners. This PR also adds a struct for transparent proxy specific configuration. Initially this is not stored as a pointer. Will revisit that decision before GA.	4 years ago
Iryna Shustava	5755c97bc7	cli: Add new `consul connect redirect-traffic` command for applying traffic redirection rules when Transparent Proxy is enabled. (#9910) ... * Add new consul connect redirect-traffic command for applying traffic redirection rules when Transparent Proxy is enabled. * Add new iptables package for applying traffic redirection rules with iptables.	4 years ago
freddygv	c6d64a8078	Stable sort cidr ranges to match on	4 years ago
freddygv	02f6768cd2	Remove kube-dns resolution since clusterip will be a tagged addr	4 years ago
R.B. Boyer	499fee73b3	connect: add toggle to globally disable wildcard outbound network access when transparent proxy is enabled (#9973) ... This adds a new config entry kind "cluster" with a single special name "cluster" where this can be controlled.	4 years ago
freddygv	f4f45af6d0	Merge master and fix upstream config protocol defaulting	4 years ago
freddygv	9f0696528b	Rename hasChains for clarity	4 years ago