64 Commits (CSLC-103-egress-gtwy-cert-tgtwy)

Author	SHA1	Message	Date
R.B. Boyer	93b164aac3	structs: add convenience methods to sort slices of ServiceName values (#13038)	3 years ago
Riddhi Shah	0c855fab98	Validate port on mesh service registration (#12881) ... Add validation to ensure connect native services have a port or socketpath specified on catalog registration. This was the only missing piece to ensure all mesh services are validated for a port (or socketpath) specification on catalog registration.	3 years ago
Chris S. Kim	9791bad136	peering: Make Upstream peer-aware (#12900) ... Adds DestinationPeer field to Upstream. Adds Peer field to UpstreamID and its string conversion functions.	3 years ago
R.B. Boyer	f507f62f3c	peering: initial sync (#12842) ... - Add endpoints related to peering: read, list, generate token, initiate peering - Update node/service/check table indexing to account for peers - Foundational changes for pushing service updates to a peer - Plumb peer name through Health.ServiceNodes path see: ENT-1765, ENT-1280, ENT-1283, ENT-1283, ENT-1756, ENT-1739, ENT-1750, ENT-1679, ENT-1709, ENT-1704, ENT-1690, ENT-1689, ENT-1702, ENT-1701, ENT-1683, ENT-1663, ENT-1650, ENT-1678, ENT-1628, ENT-1658, ENT-1640, ENT-1637, ENT-1597, ENT-1634, ENT-1613, ENT-1616, ENT-1617, ENT-1591, ENT-1588, ENT-1596, ENT-1572, ENT-1555 Co-authored-by: R.B. Boyer <rb@hashicorp.com> Co-authored-by: freddygv <freddy@hashicorp.com> Co-authored-by: Chris S. Kim <ckim@hashicorp.com> Co-authored-by: Evan Culver <eculver@hashicorp.com> Co-authored-by: Nitya Dhanushkodi <nitya@hashicorp.com>	3 years ago
R.B. Boyer	957146401e	catalog: compare node names case insensitively in more places (#12444) ... Many places in consul already treated node names case insensitively. The state store indexes already do it, but there are a few places that did a direct byte comparison which have now been corrected. One place of particular consideration is ensureCheckIfNodeMatches which is executed during snapshot restore (among other places). If a node check used a slightly different casing than the casing of the node during register then the snapshot restore here would deterministically fail. This has been fixed. Primary approach: git grep -i "node.*[!=]=.*node" -- ':!*_test.go' ':!docs' git grep -i '\[[^]]*member[^]]*\] git grep -i '\[[^]]*\(member\|name\|node\)[^]]*\]' -- ':!*_test.go' ':!website' ':!ui' ':!agent/proxycfg/testing.go:' ':!*.md'	3 years ago
freddygv	cbea3d203c	Fix race of upstreams with same passthrough ip ... Due to timing, a transparent proxy could have two upstreams to dial directly with the same address. For example: - The orders service can dial upstreams shipping and payment directly. - An instance of shipping at address 10.0.0.1 is deregistered. - Payments is scaled up and scheduled to have address 10.0.0.1. - The orders service receives the event for the new payments instance before seeing the deregistration for the shipping instance. At this point two upstreams have the same passthrough address and Envoy will reject the listener configuration. To disambiguate this commit considers the Raft index when storing passthrough addresses. In the example above, 10.0.0.1 would only be associated with the newer payments service instance.	3 years ago
R.B. Boyer	b60d89e7ef	bulk rewrite using this script ... set -euo pipefail unset CDPATH cd "$(dirname "$0")" for f in $(git grep '\brequire := require\.New(' | cut -d':' -f1 | sort -u); do echo "=== require: $f ===" sed -i '/require := require.New(t)/d' $f # require.XXX(blah) but not require.XXX(tblah) or require.XXX(rblah) sed -i 's/\brequire\.\([a-zA-Z0-9_]*\)(\([^tr]\)/require.\1(t,\2/g' $f # require.XXX(tblah) but not require.XXX(t, blah) sed -i 's/\brequire\.\([a-zA-Z0-9_]*\)(\(t[^,]\)/require.\1(t,\2/g' $f # require.XXX(rblah) but not require.XXX(r, blah) sed -i 's/\brequire\.\([a-zA-Z0-9_]*\)(\(r[^,]\)/require.\1(t,\2/g' $f gofmt -s -w $f done for f in $(git grep '\bassert := assert\.New(' | cut -d':' -f1 | sort -u); do echo "=== assert: $f ===" sed -i '/assert := assert.New(t)/d' $f # assert.XXX(blah) but not assert.XXX(tblah) or assert.XXX(rblah) sed -i 's/\bassert\.\([a-zA-Z0-9_]*\)(\([^tr]\)/assert.\1(t,\2/g' $f # assert.XXX(tblah) but not assert.XXX(t, blah) sed -i 's/\bassert\.\([a-zA-Z0-9_]*\)(\(t[^,]\)/assert.\1(t,\2/g' $f # assert.XXX(rblah) but not assert.XXX(r, blah) sed -i 's/\bassert\.\([a-zA-Z0-9_]*\)(\(r[^,]\)/assert.\1(t,\2/g' $f gofmt -s -w $f done	3 years ago
Dan Upton	1e47e3c82b	Groundwork for exposing when queries are filtered by ACLs (#11569)	3 years ago
Kyle Havlovitz	4f2cfee4b0	consul: add virtual IP generation for connect services	3 years ago
Daniel Nephin	8e9773e20b	acl: remove ACL.GetPolicy endpoint and resolve legacy acls ... And all code that was no longer used once those two were removed.	3 years ago
Dhia Ayachi	bc0e4f2f46	partition dicovery chains (#10983) ... * partition dicovery chains * fix default partition for OSS	3 years ago
Freddy	33bd9b5be8	Relax validation for expose.paths config (#10394) ... Previously we would return an error if duplicate paths were specified. This could lead to problems in cases where a user has the same path, say /healthz, on two different ports. This validation was added to signal a potential misconfiguration. Instead we will only check for duplicate listener ports, since that is what would lead to ambiguity issues when generating xDS config. In the future we could look into using a single listener and creating distinct filter chains for each path/port.	3 years ago
Daniel Nephin	ba15f92a8a	structs: fix cache keys ... So that requests are cached properly, and the cache does not return the wrong data for a request.	4 years ago
Daniel Nephin	46dfdb611f	structs: improve the interface of assertCacheInfoKeyIsComplete	4 years ago
Daniel Nephin	7c2957e24d	structs: Add more cache key tests	4 years ago
Mark Anderson	751fe7e314	Fixup more structs_test ... Signed-off-by: Mark Anderson <manderson@hashicorp.com>	4 years ago
Mark Anderson	06f0f79218	Continue working through proxy and agent ... Rework/listeners, rename makeListener Refactor, tests pass Signed-off-by: Mark Anderson <manderson@hashicorp.com>	4 years ago
freddygv	77ead5cca9	Prevent wildcard destinations for proxies and upstreams	4 years ago
freddygv	458eb41be1	Prevent synthetic upstreams without addresses from failing duplicate ip/port validation	4 years ago
Daniel Nephin	29ce5ec575	structs: fix caching of ServiceSpecificRequest when ingress=true ... The field was not being included in the cache info key. This would result in a DNS request for web.service.consul returning the same result as web.ingress.consul, when those results should not be the same.	4 years ago
kevinkengne	2e7e78999d	add completeness test for types with CacheInfo method (#9480) ... include all fields when fuzzing in tests split tests by struct type Ensure the new value for the field is different fuzzer.Fuzz could produce the same value again in some cases. Use a custom fuzz function for QueryOptions. That type is an embedded struct in the request types but only one of the fields is important to include in the cache key. Move enterpriseMetaField to an oss file so that we can change it in enterprise.	4 years ago
Daniel Nephin	b532e092dc	structs: add a namespace test for CheckServiceNode.CanRead	4 years ago
Daniel Nephin	21c21191f4	structs: add CheckServiceNode.CanRead ... And use it from the subscribe endpoint.	4 years ago
Paul Banks	7d58901ae8	Fix bad int -> string conversions caught by go vet changes in 1.15 (#8739)	4 years ago
Daniel Nephin	07ad662131	Fix panic when decoding 'Connect: null' ... Surprisingly the json Unmarshal updates the aux pointer to a nil.	4 years ago
Daniel Nephin	068b43df90	Enable gofmt simplify ... Code changes done automatically with 'gofmt -s -w'	5 years ago
freddygv	19e3954603	Move compound service names to use ServiceName type	5 years ago
Jono Sosulska	c554ba9e10	Replace whitelist/blacklist terminology with allowlist/denylist (#7971) ... * Replace whitelist/blacklist terminology with allowlist/denylist	5 years ago
Daniel Nephin	e60bb9f102	test: Remove t.Parallel() from agent/structs tests ... go test will only run tests in parallel within a single package. In this case the package test run time is exactly the same with or without t.Parallel() (~0.7s). In generally we should avoid t.Parallel() as it causes a number of problems with `go test` not reporting failure messages correctly. I encountered one of these problems, which is what prompted this change. Since `t.Parallel` is not providing any benefit in this package, this commit removes it. The change was automated with: git grep -l 't.Parallel' | xargs sed -i -e '/t.Parallel/d'	5 years ago
R.B. Boyer	b282268408	sdk: extracting testutil.RequireErrorContains from various places it was duplicated (#7753)	5 years ago
Freddy	137a2c32c6	TLS Origination for Terminating Gateways (#7671)	5 years ago
Freddy	9eb1867fbb	Terminating gateway discovery (#7571) ... * Enable discovering terminating gateways * Add TerminatingGatewayServices to state store * Use GatewayServices RPC endpoint for ingress/terminating	5 years ago
Kyle Havlovitz	c911174327	Add config entry/state for Ingress Gateways (#7483) ... * Add Ingress gateway config entry and other relevant structs * Add api package tests for ingress gateways * Embed EnterpriseMeta into ingress service struct * Add namespace fields to api module and test consul config write decoding * Don't require a port for ingress gateways * Add snakeJSON and camelJSON cases in command test * Run Normalize on service's ent metadata Sadly cannot think of a way to test this in OSS. * Every protocol requires at least 1 service * Validate ingress protocols * Update agent/structs/config_entry_gateways.go Co-authored-by: Chris Piraino <cpiraino@hashicorp.com> Co-authored-by: Freddy <freddygv@users.noreply.github.com>	5 years ago
Freddy	18d356899c	Enable CLI to register terminating gateways (#7500) ... * Enable CLI to register terminating gateways * Centralize gateway proxy configuration	5 years ago
R.B. Boyer	6adad71125	wan federation via mesh gateways (#6884) ... This is like a Möbius strip of code due to the fact that low-level components (serf/memberlist) are connected to high-level components (the catalog and mesh-gateways) in a twisty maze of references which make it hard to dive into. With that in mind here's a high level summary of what you'll find in the patch: There are several distinct chunks of code that are affected: * new flags and config options for the server * retry join WAN is slightly different * retry join code is shared to discover primary mesh gateways from secondary datacenters * because retry join logic runs in the *agent* and the results of that operation for primary mesh gateways are needed in the *server* there are some methods like `RefreshPrimaryGatewayFallbackAddresses` that must occur at multiple layers of abstraction just to pass the data down to the right layer. * new cache type `FederationStateListMeshGatewaysName` for use in `proxycfg/xds` layers * the function signature for RPC dialing picked up a new required field (the node name of the destination) * several new RPCs for manipulating a FederationState object: `FederationState:{Apply,Get,List,ListMeshGateways}` * 3 read-only internal APIs for debugging use to invoke those RPCs from curl * raft and fsm changes to persist these FederationStates * replication for FederationStates as they are canonically stored in the Primary and replicated to the Secondaries. * a special derivative of anti-entropy that runs in secondaries to snapshot their local mesh gateway `CheckServiceNodes` and sync them into their upstream FederationState in the primary (this works in conjunction with the replication to distribute addresses for all mesh gateways in all DCs to all other DCs) * a "gateway locator" convenience object to make use of this data to choose the addresses of gateways to use for any given RPC or gossip operation to a remote DC. This gets data from the "retry join" logic in the agent and also directly calls into the FSM. * RPC (`:8300`) on the server sniffs the first byte of a new connection to determine if it's actually doing native TLS. If so it checks the ALPN header for protocol determination (just like how the existing system uses the type-byte marker). * 2 new kinds of protocols are exclusively decoded via this native TLS mechanism: one for ferrying "packet" operations (udp-like) from the gossip layer and one for "stream" operations (tcp-like). The packet operations re-use sockets (using length-prefixing) to cut down on TLS re-negotiation overhead. * the server instances specially wrap the `memberlist.NetTransport` when running with gateway federation enabled (in a `wanfed.Transport`). The general gist is that if it tries to dial a node in the SAME datacenter (deduced by looking at the suffix of the node name) there is no change. If dialing a DIFFERENT datacenter it is wrapped up in a TLS+ALPN blob and sent through some mesh gateways to eventually end up in a server's :8300 port. * a new flag when launching a mesh gateway via `consul connect envoy` to indicate that the servers are to be exposed. This sets a special service meta when registering the gateway into the catalog. * `proxycfg/xds` notice this metadata blob to activate additional watches for the FederationState objects as well as the location of all of the consul servers in that datacenter. * `xds:` if the extra metadata is in place additional clusters are defined in a DC to bulk sink all traffic to another DC's gateways. For the current datacenter we listen on a wildcard name (`server.<dc>.consul`) that load balances all servers as well as one mini-cluster per node (`<node>.server.<dc>.consul`) * the `consul tls cert create` command got a new flag (`-node`) to help create an additional SAN in certs that can be used with this flavor of federation.	5 years ago
R.B. Boyer	8c596953b0	agent: ensure that we always use the same settings for msgpack (#7245) ... We set RawToString=true so that []uint8 => string when decoding an interface{}. We set the MapType so that map[interface{}]interface{} decodes to map[string]interface{}. Add tests to ensure that this doesn't break existing usages. Fixes #7223	5 years ago
Freddy	fdd10dd8b8	Expose HTTP-based paths through Connect proxy (#6446) ... Fixes: #5396 This PR adds a proxy configuration stanza called expose. These flags register listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only listening on the loopback interface, while still accepting traffic from non Connect-enabled services. Under expose there is a boolean checks flag that would automatically expose all registered HTTP and gRPC check paths. This stanza also accepts a paths list to expose individual paths. The primary use case for this functionality would be to expose paths for third parties like Prometheus or the kubelet. Listeners for requests to exposed paths are be configured dynamically at run time. Any time a proxy, or check can be registered, a listener can also be created. In this initial implementation requests to these paths are not authenticated/encrypted.	5 years ago
Mike Morris	65be58703c	connect: remove managed proxies (#6220) ... * connect: remove managed proxies implementation and all supporting config options and structs * connect: remove deprecated ProxyDestination * command: remove CONNECT_PROXY_TOKEN env var * agent: remove entire proxyprocess proxy manager * test: remove all managed proxy tests * test: remove irrelevant managed proxy note from TestService_ServerTLSConfig * test: update ContentHash to reflect managed proxy removal * test: remove deprecated ProxyDestination test * telemetry: remove managed proxy note * http: remove /v1/agent/connect/proxy endpoint * ci: remove deprecated test exclusion * website: update managed proxies deprecation page to note removal * website: remove managed proxy configuration API docs * website: remove managed proxy note from built-in proxy config * website: add note on removing proxy subdirectory of data_dir	5 years ago
R.B. Boyer	8564b6bb38	connect: validate upstreams and prevent duplicates (#6224) ... * connect: validate upstreams and prevent duplicates * Actually run Upstream.Validate() instead of ignoring it as dead code. * Prevent two upstreams from declaring the same bind address and port. It wouldn't work anyway. * Prevent two upstreams from being declared that use the same type+name+namespace+datacenter. Due to how the Upstream.Identity() function worked this ended up mostly being enforced in xDS at use-time, but it should be enforced more clearly at register-time.	5 years ago
Matt Keeler	6e65811db2	Envoy CLI bind addresses (#6107) ... * Ensure we MapWalk the proxy config in the NodeService and ServiceNode structs This gets rid of some json encoder errors in the catalog endpoints * Allow passing explicit bind addresses to envoy * Move map walking to the ConnectProxyConfig struct Any place where this struct gets JSON encoded will benefit as opposed to having to implement it everywhere. * Fail when a non-empty address is provided and not bindable * camel case * Update command/connect/envoy/envoy.go Co-Authored-By: Paul Banks <banks@banksco.de>	5 years ago
Matt Keeler	8d953f5840	Implement Mesh Gateways ... This includes both ingress and egress functionality.	5 years ago
Matt Keeler	f3d9b999ee	Add tagged addresses for services (#5965) ... This allows addresses to be tagged at the service level similar to what we allow for nodes already. The address translation that can be enabled with the `translate_wan_addrs` config was updated to take these new addresses into account as well.	6 years ago
Paul Banks	0cfb6051ea	Add integration test for central config; fix central config WIP (#5752) ... * Add integration test for central config; fix central config WIP * Add integration test for central config; fix central config WIP * Set proxy protocol correctly and begin adding upstream support * Add upstreams to service config cache key and start new notify watcher if they change. This doesn't update the tests to pass though. * Fix some merging logic get things working manually with a hack (TODO fix properly) * Simplification to not allow enabling sidecars centrally - it makes no sense without upstreams anyway * Test compile again and obvious ones pass. Lots of failures locally not debugged yet but may be flakes. Pushing up to see what CI does * Fix up service manageer and API test failures * Remove the enable command since it no longer makes much sense without being able to turn on sidecar proxies centrally * Remove version.go hack - will make integration test fail until release * Remove unused code from commands and upstream merge * Re-bump version to 1.5.0	6 years ago
Kyle Havlovitz	c269369760	Make central service config opt-in and rework the initial registration	6 years ago
Hans Hasselberg	552e150536	correct name	6 years ago
Hans Hasselberg	d4790b2827	demo nomad problem	6 years ago
Paul Banks	b29bc906ee	bugfix: use ServiceTags to generate cache key hash (#4987) ... * bugfix: use ServiceTags to generate cahce key hash * update unit test * update * remote print log * Update .gitignore * Completely deprecate ServiceTag field internally for clarity * Add explicit test for CacheInfo cases	6 years ago
Matt Keeler	f9cf0eb36e	Remaining ACL Unit Tests (#4852) ... * Add leader token upgrade test and fix various ACL enablement bugs * Update the leader ACL initialization tests. * Add a StateStore ACL tests for ACLTokenSet and ACLTokenGetBy* functions * Advertise the agents acl support status with the agent/self endpoint. * Make batch token upsert CAS’able to prevent consistency issues with token auto-upgrade * Finish up the ACL state store token tests * Finish the ACL state store unit tests Also rename some things to make them more consistent. * Do as much ACL replication testing as I can.	6 years ago
Matt Keeler	18b29c45c4	New ACLs (#4791) ... This PR is almost a complete rewrite of the ACL system within Consul. It brings the features more in line with other HashiCorp products. Obviously there is quite a bit left to do here but most of it is related docs, testing and finishing the last few commands in the CLI. I will update the PR description and check off the todos as I finish them over the next few days/week. Description At a high level this PR is mainly to split ACL tokens from Policies and to split the concepts of Authorization from Identities. A lot of this PR is mostly just to support CRUD operations on ACLTokens and ACLPolicies. These in and of themselves are not particularly interesting. The bigger conceptual changes are in how tokens get resolved, how backwards compatibility is handled and the separation of policy from identity which could lead the way to allowing for alternative identity providers. On the surface and with a new cluster the ACL system will look very similar to that of Nomads. Both have tokens and policies. Both have local tokens. The ACL management APIs for both are very similar. I even ripped off Nomad's ACL bootstrap resetting procedure. There are a few key differences though. Nomad requires token and policy replication where Consul only requires policy replication with token replication being opt-in. In Consul local tokens only work with token replication being enabled though. All policies in Nomad are globally applicable. In Consul all policies are stored and replicated globally but can be scoped to a subset of the datacenters. This allows for more granular access management. Unlike Nomad, Consul has legacy baggage in the form of the original ACL system. The ramifications of this are: A server running the new system must still support other clients using the legacy system. A client running the new system must be able to use the legacy RPCs when the servers in its datacenter are running the legacy system. The primary ACL DC's servers running in legacy mode needs to be a gate that keeps everything else in the entire multi-DC cluster running in legacy mode. So not only does this PR implement the new ACL system but has a legacy mode built in for when the cluster isn't ready for new ACLs. Also detecting that new ACLs can be used is automatic and requires no configuration on the part of administrators. This process is detailed more in the "Transitioning from Legacy to New ACL Mode" section below.	6 years ago
Pierre Souchay	51b33ef015	[Performance On Large clusters] Reduce updates on large services (#4720) ... * [Performance On Large clusters] Checks do update services/nodes only when really modified to avoid too many updates on very large clusters In a large cluster, when having a few thousands of nodes, the anti-entropy mechanism performs lots of changes (several per seconds) while there is no real change. This patch wants to improve this in order to increase Consul scalability when using many blocking requests on health for instance. * [Performance for large clusters] Only updates index of service if service is really modified * [Performance for large clusters] Only updates index of nodes if node is really modified * Added comments / ensure IsSame() has clear semantics * Avoid having modified boolean, return nil directly if stutures are Same * Fixed unstable unit tests TestLeader_ChangeServerID * Rewrite TestNode_IsSame() for better readability as suggested by @banks * Rename ServiceNode.IsSame() into IsSameService() + added unit tests * Do not duplicate TestStructs_ServiceNode_Conversions() and increase test coverage of IsSameService * Clearer documentation in IsSameService * Take into account ServiceProxy into ServiceNode.IsSameService() * Fixed IsSameService() with all new structures	6 years ago