137 Commits (7b4aca2088c441c427c0acb5b97ca07790cb46f6)

Author	SHA1	Message	Date
R.B. Boyer	9801ef8eb1	agent: enable enable_central_service_config by default (#8746)	4 years ago
R.B. Boyer	7eef25daf5	agent: when enable_central_service_config is enabled ensure agent reload doesn't revert check state to critical (#8747) ... Likely introduced when #7345 landed.	4 years ago
Daniel Nephin	c18516ad7d	Merge pull request #8680 from hashicorp/dnephin/replace-consul-opts-with-base-deps ... agent: Repalce ConsulOptions with a new struct from agent.BaseDeps	4 years ago
Daniel Nephin	cdd392d77f	agent/consul: pass dependencies directly from agent ... In an upcoming change we will need to pass a grpc.ClientConnPool from BaseDeps into Server. While looking at that change I noticed all of the existing consulOption fields are already on BaseDeps. Instead of duplicating the fields, we can create a struct used by agent/consul, and use that struct in BaseDeps. This allows us to pass along dependencies without translating them into different representations. I also looked at moving all of BaseDeps in agent/consul, however that created some circular imports. Resolving those cycles wouldn't be too bad (it was only an error in agent/consul being imported from cache-types), however this change seems a little better by starting to introduce some structure to BaseDeps. This change is also a small step in reducing the scope of Agent. Also remove some constants that were only used by tests, and move the relevant comment to where the live configuration is set. Removed some validation from NewServer and NewClient, as these are not really runtime errors. They would be code errors, which will cause a panic anyway, so no reason to handle them specially here.	4 years ago
Daniel Nephin	0bb9c318b7	http: fix tests incorrectly using HTTPAddr to get the address of the ... https server. In #8234 I changed a few tests to use TestAgent.HTTPAddr() to find the addr used in the test. Due to the way HTTPAddr() was implemented these tests were passing, but I think the pass was incidental. HTTPAddr() was not matching any servers, and was instead returning the last server, which happened to be the one these tests wanted. This commit fixes the implementation of HTTPAddr to panic if no match was found. The tests which require an HTTPS server are changed to use a new firstAddr() to look up the correct address.	4 years ago
Daniel Nephin	4c9ed41eab	Merge pull request #8554 from hashicorp/dnephin/agent-setup-persisted-tokens ... agent: move token persistence from agent into token.Store	4 years ago
Daniel Nephin	6ca45e1a61	agent: add apiServers type for managing HTTP servers ... Remove Server field from HTTPServer. The field is no longer used.	4 years ago
Daniel Nephin	330be5b740	agent/token: Move token persistence out of agent ... And into token.Store. This change isolates any awareness of token persistence in a single place. It is a small step in allowing Agent.New to accept its dependencies.	4 years ago
Daniel Nephin	72bf350069	Merge pull request #8552 from pierresouchay/reload_cache_throttling_config ... Ensure that Cache options are reloaded when `consul reload` is performed	4 years ago
Pierre Souchay	9a64d3e5fe	Also test reload of EntryFetchMaxBurst	4 years ago
Matt Keeler	f97cc0445a	Move RPC router from Client/Server and into BaseDeps (#8559) ... This will allow it to be a shared component which is needed for AutoConfig	4 years ago
Pierre Souchay	5842a902df	Tests that changes in rate limit are taken into account by agent	4 years ago
Daniel Nephin	d68edcecf4	testing: Remove all the defer os.Removeall ... Now that testutil uses t.Cleanup to remove the directory the caller no longer has to manage the removal	4 years ago
Daniel Nephin	3a4e62836b	testing: Remove TestAgent.Key and change TestAgent.DataDir ... TestAgent.Key was only used by 3 tests. Extracting it from the common helper that is used in hundreds of tests helps keep the shared part small and more focused. This required a second change (which I was planning on making anyway), which was to change the behaviour of DataDir. Now in all cases the TestAgent will use the DataDir, and clean it up once the test is complete.	4 years ago
Daniel Nephin	0738eb8596	Extract nodeID functions to a different file ... In preparation for turning them into functions. To reduce the scope of Agent, and refactor how Agent is created and started.	4 years ago
Daniel Nephin	38980ebb4c	config: Make Source an interface ... This will allow us to accept config from auto-config without needing to go through a serialziation cycle.	4 years ago
Matt Keeler	1a78cf9b4c	Ensure certificates retrieved through the cache get persisted with auto-config (#8409)	4 years ago
Matt Keeler	34034b76f5	Agent Auto Config: Implement Certificate Generation (#8360) ... Most of the groundwork was laid in previous PRs between adding the cert-monitor package to extracting the logic of signing certificates out of the connect_ca_endpoint.go code and into a method on the server. This also refactors the auto-config package a bit to split things out into multiple files.	4 years ago
Matt Keeler	83d09de230	Fix some broken code in master ... There were several PRs that while all passed CI independently, when they all got merged into the same branch caused compilation errors in test code. The main changes that caused issues where changing agent/cache.Cache.New to require a concrete options struct instead of a pointer. This broke the cert monitor tests and the catalog_list_services_test.go. Another change was made to unembed the http.Server from the agent.HTTPServer struct. That coupled with another change to add a test to ensure cache rate limiting coming from HTTP requests was working as expected caused compilation failures.	4 years ago
Pierre Souchay	505de6dc29	Added ratelimit to handle throtling cache (#8226) ... This implements a solution for #7863 It does: Add a new config cache.entry_fetch_rate to limit the number of calls/s for a given cache entry, default value = rate.Inf Add cache.entry_fetch_max_burst size of rate limit (default value = 2) The new configuration now supports the following syntax for instance to allow 1 query every 3s: command line HCL: -hcl 'cache = { entry_fetch_rate = 0.333}' in JSON { "cache": { "entry_fetch_rate": 0.333 } }	4 years ago
Matt Keeler	6e7acfa618	Add an AutoEncrypt “integration” test ... Also fix a bug where Consul could segfault if TLS was enabled but no client certificate was provided. How no one has reported this as a problem I am not sure.	4 years ago
Daniel Nephin	0279bf6fe5	Update TestAgent_GetCoordinate ... The old test case was a very specific regresion test for a case that is no longer possible. Replaced with a new test that checks the default coordinate is returned.	4 years ago
Daniel Nephin	010a609912	Fix a bunch of unparam lint issues	4 years ago
Matt Keeler	b0fcf86140	Change auto config authorizer to allow for future extension ... The envisioned changes would allow extra settings to enable dynamically defined auth methods to be used instead of or in addition to the statically defined one in the configuration.	5 years ago
Matt Keeler	3dbbd2d37d	Implement Client Agent Auto Config ... There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.	5 years ago
Daniel Nephin	068b43df90	Enable gofmt simplify ... Code changes done automatically with 'gofmt -s -w'	5 years ago
Hans Hasselberg	dfcf45c6cf	tests: use constructor instead init (#8024)	5 years ago
Pierre Souchay	9813ae512b	checks: when a service does not exists in an alias, consider it failing (#7384) ... In current implementation of Consul, check alias cannot determine if a service exists or not. Because a service without any check is semantically considered as passing, so when no healthchecks are found for an agent, the check was considered as passing. But this make little sense as the current implementation does not make any difference between: * a non-existing service (passing) * a service without any check (passing as well) In order to make it work, we have to ensure that when a check did not find any healthcheck, the service does indeed exists. If it does not, lets consider the check as failing.	5 years ago
Daniel Nephin	99eb583ebc	Replace goe/verify.Values with testify/require.Equal (#7993) ... * testing: replace most goe/verify.Values with require.Equal One difference between these two comparisons is that go/verify considers nil slices/maps to be equal to empty slices/maps, where as testify/require does not, and does not appear to provide any way to enable that behaviour. Because of this difference some expected values were changed from empty slices to nil slices, and some calls to verify.Values were left. * Remove github.com/pascaldekloe/goe/verify Reduce the number of assertion packages we use from 2 to 1	5 years ago
Daniel Nephin	600645b5f9	Add unconvert linter ... To find unnecessary type convertions	5 years ago
Pierre Souchay	2a8bf45e38	agent: show warning when enable_script_checks is enabled without safty net (#7437) ... In order to enforce a bit security on Consul agents, add a new method in agent to highlight possible security issues. This does not return an error for now, but might in the future. For now, it detects issues such as: https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ This would display this kind of messages: ``` 2020-03-11T18:27:49.873+0100 [ERROR] agent: [SECURITY] issue: error="using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/" ```	5 years ago
Daniel Nephin	e759daafdd	Rename NewTestAgentWithFields to StartTestAgent ... This function now only starts the agent. Using: git grep -l 'StartTestAgent(t, true,' | \ xargs sed -i -e 's/StartTestAgent(t, true,/StartTestAgent(t,/g'	5 years ago
Daniel Nephin	f9f6b14533	Convert the remaining calls to NewTestAgentWithFields ... After removing the t.Name() parameter with sed, convert the last few tests which use a custom name to call NewTestAgentWithFields instead.	5 years ago
Daniel Nephin	475659a132	Remove name from NewTestAgent ... Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'	5 years ago
Daniel Nephin	ad7c78f134	Remove t.Name() from TestAgent.Name ... And re-add the name to the logger so that log messages from different agents in a single can be identified.	5 years ago
Daniel Nephin	dd40a1535e	testing: reduce verbosity of output log ... Previously the log output included the test name twice and a long date format. The test output is already grouped by test, so adding the test name did not add any new information. The date and time are only useful to understand elapsed time, so using a short format should provide succident detail. Also fixed a bug in NewTestAgentWithFields where nil was returned instead of the test agent.	5 years ago
Chris Piraino	136099d834	Fix flakey health check reload test (#7490) ... This test would occasionally fail because we checked for a status of "critical" initially. This races with the actual healthcheck being run and declared passing. We instead use a ttl health check so that we don't rely on timing at all.	5 years ago
R.B. Boyer	6adad71125	wan federation via mesh gateways (#6884) ... This is like a Möbius strip of code due to the fact that low-level components (serf/memberlist) are connected to high-level components (the catalog and mesh-gateways) in a twisty maze of references which make it hard to dive into. With that in mind here's a high level summary of what you'll find in the patch: There are several distinct chunks of code that are affected: * new flags and config options for the server * retry join WAN is slightly different * retry join code is shared to discover primary mesh gateways from secondary datacenters * because retry join logic runs in the *agent* and the results of that operation for primary mesh gateways are needed in the *server* there are some methods like `RefreshPrimaryGatewayFallbackAddresses` that must occur at multiple layers of abstraction just to pass the data down to the right layer. * new cache type `FederationStateListMeshGatewaysName` for use in `proxycfg/xds` layers * the function signature for RPC dialing picked up a new required field (the node name of the destination) * several new RPCs for manipulating a FederationState object: `FederationState:{Apply,Get,List,ListMeshGateways}` * 3 read-only internal APIs for debugging use to invoke those RPCs from curl * raft and fsm changes to persist these FederationStates * replication for FederationStates as they are canonically stored in the Primary and replicated to the Secondaries. * a special derivative of anti-entropy that runs in secondaries to snapshot their local mesh gateway `CheckServiceNodes` and sync them into their upstream FederationState in the primary (this works in conjunction with the replication to distribute addresses for all mesh gateways in all DCs to all other DCs) * a "gateway locator" convenience object to make use of this data to choose the addresses of gateways to use for any given RPC or gossip operation to a remote DC. This gets data from the "retry join" logic in the agent and also directly calls into the FSM. * RPC (`:8300`) on the server sniffs the first byte of a new connection to determine if it's actually doing native TLS. If so it checks the ALPN header for protocol determination (just like how the existing system uses the type-byte marker). * 2 new kinds of protocols are exclusively decoded via this native TLS mechanism: one for ferrying "packet" operations (udp-like) from the gossip layer and one for "stream" operations (tcp-like). The packet operations re-use sockets (using length-prefixing) to cut down on TLS re-negotiation overhead. * the server instances specially wrap the `memberlist.NetTransport` when running with gateway federation enabled (in a `wanfed.Transport`). The general gist is that if it tries to dial a node in the SAME datacenter (deduced by looking at the suffix of the node name) there is no change. If dialing a DIFFERENT datacenter it is wrapped up in a TLS+ALPN blob and sent through some mesh gateways to eventually end up in a server's :8300 port. * a new flag when launching a mesh gateway via `consul connect envoy` to indicate that the servers are to be exposed. This sets a special service meta when registering the gateway into the catalog. * `proxycfg/xds` notice this metadata blob to activate additional watches for the FederationState objects as well as the location of all of the consul servers in that datacenter. * `xds:` if the extra metadata is in place additional clusters are defined in a DC to bulk sink all traffic to another DC's gateways. For the current datacenter we listen on a wildcard name (`server.<dc>.consul`) that load balances all servers as well as one mini-cluster per node (`<node>.server.<dc>.consul`) * the `consul tls cert create` command got a new flag (`-node`) to help create an additional SAN in certs that can be used with this flavor of federation.	5 years ago
Pierre Souchay	864f7efffa	agent: configuration reload preserves check's statuses for services (#7345) ... This fixes issue #7318 Between versions 1.5.2 and 1.5.3, a regression has been introduced regarding health of services. A patch #6144 had been issued for HealthChecks of nodes, but not for healthchecks of services. What happened when a reload was: 1. save all healthcheck statuses 2. cleanup everything 3. add new services with healthchecks In step 3, the state of healthchecks was taken into account locally, so at step 3, but since we cleaned up at step 2, state was lost. This PR introduces the snap parameter, so step 3 can use information from step 1	5 years ago
Chris Piraino	401221de58	Allow users to configure either unstructured or JSON logging (#7130) ... * hclog Allow users to choose between unstructured and JSON logging	5 years ago
Aestek	ba8fd8296f	Add support for dual stack IPv4/IPv6 network (#6640) ... * Use consts for well known tagged adress keys * Add ipv4 and ipv6 tagged addresses for node lan and wan * Add ipv4 and ipv6 tagged addresses for service lan and wan * Use IPv4 and IPv6 address in DNS	5 years ago
Matt Keeler	5934f803bf	Sync of OSS changes to support namespaces (#6909)	5 years ago
Matt Keeler	8f0ab0129e	Miscellaneous Fixes (#6896) ... Ensure we close the Sentinel Evaluator so as not to leak go routines Fix a bunch of test logging so that various warnings when starting a test agent go to the ltest logger and not straight to stdout. Various canned ent meta types always return a valid pointer (no more nils). This allows us to blindly deref + assign in various places. Update ACL index tracking to ensure oss -> ent upgrades will work as expected. Update ent meta parsing to include function to disallow wildcarding.	5 years ago
Freddy	60f6ec0c2f	Store check type in catalog (#6561)	5 years ago
R.B. Boyer	9566df524e	agent: cache notifications work after error if the underlying RPC returns index=1 (#6547) ... Fixes #6521 Ensure that initial failures to fetch an agent cache entry using the notify API where the underlying RPC returns a synthetic index of 1 correctly recovers when those RPCs resume working. The bug in the Cache.notifyBlockingQuery used to incorrectly "fix" the index for the next query from 0 to 1 for all queries, when it should have not done so for queries that errored. Also fixed some things that made debugging difficult: - config entry read/list endpoints send back QueryMeta headers - xds event loops don't swallow the cache notification errors	5 years ago
Freddy	fdd10dd8b8	Expose HTTP-based paths through Connect proxy (#6446) ... Fixes: #5396 This PR adds a proxy configuration stanza called expose. These flags register listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only listening on the loopback interface, while still accepting traffic from non Connect-enabled services. Under expose there is a boolean checks flag that would automatically expose all registered HTTP and gRPC check paths. This stanza also accepts a paths list to expose individual paths. The primary use case for this functionality would be to expose paths for third parties like Prometheus or the kubelet. Listeners for requests to exposed paths are be configured dynamically at run time. Any time a proxy, or check can be registered, a listener can also be created. In this initial implementation requests to these paths are not authenticated/encrypted.	5 years ago
R.B. Boyer	5882e21b2b	agent: tolerate more failure scenarios during service registration with central config enabled (#6472) ... Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.	5 years ago
R.B. Boyer	f9496dc627	sdk: add freelist tracking and ephemeral port range skipping to freeport ... This should cut down on test flakiness. Problems handled: - If you had enough parallel test cases running, the former circular approach to handling the port block could hand out the same port to multiple cases before they each had a chance to bind them, leading to one of the two tests to fail. - The freeport library would allocate out of the ephemeral port range. This has been corrected for Linux (which should cover CI). - The library now waits until a formerly-in-use port is verified to be free before putting it back into circulation.	5 years ago
Sarah Adams	001137e5e5	test: ensure all TestAgent constructions use a constructor (#6443) ... ensure all TestAgent constructions use a constructor to get start retries + test logs going to the right place Fixes #6435	5 years ago
Sarah Adams	74461406e0	remove funky panic/recover in agent tests (#6442)	5 years ago