76 Commits (564288adcc24027b4d671d3c21a9b24cad4d6f43)

Author	SHA1	Message	Date
Matt Keeler	34b53e7099	A few misc fixes found by go vet	6 years ago
Matt Keeler	18b29c45c4	New ACLs (#4791) ... This PR is almost a complete rewrite of the ACL system within Consul. It brings the features more in line with other HashiCorp products. Obviously there is quite a bit left to do here but most of it is related docs, testing and finishing the last few commands in the CLI. I will update the PR description and check off the todos as I finish them over the next few days/week. Description At a high level this PR is mainly to split ACL tokens from Policies and to split the concepts of Authorization from Identities. A lot of this PR is mostly just to support CRUD operations on ACLTokens and ACLPolicies. These in and of themselves are not particularly interesting. The bigger conceptual changes are in how tokens get resolved, how backwards compatibility is handled and the separation of policy from identity which could lead the way to allowing for alternative identity providers. On the surface and with a new cluster the ACL system will look very similar to that of Nomads. Both have tokens and policies. Both have local tokens. The ACL management APIs for both are very similar. I even ripped off Nomad's ACL bootstrap resetting procedure. There are a few key differences though. Nomad requires token and policy replication where Consul only requires policy replication with token replication being opt-in. In Consul local tokens only work with token replication being enabled though. All policies in Nomad are globally applicable. In Consul all policies are stored and replicated globally but can be scoped to a subset of the datacenters. This allows for more granular access management. Unlike Nomad, Consul has legacy baggage in the form of the original ACL system. The ramifications of this are: A server running the new system must still support other clients using the legacy system. A client running the new system must be able to use the legacy RPCs when the servers in its datacenter are running the legacy system. The primary ACL DC's servers running in legacy mode needs to be a gate that keeps everything else in the entire multi-DC cluster running in legacy mode. So not only does this PR implement the new ACL system but has a legacy mode built in for when the cluster isn't ready for new ACLs. Also detecting that new ACLs can be used is automatic and requires no configuration on the part of administrators. This process is detailed more in the "Transitioning from Legacy to New ACL Mode" section below.	6 years ago
Jack Pearkes	8c684db488	New command: consul debug (#4754) ... * agent/debug: add package for debugging, host info * api: add v1/agent/host endpoint * agent: add v1/agent/host endpoint * command/debug: implementation of static capture * command/debug: tests and only configured targets * agent/debug: add basic test for host metrics * command/debug: add methods for dynamic data capture * api: add debug/pprof endpoints * command/debug: add pprof * command/debug: timing, wg, logs to disk * vendor: add gopsutil/disk * command/debug: add a usage section * website: add docs for consul debug * agent/host: require operator:read * api/host: improve docs and no retry timing * command/debug: fail on extra arguments * command/debug: fixup file permissions to 0644 * command/debug: remove server flags * command/debug: improve clarity of usage section * api/debug: add Trace for profiling, fix profile * command/debug: capture profile and trace at the same time * command/debug: add index document * command/debug: use "clusters" in place of members * command/debug: remove address in output * command/debug: improve comment on metrics sleep * command/debug: clarify usage * agent: always register pprof handlers and protect This will allow us to avoid a restart of a target agent for profiling by always registering the pprof handlers. Given this is a potentially sensitive path, it is protected with an operator:read ACL and enable debug being set to true on the target agent. enable_debug still requires a restart. If ACLs are disabled, enable_debug is sufficient. * command/debug: use trace.out instead of .prof More in line with golang docs. * agent: fix comment wording * agent: wrap table driven tests in t.run()	6 years ago
Freddy	524192c919	Improve resilience of api pkg tests (#4676) ... * Add function to wait for serfHealth in api tests * Disable connect when creating semaphore test clients * Wait for serfHealth when creating sessions in their tests * Add helper functions to create lock/semaphore sessions without checks * Log passing tests to prevent timeout in Travis due to lack of output	6 years ago
Pierre Souchay	2f76019847	Fixed message: testutil/server.go:317: missing ... in args forwarded to print-like function	6 years ago
Paul Banks	597e55e8e2	Misc test fixes	7 years ago
Paul Banks	96c416012e	Misc rebase and test fixes	7 years ago
Paul Banks	4aeab3897c	Fixed many tests after rebase. Some still failing and seem unrelated to any connect changes.	7 years ago
Paul Banks	e0e12e165b	TLS watching integrated into Service with some basic tests. ... There are also a lot of small bug fixes found when testing lots of things end-to-end for the first time and some cleanup now it's integrated with real CA code.	7 years ago
Josh Soref	94835a2715	Spelling (#3958) ... * spelling: another * spelling: autopilot * spelling: beginning * spelling: circonus * spelling: default * spelling: definition * spelling: distance * spelling: encountered * spelling: enterprise * spelling: expands * spelling: exits * spelling: formatting * spelling: health * spelling: hierarchy * spelling: imposed * spelling: independence * spelling: inspect * spelling: last * spelling: latest * spelling: client * spelling: message * spelling: minimum * spelling: notify * spelling: nonexistent * spelling: operator * spelling: payload * spelling: preceded * spelling: prepared * spelling: programmatically * spelling: required * spelling: reconcile * spelling: responses * spelling: request * spelling: response * spelling: results * spelling: retrieve * spelling: service * spelling: significantly * spelling: specifies * spelling: supported * spelling: synchronization * spelling: synchronous * spelling: themselves * spelling: unexpected * spelling: validations * spelling: value	7 years ago
Frank Schroeder	c94751ad43	test: replace porter tool with freeport lib ... This patch removes the porter tool which hands out free ports from a given range with a library which does the same thing. The challenge for acquiring free ports in concurrent go test runs is that go packages are tested concurrently and run in separate processes. There has to be some inter-process synchronization in preventing processes allocating the same ports. freeport allocates blocks of ports from a range expected to be not in heavy use and implements a system-wide mutex by binding to the first port of that block for the lifetime of the application. Ports are then provided sequentially from that block and are tested on localhost before being returned as available.	7 years ago
Alex Dadgar	c7a65b4587	Testutil falls back to random ports w/o porter (#3604) ... * Testutil falls back to random ports w/o porter This PR allows the testutil server to be used without porter. * Adds sterner-sounding fallback comments.	7 years ago
Frank Schröder	12216583a1	New config parser, HCL support, multiple bind addrs (#3480) ... * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt	7 years ago
James Phillips	b1a15e0c3d	Adds open source side of network segments (feature is Enterprise-only).	7 years ago
James Phillips	1791d99a10	Adds new config to make script checks opt-in, updates documentation. (#3284)	7 years ago
Frank Schroeder	cd0b0b35c9	api: do not block in waitForLeader since the retry loop is handling this	8 years ago
James Phillips	ebe032a75c	Adds a temporary band-aid to the API tests. ... Once we can run the agent directly in-process with random ports from the kernel, and the special retry just in memberlist, we shouldn't need this.	8 years ago
Frank Schroeder	1d5b56eec1	test: shutdown server properly	8 years ago
Frank Schroeder	f44c4e769b	test: bump ReadyTimeout to 10s	8 years ago
Frank Schroeder	65b5c51ec7	test: add helper for ioutil.TempDir/TempFile ... This creates a simplified helper for temporary directories and files. All path names are prefixed with the name of the current test. All files and directories are stored either in /tmp/consul-test or /tmp if the former could not be created. Using the system temp dir breaks some tests on macOS where the unix socket path becomes too long.	8 years ago
Frank Schroeder	3522a41bad	test: include test name in temp file/dir ... This helps identifying hanging tests by looking at the process list.	8 years ago
Cyril Gaudin	417ffb8893	test: Make testserver start timeout configurable.	8 years ago
Frank Schroeder	c3d6814149	test: Drop WaitForResult from testutil and testrpc	8 years ago
Kyle Havlovitz	cd56a5ebdd	Add separate option for verifying incoming HTTPS traffic (#2974) ... * Add separate option for verifying incoming HTTPS traffic	8 years ago
Frank Schroeder	cf3ec1cf5c	golint: Rename fields and structs	8 years ago
Kyle Havlovitz	ae6bf56ee1	Add tls client options to api/cli	8 years ago
James Phillips	59a599a14b	Cleans up a stray mark and fixes unit tests. ... Ended up removing the leader_test.go server address change test as part of this. The join was failing becase we were using a new node name with the new logic here, but realized this was hitting some of the memberlist conflict logic and not working as we expected. We need some additional work to fully support address changes, so removed the test for now.	8 years ago
James Phillips	4d3f3ea8d2	Changes verson 8 ACLs to opt-out.	8 years ago
James Phillips	15e7247131	Fixes test server startup returning wrong error. ... Seen in https://travis-ci.org/hashicorp/consul/builds/214521317.	8 years ago
Seth Vargo	1dc1f72fb6	Add RPC	8 years ago
Seth Vargo	3d830b3676	Reduce coupling with testing.T ... This reduces the coupling with testing.T, allowing many of the server's startup-related functions to return an error. This makes them more re-usable.	8 years ago
Kyle Havlovitz	9b4497de09	Cleaned up and reorganized some autopilot-related code	8 years ago
Kyle Havlovitz	f389b59fd5	Fix up command and api tests	8 years ago
Jeff Mitchell	44733caf5c	Update unix dial functions to use DialContext with new go-cleanhttp	8 years ago
James Phillips	14c6d009cc	Drop testutil wait to 2 seconds. ... There's likely a race (related to https://github.com/hashicorp/consul/issues/2644) where the catalog update might be in but the leader tracking doesn't report a leader, so this blocks forever and then times out. As a workaround we can lower the query wait time to always allow for a few retries.	8 years ago
James Phillips	dcff5137f9	Adds a little wait after we get the event we were seeking.	8 years ago
James Phillips	8493d239ac	Run the waited-for function before sleeping, and ramp up the sleep exponentially.	8 years ago
James Phillips	8c33ddbee2	Drops the unit test wait time back down and turns initial wait into blocking query.	8 years ago
Seth Vargo	c0b6c7ad3e	Check to see if TaggedAddresses have been populated ... This ensures the node's anti-entropy checks have finished before telling the client Consul is ready.	8 years ago
Seth Vargo	624b7daba4	Omit empty node_meta	8 years ago
Kyle Havlovitz	f48f105949	Minor formatting tweaks as a follow-up to #2654	8 years ago
Seth Vargo	8bcfa9d3cc	Do not skip tests when Consul is missing ... This bit me on CI. The current behavior of the testutil server is to skip if consul isn't present. When lots of output is scrolling by, you're likely to miss the message that the test was skipped. Instead, I propose that we hard fatal if consul doesn't exist, and upstream consumers can skip the tests if they want.	8 years ago
Kyle Havlovitz	87c0283bb1	Update client api and docs for node metadata	8 years ago
Seth Vargo	9010b0e0f4	Use a random port instead of idx in testutil ... The testutil server uses an atomic incrementer to generate unique port numbers. This works great until tests are run in parallel, _across packages_. Because each package starts at the same "offset" idx, they collide. One way to overcome this is to run each packages' test in isolation, but that makes the test suite much longer as it does not maximize parallelization. Alternatively, instead of having "predictable" ports, we can let the OS choose a random open port automatically. This still has a (albeit smaller) race condition in that the OS could return an open port twice, before the server has a chance to actually start and occupy said port. In practice, I have not been able to hit this race condition, so it either doesn't happen or it happens far less frequently that the existing implementation. I'm not sure how I feel about the panic, but this is just test code, so I'm including to say it's okay?	8 years ago
Kyle Havlovitz	bd69c6d871	Add reload/leave http endpoints (#2516)	8 years ago
Kyle Havlovitz	dd3368c19e	Add keyring http endpoints	8 years ago
James Phillips	57db4bcce6	Adds performance tuning capability for Raft, detuned defaults, and supplemental docs.	8 years ago
Sean Chittenden	e63d3a1275	Update Check API to use constants ... Use constants where appropriate to advocate their use. Also add a deprecation notice re: `updateTTL`.	9 years ago
kaskavalci	556f20b8c3	Make testutil.TestServer work correctly on Windows.	9 years ago
James Phillips	7510f00100	Merge pull request #1846 from TeaBough/master ... Define a TestingT interface in server.go	9 years ago