69 Commits (7c32c946964aa23631dbee145937e86ae8bdb2cc)

Author	SHA1	Message	Date
R.B. Boyer	af04851637	peering: move peer replication to the external gRPC port (#13698) ... Peer replication is intended to be between separate Consul installs and effectively should be considered "external". This PR moves the peer stream replication bidirectional RPC endpoint to the external gRPC server and ensures that things continue to function.	2 years ago
R.B. Boyer	0d6d16ddfb	add general runstep test helper instead of copying it all over the place (#13013)	3 years ago
DanStough	95250e7915	Update go version to 1.18.1	3 years ago
R.B. Boyer	80dfcb1bcd	raft: update to v1.3.5 (#12325) ... This includes closing some leadership transfer gaps and adding snapshot restore progress logging.	3 years ago
Daniel Nephin	b058845110	sdk: add TestLogLevel for setting log level in tests ... And default log level to WARN.	3 years ago
Ashwin Venkatesh	7568f3a102	Add support for 'Partition' and 'RetryJoin' (#12126) ... - Adding a 'Partition' and 'RetryJoin' command allows test cases where one would like to spin up a Consul Agent in a non-default partition to test use-cases that are common when enabling Admin Partition on Kubernetes.	3 years ago
Dan Upton	ca3aca92c4	[OSS] Remove remaining references to master (#11827)	3 years ago
Daniel Nephin	4185045a7f	sdk/freeport: rename Port to GetOne ... For better consistency with GetN	3 years ago
Daniel Nephin	ba41db5c4e	sdk: upgrade freeport godoc ... To clarify how it works, and when it is important to use.	3 years ago
Daniel Nephin	5a61893642	testing: use httptest with freeport	3 years ago
Daniel Nephin	8c7475d95e	sdk: add freeport functions that use t.Cleanup	3 years ago
R.B. Boyer	1e02460bd1	re-run gofmt on 1.17 (#11579) ... This should let freshly recompiled golangci-lint binaries using Go 1.17 pass 'make lint'	3 years ago
Nitya Dhanushkodi	139c4eb844	command/redirect_traffic: Redirect DNS requests to Consul if -consul-dns-ip is passed in (#11480) ... * command/redirect_traffic: add rules to redirect DNS to Consul. Currently uses a hack to get the consul dns service ip, and this hack only works when the service is deployed in the same namespace as consul. * command/redirect_traffic: redirect DNS to Consul when -consul-dns-ip is passed in * Add unit tests to Consul DNS IP table redirect rules Co-authored-by: Ashwin Venkatesh <ashwin@hashicorp.com> Co-authored-by: Iryna Shustava <ishustava@users.noreply.github.com>	3 years ago
FFMMM	4ddf973a31	add root_cert_ttl option for consul connect, vault ca providers (#11428) ... * add root_cert_ttl option for consul connect, vault ca providers Signed-off-by: FFMMM <FFMMM@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Chris S. Kim <ckim@hashicorp.com> * add changelog, pr feedback Signed-off-by: FFMMM <FFMMM@users.noreply.github.com> * Update .changelog/11428.txt, more docs Co-authored-by: Daniel Nephin <dnephin@hashicorp.com> * Update website/content/docs/agent/options.mdx Co-authored-by: Kyle Havlovitz <kylehav@gmail.com> Co-authored-by: Chris S. Kim <ckim@hashicorp.com> Co-authored-by: Daniel Nephin <dnephin@hashicorp.com> Co-authored-by: Kyle Havlovitz <kylehav@gmail.com>	3 years ago
Freddy	e18f3c1f6d	Update error texts (#11022) ... Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>	3 years ago
Sergey Matyukevich	06f3ccebce	Allow configuring graceful stop in testutil (#10566) ... * Allow configuring graceful stop in testutil Signed-off-by: Sergey Matyukevich <s.matyukevich@gmail.com> * add a changelog Co-authored-by: Dhia Ayachi <dhia@hashicorp.com>	3 years ago
Iryna Shustava	95305881ce	cli/sdk: Allow applying redirect-traffic rules in a provided Linux namespace (#10564)	3 years ago
R.B. Boyer	d7fb3543fc	sdk: Stop making a special /tmp/consul-test directory for testutil.TempFile and testutil.TempDir (#10494)	3 years ago
David Bariod	797bfb7b51	remove a race condition in sdk testutil server stop (#10342) ... * remove a race condition in sdk testutil server stop * ensure the process has actually stopped	4 years ago
Matt Keeler	caafc02449	hcs-1936: Prepare for adding license auto-retrieval to auto-config in enterprise	4 years ago
Matt Keeler	234d0a3c2a	Preparation for changing where license management is done.	4 years ago
Daniel Nephin	1618912cf6	Fix some test flakes ... - return errors in TestAgent.Start so that the retry works correctly - remove duplicate logging, the error is returned already - add a missing t.Helper() to retry.Run - properly set a.Agent to nil so that subsequent retry attempts will actually try to start	4 years ago
Iryna Shustava	8dffb89131	Implement traffic redirection exclusion based on proxy config and user-provided values (#10134) ... * Use proxy outbound port from TransparentProxyConfig if provided * If -proxy-id is provided to the redirect-traffic command, exclude any listener ports from inbound traffic redirection. This includes envoy_prometheus_bind_addr, envoy_stats_bind_addr, and the ListenerPort from the Expose configuration. * Allow users to provide additional inbound and outbound ports, outbound CIDRs and additional user IDs to be excluded from traffic redirection. This affects both the traffic-redirect command and the iptables SDK package.	4 years ago
Daniel Nephin	3a27fce2ad	submatview: fix godoc and comment typos ... Co-authored-by: Paul Banks <banks@banksco.de>	4 years ago
Daniel Nephin	a2986ebb9c	sdk/retry: support ending the iteration early ... I've found this feature to be very useful in https://pkg.go.dev/gotest.tools/v3/poll#WaitOn I have encountered a few cases where I wanted that same support, so this commit adds it.	4 years ago
Daniel Nephin	034c5c5f8c	sdk/retry: remove the need to pass args to NextOr	4 years ago
Daniel Nephin	5de0a452cf	sdk/retry: a few small debug improvements ... On a few occasions I've had to read timeout stack traces for tests and noticed that retry.Run runs the function in a goroutine. This makes debuging a timeout more difficult because the gourinte of the retryable function is disconnected from the stack of the actual test. It requires searching through the entire stack trace to find the other goroutine. By using panic instead of runtime.Goexit() we remove the need for a separate goroutine. Also a few other small improvements: * add `R.Helper` so that an assertion function can be used with both testing.T and retry.R. * Pass t to `Retryer.NextOr`, and call `t.Helper` in a number of places so that the line number reported by `t.Log` is the line in the test where `retry.Run` was called, instead of some line in `retry.go` that is not relevant to the failure. * improve the implementation of `dedup` by removing the need to iterate twice. Instad track the lines and skip any duplicate when writing to the buffer.	4 years ago
Iryna Shustava	5755c97bc7	cli: Add new `consul connect redirect-traffic` command for applying traffic redirection rules when Transparent Proxy is enabled. (#9910) ... * Add new consul connect redirect-traffic command for applying traffic redirection rules when Transparent Proxy is enabled. * Add new iptables package for applying traffic redirection rules with iptables.	4 years ago
John Eikenberry	1266bfd65b	[SDK] change all cases of *testing.T to testing.TB ... Using the interface opens up the use of all methods to benchmarks as well as tests.	4 years ago
Shengjing Zhu	b931e989b8	sdk: read procfs instead of using sysctl command ... This drops the dependency of sysctl command	4 years ago
R.B. Boyer	a2a8e9c783	connect: intentions are now managed as a new config entry kind "service-intentions" (#8834) ... - Upgrade the ConfigEntry.ListAll RPC to be kind-aware so that older copies of consul will not see new config entries it doesn't understand replicate down. - Add shim conversion code so that the old API/CLI method of interacting with intentions will continue to work so long as none of these are edited via config entry endpoints. Almost all of the read-only APIs will continue to function indefinitely. - Add new APIs that operate on individual intentions without IDs so that the UI doesn't need to implement CAS operations. - Add a new serf feature flag indicating support for intentions-as-config-entries. - The old line-item intentions way of interacting with the state store will transparently flip between the legacy memdb table and the config entry representations so that readers will never see a hiccup during migration where the results are incomplete. It uses a piece of system metadata to control the flip. - The primary datacenter will begin migrating intentions into config entries on startup once all servers in the datacenter are on a version of Consul with the intentions-as-config-entries feature flag. When it is complete the old state store representations will be cleared. We also record a piece of system metadata indicating this has occurred. We use this metadata to skip ALL of this code the next time the leader starts up. - The secondary datacenters continue to run the old intentions replicator until all servers in the secondary DC and primary DC support intentions-as-config-entries (via serf flag). Once this condition it met the old intentions replicator ceases. - The secondary datacenters replicate the new config entries as they are migrated in the primary. When they detect that the primary has zeroed it's old state store table it waits until all config entries up to that point are replicated and then zeroes its own copy of the old state store table. We also record a piece of system metadata indicating this has occurred. We use this metadata to skip ALL of this code the next time the leader starts up.	4 years ago
Paul Burlumi	bb2b5dd871	testutil: use TestingTB instead of testing.TB	4 years ago
R.B. Boyer	7cde9c6788	sdk: also print test agent logs in verbose mode (#8616)	4 years ago
Daniel Nephin	51b08c645b	Merge pull request #8514 from hashicorp/dnephin/testing-improvements-1 ... testing: small improvements to TestSessionCreate and testutil.retry	4 years ago
Daniel Nephin	16217fe9b9	testing: use t.Cleanup in testutil.TempFile ... So that it has the same behaviour as TempDir. Also remove the now unnecessary 'defer os.Remove'	4 years ago
Daniel Nephin	587dcebfca	Use t.Helper in testutil/retry	4 years ago
Daniel Nephin	070e843113	testutil: Add t.Cleanup to TempDir ... TempDir registers a Cleanup so that the directory is always removed. To disable to cleanup, set the TEST_NOCLEANUP env var.	4 years ago
Daniel Nephin	fe2f80c3a1	Use SIGABRT to get a stack trace when the timeout is hit	4 years ago
Daniel Nephin	8e67d8eaeb	sdk: mitigate api test timeout ... Occasionally we are seeing the go-test-api job timeout at 10 minutes. Looking at the stack trace I saw the following: 1. Lots of tests blocked on server.Stop in NewTestServerConfigT. This suggests that SIGINT is being sent to the server, but the server is not properly shutting down. 2. Over 20k goroutines that look like this: goroutine 16355 [select, 8 minutes]: net/http.(*persistConn).readLoop(0xc004270240) /usr/local/go/src/net/http/transport.go:2099 +0x99e created by net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:1647 +0xc56 Issue 1 seems to be the main problem, but debugging that directly is not possible because our buffered logs do not get sent when the tests timeout. To mitigate this problem I've added a timeout to the cmd.Wait() to force kill the process and return an error. Unfortunately because we retry this operation, we still may not see the cause because the next attempt will likely pass. I'm tempted to remove the retry around NewTestServerConfigT. Issue 2 seems to be caused by not closing the response body. Since the request is performed many times in a loop, many goroutines are created and are not closed until the response body is closed.	4 years ago
Daniel Nephin	51efba2c7d	testutil: NewLogBuffer - buffer logs until a test fails ... Replaces #7559 Running tests in parallel, with background goroutines, results in test output not being associated with the correct test. `go test` does not make any guarantees about output from goroutines being attributed to the correct test case. Attaching log output from background goroutines also cause data races. If the goroutine outlives the test, it will race with the test being marked done. Previously this was noticed as a panic when logging, but with the race detector enabled it is shown as a data race. The previous solution did not address the problem of correct test attribution because test output could still be hidden when it was associated with a test that did not fail. You would have to look at all of the log output to find the relevant lines. It also made debugging test failures more difficult because each log line was very long. This commit attempts a new approach. Instead of printing all the logs, only print when a test fails. This should work well when there are a small number of failures, but may not work well when there are many test failures at the same time. In those cases the failures are unlikely a result of a specific test, and the log output is likely less useful. All of the logs are printed from the test goroutine, so they should be associated with the correct test. Also removes some test helpers that were not used, or only had a single caller. Packages which expose many functions with similar names can be difficult to use correctly. Related: https://github.com/golang/go/issues/38458 (may be fixed in go1.15) https://github.com/golang/go/issues/38382#issuecomment-612940030	4 years ago
Iryna Shustava	03366573d7	sdk: Use /v1/status/leader endpoint when starting a test server (#8192) ... Switch from /v1/agent/self to /v1/status/leader when checking if the test server has come up successfully in the waitForAPI function. Previously, the test server was relying (probably not intentionally) on the default value of the acl_enforce_version_8 in the TestConfig, which was false. So if you create a test server and enabled ACLs, they would not be enforced and the server would be able to come up pretty quickly because /v1/agent/self would return a 200 status pretty much as soon as the agent is running and most likely before leader election is finished. Now that we have removed acl_enforce_version_8 property (equivalent to being true by default) if you've created a test server with ACLs enabled, it will need to wait for leader election and for ACLs to be initialized before it'll get a successful response from the /v1/agent/self. Note: With this change, waitForAPI function no longer requires a 200 response status from the v1/status/leader endpoint. This is because in some tests, namely TestAPI_AgentLeave, we are only running clients, and this endpoint returns a 500 status.	4 years ago
R.B. Boyer	ffb9c7d6f7	acl: remove the deprecated `acl_enforce_version_8` option (#7991) ... Fixes #7292	5 years ago
R.B. Boyer	b282268408	sdk: extracting testutil.RequireErrorContains from various places it was duplicated (#7753)	5 years ago
Pierre Souchay	5703aadb59	[BUGFIX] Fix race condition in freeport (#7567) ... This removes a race condition in reset since pendingPorts can be set to nil in reset() If ticker is hit at wrong time, it would crash the unit test. We ensure in reset to avoid this race condition by cancelling the goroutine using killTicker chan. We also properly clean up eveything, so garbage collector can work as expected. To reproduce existing bug: `while go test -timeout 30s github.com/hashicorp/consul/sdk/freeport -run '^(Test.*)$'; do go clean -testcache; done` Will crash after a few 10s runs on my machine. Error could be seen in unit tests sometimes: [INFO] freeport: resetting the freeport package state panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1125536] goroutine 25 [running]: container/list.(*List).Len(...) /usr/local/Cellar/go/1.14/libexec/src/container/list/list.go:66 github.com/hashicorp/consul/sdk/freeport.checkFreedPortsOnce() /Users/p.souchay/go/src/github.com/hashicorp/consul/sdk/freeport/freeport.go:157 +0x86 github.com/hashicorp/consul/sdk/freeport.checkFreedPorts() /Users/p.souchay/go/src/github.com/hashicorp/consul/sdk/freeport/freeport.go:147 +0x71 created by github.com/hashicorp/consul/sdk/freeport.initialize /Users/p.souchay/go/src/github.com/hashicorp/consul/sdk/freeport/freeport.go:113 +0x2cf FAIL github.com/hashicorp/consul/sdk/freeport 1.607s	5 years ago
Pierre Souchay	952b60abaa	Proper detection of ephemeral ports on Mac OS (#7539) ... Use systemctl to properly detect ephemeral ports on Mac OS (aka darwin) by fetching systemctl values: * net.inet.ip.portrange.first * net.inet.ip.portrange.last This will avoid the message: `[INFO] freeport: ephemeral port range detection not configured for GOOS="darwin"` and properly detect the correct port range	5 years ago
Daniel Nephin	3f8578f1e1	ci: Use golangci-lint for linting ... Using golangci-lint has a number of advantages: - adding new linters becomes much easier, its a couple lines of yaml config instead of more bash scripting - it enables whitelisting of issues using inline comments or regex - when running multiple linters less work is done. The parsed source can be reused by multiple linters - linters are run in parallel to reduce CI runtime.	5 years ago
Matt Keeler	dfb0177dbc	Testing updates to support namespaced testing of the agent/xds… (#7185) ... * Various testing updates to support namespaced testing of the agent/xds package * agent/proxycfg package updates to support better namespace testing	5 years ago
Chris Piraino	401221de58	Allow users to configure either unstructured or JSON logging (#7130) ... * hclog Allow users to choose between unstructured and JSON logging	5 years ago
Matt Keeler	b57c2b78fd	Unflake the TestAPI_AgentConnectCALeaf test (#7142) ... * Unflake the TestAPI_AgentConnectCALeaf test * Modify the WaitForActiveCARoot to actually verify that at least one root exists Also verify that the active root id field is set	5 years ago
Chris Piraino	336f62fa99	Fix up formatting in sdk package (#7109)	5 years ago