111 Commits (NET-3059/acl-agent-api)

Author	SHA1	Message	Date
Ronald Ekambi	299ff67516	Add copyright headers non ui files	2 years ago
Matt Keeler	085c0addc0	Protobuf Refactoring for Multi-Module Cleanliness (#16302) ... Protobuf Refactoring for Multi-Module Cleanliness This commit includes the following: Moves all packages that were within proto/ to proto/private Rewrites imports to account for the packages being moved Adds in buf.work.yaml to enable buf workspaces Names the proto-public buf module so that we can override the Go package imports within proto/buf.yaml Bumps the buf version dependency to 1.14.0 (I was trying out the version to see if it would get around an issue - it didn't but it also doesn't break things and it seemed best to keep up with the toolchain changes) Why: In the future we will need to consume other protobuf dependencies such as the Google HTTP annotations for openapi generation or grpc-gateway usage. There were some recent changes to have our own ratelimiting annotations. The two combined were not working when I was trying to use them together (attempting to rebase another branch) Buf workspaces should be the solution to the problem Buf workspaces means that each module will have generated Go code that embeds proto file names relative to the proto dir and not the top level repo root. This resulted in proto file name conflicts in the Go global protobuf type registry. The solution to that was to add in a private/ directory into the path within the proto/ directory. That then required rewriting all the imports. Is this safe? AFAICT yes The gRPC wire protocol doesn't seem to care about the proto file names (although the Go grpc code does tack on the proto file name as Metadata in the ServiceDesc) Other than imports, there were no changes to any generated code as a result of this.	2 years ago
Semir Patel	40c0bb24ae	emit metrics for global rate limiting (#15891)	2 years ago
Pier-Luc Caron St-Pierre	76fc2f6562	connect: Add support for ConsulResolver to specifies a filter expression (#15659) ... * connect: Add support for ConsulResolver to specifies a filter expression	2 years ago
Kyle Schochenmaier	bf0f61a878	removes ioutil usage everywhere which was deprecated in go1.16 (#15297) ... * update go version to 1.18 for api and sdk, go mod tidy * removes ioutil usage everywhere which was deprecated in go1.16 in favour of io and os packages. Also introduces a lint rule which forbids use of ioutil going forward. Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>	2 years ago
Chris S. Kim	bde57c0dd0	Regenerate files according to 1.19.2 formatter	2 years ago
Chris S. Kim	7dd16ed4fb	Add retries and debugging to flaky test	2 years ago
Chris S. Kim	7f2732e12c	Ensure connections are closed before WaitGroup marked as done ... The previous ordering of defers meant the listener's connWG could fire and wake up other goroutines before the connection closed. Unsure if this caused any real bugs but this commit should make the code more correct.	2 years ago
Chris S. Kim	a5fe2125e9	Remove unnecessary goroutine in flaky test ... The watch is established in a background goroutine and the first assertion proves that the watcher is active so there is no reason for the update to happen in a racy goroutine. Note that this does not completely remove the race condition as the first call to testGetConfigValTimeout could time out before a config is returned.	2 years ago
Matt Keeler	ead8e4a200	Fix race during proxy closing (#13283) ... p.service is written to within the Serve method. The Serve method also waits for the stopChan to be closed. The race was between Close being called on the proxy causing Close on the service which was written to around the same time in the Serve method. The fix is to have Serve be responsible for closing p.service.	3 years ago
cskh	364d4f5efe	Retry on bad dogstatsd connection (#13091) ... - Introduce a new telemetry configurable parameter retry_failed_connection. User can set the value to true to let consul agent continue its start process on failed connection to datadog server. When set to false, agent will stop on failed start. The default behavior is true. Co-authored-by: Dan Upton <daniel@floppy.co> Co-authored-by: Evan Culver <eculver@users.noreply.github.com>	3 years ago
DanStough	95250e7915	Update go version to 1.18.1	3 years ago
Eric	e4b4f175ed	Bump go-control-plane ... * `go get cloud.google.com/go@v0.59.0` * `go get github.com/envoyproxy/go-control-plane@v0.9.9` * `make envoy-library` * Bumpprotoc to 3.15.8	3 years ago
R.B. Boyer	b60d89e7ef	bulk rewrite using this script ... set -euo pipefail unset CDPATH cd "$(dirname "$0")" for f in $(git grep '\brequire := require\.New(' | cut -d':' -f1 | sort -u); do echo "=== require: $f ===" sed -i '/require := require.New(t)/d' $f # require.XXX(blah) but not require.XXX(tblah) or require.XXX(rblah) sed -i 's/\brequire\.\([a-zA-Z0-9_]*\)(\([^tr]\)/require.\1(t,\2/g' $f # require.XXX(tblah) but not require.XXX(t, blah) sed -i 's/\brequire\.\([a-zA-Z0-9_]*\)(\(t[^,]\)/require.\1(t,\2/g' $f # require.XXX(rblah) but not require.XXX(r, blah) sed -i 's/\brequire\.\([a-zA-Z0-9_]*\)(\(r[^,]\)/require.\1(t,\2/g' $f gofmt -s -w $f done for f in $(git grep '\bassert := assert\.New(' | cut -d':' -f1 | sort -u); do echo "=== assert: $f ===" sed -i '/assert := assert.New(t)/d' $f # assert.XXX(blah) but not assert.XXX(tblah) or assert.XXX(rblah) sed -i 's/\bassert\.\([a-zA-Z0-9_]*\)(\([^tr]\)/assert.\1(t,\2/g' $f # assert.XXX(tblah) but not assert.XXX(t, blah) sed -i 's/\bassert\.\([a-zA-Z0-9_]*\)(\(t[^,]\)/assert.\1(t,\2/g' $f # assert.XXX(rblah) but not assert.XXX(r, blah) sed -i 's/\bassert\.\([a-zA-Z0-9_]*\)(\(r[^,]\)/assert.\1(t,\2/g' $f gofmt -s -w $f done	3 years ago
Daniel Nephin	4185045a7f	sdk/freeport: rename Port to GetOne ... For better consistency with GetN	3 years ago
Daniel Nephin	e8312d6b5a	testing: remove unnecessary calls to freeport ... Previously we believe it was necessary for all code that required ports to use freeport to prevent conflicts. https://github.com/dnephin/freeport-test shows that it is actually save to use port 0 (`127.0.0.1:0`) as long as it is passed directly to `net.Listen`, and the listener holds the port for as long as it is needed. This works because freeport explicitly avoids the ephemeral port range, and port 0 always uses that range. As you can see from the test output of https://github.com/dnephin/freeport-test, the two systems never use overlapping ports. This commit converts all uses of freeport that were being passed directly to a net.Listen to use port 0 instead. This allows us to remove a bit of wrapping we had around httptest, in a couple places.	3 years ago
Daniel Nephin	d795a73f78	testing: use the new freeport interfaces	3 years ago
Dhia Ayachi	1950ebbe1f	oss portion of ent #1069 (#10883)	3 years ago
jkirschner-hashicorp	5f73de6fbc	Merge pull request #10560 from jkirschner-hashicorp/change-sane-to-reasonable ... Replace use of 'sane' where appropriate	3 years ago
Jared Kirschner	bd536151e1	Replace use of 'sane' where appropriate ... HashiCorp voice, style, and language guidelines recommend avoiding ableist language unless its reference to ability is accurate in a particular use.	3 years ago
R.B. Boyer	ed8a901be7	connect: include optional partition prefixes in SPIFFE identifiers (#10507) ... NOTE: this does not include any intentions enforcement changes yet	3 years ago
R.B. Boyer	ca0a58ff71	connect/proxy: fixes logic bug preventing builtin/native proxy from starting upstream listeners (#10486) ... Fixes #10480 Also fixed a data race in the `connect/proxy` package that was unearthed by the tests changed for this bugfix.	3 years ago
Daniel Nephin	0624c75c56	testing: slightly better comparison for x509.CertPool	4 years ago
Mark Anderson	8040f91a43	Add support for downstreams ... Enhance config by adding SocketPath and LocalSocketPath config values Supports syntax of the form: ``` services { name = "sock_forwarder" id = "sock_forwarder.1" socket_path = "/tmp/downstream_3.sock" connect { sidecar_service { proxy { local_service_socket_path = "/tmp/downstream.sock" } } } } ``` Signed-off-by: Mark Anderson <manderson@hashicorp.com>	4 years ago
Mark Anderson	06f0f79218	Continue working through proxy and agent ... Rework/listeners, rename makeListener Refactor, tests pass Signed-off-by: Mark Anderson <manderson@hashicorp.com>	4 years ago
Daniel Nephin	d18a03b07f	connect/proxy: fix a number of problems with Listener ... We noticed that TestUpstreamListener would deadlock sometimes when run with the race detector. While debugging this issue I found and fixed the following problems. 1. the net.Listener was not being closed properly when Listener.Stop was called. This caused the Listener.Serve goroutine to run forever. Fixed by storing a reference to net.Listener and closing it properly when Listener.Stop is called. 2. call connWG.Add in the correct place. WaitGroup.Add must be called before starting a goroutine, not from inside the goroutine. 3. Set metrics config EnableRuntimeMetrics to `false` so that we don't start a background goroutine in each test for no reason. There is no way to shutdown this goroutine, and it was an added distraction while debugging these timeouts. 5. two tests were calling require.NoError from a goroutine. require.NoError calls t.FailNow, which MUST be called from the main test goroutine. Instead use t.Errorf, which can be called from other goroutines and will still fail the test. 6. `assertCurrentGaugeValue` wass breaking out of a for loop, which would cause the `RWMutex.RUnlock` to be missed. Fixed by calling unlock before `break`. The core issue of a deadlock was fixed by https://github.com/armon/go-metrics/pull/124.	4 years ago
Daniel Nephin	146e1d3878	connect/proxy: remove t.Parallel from tests ... These tests run in under 10ms, t.Parallel provides no value and makes debuging failures more difficult.	4 years ago
Daniel Nephin	3e20bd25bd	connect: fix test for go1.16 ... There is no way to compare x509.CertPools now that it has an unexpected function field. This comparison is as close as we can get. See https://github.com/golang/go/issues/26614 for a related issue.	4 years ago
Florian Apolloner	c01922d40a	Allow passing ALPN next protocols down to connect services. Fixes #4466. (#9920) ... * Allow passing ALPN next protocols down to connect services. Fixes #4466. * Update connect/proxy/proxy_test.go Co-authored-by: Paul Banks <banks@banksco.de> Co-authored-by: Paul Banks <banks@banksco.de>	4 years ago
Daniel Nephin	b9e60c0775	testing: skip slow tests with -short ... Add a skip condition to all tests slower than 100ms. This change was made using `gotestsum tool slowest` with data from the last 3 CI runs of master. See https://github.com/gotestyourself/gotestsum#finding-and-skipping-slow-tests With this change: ``` $ time go test -count=1 -short ./agent ok github.com/hashicorp/consul/agent 0.743s real 0m4.791s $ time go test -count=1 -short ./agent/consul ok github.com/hashicorp/consul/agent/consul 4.229s real 0m8.769s ```	4 years ago
Daniel Nephin	839429eb40	Merge pull request #9160 from hashicorp/dnephin/go-test-race-in-to-out-list ... ci: change go-test-race package list to exclude list	4 years ago
Kit Patella	5e0e4098c9	push prometheus sink definiitons into prometheus.PrometheusOpts	4 years ago
Kit Patella	24a2471029	first pass on agent-configured prometheusDefs and adding defs for every consul metric	4 years ago
Daniel Nephin	b27457dac8	ci: go-test-race switch to exclude list ... Most packages should pass the race detector. An exclude list ensures that new packages are automatically tested with -race. Also fix a couple small test races to allow more packages to be tested. Returning readyCh requires a lock because it can be set to nil, and setting it to nil will race without the lock. Move the TestServer.Listening calls around so that they properly guard setting TestServer.l. Otherwise it races. Remove t.Parallel in a small package. The entire package tests run in a few seconds, so t.Parallel does very little. In auto-config, wait for the AutoConfig.run goroutine to stop before calling readPersistedAutoConfig. Without this change there was a data race on reading ac.config.	4 years ago
R.B. Boyer	a2a8e9c783	connect: intentions are now managed as a new config entry kind "service-intentions" (#8834) ... - Upgrade the ConfigEntry.ListAll RPC to be kind-aware so that older copies of consul will not see new config entries it doesn't understand replicate down. - Add shim conversion code so that the old API/CLI method of interacting with intentions will continue to work so long as none of these are edited via config entry endpoints. Almost all of the read-only APIs will continue to function indefinitely. - Add new APIs that operate on individual intentions without IDs so that the UI doesn't need to implement CAS operations. - Add a new serf feature flag indicating support for intentions-as-config-entries. - The old line-item intentions way of interacting with the state store will transparently flip between the legacy memdb table and the config entry representations so that readers will never see a hiccup during migration where the results are incomplete. It uses a piece of system metadata to control the flip. - The primary datacenter will begin migrating intentions into config entries on startup once all servers in the datacenter are on a version of Consul with the intentions-as-config-entries feature flag. When it is complete the old state store representations will be cleared. We also record a piece of system metadata indicating this has occurred. We use this metadata to skip ALL of this code the next time the leader starts up. - The secondary datacenters continue to run the old intentions replicator until all servers in the secondary DC and primary DC support intentions-as-config-entries (via serf flag). Once this condition it met the old intentions replicator ceases. - The secondary datacenters replicate the new config entries as they are migrated in the primary. When they detect that the primary has zeroed it's old state store table it waits until all config entries up to that point are replicated and then zeroes its own copy of the old state store table. We also record a piece of system metadata indicating this has occurred. We use this metadata to skip ALL of this code the next time the leader starts up.	4 years ago
Matt Keeler	3dbbd2d37d	Implement Client Agent Auto Config ... There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.	5 years ago
Daniel Nephin	c88fae0aac	ci: Add staticcheck and fix most errors ... Three of the checks are temporarily disabled to limit the size of the diff, and allow us to enable all the other checks in CI. In a follow up we can fix the issues reported by the other checks one at a time, and enable them.	5 years ago
Daniel Nephin	f9f6b14533	Convert the remaining calls to NewTestAgentWithFields ... After removing the t.Name() parameter with sed, convert the last few tests which use a custom name to call NewTestAgentWithFields instead.	5 years ago
Daniel Nephin	475659a132	Remove name from NewTestAgent ... Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'	5 years ago
R.B. Boyer	80b1165976	fix use of hclog logger (#7264)	5 years ago
Chris Piraino	401221de58	Allow users to configure either unstructured or JSON logging (#7130) ... * hclog Allow users to choose between unstructured and JSON logging	5 years ago
Paul Banks	87699eca2f	Fix support for RSA CA keys in Connect. (#6638) ... * Allow RSA CA certs for consul and vault providers to correctly sign EC leaf certs. * Ensure key type ad bits are populated from CA cert and clean up tests * Add integration test and fix error when initializing secondary CA with RSA key. * Add more tests, fix review feedback * Update docs with key type config and output * Apply suggestions from code review Co-Authored-By: R.B. Boyer <rb@hashicorp.com>	5 years ago
R.B. Boyer	af01d397a5	connect: don't colon-hex-encode the AuthorityKeyId and SubjectKeyId fields in connect certs (#6492) ... The fields in the certs are meant to hold the original binary representation of this data, not some ascii-encoded version. The only time we should be colon-hex-encoding fields is for display purposes or marshaling through non-TLS mediums (like RPC).	5 years ago
R.B. Boyer	f9496dc627	sdk: add freelist tracking and ephemeral port range skipping to freeport ... This should cut down on test flakiness. Problems handled: - If you had enough parallel test cases running, the former circular approach to handling the port block could hand out the same port to multiple cases before they each had a chance to bind them, leading to one of the two tests to fail. - The freeport library would allocate out of the ephemeral port range. This has been corrected for Linux (which should cover CI). - The library now waits until a formerly-in-use port is verified to be free before putting it back into circulation.	5 years ago
Mike Morris	65be58703c	connect: remove managed proxies (#6220) ... * connect: remove managed proxies implementation and all supporting config options and structs * connect: remove deprecated ProxyDestination * command: remove CONNECT_PROXY_TOKEN env var * agent: remove entire proxyprocess proxy manager * test: remove all managed proxy tests * test: remove irrelevant managed proxy note from TestService_ServerTLSConfig * test: update ContentHash to reflect managed proxy removal * test: remove deprecated ProxyDestination test * telemetry: remove managed proxy note * http: remove /v1/agent/connect/proxy endpoint * ci: remove deprecated test exclusion * website: update managed proxies deprecation page to note removal * website: remove managed proxy configuration API docs * website: remove managed proxy note from built-in proxy config * website: add note on removing proxy subdirectory of data_dir	5 years ago
Todd Radel	2552f4a11a	connect: Support RSA keys in addition to ECDSA (#6055) ... Support RSA keys in addition to ECDSA	5 years ago
Freddy	5873c56a03	Flaky test overhaul (#6100)	5 years ago
Hans Hasselberg	33a7df3330	tls: auto_encrypt enables automatic RPC cert provisioning for consul clients (#5597)	5 years ago
Pierre Souchay	4a4c63bda0	Ensure Consul is IPv6 compliant (#5468)	6 years ago
Matt Keeler	222afeae4c	Move the watch package into the api module (#5664) ... * Move the watch package into the api module It was already just a thin wrapper around the API anyways. The biggest change was to the testing. Instead of using a test agent directly from the agent package it now uses the binary on the PATH just like the other API tests. The other big changes were to fix up the connect based watch tests so that we didn’t need to pull in the connect package (and therefore all of Consul)	6 years ago