consul

Commit Graph

Author	SHA1	Message	Date
R.B. Boyer	72f991d8d3	agent: remove agent cache dependency from service mesh leaf certificate management (#17075 ) * agent: remove agent cache dependency from service mesh leaf certificate management This extracts the leaf cert management from within the agent cache. This code was produced by the following process: 1. All tests in agent/cache, agent/cache-types, agent/auto-config, agent/consul/servercert were run at each stage. - The tests in agent matching .Leaf were run at each stage. - The tests in agent/leafcert were run at each stage after they existed. 2. The former leaf cert Fetch implementation was extracted into a new package behind a "fake RPC" endpoint to make it look almost like all other cache type internals. 3. The old cache type was shimmed to use the fake RPC endpoint and generally cleaned up. 4. I selectively duplicated all of Get/Notify/NotifyCallback/Prepopulate from the agent/cache.Cache implementation over into the new package. This was renamed as leafcert.Manager. - Code that was irrelevant to the leaf cert type was deleted (inlining blocking=true, refresh=false) 5. Everything that used the leaf cert cache type (including proxycfg stuff) was shifted to use the leafcert.Manager instead. 6. agent/cache-types tests were moved and gently replumbed to execute as-is against a leafcert.Manager. 7. Inspired by some of the locking changes from derek's branch I split the fat lock into N+1 locks. 8. The waiter chan struct{} was eventually replaced with a singleflight.Group around cache updates, which was likely the biggest net structural change. 9. The awkward two layers or logic produced as a byproduct of marrying the agent cache management code with the leaf cert type code was slowly coalesced and flattened to remove confusion. 10. The .Leaf tests from the agent package were copied and made to work directly against a leafcert.Manager to increase direct coverage. I have done a best effort attempt to port the previous leaf-cert cache type's tests over in spirit, as well as to take the e2e-ish tests in the agent package with Leaf in the test name and copy those into the agent/leafcert package to get more direct coverage, rather than coverage tangled up in the agent logic. There is no net-new test coverage, just coverage that was pushed around from elsewhere.	1 year ago
Ronald	94ec4eb2f4	copyright headers for agent folder (#16704 ) * copyright headers for agent folder * Ignore test data files * fix proto files and remove headers in agent/uiserver folder * ignore deep-copy files	2 years ago
Matt Keeler	085c0addc0	Protobuf Refactoring for Multi-Module Cleanliness (#16302 ) Protobuf Refactoring for Multi-Module Cleanliness This commit includes the following: Moves all packages that were within proto/ to proto/private Rewrites imports to account for the packages being moved Adds in buf.work.yaml to enable buf workspaces Names the proto-public buf module so that we can override the Go package imports within proto/buf.yaml Bumps the buf version dependency to 1.14.0 (I was trying out the version to see if it would get around an issue - it didn't but it also doesn't break things and it seemed best to keep up with the toolchain changes) Why: In the future we will need to consume other protobuf dependencies such as the Google HTTP annotations for openapi generation or grpc-gateway usage. There were some recent changes to have our own ratelimiting annotations. The two combined were not working when I was trying to use them together (attempting to rebase another branch) Buf workspaces should be the solution to the problem Buf workspaces means that each module will have generated Go code that embeds proto file names relative to the proto dir and not the top level repo root. This resulted in proto file name conflicts in the Go global protobuf type registry. The solution to that was to add in a private/ directory into the path within the proto/ directory. That then required rewriting all the imports. Is this safe? AFAICT yes The gRPC wire protocol doesn't seem to care about the proto file names (although the Go grpc code does tack on the proto file name as Metadata in the ServiceDesc) Other than imports, there were no changes to any generated code as a result of this.	2 years ago
Matt Keeler	5afd4657ec	Protobuf Modernization (#15949 ) * Protobuf Modernization Remove direct usage of golang/protobuf in favor of google.golang.org/protobuf Marshallers (protobuf and json) needed some changes to account for different APIs. Moved to using the google.golang.org/protobuf/types/known/* for the well known types including replacing some custom Struct manipulation with whats available in the structpb well known type package. This also updates our devtools script to install protoc-gen-go from the right location so that files it generates conform to the correct interfaces. * Fix go-mod-tidy make target to work on all modules	2 years ago
Paul Glass	619032cfcd	Deprecate -join and -join-wan (#15598 )	2 years ago
Kyle Schochenmaier	bf0f61a878	removes ioutil usage everywhere which was deprecated in go1.16 (#15297 ) * update go version to 1.18 for api and sdk, go mod tidy * removes ioutil usage everywhere which was deprecated in go1.16 in favour of io and os packages. Also introduces a lint rule which forbids use of ioutil going forward. Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>	2 years ago
Chris S. Kim	bde57c0dd0	Regenerate files according to 1.19.2 formatter	2 years ago
R.B. Boyer	f507f62f3c	peering: initial sync (#12842 ) - Add endpoints related to peering: read, list, generate token, initiate peering - Update node/service/check table indexing to account for peers - Foundational changes for pushing service updates to a peer - Plumb peer name through Health.ServiceNodes path see: ENT-1765, ENT-1280, ENT-1283, ENT-1283, ENT-1756, ENT-1739, ENT-1750, ENT-1679, ENT-1709, ENT-1704, ENT-1690, ENT-1689, ENT-1702, ENT-1701, ENT-1683, ENT-1663, ENT-1650, ENT-1678, ENT-1628, ENT-1658, ENT-1640, ENT-1637, ENT-1597, ENT-1634, ENT-1613, ENT-1616, ENT-1617, ENT-1591, ENT-1588, ENT-1596, ENT-1572, ENT-1555 Co-authored-by: R.B. Boyer <rb@hashicorp.com> Co-authored-by: freddygv <freddy@hashicorp.com> Co-authored-by: Chris S. Kim <ckim@hashicorp.com> Co-authored-by: Evan Culver <eculver@hashicorp.com> Co-authored-by: Nitya Dhanushkodi <nitya@hashicorp.com>	3 years ago
Eric	e4b4f175ed	Bump go-control-plane * `go get cloud.google.com/go@v0.59.0` * `go get github.com/envoyproxy/go-control-plane@v0.9.9` * `make envoy-library` * Bumpprotoc to 3.15.8	3 years ago
Mike Morris	f8a2ae2606	agent: convert listener config to TLS types (#12522 ) * tlsutil: initial implementation of types/TLSVersion tlsutil: add test for parsing deprecated agent TLS version strings tlsutil: return TLSVersionInvalid with error tlsutil: start moving tlsutil cipher suite lookups over to types/tls tlsutil: rename tlsLookup to ParseTLSVersion, add cipherSuiteLookup agent: attempt to use types in runtime config agent: implement b.tlsVersion validation in config builder agent: fix tlsVersion nil check in builder tlsutil: update to renamed ParseTLSVersion and goTLSVersions tlsutil: fixup TestConfigurator_CommonTLSConfigTLSMinVersion tlsutil: disable invalid config parsing tests tlsutil: update tests auto_config: lookup old config strings from base.TLSMinVersion auto_config: update endpoint tests to use TLS types agent: update runtime_test to use TLS types agent: update TestRuntimeCinfig_Sanitize.golden agent: update config runtime tests to expect TLS types * website: update Consul agent tls_min_version values * agent: fixup TLS parsing and compilation errors * test: fixup lint issues in agent/config_runtime_test and tlsutil/config_test * tlsutil: add CHACHA20_POLY1305 cipher suites to goTLSCipherSuites * test: revert autoconfig tls min version fixtures to old format * types: add TLSVersions public function * agent: add warning for deprecated TLS version strings * agent: move agent config specific logic from tlsutil.ParseTLSVersion into agent config builder * tlsutil(BREAKING): change default TLS min version to TLS 1.2 * agent: move ParseCiphers logic from tlsutil into agent config builder * tlsutil: remove unused CipherString function * agent: fixup import for types package * Revert "tlsutil: remove unused CipherString function" This reverts commit `6ca7f6f58d`. * agent: fixup config builder and runtime tests * tlsutil: fixup one remaining ListenerConfig -> ProtocolConfig * test: move TLS cipher suites parsing test from tlsutil into agent config builder tests * agent: remove parseCiphers helper from auto_config_endpoint_test * test: remove unused imports from tlsutil * agent: remove resolved FIXME comment * tlsutil: remove TODO and FIXME in cipher suite validation * agent: prevent setting inherited cipher suite config when TLS 1.3 is specified * changelog: add entry for converting agent config to TLS types * agent: remove FIXME in runtime test, this is covered in builder tests with invalid tls9 value now * tlsutil: remove config tests for values checked at agent config builder boundary * tlsutil: remove tls version check from loadProtocolConfig * tlsutil: remove tests and TODOs for logic checked in TestBuilder_tlsVersion and TestBuilder_tlsCipherSuites * website: update search link for supported Consul agent cipher suites * website: apply review suggestions for tls_min_version description * website: attempt to clean up markdown list formatting for tls_min_version * website: moar linebreaks to fix tls_min_version formatting * Revert "website: moar linebreaks to fix tls_min_version formatting" This reverts commit `3858592742`. * autoconfig: translate old values for TLSMinVersion * agent: rename var for translated value of deprecated TLS version value * Update agent/config/deprecated.go Co-authored-by: Dan Upton <daniel@floppy.co> * agent: fix lint issue * agent: fixup deprecated config test assertions for updated warning Co-authored-by: Dan Upton <daniel@floppy.co>	3 years ago
Dhia Ayachi	72a997242b	split `pbcommon` to `pbcommon` and `pbcommongogo` (#12587 ) * mogify needed pbcommon structs * mogify needed pbconnect structs * fix compilation errors and make config_translate_test pass * add missing file * remove redundant oss func declaration * fix EnterpriseMeta to copy the right data for enterprise * rename pbcommon package to pbcommongogo * regenerate proto and mog files * add missing mog files * add pbcommon package * pbcommon no mog * fix enterprise meta code generation * fix enterprise meta code generation (pbcommongogo) * fix mog generation for gogo * use `protoc-go-inject-tag` to inject tags * rename proto package * pbcommon no mog * use `protoc-go-inject-tag` to inject tags * add non gogo proto to make file * fix proto get	3 years ago
Dan Upton	b36d4e16b6	Support per-listener TLS configuration ⚙️ (#12504 ) Introduces the capability to configure TLS differently for Consul's listeners/ports (i.e. HTTPS, gRPC, and the internal multiplexed RPC port) which is useful in scenarios where you may want the HTTPS or gRPC interfaces to present a certificate signed by a well-known/public CA, rather than the certificate used for internal communication which must have a SAN in the form `server.<dc>.consul`.	3 years ago
Eric	eea8300187	Remove the stdduration gogo extension	3 years ago
Dan Upton	7efab269c0	Rename `Master` and `AgentMaster` fields in config protobuf (#11764 )	3 years ago
Dan Upton	d8afd2f6c8	Rename `master` and `agent_master` ACL tokens in the config file format (#11665 )	3 years ago
R.B. Boyer	db91cbf484	auto-config: ensure the feature works properly with partitions (#11699 )	3 years ago
R.B. Boyer	1e02460bd1	re-run gofmt on 1.17 (#11579 ) This should let freshly recompiled golangci-lint binaries using Go 1.17 pass 'make lint'	3 years ago
freddygv	53ea1f634a	Ensure partition is handled by auto-encrypt	3 years ago
Daniel Nephin	d5498770fa	acl: make ACLDisabledTTL a constant This field was never user-configurable. We always overwrote the value with 120s from NonUserSource. However, we also never copied the value from RuntimeConfig to consul.Config, So the value in NonUserSource was always ignored, and we used the default value of 30s set by consul.DefaultConfig. All of this code is an unnecessary distraction because a user can not actually configure this value. This commit removes the fields and uses a constant value instad. Someone attempting to set acl.disabled_ttl in their config will now get an error about an unknown field, but previously the value was completely ignored, so the new behaviour seems more correct. We have to keep this field in the AutoConfig response for backwards compatibility, but the value will be ignored by the client, so it doesn't really matter what value we set.	3 years ago
R.B. Boyer	188e8dc51f	agent/structs: add a bunch more EnterpriseMeta helper functions to help with partitioning (#10669 )	3 years ago
R.B. Boyer	ed8a901be7	connect: include optional partition prefixes in SPIFFE identifiers (#10507 ) NOTE: this does not include any intentions enforcement changes yet	3 years ago
R.B. Boyer	a2876453a5	connect/ca: cease including the common name field in generated certs (#10424 ) As part of this change, we ensure that the SAN extensions are marked as critical when the subject is empty so that AWS PCA tolerates the loss of common names well and continues to function as a Connect CA provider. Parts of this currently hack around a bug in crypto/x509 and can be removed after https://go-review.googlesource.com/c/go/+/329129 lands in a Go release. Note: the AWS PCA tests do not run automatically, but the following passed locally for me: ENABLE_AWS_PCA_TESTS=1 go test ./agent/connect/ca -run TestAWS	3 years ago
Daniel Nephin	2aad3f80fb	tlsutil: reduce interface provided to auto-config Replace two methods with a single one that returns the cert. This moves more of the logic into the single caller (auto-config). tlsutil.Configurator is widely used. By keeping it smaller and focused only on storing and returning TLS config, we make the code easier to follow. These two methods were more related to auto-config than to tlsutil, so reducing the interface moves the logic closer to the feature that requires it.	3 years ago
Matt Keeler	caafc02449	hcs-1936: Prepare for adding license auto-retrieval to auto-config in enterprise	4 years ago
Daniel Nephin	97a577502d	config: improve the interface of Load This commit reduces the interface to Load() a bit, in preparation for unexporting NewBuilder and having everything call Load. The three arguments are reduced to a single argument by moving the other two into the options struct. The three return values are reduced to two by moving the RuntimeConfig and Warnings into a LoadResult struct.	4 years ago
Matt Keeler	4fa884ee3d	Fix flaky test by marking mock expectations as optional (#9596 ) These expectations are optional because in a slow CI environment the deadline to cancell the context might occur before the go routine reaches issuing the RPC. Either way we are successfully ensuring context cancellation is working.	4 years ago
Daniel Nephin	b9e60c0775	testing: skip slow tests with -short Add a skip condition to all tests slower than 100ms. This change was made using `gotestsum tool slowest` with data from the last 3 CI runs of master. See https://github.com/gotestyourself/gotestsum#finding-and-skipping-slow-tests With this change: ``` $ time go test -count=1 -short ./agent ok github.com/hashicorp/consul/agent 0.743s real 0m4.791s $ time go test -count=1 -short ./agent/consul ok github.com/hashicorp/consul/agent/consul 4.229s real 0m8.769s ```	4 years ago
Daniel Nephin	b27457dac8	ci: go-test-race switch to exclude list Most packages should pass the race detector. An exclude list ensures that new packages are automatically tested with -race. Also fix a couple small test races to allow more packages to be tested. Returning readyCh requires a lock because it can be set to nil, and setting it to nil will race without the lock. Move the TestServer.Listening calls around so that they properly guard setting TestServer.l. Otherwise it races. Remove t.Parallel in a small package. The entire package tests run in a few seconds, so t.Parallel does very little. In auto-config, wait for the AutoConfig.run goroutine to stop before calling readPersistedAutoConfig. Without this change there was a data race on reading ac.config.	4 years ago
Daniel Nephin	e54567223b	lib/retry: Refactor to reduce the interface surface Reduce Jitter to one function Rename NewRetryWaiter Fix a bug in calculateWait where maxWait was applied before jitter, which would make it possible to wait longer than maxWait.	4 years ago
Daniel Nephin	ca26dfb4a2	lib/retry: extract a new package from lib	4 years ago
Daniel Nephin	629e4aaa65	config: use token.Config for ACLToken config Using the target Config struct reduces the amount of copying and translating of configuration structs.	4 years ago
Matt Keeler	91d680b830	Merge of auto-config and auto-encrypt code (#8523 ) auto-encrypt is now handled as a special case of auto-config. This also is moving all the cert-monitor code into the auto-config package.	4 years ago
Daniel Nephin	d68edcecf4	testing: Remove all the defer os.Removeall Now that testutil uses t.Cleanup to remove the directory the caller no longer has to manage the removal	4 years ago
Daniel Nephin	37eacf8192	auto-config: reduce awareness of config This is a small step to allowing Agent to accept its dependencies instead of creating them in New. There were two fields in autoconfig.Config that were used exclusively to load config. These were replaced with a single function, allowing us to move LoadConfig back to the config package. Also removed the WithX functions for building a Config. Since these were simple assignment, it appeared we were not getting much value from them.	4 years ago
Daniel Nephin	4297a8ba07	auto-config: Avoid the marshal/unmarshal cycle in auto-config Use a LiteralConfig and return a config.Config from translate.	4 years ago
Daniel Nephin	38980ebb4c	config: Make Source an interface This will allow us to accept config from auto-config without needing to go through a serialziation cycle.	4 years ago
Matt Keeler	1a78cf9b4c	Ensure certificates retrieved through the cache get persisted with auto-config (#8409 )	4 years ago
Matt Keeler	34034b76f5	Agent Auto Config: Implement Certificate Generation (#8360 ) Most of the groundwork was laid in previous PRs between adding the cert-monitor package to extracting the logic of signing certificates out of the connect_ca_endpoint.go code and into a method on the server. This also refactors the auto-config package a bit to split things out into multiple files.	4 years ago
Matt Keeler	2713c0e682	Refactor the agentpb package (#8362 ) First move the whole thing to the top-level proto package name. Secondly change some things around internally to have sub-packages.	4 years ago
Matt Keeler	d2e4869c7c	Refactor AutoConfig RPC to not have a direct dependency on the Server type Instead it has an interface which can be mocked for better unit testing that is deterministic and not prone to flakiness.	4 years ago
Matt Keeler	be576c9737	Use the DNS and IP SANs from the auto config stanza when set	4 years ago
Matt Keeler	e8b39dd255	Overhaul the auto-config translation This fixes some issues around spurious warnings about using enterprise configuration in OSS.	4 years ago
Matt Keeler	4a5b352c18	Require enabling TLS to enable Auto Config (#8159 ) On the servers they must have a certificate. On the clients they just have to set verify_outgoing to true to attempt TLS connections for RPCs. Eventually we may relax these restrictions but right now all of the settings we push down (acl tokens, acl related settings, certificates, gossip key) are sensitive and shouldn’t be transmitted over an unencrypted connection. Our guides and docs should recoommend verify_server_hostname on the clients as well. Another reason to do this is weird things happen when making an insecure RPC when TLS is not enabled. Basically it tries TLS anyways. We should probably fix that to make it clearer what is going on.	4 years ago
Matt Keeler	d6e05482ab	Allow cancelling startup when performing auto-config (#8157 ) Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>	4 years ago
Matt Keeler	3dbbd2d37d	Implement Client Agent Auto Config There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.	4 years ago

45 Commits (47d445d680b6a6a2670b70dde74e9868caa35da5)