156 Commits (43bd3512f0571214dd32da4b079c3a842d7b11eb)

Author	SHA1	Message	Date
John Eikenberry	e8eec1fa80	add provider ca auth support for kubernetes ... Adds support for Kubernetes jwt/token file based auth. Only needs to read the file and save the contents as the jwt/token.	2 years ago
John Eikenberry	4211069080	add provider ca support for jwt file base auth ... Adds support for a jwt token in a file. Simply reads the file and sends the read in jwt along to the vault login. It also supports a legacy mode with the jwt string being passed directly. In which case the path is made optional.	2 years ago
John Eikenberry	4f2d9a91e5	add provider ca auth-method support for azure ... Does the required dance with the local HTTP endpoint to get the required data for the jwt based auth setup in Azure. Keeps support for 'legacy' mode where all login data is passed on via the auth methods parameters. Refactored check for hardcoded /login fields.	2 years ago
John Eikenberry	ed7367b6f4	remove redundant vault api retry logic (#16143) ... remove redundant vault api retry logic We upgraded Vault API module version to a version that has built-in retry logic. So this code is no longer necessary. Also add mention of re-configuring the provider in comments.	2 years ago
John Eikenberry	5c836f2aa9	fix goroutine leak in renew testing (#16142) ... fix goroutine leak in renew testing Test overwrote the stopWatcher() function variable for the test without keeping and calling the original value. The original value is the function that stops the goroutine... so it needs to be called.	2 years ago
Chris Thain	2f4c8e50f2	Support Vault agent auth config for AWS/GCP CA provider auth (#15970)	2 years ago
Dan Upton	7a55de375c	xds: don't attempt to load-balance sessions for local proxies (#15789) ... Previously, we'd begin a session with the xDS concurrency limiter regardless of whether the proxy was registered in the catalog or in the server's local agent state. This caused problems for users who run `consul connect envoy` directly against a server rather than a client agent, as the server's locally registered proxies wouldn't be included in the limiter's capacity. Now, the `ConfigSource` is responsible for beginning the session and we only do so for services in the catalog. Fixes: https://github.com/hashicorp/consul/issues/15753	2 years ago
R.B. Boyer	5af94fb2a0	connect: use -dev-no-store-token for test vaults to reduce source of flakes (#15691) ... It turns out that by default the dev mode vault server will attempt to interact with the filesystem to store the provided root token. If multiple vault instances are running they'll all awkwardly share the filesystem and if timing results in one server stopping while another one is starting then the starting one will error with: Error initializing Dev mode: rename /home/circleci/.vault-token.tmp /home/circleci/.vault-token: no such file or directory This change uses `-dev-no-store-token` to bypass that source of flakes. Also the stdout/stderr from the vault process is included if the test fails. The introduction of more `t.Parallel` use in https://github.com/hashicorp/consul/pull/15669 increased the likelihood of this failure, but any of the tests with multiple vaults in use (or running multiple package tests in parallel that all use vault) were eventually going to flake on this.	2 years ago
R.B. Boyer	900584ca82	connect: ensure all vault connect CA tests use limited privilege tokens (#15669) ... All of the current integration tests where Vault is the Connect CA now use non-root tokens for the test. This helps us detect privilege changes in the vault model so we can keep our guides up to date. One larger change was that the RenewIntermediate function got refactored slightly so it could be used from a test, rather than the large duplicated function we were testing in a test which seemed error prone.	2 years ago
R.B. Boyer	4940a728ab	Detect Vault 1.11+ import in secondary datacenters and update default issuer (#15661) ... The fix outlined and merged in #15253 fixed the issue as it occurs in the primary DC. There is a similar issue that arises when vault is used as the Connect CA in a secondary datacenter that is fixed by this PR. Additionally: this PR adds support to run the existing suite of vault related integration tests against the last 4 versions of vault (1.9, 1.10, 1.11, 1.12)	2 years ago
Chris S. Kim	c9ec9fa320	Fix Vault managed intermediate PKI bug (#15525)	2 years ago
Alexander Scheel	2b90307f6d	Detect Vault 1.11+ import, update default issuer (#15253) ... Consul used to rely on implicit issuer selection when calling Vault endpoints to issue new CSRs. Vault 1.11+ changed that behavior, which caused Consul to check the wrong (previous) issuer when renewing its Intermediate CA. This patch allows Consul to explicitly set a default issuer when it detects that the response from Vault is 1.11+. Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com> Co-authored-by: Chris S. Kim <ckim@hashicorp.com>	2 years ago
Kyle Schochenmaier	bf0f61a878	removes ioutil usage everywhere which was deprecated in go1.16 (#15297) ... * update go version to 1.18 for api and sdk, go mod tidy * removes ioutil usage everywhere which was deprecated in go1.16 in favour of io and os packages. Also introduces a lint rule which forbids use of ioutil going forward. Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>	2 years ago
Kyle Havlovitz	d122108992	Warn instead of returning an error when intermediate mount tune permission is missing	2 years ago
Kyle Havlovitz	d67bccd210	Update intermediate pki mount/role when reconfiguring Vault provider	2 years ago
Freddy	74ca6406ea	Configure upstream TLS context with peer root certs (#13321) ... For mTLS to work between two proxies in peered clusters with different root CAs, proxies need to configure their outbound listener to use different root certificates for validation. Up until peering was introduced proxies would only ever use one set of root certificates to validate all mesh traffic, both inbound and outbound. Now an upstream proxy may have a leaf certificate signed by a CA that's different from the dialing proxy's. This PR makes changes to proxycfg and xds so that the upstream TLS validation uses different root certificates depending on which cluster is being dialed.	3 years ago
Mark Anderson	c6ff4ba7d8	Support vault namespaces in connect CA (#12904) ... * Support vault namespaces in connect CA Follow on to some missed items from #12655 From an internal ticket "Support standard "Vault namespace in the path" semantics for Connect Vault CA Provider" Vault allows the namespace to be specified as a prefix in the path of a PKI definition, but our usage of the Vault API includes calls that don't support a namespaced key. In particular the sys.* family of calls simply appends the key, instead of prefixing the namespace in front of the path. Unfortunately it is difficult to reliably parse a path with a namespace; only vault knows what namespaces are present, and the '/' separator can be inside a key name, as well as separating path elements. This is in use in the wild; for example 'dc1/intermediate-key' is a relatively common naming schema. Instead we add two new fields: RootPKINamespace and IntermediatePKINamespace, which are the absolute namespace paths 'prefixed' in front of the respective PKI Paths. Signed-off-by: Mark Anderson <manderson@hashicorp.com>	3 years ago
R.B. Boyer	4274e67b47	chore: upgrade mockery to v2 and regenerate (#12836)	3 years ago
John Murret	a1117261df	set vault namespaces on vault client prior to logging in with the vault auth method	3 years ago
Mark Anderson	018edc222e	Avoid using sys/mounts to enable namespaces (#12655) ... * Avoid doing list of /sys/mounts From an internal ticket "Support standard "Vault namespace in the path" semantics for Connect Vault CA Provider" Vault allows the namespace to be specified as a prefix in the path of a PKI definition, but this doesn't currently work for ```IntermediatePKIPath``` specifications, because we attempt to list all of the paths to check if ours is already defined. This doesn't really work in a namespaced world. This changes the IntermediatePKIPath code to follow the same pattern as the root key, where we directly get the key rather than listing. This code is difficult to write automated tests for because it relies on features of Vault Enterprise, which isn't currently part of our test framework, so it was tested manually. Signed-off-by: Mark Anderson <manderson@hashicorp.com> * add changelog Signed-off-by: Mark Anderson <manderson@hashicorp.com>	3 years ago
Connor	922619dfc3	Fix leaked Vault LifetimeRenewers (#12607) ... * Fix leaked Vault LifetimeRenewers When the Vault CA Provider is reconfigured we do not stop the LifetimeRenewers which can cause them to leak until the Consul processes recycles. On Configure execute stopWatcher if it exists and is not nil before starting a new renewal * Add jitter before restarting the LifetimeWatcher If we fail to login to Vault or our token is no longer valid we can overwhelm a Vault instance with many requests very quickly by restarting the LifetimeWatcher. Before restarting the LifetimeWatcher provide a backoff time of 1 second or less. * Use a retry.Waiter instead of RandomStagger * changelog * gofmt'd * Swap out bool for atomic.Unit32 in test * Provide some extra clarification in comment and changelog	3 years ago
Daniel Nephin	1f00ede559	ca: require that tests that use Vault are named correctly ... Previously we were using two different criteria to decide where to run a test. The main `go-test` job would skip Vault tests based on the presence of the `vault` binary, but the `test-connect-ca-providers` job would run tests based on the name. This led to a scenario where a test may never run in CI. To fix this problem I added a name check to the function we use to skip the test. This should ensure that any test that requires vault is named correctly to be run as part of the `test-connect-ca-providers` job. At the same time I relaxed the regex we use. I verified this runs the same tests using `go test --list Vault`. I made this change because a bunch of tests in `agent/connect/ca` used `Vault` in the name, without the underscores. Instead of changing a bunch of test names, this seemed easier. With this approach, the worst case is that we run a few extra tests in the `test-connect-ca-providers` job, which doesn't seem like a problem.	3 years ago
Daniel Nephin	6b679aa9d4	Update TODOs to reference an issue with more details ... And remove a no longer needed TODO	3 years ago
Daniel Nephin	5e8ea2a039	ca: add a test for secondary with external CA	3 years ago
Daniel Nephin	42ec34d101	ca: examine the full chain in newCARoot ... make TestNewCARoot much more strict compare the full result instead of only a few fields. add a test case with 2 and 3 certificates in the pem	3 years ago
Daniel Nephin	71f3ae04e2	ca: small docs improvements	3 years ago
Daniel Nephin	86994812ed	ca: cleanup validateSetIntermediate	3 years ago
Daniel Nephin	c1c1580bf8	ca: only return the leaf cert from Sign in vault provider ... The interface is documented as 'Sign will only return the leaf', and the other providers only return the leaf. It seems like this was added during the initial implementation, so is likely just something we missed. It doesn't break anything , but it does cause confusing cert chains in the API response which could break something in the future.	3 years ago
FFMMM	78264a8030	Vendor in rpc mono repo for net/rpc fork, go-msgpack, msgpackrpc. (#12311) ... This commit syncs ENT changes to the OSS repo. Original commit details in ENT: ``` commit 569d25f7f4578981c3801e6e067295668210f748 Author: FFMMM <FFMMM@users.noreply.github.com> Date: Thu Feb 10 10:23:33 2022 -0800 Vendor fork net rpc (#1538) * replace net/rpc w consul-net-rpc/net/rpc Signed-off-by: FFMMM <FFMMM@users.noreply.github.com> * replace msgpackrpc and go-msgpack with fork from mono repo Signed-off-by: FFMMM <FFMMM@users.noreply.github.com> * gofmt all files touched Signed-off-by: FFMMM <FFMMM@users.noreply.github.com> ``` Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>	3 years ago
Daniel Nephin	51b0f82d0e	Make test more readable ... And fix typo	3 years ago
Daniel Nephin	608597c7b6	ca: relax and move private key type/bit validation for vault ... This commit makes two changes to the validation. Previously we would call this validation in GenerateRoot, which happens both on initialization (when a follower becomes leader), and when a configuration is updated. We only want to do this validation during config update so the logic was moved to the UpdateConfiguration function. Previously we would compare the config values against the actual cert. This caused problems when the cert was created manually in Vault (not created by Consul). Now we compare the new config against the previous config. Using a already created CA cert should never error now. Adding the key bit and types to the config should only error when the previous values were not the defaults.	3 years ago
Daniel Nephin	7839b2d7e0	ca: add a test that uses an intermediate CA as the primary CA ... This test found a bug in the secondary. We were appending the root cert to the PEM, but that cert was already appended. This was failing validation in Vault here: https://github.com/hashicorp/vault/blob/sdk/v0.3.0/sdk/helper/certutil/types.go#L329 Previously this worked because self signed certs have the same SubjectKeyID and AuthorityKeyID. So having the same self-signed cert repeated doesn't fail that check. However with an intermediate that is not self-signed, those values are different, and so we fail the check. A test I added in a previous commit should show that this continues to work with self-signed root certs as well.	3 years ago
Daniel Nephin	9b7468f99e	ca/provider: remove ActiveRoot from Provider	3 years ago
Daniel Nephin	c2b9c81a55	ca: update MockProvider for new interface	3 years ago
Daniel Nephin	f05bad4a1d	ca: update GenerateRoot godoc	3 years ago
R.B. Boyer	b60d89e7ef	bulk rewrite using this script ... set -euo pipefail unset CDPATH cd "$(dirname "$0")" for f in $(git grep '\brequire := require\.New(' | cut -d':' -f1 | sort -u); do echo "=== require: $f ===" sed -i '/require := require.New(t)/d' $f # require.XXX(blah) but not require.XXX(tblah) or require.XXX(rblah) sed -i 's/\brequire\.\([a-zA-Z0-9_]*\)(\([^tr]\)/require.\1(t,\2/g' $f # require.XXX(tblah) but not require.XXX(t, blah) sed -i 's/\brequire\.\([a-zA-Z0-9_]*\)(\(t[^,]\)/require.\1(t,\2/g' $f # require.XXX(rblah) but not require.XXX(r, blah) sed -i 's/\brequire\.\([a-zA-Z0-9_]*\)(\(r[^,]\)/require.\1(t,\2/g' $f gofmt -s -w $f done for f in $(git grep '\bassert := assert\.New(' | cut -d':' -f1 | sort -u); do echo "=== assert: $f ===" sed -i '/assert := assert.New(t)/d' $f # assert.XXX(blah) but not assert.XXX(tblah) or assert.XXX(rblah) sed -i 's/\bassert\.\([a-zA-Z0-9_]*\)(\([^tr]\)/assert.\1(t,\2/g' $f # assert.XXX(tblah) but not assert.XXX(t, blah) sed -i 's/\bassert\.\([a-zA-Z0-9_]*\)(\(t[^,]\)/assert.\1(t,\2/g' $f # assert.XXX(rblah) but not assert.XXX(r, blah) sed -i 's/\bassert\.\([a-zA-Z0-9_]*\)(\(r[^,]\)/assert.\1(t,\2/g' $f gofmt -s -w $f done	3 years ago
Daniel Nephin	4116a143e0	fix misleading errors on vault shutdown	3 years ago
Daniel Nephin	214dcf8d0d	ca: use the real FSM operation in tests ... Previously we had a couple copies that reproduced the FSM operation. These copies introduce risk that the test does not accurately match production. This PR removes the test versions of the FSM operation, and exports the real production FSM operation so that it can be used in tests. The consul provider tests did need to change because of this. Previously we would return a hardcoded value of 2, but in production this value is always incremented.	3 years ago
Daniel Nephin	d795a73f78	testing: use the new freeport interfaces	3 years ago
Daniel Nephin	b92084b8e8	ca: reduce consul provider backend interface a bit ... This makes it easier to fake, which will allow me to use the ConsulProvider as an 'external PKI' to test a customer setup where the actual root CA is not the root we use for the Consul CA. Replaces a call to the state store to fetch the clusterID with the clusterID field already available on the built-in provider.	3 years ago
Iryna Shustava	0ee456649f	connect: Support auth methods for the vault connect CA provider (#11573) ... * Support vault auth methods for the Vault connect CA provider * Rotate the token (re-authenticate to vault using auth method) when the token can no longer be renewed	3 years ago
Daniel Nephin	b4080bc0dc	ca: use the cluster ID passed to the primary ... instead of fetching it from the state store.	3 years ago
Daniel Nephin	b9ab9bae12	ca: accept only the cluster ID to SpiffeIDSigningForCluster ... To make it more obivous where ClusterID is used, and remove the need to create a struct when only one field is used.	3 years ago
Connor	efe4b21287	Support Vault Namespaces explicitly in CA config (#11477) ... * Support Vault Namespaces explicitly in CA config If there is a Namespace entry included in the Vault CA configuration, set it as the Vault Namespace on the Vault client Currently the only way to support Vault namespaces in the Consul CA config is by doing one of the following: 1) Set the VAULT_NAMESPACE environment variable which will be picked up by the Vault API client 2) Prefix all Vault paths with the namespace Neither of these are super pleasant. The first requires direct access and modification to the Consul runtime environment. It's possible and expected, not super pleasant. The second requires more indepth knowledge of Vault and how it uses Namespaces and could be confusing for anyone without that context. It also infers that it is not supported * Add changelog * Remove fmt.Fprint calls * Make comment clearer * Add next consul version to website docs * Add new test for default configuration * go mod tidy * Add skip if vault not present * Tweak changelog text	3 years ago
FFMMM	61bd417a82	plumb thru root cert tll to the aws ca provider (#11449) ... * plumb thru root cert ttl to the aws ca provider Signed-off-by: FFMMM <FFMMM@users.noreply.github.com> * Update .changelog/11449.txt Co-authored-by: Dhia Ayachi <dhia@hashicorp.com> Co-authored-by: Dhia Ayachi <dhia@hashicorp.com>	3 years ago
FFMMM	6004a21f35	fix aws pca certs (#11470) ... Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>	3 years ago
FFMMM	4ddf973a31	add root_cert_ttl option for consul connect, vault ca providers (#11428) ... * add root_cert_ttl option for consul connect, vault ca providers Signed-off-by: FFMMM <FFMMM@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Chris S. Kim <ckim@hashicorp.com> * add changelog, pr feedback Signed-off-by: FFMMM <FFMMM@users.noreply.github.com> * Update .changelog/11428.txt, more docs Co-authored-by: Daniel Nephin <dnephin@hashicorp.com> * Update website/content/docs/agent/options.mdx Co-authored-by: Kyle Havlovitz <kylehav@gmail.com> Co-authored-by: Chris S. Kim <ckim@hashicorp.com> Co-authored-by: Daniel Nephin <dnephin@hashicorp.com> Co-authored-by: Kyle Havlovitz <kylehav@gmail.com>	3 years ago
Daniel Nephin	b2f49279e2	ca: split Primary/Secondary Provider ... To make it more clear which methods are necessary for each scenario. This can also prevent problems which force all DCs to use the same Vault instance, which is currently a problem.	3 years ago
Dhia Ayachi	58bd817336	check expiry date of the root/intermediate before using it to sign a leaf (#10500) ... * ca: move provider creation into CAManager This further decouples the CAManager from Server. It reduces the interface between them and removes the need for the SetLogger method on providers. * ca: move SignCertificate to CAManager To reduce the scope of Server, and keep all the CA logic together * ca: move SignCertificate to the file where it is used * auto-config: move autoConfigBackend impl off of Server Most of these methods are used exclusively for the AutoConfig RPC endpoint. This PR uses a pattern that we've used in other places as an incremental step to reducing the scope of Server. * fix linter issues * check error when `raftApplyMsgpack` * ca: move SignCertificate to CAManager To reduce the scope of Server, and keep all the CA logic together * check expiry date of the intermediate before using it to sign a leaf * fix typo in comment Co-authored-by: Kyle Havlovitz <kylehav@gmail.com> * Fix test name * do not check cert start date * wrap error to mention it is the intermediate expired * Fix failing test * update comment Co-authored-by: Daniel Nephin <dnephin@hashicorp.com> * use shim to avoid sleep in test * add root cert validation * remove duplicate code * Revert "fix linter issues" This reverts commit 6356302b54. * fix import issue * gofmt leader_connect_ca * add changelog entry * update error message Co-authored-by: Freddy <freddygv@users.noreply.github.com> * fix error message in test Co-authored-by: Daniel Nephin <dnephin@hashicorp.com> Co-authored-by: Kyle Havlovitz <kylehav@gmail.com> Co-authored-by: Freddy <freddygv@users.noreply.github.com>	3 years ago
R.B. Boyer	6c47efd532	connect/ca: ensure edits to the key type/bits for the connect builtin CA will regenerate the roots (#10330) ... progress on #9572	3 years ago