5156 Commits (62005369b54c63ccd7842c001de740ff3795d02c)

Author	SHA1	Message	Date
Dan Stough	33d898b857	[OSS] test: improve xDS listener code coverage (#18138) ... test: improve xDS listener code coverage	1 year ago
Ronald	bcc6a9d752	Use JWT-auth filter in metadata mode & Delegate validation to RBAC filter (#18062) ... ### Description <!-- Please describe why you're making this change, in plain English. --> - Currently the jwt-auth filter doesn't take into account the service identity when validating jwt-auth, it only takes into account the path and jwt provider during validation. This causes issues when multiple source intentions restrict access to an endpoint with different JWT providers. - To fix these issues, rather than use the JWT auth filter for validation, we use it in metadata mode and allow it to forward the successful validated JWT token payload to the RBAC filter which will make the decisions. This PR ensures requests with and without JWT tokens successfully go through the jwt-authn filter. The filter however only forwards the data for successful/valid tokens. On the RBAC filter level, we check the payload for claims and token issuer + existing rbac rules. ### Testing & Reproduction steps <!-- * In the case of bugs, describe how to replicate * If any manual tests were done, document the steps and the conditions to replicate * Call out any important/ relevant unit tests, e2e tests or integration tests you have added or are adding --> - This test covers a multi level jwt requirements (requirements at top level and permissions level). It also assumes you have envoy running, you have a redis and a sidecar proxy service registered, and have a way to generate jwks with jwt. I mostly use: https://www.scottbrady91.com/tools/jwt for this. - first write your proxy defaults ``` Kind = "proxy-defaults" name = "global" config { protocol = "http" } ``` - Create two providers ``` Kind = "jwt-provider" Name = "auth0" Issuer = "https://ronald.local" JSONWebKeySet = { Local = { JWKS = "eyJrZXlzIjog....." } } ``` ``` Kind = "jwt-provider" Name = "okta" Issuer = "https://ronald.local" JSONWebKeySet = { Local = { JWKS = "eyJrZXlzIjogW3...." } } ``` - add a service intention ``` Kind = "service-intentions" Name = "redis" JWT = { Providers = [ { Name = "okta" }, ] } Sources = [ { Name = "*" Permissions = [{ Action = "allow" HTTP = { PathPrefix = "/workspace" } JWT = { Providers = [ { Name = "okta" VerifyClaims = [ { Path = ["aud"] Value = "my_client_app" }, { Path = ["sub"] Value = "5be86359073c434bad2da3932222dabe" } ] }, ] } }, { Action = "allow" HTTP = { PathPrefix = "/" } JWT = { Providers = [ { Name = "auth0" }, ] } }] } ] ``` - generate 3 jwt tokens: 1 from auth0 jwks, 1 from okta jwks with different claims than `/workspace` expects and 1 with correct claims - connect to your envoy (change service and address as needed) to view logs and potential errors. You can add: `-- --log-level debug` to see what data is being forwarded ``` consul connect envoy -sidecar-for redis1 -grpc-addr 127.0.0.1:8502 ``` - Make the following requests: ``` curl -s -H "Authorization: Bearer $Auth0_TOKEN" --insecure --cert leaf.cert --key leaf.key --cacert connect-ca.pem https://localhost:20000/workspace -v RBAC filter denied curl -s -H "Authorization: Bearer $Okta_TOKEN_with_wrong_claims" --insecure --cert leaf.cert --key leaf.key --cacert connect-ca.pem https://localhost:20000/workspace -v RBAC filter denied curl -s -H "Authorization: Bearer $Okta_TOKEN_with_correct_claims" --insecure --cert leaf.cert --key leaf.key --cacert connect-ca.pem https://localhost:20000/workspace -v Successful request ``` ### TODO * [x] Update test coverage * [ ] update integration tests (follow-up PR) * [x] appropriate backport labels added	1 year ago
Poonam Jadhav	5930518489	fix: update delegateMock used in ENT (#18149) ... ### Description <!-- Please describe why you're making this change, in plain English. --> The mock is used in `http_ent_test` file which caused lint failures. For OSS->ENT parity adding the same change here. ### Links <!-- Include any links here that might be helpful for people reviewing your PR (Tickets, GH issues, API docs, external benchmarks, tools docs, etc). If there are none, feel free to delete this section. Please be mindful not to leak any customer or confidential information. HashiCorp employees may want to use our internal URL shortener to obfuscate links. --> Identified in OSS->ENT [merge PR](https://github.com/hashicorp/consul-enterprise/pull/6328) ### PR Checklist * [ ] ~updated test coverage~ * [ ] ~external facing docs updated~ * [x] appropriate backport labels added * [ ] ~not a security concern~	1 year ago
wangxinyi7	e7194787a7	re org resource type registry (#18133)	1 year ago
John Murret	691bc9673a	add a conditional around setting LANFilter.AllSegments to make sure it is valid (#18139) ... ### Description This is to correct a code problem because this assumes all segments, but when you get to Enterprise, you can be in partition that is not the default partition, in which case specifying all segments does not validate and fails. This is to correct the setting of this filter with `AllSegments` to `true` to only occur when in the the `default` partition. ### Testing & Reproduction steps <!-- * In the case of bugs, describe how to replicate * If any manual tests were done, document the steps and the conditions to replicate * Call out any important/ relevant unit tests, e2e tests or integration tests you have added or are adding --> ### Links <!-- Include any links here that might be helpful for people reviewing your PR (Tickets, GH issues, API docs, external benchmarks, tools docs, etc). If there are none, feel free to delete this section. Please be mindful not to leak any customer or confidential information. HashiCorp employees may want to use our internal URL shortener to obfuscate links. --> ### PR Checklist * [ ] updated test coverage * [ ] external facing docs updated * [ ] appropriate backport labels added * [ ] not a security concern	1 year ago
Chris S. Kim	747a4c73c1	Fix bug with Vault CA provider (#18112) ... Updating RootPKIPath but not IntermediatePKIPath would not update leaf signing certs with the new root. Unsure if this happens in practice but manual testing showed it is a bug that would break mesh and agent connections once the old root is pruned.	1 year ago
Poonam Jadhav	5208ea90e4	NET-4657/add resource service client (#18053) ... ### Description <!-- Please describe why you're making this change, in plain English. --> Dan had already started on this [task](https://github.com/hashicorp/consul/pull/17849) which is needed to start building the HTTP APIs. This just needed some cleanup to get it ready for review. Overview: - Rename `internalResourceServiceClient` to `insecureResourceServiceClient` for name consistency - Configure a `secureResourceServiceClient` with auth enabled ### PR Checklist * [ ] ~updated test coverage~ * [ ] ~external facing docs updated~ * [x] appropriate backport labels added * [ ] ~not a security concern~	1 year ago
Vijay	2f20c77e4d	Displays Consul version of each nodes in UI nodes section (#17754) ... * update UINodes and UINodeInfo response with consul-version info added as NodeMeta, fetched from serf members * update test cases TestUINodes, TestUINodeInfo * added nil check for map * add consul-version in local agent node metadata * get consul version from serf member and add this as node meta in catalog register request * updated ui mock response to include consul versions as node meta * updated ui trans and added version as query param to node list route * updates in ui templates to display consul version with filter and sorts * updates in ui - model class, serializers,comparators,predicates for consul version feature * added change log for Consul Version Feature * updated to get version from consul service, if for some reason not available from serf * updated changelog text * updated dependent testcases * multiselection version filter * Update agent/consul/state/catalog.go comments updated Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com> --------- Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com>	1 year ago
Tom Davies	f472164f05	Pass configured role name to Vault for AWS auth in Connect CA (#17885)	1 year ago
Dan Stough	da79997f3d	test: fix FIPS inline cert test message (#18076)	1 year ago
Dan Stough	1b08626358	[OSS] Fix initial_fetch_timeout to wait for all xDS resources (#18024) ... * fix(connect): set initial_fetch_time to wait indefinitely * changelog * PR feedback 1	1 year ago
Fulvio	f4b08040fd	Add verify server hostname to tls default (#17155)	1 year ago
Ronald	ada3938115	Add first integration test for jwt auth with intention (#18005)	1 year ago
Poonam Jadhav	8af4ad178c	feat: include nodes count in operator usage endpoint and cli command (#17939) ... * feat: update operator usage api endpoint to include nodes count * feat: update operator usange cli command to includes nodes count	1 year ago
Derek Menteer	0094dbf312	Fix incorrect protocol for transparent proxy upstreams. (#17894) ... This PR fixes a bug that was introduced in: https://github.com/hashicorp/consul/pull/16021 A user setting a protocol in proxy-defaults would cause tproxy implicit upstreams to not honor the upstream service's protocol set in its `ServiceDefaults.Protocol` field, and would instead always use the proxy-defaults value. Due to the fact that upstreams configured with "tcp" can successfully contact upstream "http" services, this issue was not recognized until recently (a proxy-defaults with "tcp" and a listening service with "http" would make successful requests, but not the opposite). As a temporary work-around, users experiencing this issue can explicitly set the protocol on the `ServiceDefaults.UpstreamConfig.Overrides`, which should take precedence. The fix in this PR removes the proxy-defaults protocol from the wildcard upstream that tproxy uses to configure implicit upstreams. When the protocol was included, it would always overwrite the value during discovery chain compilation, which was not correct. The discovery chain compiler also consumes proxy defaults to determine the protocol, so simply excluding it from the wildcard upstream config map resolves the issue.	1 year ago
Ronald	80394278b8	Expose JWKS cluster config through JWTProviderConfigEntry (#17978) ... * Expose JWKS cluster config through JWTProviderConfigEntry * fix typos, rename trustedCa to trustedCA	1 year ago
Chris Thain	0b1299c28d	Remove duplicate and unused newDecodeConfigEntry func (#17979)	1 year ago
Chris S. Kim	50a9d1b696	Remove POC code (#17974)	1 year ago
Ashesh Vidyut	2af6bc434a	feature - [NET - 4005] - [Supportability] Reloadable Configuration - enable_debug (#17565) ... * # This is a combination of 9 commits. # This is the 1st commit message: init without tests # This is the commit message #2: change log # This is the commit message #3: fix tests # This is the commit message #4: fix tests # This is the commit message #5: added tests # This is the commit message #6: change log breaking change # This is the commit message #7: removed breaking change # This is the commit message #8: fix test # This is the commit message #9: keeping the test behaviour same * # This is a combination of 12 commits. # This is the 1st commit message: init without tests # This is the commit message #2: change log # This is the commit message #3: fix tests # This is the commit message #4: fix tests # This is the commit message #5: added tests # This is the commit message #6: change log breaking change # This is the commit message #7: removed breaking change # This is the commit message #8: fix test # This is the commit message #9: keeping the test behaviour same # This is the commit message #10: made enable debug atomic bool # This is the commit message #11: fix lint # This is the commit message #12: fix test true enable debug * parent 10f500e895 author absolutelightning <ashesh.vidyut@hashicorp.com> 1687352587 +0530 committer absolutelightning <ashesh.vidyut@hashicorp.com> 1687352592 +0530 init without tests change log fix tests fix tests added tests change log breaking change removed breaking change fix test keeping the test behaviour same made enable debug atomic bool fix lint fix test true enable debug using enable debug in agent as atomic bool test fixes fix tests fix tests added update on correct locaiton fix tests fix reloadable config enable debug fix tests fix init and acl 403 * revert commit	1 year ago
Ronald	1512ea307e	Dynamically create jwks clusters for jwt-providers (#17944)	1 year ago
Ranjandas	1b1f33f224	Fixes Secondary ConnectCA update (#17846) ... This fixes a bug that was identified which resulted in subsequent ConnectCA configuration update not to persist in the cluster.	1 year ago
John Maguire	67a239a821	Ensure RSA keys are at least 2048 bits in length (#17911) ... * Ensure RSA keys are at least 2048 bits in length * Add changelog * update key length check for FIPS compliance * Fix no new variables error and failing to return when error exists from validating * clean up code for better readability * actually return value	1 year ago
Ronald	767ef2dd4c	Allow service identity tokens the ability to read jwt-providers (#17893) ... * Allow service identity tokens the ability to read jwt-providers * more tests * service_prefix tests	1 year ago
Alex Simenduev	33a2d90852	Fix a bug that wrongly trims domains when there is an overlap with DC name (#17160) ... * Fix a bug that wrongly trims domains when there is an overlap with DC name Before this change, when DC name and domain/alt-domain overlap, the domain name incorrectly trimmed from the query. Example: Given: datacenter = dc-test, alt-domain = test.consul. Querying for "test-node.node.dc-test.consul" will faile, because the code was trimming "test.consul" instead of just ".consul" This change, fixes the issue by adding dot (.) before trimming * trimDomain: ensure domain trimmed without modyfing original domains * update changelog --------- Co-authored-by: Dhia Ayachi <dhia@hashicorp.com>	1 year ago
Dan Upton	b117eb0126	resource: enforce consistent naming of resource types (#17611) ... For consistency, resource type names must follow these rules: - `Group` must be snake case, and in most cases a single word. - `GroupVersion` must be lowercase, start with a "v" and end with a number. - `Kind` must be pascal case. These were chosen because they map to our protobuf type naming conventions.	1 year ago
cskh	f16c5d87ab	watch: support -filter for consul watch: checks, services, nodes, service (#17780) ... * watch: support -filter for watch checks * Add filter for watch nodes, services, and service - unit test added - Add changelog - update doc	1 year ago
Chris Thain	366bd6f89f	ext-authz Envoy extension: support `localhost` as a valid target URI. (#17821)	1 year ago
Chris S. Kim	a4653de8da	CA provider doc updates and Vault provider minor update (#17831) ... Update CA provider docs Clarify that providers can differ between primary and secondary datacenters Provide a comparison chart for consul vs vault CA providers Loosen Vault CA provider validation for RootPKIPath Update Vault CA provider documentation	1 year ago
George Bolo	82441a27fa	fixes #17732 - AccessorID in request body should be optional when updating ACL token (#17739) ... * AccessorID in request body should be optional when updating ACL token * add a test case * fix test case * add changelog entry for PR #17739	1 year ago
Eric Haberkorn	a3ba559149	Make locality aware routing xDS changes (#17826)	1 year ago
Paul Glass	d2363eb711	Test permissive mTLS filter chain not configured with tproxy disabled (#17747)	1 year ago
chappie	5352ccf8ed	HCP Add node id/name to config (#17750)	1 year ago
Ronald	5f95f5f6d8	Stop referenced jwt providers from being deleted (#17755) ... * Stop referenced jwt providers from being deleted	1 year ago
Michael Zalimeni	265c003033	Add Patch index to Prop Override validation errors (#17777) ... When a patch is found invalid, include its index for easier debugging when multiple patches are provided.	1 year ago
Michael Zalimeni	f9aa7aebb3	Property Override validation improvements (#17759) ... * Reject inbound Prop Override patch with Services Services filtering is only supported for outbound TrafficDirection patches. * Improve Prop Override unexpected type validation - Guard against additional invalid parent and target types - Add specific error handling for Any fields (unsupported)	1 year ago
Derek Menteer	04edace1de	Fix issue with streaming service health watches. (#17775) ... Fix issue with streaming service health watches. This commit fixes an issue where the health streams were unaware of service export changes. Whenever an exported-services config entry is modified, it is effectively an ACL change. The bug would be triggered by the following situation: - no services are exported - an upstream watch to service X is spawned - the streaming backend filters out data for service X (due to lack of exports) - service X is finally exported In the situation above, the streaming backend does not trigger a refresh of its data. This means that any events that were supposed to have been received prior to the export are NOT backfilled, and the watches never see service X spawning. We currently have decided to not trigger a stream refresh in this situation due to the potential for a thundering herd effect (touching exports would cause a re-fetch of all watches for that partition, potentially). Therefore, a local blocking-query approach was added by this commit for agentless. It's also worth noting that the streaming subscription is currently bypassed most of the time with agentful, because proxycfg has a `req.Source.Node != ""` which prevents the `streamingEnabled` check from passing. This means that while agents should technically have this same issue, they don't experience it with mesh health watches. Note that this is a temporary fix that solves the issue for proxycfg, but not service-discovery use cases.	1 year ago
Eric Haberkorn	0994ccf162	validate localities on agent configs and registration endpoints (#17712)	1 year ago
chappie	7ab287c1d5	Add truncation to body (#17723)	1 year ago
Chris Thain	9289e680d6	OSS merge: Update error handling login when applying extensions (#17740)	1 year ago
Ashesh Vidyut	fa40654885	[NET-3865] [Supportability] Additional Information in the output of 'consul operator raft list-peers' (#17582) ... * init * fix tests * added -detailed in docs * added change log * fix doc * checking for entry in map * fix tests * removed detailed flag * removed detailed flag * revert unwanted changes * removed unwanted changes * updated change log * pr review comment changes * pr comment changes single API instead of two * fix change log * fix tests * fix tests * fix test operator raft endpoint test * Update .changelog/17582.txt Co-authored-by: Semir Patel <semir.patel@hashicorp.com> * nits * updated docs --------- Co-authored-by: Semir Patel <semir.patel@hashicorp.com>	1 year ago
R.B. Boyer	72f991d8d3	agent: remove agent cache dependency from service mesh leaf certificate management (#17075) ... * agent: remove agent cache dependency from service mesh leaf certificate management This extracts the leaf cert management from within the agent cache. This code was produced by the following process: 1. All tests in agent/cache, agent/cache-types, agent/auto-config, agent/consul/servercert were run at each stage. - The tests in agent matching .*Leaf were run at each stage. - The tests in agent/leafcert were run at each stage after they existed. 2. The former leaf cert Fetch implementation was extracted into a new package behind a "fake RPC" endpoint to make it look almost like all other cache type internals. 3. The old cache type was shimmed to use the fake RPC endpoint and generally cleaned up. 4. I selectively duplicated all of Get/Notify/NotifyCallback/Prepopulate from the agent/cache.Cache implementation over into the new package. This was renamed as leafcert.Manager. - Code that was irrelevant to the leaf cert type was deleted (inlining blocking=true, refresh=false) 5. Everything that used the leaf cert cache type (including proxycfg stuff) was shifted to use the leafcert.Manager instead. 6. agent/cache-types tests were moved and gently replumbed to execute as-is against a leafcert.Manager. 7. Inspired by some of the locking changes from derek's branch I split the fat lock into N+1 locks. 8. The waiter chan struct{} was eventually replaced with a singleflight.Group around cache updates, which was likely the biggest net structural change. 9. The awkward two layers or logic produced as a byproduct of marrying the agent cache management code with the leaf cert type code was slowly coalesced and flattened to remove confusion. 10. The .*Leaf tests from the agent package were copied and made to work directly against a leafcert.Manager to increase direct coverage. I have done a best effort attempt to port the previous leaf-cert cache type's tests over in spirit, as well as to take the e2e-ish tests in the agent package with Leaf in the test name and copy those into the agent/leafcert package to get more direct coverage, rather than coverage tangled up in the agent logic. There is no net-new test coverage, just coverage that was pushed around from elsewhere.	1 year ago
Eric Haberkorn	0a1efe73f3	Refactor disco chain prioritize by locality structs (#17696) ... This includes prioritize by localities on disco chain targets rather than resolvers, allowing different targets within the same partition to have different policies.	1 year ago
Dan Stough	bba5cd8455	fix: stop peering delete routine on leader loss (#17483)	1 year ago
Chris Thain	a8f1350835	ENT merge of ext-authz extension updates (#17684)	1 year ago
Chris Thain	c04c122ef3	Default `ProxyType` for builtin extensions (#17657)	1 year ago
Nathan Coleman	1074252361	api-gateway: stop adding all header filters to virtual host when generating xDS (#17644) ... * Add header filter to api-gateway xDS golden test * Stop adding all header filters to virtual host when generating xDS for api-gateway * Regenerate xDS golden file for api-gateway w/ header filter	1 year ago
Matt Keeler	baaf6d84c7	Add generic experiments configuration and use it to enable catalog v2 resources (#17604) ... * Add generic experiments configuration and use it to enable catalog v2 resources * Run formatting with -s as CI will validate that this has been done	1 year ago
R.B. Boyer	ec347ef01d	sort some imports that are wonky between oss and ent (#17637)	1 year ago
Andrew Stucki	3cb70566a9	[API Gateway] Fix rate limiting for API gateways (#17631) ... * [API Gateway] Fix rate limiting for API gateways * Add changelog * Fix failing unit tests * Fix operator usage tests for api package	1 year ago
Andrew Stucki	9a4f503b2b	[API Gateway] Fix trust domain for external peered services in synthesis code (#17609) ... * [API Gateway] Fix trust domain for external peered services in synthesis code * Add changelog	1 year ago