Change logging of registered v2 resource endpoints to add /api prefix (#20352)
* Change logging of registered v2 resource endpoints to add /api prefix
Previous:
agent.http: Registered resource endpoint: endpoint=/demo/v1/executive
New:
agent.http: Registered resource endpoint: endpoint=/api/demo/v1/executive
This reduces confusion when attempting to call the APIs after looking at
the logs.
NET-5824 Exported services api (#20015)
* Exported services api implemented
* Tests added, refactored code
* Adding server tests
* changelog added
* Proto gen added
* Adding codegen changes
* changing url, response object
* Fixing lint error by having namespace and partition directly
* Tests changes
* refactoring tests
* Simplified uniqueness logic for exported services, sorted the response in order of service name
* Fix lint errors, refactored code
* [Cloud][CC-6925] Updates to pushing server state (#19682)
* Upgrade hcp-sdk-go to latest version v0.73
Changes:
- go get github.com/hashicorp/hcp-sdk-go
- go mod tidy
* From upgrade: regenerate protobufs for upgrade from 1.30 to 1.31
Ran: `make proto`
Slack: https://hashicorp.slack.com/archives/C0253EQ5B40/p1701105418579429
* From upgrade: fix mock interface implementation
After upgrading, there is the following compile error:
cannot use &mockHCPCfg{} (value of type *mockHCPCfg) as "github.com/hashicorp/hcp-sdk-go/config".HCPConfig value in return statement: *mockHCPCfg does not implement "github.com/hashicorp/hcp-sdk-go/config".HCPConfig (missing method Logout)
Solution: update the mock to have the missing Logout method
* From upgrade: Lint: remove usage of deprecated req.ServerState.TLS
Due to upgrade, linting is erroring due to usage of a newly deprecated field
22:47:56 [consul]: make lint
--> Running golangci-lint (.)
agent/hcp/testing.go:157:24: SA1019: req.ServerState.TLS is deprecated: use server_tls.internal_rpc instead. (staticcheck)
time.Until(time.Time(req.ServerState.TLS.CertExpiry)).Hours()/24,
^
* From upgrade: adjust oidc error message
From the upgrade, this test started failing:
=== FAIL: internal/go-sso/oidcauth TestOIDC_ClaimsFromAuthCode/failed_code_exchange (re-run 2) (0.01s)
oidc_test.go:393: unexpected error: Provider login failed: Error exchanging oidc code: oauth2: "invalid_grant" "unexpected auth code"
Prior to the upgrade, the error returned was:
```
Provider login failed: Error exchanging oidc code: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_grant\",\"error_description\":\"unexpected auth code\"}\n
```
Now the error returned is as below and does not contain "cannot fetch token"
```
Provider login failed: Error exchanging oidc code: oauth2: "invalid_grant" "unexpected auth code"
```
* Update AgentPushServerState structs with new fields
HCP-side changes for the new fields are in:
https://github.com/hashicorp/cloud-global-network-manager-service/pull/1195/files
* Minor refactor for hcpServerStatus to abstract tlsInfo into struct
This will make it easier to set the same tls-info information to both
- status.TLS (deprecated field)
- status.ServerTLSMetadata (new field to use instead)
* Update hcpServerStatus to parse out information for new fields
Changes:
- Improve error message and handling (encountered some issues and was confused)
- Set new field TLSInfo.CertIssuer
- Collect certificate authority metadata and set on TLSInfo.CertificateAuthorities
- Set TLSInfo on both server.TLS and server.ServerTLSMetadata.InternalRPC
* Update serverStatusToHCP to convert new fields to GNM rpc
* Add changelog
* Feedback: connect.ParseCert, caCerts
* Feedback: refactor and unit test server status
* Feedback: test to use expected struct
* Feedback: certificate with intermediate
* Feedback: catch no leaf, remove expectedErr
* Feedback: update todos with jira ticket
* Feedback: mock tlsConfigurator
* make proto for additional file in 1.17 not in main
test: add test helper to detect if the current build is enterprise (#19201)
This can help conditionally change test behavior if the test is executing in enterprise without the need to split the test with build tags.
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
Introduce randomized timings and reproducible randomization into controller integration tests.
As the V2 architecture hinges on eventual consistency and controllers reconciling the existing state in response to writes, there are potential issues we could run into regarding ordering and timing of operations. We want to be able to guarantee that given a set of resources the system will always eventually get to the desired correct state. The order of resource writes and delays in performing those writes should not alter the final outcome of reaching the desired state.
To that end, this commit introduces arbitrary randomized delays before performing resources writes into the `resourcetest.Client`. Its `PublishResources` method was already randomizing the order of resource writes. By default, no delay is added to normal writes and deletes but tests can opt-in via either passing hard coded options when creating the `resourcetest.Client` or using the `resourcetest.ConfigureTestCLIFlags` function to allow processing of CLI parameters.
In addition to allowing configurability of the request delay min and max, the client also has a configurable random number generator seed. When Using the CLI parameter helpers, a test log will be written noting the currently used settings. If the test fails then you can reproduce the same delays and order randomizations by providing the seed during the previous test failure.
# Conflicts:
# internal/catalog/catalogtest/run_test.go
# internal/catalog/catalogtest/test_integration_v2beta1.go
# test/integration/consul-container/test/catalog/catalog_test.go
Co-authored-by: Matt Keeler <mjkeeler7@gmail.com>
* backport of commit f142c77956
* backport of commit 79eadd1afd
* backport of commit d269c63a99
* backport of commit 532aad4dbf
* backport of commit 2de83d470f
* backport of commit c8af477a30
* NET-6080 - xds controller golden file inputs into xds resources - destinations (#19244)
* NET-5397 - wire up golden tests from sidecar-proxy controller for xds controller and xdsv2
* WIP
* WIP
* everything matching except leafCerts. need to mock those
* single port destinations working except mixed destinations
* golden test input to xds controller tests for destinations
* proposed fix for failover group naming errors
* clean up test to use helper.
* clean up test to use helper.
* fix test file
* add docstring for test function.
* add docstring for test function.
* fix linting error
* fixing test after route fix merged into main
* first source test works
* WIP
* modify all source files
* source tests pass
* fixing tests after bug fix in main
* got first destination working.
* adding destinations
* fix docstring for test
* fixing tests after bug fix in main
---------
Co-authored-by: John Murret <john.murret@hashicorp.com>
* server: run the api checks against the path without params (#19205)
* Clone proto into deepcopy correctly (#19204)
* chore: update version and nightly CI for 1.17 (#19208)
Update version file to 1.18-dev, and replace 1.13 nightly test with
1.17.
* mesh: add validation hook to proxy configuration (#19186)
* mesh: add more validations to Destinations resource (#19202)
* catalog, mesh: implement missing ACL hooks (#19143)
This change adds ACL hooks to the remaining catalog and mesh resources, excluding any computed ones. Those will for now continue using the default operator:x permissions.
It refactors a lot of the common testing functions so that they can be re-used between resources.
There are also some types that we don't yet support (e.g. virtual IPs) that this change adds ACL hooks to for future-proofing.
* NET-5073 - ProxyConfiguration: implement various connection options (#19187)
* NET-5073 - ProxyConfiguration: implement various connection options
* PR feedback - LocalConnection and InboundConnection do not affect exposed routes. configure L7 route destinations. fix connection proto sequence numbers.
* add timeout to L7 Route Destinations
* Relplat 897 copywrite bot workarounds (#19200)
Co-authored-by: Ronald Ekambi <ronekambi@gmail.com>
* mesh: add xRoute ACL hook tenancy tests (#19177)
Enhance the xRoute ACL hook tests to cover tenanted situations.
These tests will only execute in enterprise.
* resource: enforce lowercase v2 resource names (#19218)
* mesh: add DestinationPolicy ACL hook tenancy tests (#19178)
Enhance the DestinationPolicy ACL hook tests to cover tenanted situations.
These tests will only execute in enterprise.
* catalog: add FailoverPolicy ACL hook tenancy test (#19179)
* docs: Multi-port corrections (#19224)
* typo fixes and instruction corrections
* typo
* link path correction
* Add reason why port 53 is not used by default (#19222)
* Update dns-configuration.mdx
* Update website/content/docs/services/discovery/dns-configuration.mdx
Co-authored-by: Tu Nguyen <im2nguyen@users.noreply.github.com>
* v2tenancy: rename v1alpha1 -> v2beta1 (#19227)
* [NET-5944] security: Update Go version to 1.20.10 and `x/net` to 0.17.0 (#19225)
* Bump golang.org/x/net to 0.17.0
This resolves [CVE-2023-39325](https://nvd.nist.gov/vuln/detail/CVE-2023-39325)
/ [CVE-2023-44487](https://nvd.nist.gov/vuln/detail/CVE-2023-44487).
* Update Go version to 1.20.10
This resolves [CVE-2023-39325](https://nvd.nist.gov/vuln/detail/CVE-2023-39325)
/ [CVE-2023-44487](https://nvd.nist.gov/vuln/detail/CVE-2023-44487)
(`net/http`).
* NET-6097 - sidecar proxy controller - give name to first failover policy target (#19239)
* Cc 5545: Upgrade HDS packages and modifiers (#19226)
* Upgrade @hashicorp/design-system-tokens to 1.9.0
* Upgrade @hashicorp/design-system-components to 1.8.1
* Upgrade @hashicorp/design-system-components and ember-in-viewport
* Explicitly install ember-modifier@4.1.0
* rename copy-button
* Fix how cleanup is done in with-copyable
* Update aria-menu modifier for new structure
* Update css-prop modifier to new structure
* Convert did-upsert to regular class modifier
* Update notification modifier for new structure
* Update on-oustside modifier for new structure
* Move destroy handler registration in with-copyable
* Update style modifier for new structure
* Update validate modifier for new structure
* Guard against setting on destroyed object
* Upgrade @hashicorp/design-system-components to 2.14.1
* Remove debugger
* Guard against null in aria-menu
* Fix undefined hash in validate addon
* Upgrade ember-on-resize-modifier
* Fix copy button import, missing import and array destructuring
---------
Co-authored-by: wenincode <tyler.wendlandt@hashicorp.com>
* [NET-5810] CE changes for multiple virtual hosts (#19246)
CE changes for multiple virtual hosts
* Net 4893- Ensure we're testing all the latest versions of Vault/Nomad (#19119)
* NET-5592 - update Nomad integration testing
* NET-4893: Ensure we're testing all the latest versions of Vault/Nomad
* docs: Fix example control-plane-request-limit HCL and JSON (#19105)
The control-plane-request-limit config entry does not support
specifying parameter names in snake case format.
This commit updates the HCL and JSON examples to use the supported
camel case key format.
* test: add 1.17 nightly integrations test (#19253)
* fix expose paths (#19257)
When testing adding http probes to apps, I ran into some issues which I fixed here:
- The listener should be listening on the exposed listener port, updated that.
- The listener and route names were pointing to the path of the exposed path. In my test, the path was "/" resulting in an empty string path. Also, the path may not be unique across exposed path listeners, so I decided to use the path+exposed port as the unique identifier.
* docs: Multiport HCP constraint update (#19261)
* Add sentence
* link text adjustment
* docs: Fix multi-port install (#19262)
* Update configure.mdx
Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com>
* Prevent circular dependencies between v2 resources and generate a mermaid diagram with their dependencies (#19230)
* build(docker): always publish full and minor version tags for dev images (#19278)
* fix nightly integration test: envoy version and n-2 version (#19286)
* [NET-6221] Ensure LB policy set for locality-aware routing (CE) (#19283)
Ensure LB policy set for locality-aware routing (CE)
`overprovisioningFactor` should be overridden with the expected value
(100,000) when there are multiple endpoint groups. Update code and
tests to enforce this.
This is an Enterprise feature. This commit represents the CE portions of
the change; tests are added in the corresponding `consul-enterprise`
change.
* fix: allow snake case keys for ip based rate limit config entry (#19277)
* fix: allow snake case keys for ip based rate limit config entry
* chore: add changelog
* reformatted the JSON schema server conf ref (#19288)
* acls,catalog,mesh: properly authorize workload selectors on writes (#19260)
To properly enforce writes on resources that have workload selectors with prefixes, we need another service authorization rule that allows us to check whether read is allowed within a given prefix. Specifically we need to only allow writes if the policy prefix allows for a wider set of names than the prefix selector on the resource. We should also not allow policies with exact names for prefix matches.
Part of [NET-3993]
* NET-6239: Temporarily disable verify envoy check (#19299)
* skip verify envoy version
* cleanup
* Update supported Envoy versions (#19276)
* mesh: provide missing domain to route configurations in ProxyStateTemplate (#19298)
* add empty domains
* update unit tests
* enable verify envoy script (#19303)
* Vault CA bugfixes (#19285)
* Re-add retry logic to Vault token renewal
* Fix goroutine leak
* Add test for detecting goroutine leak
* Add changelog
* Rename tests
* Add comment
* Backout Envoy 1.28.0 (#19306)
* added ent to ce downgrade changes (#19311)
* added ent to ce downgrade changes
* added changelog
* added busl headers
* skip envoy version check in ci (#19315)
* Tenancy Bridge v2 (#19220)
* tenancy bridge v2 for v2 resources
* add missing copywrite headers
* remove branch name causing conflicts (#19319)
* mesh: ensure route configs are named uniquely per port (#19323)
* [NET-5327] Templated policies api/cli docs (#19270)
* More templated policies docs (#19312)
[NET-5327]More templated policies docs
* Fixing docs to add more templated policies references (#19335)
* Upgrade Consul UI to Node 18 (#19252)
* Upgrading node to node 18
* Ensure we're on latest version of yarn as well
* add comma to make frontend tests run
* Use Node 18 Alpine image in UI build dockerfile
* delete package-lock.json
---------
Co-authored-by: wenincode <tyler.wendlandt@hashicorp.com>
Co-authored-by: Ella Cai <ella.y.cai@gmail.com>
Co-authored-by: Ella Cai <ella@hashicorp.com>
* resource: default peername to local in list endpoints (#19340)
* NET-5397 - wire up golden tests from sidecar-proxy controller for xds controller and xdsv2
* WIP
* WIP
* everything matching except leafCerts. need to mock those
* single port destinations working except mixed destinations
* golden test input to xds controller tests for destinations
* proposed fix for failover group naming errors
* clean up test to use helper.
* clean up test to use helper.
* fix test file
* add docstring for test function.
* add docstring for test function.
* fix linting error
* fixing test after route fix merged into main
* first source test works
* WIP
* modify all source files
* backport of commit 17a76f6e24
* backport of commit de5a7c0e11
* backport of commit eb8e27cc48
---------
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
Co-authored-by: Ashwin Venkatesh <ashwin@hashicorp.com>
Co-authored-by: Michael Zalimeni <michael.zalimeni@hashicorp.com>
Co-authored-by: Iryna Shustava <ishustava@users.noreply.github.com>
Co-authored-by: John Murret <john.murret@hashicorp.com>
Co-authored-by: modrake <12264057+modrake@users.noreply.github.com>
Co-authored-by: Ronald Ekambi <ronekambi@gmail.com>
Co-authored-by: Semir Patel <semir.patel@hashicorp.com>
Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com>
Co-authored-by: David Yu <dyu@hashicorp.com>
Co-authored-by: Tu Nguyen <im2nguyen@users.noreply.github.com>
Co-authored-by: Chris Hut <tophernuts@gmail.com>
Co-authored-by: wenincode <tyler.wendlandt@hashicorp.com>
Co-authored-by: John Maguire <john.maguire@hashicorp.com>
Co-authored-by: Sophie Gairo <97480023+sophie-gairo@users.noreply.github.com>
Co-authored-by: Blake Covarrubias <blake@covarrubi.as>
Co-authored-by: Dan Stough <dan.stough@hashicorp.com>
Co-authored-by: Nitya Dhanushkodi <nitya@hashicorp.com>
Co-authored-by: Eric Haberkorn <erichaberkorn@gmail.com>
Co-authored-by: cskh <hui.kang@hashicorp.com>
Co-authored-by: Poonam Jadhav <poonam.jadhav@hashicorp.com>
Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
Co-authored-by: Anita Akaeze <anita.akaeze@hashicorp.com>
Co-authored-by: Chris Thain <32781396+cthain@users.noreply.github.com>
Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
Co-authored-by: aahel <aahel.guha@hashicorp.com>
Co-authored-by: Dhia Ayachi <dhia@hashicorp.com>
Co-authored-by: Ronald <roncodingenthusiast@users.noreply.github.com>
Co-authored-by: Ella Cai <ella.y.cai@gmail.com>
Co-authored-by: Ella Cai <ella@hashicorp.com>
* server: run the api checks against the path without params (#19205)
* Clone proto into deepcopy correctly (#19204)
* chore: update version and nightly CI for 1.17 (#19208)
Update version file to 1.18-dev, and replace 1.13 nightly test with
1.17.
* mesh: add validation hook to proxy configuration (#19186)
* mesh: add more validations to Destinations resource (#19202)
* catalog, mesh: implement missing ACL hooks (#19143)
This change adds ACL hooks to the remaining catalog and mesh resources, excluding any computed ones. Those will for now continue using the default operator:x permissions.
It refactors a lot of the common testing functions so that they can be re-used between resources.
There are also some types that we don't yet support (e.g. virtual IPs) that this change adds ACL hooks to for future-proofing.
* NET-5073 - ProxyConfiguration: implement various connection options (#19187)
* NET-5073 - ProxyConfiguration: implement various connection options
* PR feedback - LocalConnection and InboundConnection do not affect exposed routes. configure L7 route destinations. fix connection proto sequence numbers.
* add timeout to L7 Route Destinations
* Relplat 897 copywrite bot workarounds (#19200)
Co-authored-by: Ronald Ekambi <ronekambi@gmail.com>
* mesh: add xRoute ACL hook tenancy tests (#19177)
Enhance the xRoute ACL hook tests to cover tenanted situations.
These tests will only execute in enterprise.
* resource: enforce lowercase v2 resource names (#19218)
* mesh: add DestinationPolicy ACL hook tenancy tests (#19178)
Enhance the DestinationPolicy ACL hook tests to cover tenanted situations.
These tests will only execute in enterprise.
* catalog: add FailoverPolicy ACL hook tenancy test (#19179)
* docs: Multi-port corrections (#19224)
* typo fixes and instruction corrections
* typo
* link path correction
* Add reason why port 53 is not used by default (#19222)
* Update dns-configuration.mdx
* Update website/content/docs/services/discovery/dns-configuration.mdx
Co-authored-by: Tu Nguyen <im2nguyen@users.noreply.github.com>
* v2tenancy: rename v1alpha1 -> v2beta1 (#19227)
* [NET-5944] security: Update Go version to 1.20.10 and `x/net` to 0.17.0 (#19225)
* Bump golang.org/x/net to 0.17.0
This resolves [CVE-2023-39325](https://nvd.nist.gov/vuln/detail/CVE-2023-39325)
/ [CVE-2023-44487](https://nvd.nist.gov/vuln/detail/CVE-2023-44487).
* Update Go version to 1.20.10
This resolves [CVE-2023-39325](https://nvd.nist.gov/vuln/detail/CVE-2023-39325)
/ [CVE-2023-44487](https://nvd.nist.gov/vuln/detail/CVE-2023-44487)
(`net/http`).
* NET-6097 - sidecar proxy controller - give name to first failover policy target (#19239)
* Cc 5545: Upgrade HDS packages and modifiers (#19226)
* Upgrade @hashicorp/design-system-tokens to 1.9.0
* Upgrade @hashicorp/design-system-components to 1.8.1
* Upgrade @hashicorp/design-system-components and ember-in-viewport
* Explicitly install ember-modifier@4.1.0
* rename copy-button
* Fix how cleanup is done in with-copyable
* Update aria-menu modifier for new structure
* Update css-prop modifier to new structure
* Convert did-upsert to regular class modifier
* Update notification modifier for new structure
* Update on-oustside modifier for new structure
* Move destroy handler registration in with-copyable
* Update style modifier for new structure
* Update validate modifier for new structure
* Guard against setting on destroyed object
* Upgrade @hashicorp/design-system-components to 2.14.1
* Remove debugger
* Guard against null in aria-menu
* Fix undefined hash in validate addon
* Upgrade ember-on-resize-modifier
* Fix copy button import, missing import and array destructuring
---------
Co-authored-by: wenincode <tyler.wendlandt@hashicorp.com>
* [NET-5810] CE changes for multiple virtual hosts (#19246)
CE changes for multiple virtual hosts
* Net 4893- Ensure we're testing all the latest versions of Vault/Nomad (#19119)
* NET-5592 - update Nomad integration testing
* NET-4893: Ensure we're testing all the latest versions of Vault/Nomad
* docs: Fix example control-plane-request-limit HCL and JSON (#19105)
The control-plane-request-limit config entry does not support
specifying parameter names in snake case format.
This commit updates the HCL and JSON examples to use the supported
camel case key format.
* test: add 1.17 nightly integrations test (#19253)
* fix expose paths (#19257)
When testing adding http probes to apps, I ran into some issues which I fixed here:
- The listener should be listening on the exposed listener port, updated that.
- The listener and route names were pointing to the path of the exposed path. In my test, the path was "/" resulting in an empty string path. Also, the path may not be unique across exposed path listeners, so I decided to use the path+exposed port as the unique identifier.
* docs: Multiport HCP constraint update (#19261)
* Add sentence
* link text adjustment
* docs: Fix multi-port install (#19262)
* Update configure.mdx
Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com>
* Prevent circular dependencies between v2 resources and generate a mermaid diagram with their dependencies (#19230)
* build(docker): always publish full and minor version tags for dev images (#19278)
* fix nightly integration test: envoy version and n-2 version (#19286)
* [NET-6221] Ensure LB policy set for locality-aware routing (CE) (#19283)
Ensure LB policy set for locality-aware routing (CE)
`overprovisioningFactor` should be overridden with the expected value
(100,000) when there are multiple endpoint groups. Update code and
tests to enforce this.
This is an Enterprise feature. This commit represents the CE portions of
the change; tests are added in the corresponding `consul-enterprise`
change.
* fix: allow snake case keys for ip based rate limit config entry (#19277)
* fix: allow snake case keys for ip based rate limit config entry
* chore: add changelog
* reformatted the JSON schema server conf ref (#19288)
* acls,catalog,mesh: properly authorize workload selectors on writes (#19260)
To properly enforce writes on resources that have workload selectors with prefixes, we need another service authorization rule that allows us to check whether read is allowed within a given prefix. Specifically we need to only allow writes if the policy prefix allows for a wider set of names than the prefix selector on the resource. We should also not allow policies with exact names for prefix matches.
Part of [NET-3993]
* NET-6239: Temporarily disable verify envoy check (#19299)
* skip verify envoy version
* cleanup
* Update supported Envoy versions (#19276)
* mesh: provide missing domain to route configurations in ProxyStateTemplate (#19298)
* add empty domains
* update unit tests
* enable verify envoy script (#19303)
* Vault CA bugfixes (#19285)
* Re-add retry logic to Vault token renewal
* Fix goroutine leak
* Add test for detecting goroutine leak
* Add changelog
* Rename tests
* Add comment
* Backout Envoy 1.28.0 (#19306)
* added ent to ce downgrade changes (#19311)
* added ent to ce downgrade changes
* added changelog
* added busl headers
* skip envoy version check in ci (#19315)
* Tenancy Bridge v2 (#19220)
* tenancy bridge v2 for v2 resources
* add missing copywrite headers
* remove branch name causing conflicts (#19319)
* mesh: ensure route configs are named uniquely per port (#19323)
* [NET-5327] Templated policies api/cli docs (#19270)
* More templated policies docs (#19312)
[NET-5327]More templated policies docs
* Fixing docs to add more templated policies references (#19335)
* Upgrade Consul UI to Node 18 (#19252)
* Upgrading node to node 18
* Ensure we're on latest version of yarn as well
* add comma to make frontend tests run
* Use Node 18 Alpine image in UI build dockerfile
* delete package-lock.json
---------
Co-authored-by: wenincode <tyler.wendlandt@hashicorp.com>
Co-authored-by: Ella Cai <ella.y.cai@gmail.com>
Co-authored-by: Ella Cai <ella@hashicorp.com>
* resource: default peername to local in list endpoints (#19340)
* NET-5397 - wire up golden tests from sidecar-proxy controller for xds controller and xdsv2
* WIP
* WIP
* everything matching except leafCerts. need to mock those
* single port destinations working except mixed destinations
* golden test input to xds controller tests for destinations
* proposed fix for failover group naming errors
* clean up test to use helper.
* clean up test to use helper.
* fix test file
* add docstring for test function.
* add docstring for test function.
* fix linting error
* backport of commit b5fa4f29c8
* backport of commit e7ff94b3a6
---------
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
Co-authored-by: Ashwin Venkatesh <ashwin@hashicorp.com>
Co-authored-by: Michael Zalimeni <michael.zalimeni@hashicorp.com>
Co-authored-by: Iryna Shustava <ishustava@users.noreply.github.com>
Co-authored-by: John Murret <john.murret@hashicorp.com>
Co-authored-by: modrake <12264057+modrake@users.noreply.github.com>
Co-authored-by: Ronald Ekambi <ronekambi@gmail.com>
Co-authored-by: Semir Patel <semir.patel@hashicorp.com>
Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com>
Co-authored-by: David Yu <dyu@hashicorp.com>
Co-authored-by: Tu Nguyen <im2nguyen@users.noreply.github.com>
Co-authored-by: Chris Hut <tophernuts@gmail.com>
Co-authored-by: wenincode <tyler.wendlandt@hashicorp.com>
Co-authored-by: John Maguire <john.maguire@hashicorp.com>
Co-authored-by: Sophie Gairo <97480023+sophie-gairo@users.noreply.github.com>
Co-authored-by: Blake Covarrubias <blake@covarrubi.as>
Co-authored-by: Dan Stough <dan.stough@hashicorp.com>
Co-authored-by: Nitya Dhanushkodi <nitya@hashicorp.com>
Co-authored-by: Eric Haberkorn <erichaberkorn@gmail.com>
Co-authored-by: cskh <hui.kang@hashicorp.com>
Co-authored-by: Poonam Jadhav <poonam.jadhav@hashicorp.com>
Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
Co-authored-by: Anita Akaeze <anita.akaeze@hashicorp.com>
Co-authored-by: Chris Thain <32781396+cthain@users.noreply.github.com>
Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
Co-authored-by: aahel <aahel.guha@hashicorp.com>
Co-authored-by: Dhia Ayachi <dhia@hashicorp.com>
Co-authored-by: Ronald <roncodingenthusiast@users.noreply.github.com>
Co-authored-by: Ella Cai <ella.y.cai@gmail.com>
Co-authored-by: Ella Cai <ella@hashicorp.com>
mesh: add DestinationPolicy ACL hook tenancy tests (#19178)
Enhance the DestinationPolicy ACL hook tests to cover tenanted situations.
These tests will only execute in enterprise.
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
* server: run the api checks against the path without params (#19205)
* Clone proto into deepcopy correctly (#19204)
* chore: update version and nightly CI for 1.17 (#19208)
Update version file to 1.18-dev, and replace 1.13 nightly test with
1.17.
* mesh: add validation hook to proxy configuration (#19186)
* mesh: add more validations to Destinations resource (#19202)
* catalog, mesh: implement missing ACL hooks (#19143)
This change adds ACL hooks to the remaining catalog and mesh resources, excluding any computed ones. Those will for now continue using the default operator:x permissions.
It refactors a lot of the common testing functions so that they can be re-used between resources.
There are also some types that we don't yet support (e.g. virtual IPs) that this change adds ACL hooks to for future-proofing.
* NET-5073 - ProxyConfiguration: implement various connection options
* PR feedback - LocalConnection and InboundConnection do not affect exposed routes. configure L7 route destinations. fix connection proto sequence numbers.
* backport of commit c9c1b86789
* backport of commit 44c6c8c896
---------
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
Co-authored-by: Ashwin Venkatesh <ashwin@hashicorp.com>
Co-authored-by: Michael Zalimeni <michael.zalimeni@hashicorp.com>
Co-authored-by: Iryna Shustava <ishustava@users.noreply.github.com>
Co-authored-by: John Murret <john.murret@hashicorp.com>
catalog, mesh: implement missing ACL hooks (#19143)
This change adds ACL hooks to the remaining catalog and mesh resources, excluding any computed ones. Those will for now continue using the default operator:x permissions.
It refactors a lot of the common testing functions so that they can be re-used between resources.
There are also some types that we don't yet support (e.g. virtual IPs) that this change adds ACL hooks to for future-proofing.
Co-authored-by: Iryna Shustava <ishustava@users.noreply.github.com>
* server: run the api checks against the path without params (#19205)
* Clone proto into deepcopy correctly (#19204)
* mesh: add validation hook to proxy configuration
* backport of commit b08d9d4b47
* backport of commit 55b9363539
---------
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
Co-authored-by: Ashwin Venkatesh <ashwin@hashicorp.com>
Co-authored-by: Iryna Shustava <iryna@hashicorp.com>
This implements the Filter field on pbcatalog.WorkloadSelector to be
a post-fetch in-memory filter using the https://github.com/hashicorp/go-bexpr
expression language to filter resources based on their envelope metadata fields.
All existing usages of WorkloadSelector should be able to make use of the filter.
* xdsv2: support l7 by adding xfcc policy/headers, tweaking routes, and make a bunch of listeners l7 tests pass
* sidecarproxycontroller: add l7 local app support
* trafficpermissions: make l4 traffic permissions work on l7 workloads
* rename route name field for consistency with l4 cluster name field
* resolve conflicts and rebase
* fix: ensure route name is used in l7 destination route name as well. previously it was only in the route names themselves, now the route name and l7 destination route name line up
Sometimes workloads could come with unspecified protocols such as when running on Kubernetes. Currently, if this is the case, we will just default to tcp protocol.
However, to make sidecar-proxy controller work with l7 protocols we should instead inherit the protocol from service. This change adds tracking for services that a workload is part of and attempts to inherit the protocol whenever services a workload is part of doesn't have conflicting protocols.
This change builds on #19043 and #19067 and updates the sidecar controller to use those computed resources. This achieves several benefits:
* The cache is now simplified which helps us solve for previous bugs (such as multiple Upstreams/Destinations targeting the same service would overwrite each other)
* We no longer need proxy config cache
* We no longer need to do merging of proxy configs as part of the controller logic
* Controller watches are simplified because we no longer need to have complex mapping using cache and can instead use the simple ReplaceType mapper.
It also makes several other improvements/refactors:
* Unifies all caches into one. This is because originally the caches were more independent, however, now that they need to interact with each other it made sense to unify them where sidecar proxy controller uses one cache with 3 bimappers
* Unifies cache and mappers. Mapper already needed all caches anyway and so it made sense to make the cache do the mapping also now that the cache is unified.
* Gets rid of service endpoints watches. This was needed to get updates in a case when service's identities have changed and we need to update proxy state template's spiffe IDs for those destinations. This will however generate a lot of reconcile requests for this controller as service endpoints objects can change a lot because they contain workload's health status. This is solved by adding a status to the service object tracking "bound identities" and have service endpoints controller update it. Having service's status updated allows us to get updates in the sidecar proxy controller because it's already watching service objects
* Add a watch for workloads. We need it so that we get updates if workload's ports change. This also ensures that we update cached identities in case workload's identity changes.
This commit adds a new type ComputedDestinations that will contain all destinations from any Destinations resources and will be name-aligned with a workload. This also adds an explicit-destinations controller that computes these resources.
This is needed to simplify the tracking we need to do currently in the sidecar-proxy controller and makes it easier to query all explicit destinations that apply to a workload.
* Introduce a new type `ComputedProxyConfiguration` and add a controller for it. This is needed for two reasons. The first one is that external integrations like kubernetes may need to read the fully computed and sorted proxy configuration per workload. The second reasons is that it makes sidecar-proxy controller logic quite a bit simpler as it no longer needs to do this.
* Generalize workload selection mapper and fix a bug where it would delete IDs from the tree if only one is left after a removal is done.
Luke Kysow