Commit Graph

328 Commits (ab8f23478a4b0183b2d740a73b28946e24805613)

Author SHA1 Message Date
John Maguire 24e9603d9b
Fix Gatewayproxy Controller and Re-Enable APIGW v2 Controller (#20508)
re-enable apigw controller, fix typo in key name for metadata for
gatewayproxy
2024-02-06 18:55:55 +00:00
Matt Keeler 49e6c0232d
Panic for unregistered types (#20476)
* Panic when controllers attempt to make invalid requests to the resource service

This will help to catch bugs in tests that could cause infinite errors to be emitted.

* Disable the API GW v2 controller

With the previous commit, this would cause a server to panic due to watching a type which has not yet been created/registered.

* Ensure that a test server gets the full type registry instead of constructing its own

* Skip TestServer_ControllerDependencies

* Fix peering tests so that they use the full resource registry.
2024-02-06 11:23:06 -05:00
John Maguire 54c974748e
[NET-7280] Add APIGW support to the gatewayproxy controller (#20484)
* Add APIGW support to the gatewayproxy controller

* update copywrite headers
2024-02-06 11:03:37 -05:00
Tauhid Anjum 88b8a1cc36
NET-6776 - Update Routes controller to use ComputedFailoverPolicy CE (#20496)
Update Routes controller to use ComputedFailoverPolicy
2024-02-06 13:28:18 +05:30
R.B. Boyer deca6a49bd
catalog: improve the bound workload identity encoding on services (#20458)
The endpoints controller currently encodes the list of unique workload identities 
referenced by all workload matched by a Service into a special data-bearing 
status condition on that Service. This allows a downstream controller to avoid an 
expensive watch on the ServiceEndpoints type just to get this data.

The current encoding does not lend itself well to machine parsing, which is what 
the field is meant for, so this PR simplifies the encoding from:

    "blah blah: " + strings.Join(ids, ",") + "."

to

    strings.Join(ids, ",")

It also provides an exported utility function to easily extract this data.
2024-02-02 16:28:39 -06:00
Nick Ethier 9d4ad74a63
internal/hcp: prevent write loop on telemetrystate resource updates (#20435)
* internal/hcp: prevent write loop on telemetrystate resource updates

* Update controller.go

Co-authored-by: Nick Cellino <nick.cellino@hashicorp.com>

* internal/hcp: add assertion for looping controller

---------

Co-authored-by: Nick Cellino <nick.cellino@hashicorp.com>
2024-02-02 16:28:20 -05:00
R.B. Boyer c029b20615
v2: ensure the controller caches are fully populated before first use (#20421)
The new controller caches are initialized before the DependencyMappers or the 
Reconciler run, but importantly they are not populated. The expectation is that 
when the WatchList call is made to the resource service it will send an initial 
snapshot of all resources matching a single type, and then perpetually send 
UPSERT/DELETE events afterward. This initial snapshot will cycle through the 
caching layer and will catch it up to reflect the stored data.

Critically the dependency mappers and reconcilers will race against the restoration 
of the caches on server startup or leader election. During this time it is possible a
 mapper or reconciler will use the cache to lookup a specific relationship and 
not find it. That very same reconciler may choose to then recompute some 
persisted resource and in effect rewind it to a prior computed state.

Change

- Since we are updating the behavior of the WatchList RPC, it was aligned to 
  match that of pbsubscribe and pbpeerstream using a protobuf oneof instead of the enum+fields option.

- The WatchList rpc now has 3 alternating response events: Upsert, Delete, 
  EndOfSnapshot. When set the initial batch of "snapshot" Upserts sent on a new 
  watch, those operations will be followed by an EndOfSnapshot event before beginning 
  the never-ending sequence of Upsert/Delete events.

- Within the Controller startup code we will launch N+1 goroutines to execute WatchList 
  queries for the watched types. The UPSERTs will be applied to the nascent cache
   only (no mappers will execute).

- Upon witnessing the END operation, those goroutines will terminate.

- When all cache priming routines complete, then the normal set of N+1 long lived 
watch routines will launch to officially witness all events in the system using the 
primed cached.
2024-02-02 15:11:05 -06:00
Eric Haberkorn 543c6a30af
Trigger the V1 Compat exported-services Controller when V1 Config Entries are Updated (#20456)
* Trigger the v1 compat exported-services controller when the v1 config entry is modified.

* Hook up exported-services config entries to the event publisher.
* Add tests to the v2 exported services shim.
* Use the local materializer trigger updates on the v1 compat exported services controller when exported-services config entries are modified.

* stop sleeping when context is cancelled
2024-02-02 15:30:04 -05:00
Eric Haberkorn d0243b618d
Change the multicluster group to v2 (#20430) 2024-02-01 12:08:26 -05:00
Melisa Griffin 7c00d396cf
[NET-6417] Add validation of MeshGateway name + listeners (#20425)
* Add validation of MeshGateway name + listeners

* Adds test for ValidateMeshGateway

* Fixes data fetcher test for gatewayproxy

---------

Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>
2024-01-31 18:47:57 -05:00
Nick Ethier 383d92e9ab
hcp.v2.TelemetryState resource and controller implementation (#20257)
* pbhcp: add TelemetryState resource

* agent/hcp: add GetObservabilitySecrets to client

* internal/hcp: add TelemetryState controller logic

* hcp/telemetry-state: added config options for hcp sdk and debug key to skip deletion during reconcile

* pbhcp: update proto documentation

* hcp: address PR feedback, additional validations and code cleanup

* internal/hcp: fix type sig change in test

* update testdata/v2-resource-dependencies
2024-01-31 14:47:05 -05:00
Nathan Coleman 74e4200d07
[NET-6429] Program ProxyStateTemplate to route cross-partition traffi… (#20410)
[NET-6429] Program ProxyStateTemplate to route cross-partition traffic to the correct destination mesh gateway

* Program mesh port to route wildcarded gateway SNI to the appropriate remote partition's mesh gateway

* Update target + route ports in service endpoint refs when building PST

* Use proper name of local datacenter when constructing SNI for gateway target

* Use destination identities for TLS when routing L4 traffic through the mesh gateway

* Use new constants, move comment to correct location

* Use new constants for port names

* Update test assertions

* Undo debug logging change
2024-01-31 10:46:04 -05:00
Ronald 8799c36410
[NET-6231] Handle Partition traffic permissions when reconciling traffic permissions (#20408)
[NET-6231] Partition traffic permissions

Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
2024-01-30 22:14:32 +00:00
Chris S. Kim 7cc88a1577
Handle NamespaceTrafficPermissions when reconciling TrafficPermissions (#20407) 2024-01-30 21:31:25 +00:00
Nathan Coleman 21b3c18d5d
Use a full EndpointRef on ComputedRoutes targets instead of just the ID (#20400)
* Use a full EndpointRef on ComputedRoutes targets instead of just the ID

Today, the `ComputedRoutes` targets have the appropriate ID set for their `ServiceEndpoints` reference; however, the `MeshPort` and `RoutePort` are assumed to be that of the target when adding the endpoints reference in the sidecar's `ProxyStateTemplate`.

This is problematic when the target lives behind a `MeshGateway` and the `Mesh/RoutePort` used in the sidecar's `ProxyStateTemplate` should be that of the `MeshGateway` instead of the target.

Instead of assuming the `MeshPort` and `RoutePort` when building the `ProxyStateTemplate` for the sidecar, let's just add the full `EndpointRef` -- including the ID and the ports -- when hydrating the computed destinations.

* Make sure the UID from the existing ServiceEndpoints makes it onto ComputedRoutes

* Update test assertions

* Undo confusing whitespace change

* Remove one-line function wrapper

* Use plural name for endpoints ref

* Add constants for gateway name, kind and port names
2024-01-30 16:25:44 -05:00
Ronald 783f33db3b
[NET-7074] Exported Services typo fix (#20402) 2024-01-30 21:08:36 +00:00
Ganesh S 4ca6573384
Add status for exported services controller (#20376) 2024-01-30 22:20:09 +05:30
Melissa Kam b0e87dbe13
[CC-7049] Stop the HCP manager when link is deleted (#20351)
* Add Stop method to telemetry provider

Stop the main loop of the provider and set the config
to disabled.

* Add interface for telemetry provider

Added for easier testing. Also renamed Run to Start, which better
fits with Stop.

* Add Stop method to HCP manager

* Add manager interface, rename implementation

Add interface for easier testing, rename existing Manager to HCPManager.

* Stop HCP manager in link Finalizer

* Attempt to cleanup if resource has been deleted

The link should be cleaned up by the finalizer, but there's an edge
case in a multi-server setup where the link is fully deleted on one
server before the other server reconciles. This will cover the case
where the reconcile happens after the resource is deleted.

* Add a delete mananagement token function

Passes a function to the HCP manager that deletes the management token
that was initially created by the manager.

* Delete token as part of stopping the manager

* Lock around disabling config, remove descriptions
2024-01-30 09:40:36 -06:00
Melissa Kam 3b9bb8d6f9
[CC-7044] Start HCP manager as part of link creation (#20312)
* Check for ACL write permissions on write

Link eventually will be creating a token, so require acl:write.

* Convert Run to Start, only allow to start once

* Always initialize HCP components at startup

* Support for updating config and client

* Pass HCP manager to controller

* Start HCP manager in link resource

Start as part of link creation rather than always starting. Update
the HCP manager with values from the link before starting as well.

* Fix metrics sink leaked goroutine

* Remove the hardcoded disabled hostname prefix

The HCP metrics sink will always be enabled, so the length of sinks will
always be greater than zero. This also means that we will also always
default to prefixing metrics with the hostname, which is what our
documentation states is the expected behavior anyway.

* Add changelog

* Check and set running status in one method

* Check for primary datacenter, add back test

* Clarify merge reasoning, fix timing issue in test

* Add comment about controller placement

* Expand on breaking change, fix typo in changelog
2024-01-29 16:31:44 -06:00
Matt Keeler d350115e7f
Fix filename with two periods (#20389) 2024-01-29 15:38:40 -05:00
Matt Keeler 34a32d4ce5
Remove V2 PeerName field from pbresource.Tenancy (#19865)
The peer name will eventually show up elsewhere in the resource. For now though this rips it out of where we don’t want it to be.
2024-01-29 15:08:31 -05:00
Nitya Dhanushkodi 92aab7ea31
[NET-5586][rebased] v2: Support virtual port references in config (#20371)
[OG Author: michael.zalimeni@hashicorp.com, rebase needed a separate PR]

* v2: support virtual port in Service port references

In addition to Service target port references, allow users to specify a
port by stringified virtual port value. This is useful in environments
such as Kubernetes where typical configuration is written in terms of
Service virtual ports rather than workload (pod) target port names.

Retaining the option of referencing target ports by name supports VMs,
Nomad, and other use cases where virtual ports are not used by default.

To support both uses cases at once, we will strictly interpret port
references based on whether the value is numeric. See updated
`ServicePort` docs for more details.

* v2: update service ref docs for virtual port support

Update proto and generated .go files with docs reflecting virtual port
reference support.

* v2: add virtual port references to L7 topo test

Add coverage for mixed virtual and target port references to existing
test.

* update failover policy controller tests to work with computed failover policy and assert error conditions against FailoverPolicy and ComputedFailoverPolicy resources

* accumulate services; don't overwrite them in enterprise

---------

Co-authored-by: Michael Zalimeni <michael.zalimeni@hashicorp.com>
Co-authored-by: R.B. Boyer <rb@hashicorp.com>
2024-01-29 10:43:41 -08:00
Chris S. Kim a2d50af1bd
Fix panic on error (#20377) 2024-01-29 17:44:13 +00:00
Dan Stough 0ca7313b07
feat(v2dns): add PTR query support (#20362) 2024-01-29 11:40:10 -05:00
Nathan Coleman 27aecdb8cc
[NET-5075] Implement mesh gateway mode for explicit destinations (#20361) 2024-01-26 17:17:18 -05:00
Nitya Dhanushkodi 0ec7bddb9a
[Net-5594][Net-7466] v2: Only route to endpoints that implement the port being routed to, and make xdscontroller and xdsv2 golden tests use tenancy (#20356)
* If a workload does not implement a port, it should not be included in the list of endpoints for the Envoy cluster for that port.

* Adds tenancy tests for xds controller and xdsv2 resource generation, and adds all those files.

* The original change in this PR was for filtering the list of endpoints by the port being routed to (bullet 1). Since I made changes to sidecarproxycontroller golden files, I realized some of the golden files were unused because of the tenancy changes, so when I deleted those, that broke xds controller tests which weren't correctly using tenancy. So when I fixed that, then the xdsv2 tests broke, so I added tenancy support there too. So now, from sidecarproxy controller -> xds controller -> xdsv2 we now have tenancy support and all the golden files are lined up.
2024-01-26 10:07:21 -08:00
sarahalsmiller 37ebaa6920
Net 7155- Consul API Gateway Controller Stub Work (#20324)
* API Gateway proto

* fix lint issue

* new line

* run make proto format

* checkpoint

* stub

* Update internal/mesh/internal/controllers/apigateways/controller.go
2024-01-25 23:16:20 +00:00
Luke Kysow 840f11a0c5
Change logging of registered v2 resource endpoints to add /api prefix (#20352)
* Change logging of registered v2 resource endpoints to add /api prefix

Previous:

    agent.http: Registered resource endpoint: endpoint=/demo/v1/executive

New:

    agent.http: Registered resource endpoint: endpoint=/api/demo/v1/executive

This reduces confusion when attempting to call the APIs after looking at
the logs.
2024-01-25 14:18:54 -08:00
Nick Cellino ec0df00fc1
Add finalizer to link resource (#20321)
* Add finalizer to link resource

* Update internal/hcp/internal/controllers/link/controller.go

Co-authored-by: Semir Patel <semir.patel@hashicorp.com>

* Address PR style feedback

---------

Co-authored-by: Semir Patel <semir.patel@hashicorp.com>
2024-01-25 12:27:36 -05:00
John Landa 65920dccf4
move deny action to enterprise only for traffic permissions (#20313)
Add missing import

Add explicit enum case for deny action

Remove extra comments

Add build tags to ent and ce tests

Add copyright headers for the ce files

Fix case statements for ce validator

Remove ce tests with Deny traffic permissions

Fix more integration tests

Split more ce and ent tests, add back ent deny tests for traffic permissions controller

temp rename before rebase

Readd ent deny tests for traffic permissions controller
2024-01-24 13:01:22 -07:00
John Maguire cfe4d59938
[NET-7265] Panic when passing an incorrect type to the data fetcher for gatewayproxy (#20238)
* panic when passing an incorrect type to the data fetcher

* Add assertions for sidecarproxy datafetcher as well

* rename assertion function

* Add in comments to ensure devs know about potential panics for using
invalid types

* fix method call
2024-01-24 14:16:56 -05:00
Melissa Kam 7900544249
[CC-7063] Fetch HCP agent bootstrap config in Link reconciler (#20306)
* Move config-dependent methods to separate package

In order to reuse the fetching and file creation part of the
bootstrap package, move the code that would cause cyclical
dependencies to a different package.

* Export needed bootstrap methods and variables

Also add back validating persisted config and update tests.

* Add support to check for just management token

Add a new method that fetches the bootstrap configuration only if
there isn't a valid management token file instead of checking for
all the hcp-config files.

* Pass data dir as a dependency to link controller

The link controller needs to check the data directory for
the hcp-config files.

* Fetch bootstrap config for token in controller

Load the management token when reconciling a link resource, which will
fetch the agent boostrap configuration if the token is not already
persisted locally. Skip this step if the cluster is in read-only mode.

* Validate resource ID format in link creation

* Handle unauthorized and forbidden errors

Check for 401 and 403s when making GNM requests, exit bootstrap fetch
loop and return specific failure statuses for link.

* Move test function to a testing file

* Log load and status write errors
2024-01-24 09:51:43 -06:00
aahel 3446eb3b1b
added computed failover controller (#20329)
* added computed failover controller

* removed some uncessary changes

* removed uncessary changes

* minor refactor

* minor refactor fmt

* added copyright
2024-01-24 11:50:27 +05:30
skpratt 0abf8f8426
Net 5092/internal l7 traffic permissions (#20276)
* wire up L7 Traffic Permissions

* testing

* update comment
2024-01-23 20:07:58 -06:00
skpratt 44bcda8523
Net 7074/decentralized exported services management (#20318)
* Add decentralized management of V1 exported-services config entries using V2 multicluster resources.

* cleanup

---------

Co-authored-by: Matt Keeler <mjkeeler7@gmail.com>
2024-01-23 19:44:10 -06:00
Tauhid Anjum b37fe80eee
Net 6774 Make Sameness Groups Work With Traffic Permissions CE (#20316)
* Make Sameness Groups Work With Traffic Permissions

* Fix controller dependency
2024-01-23 13:23:03 +05:30
Tauhid Anjum 5d294b26d3
NET-5824 Exported services api (#20015)
* Exported services api implemented

* Tests added, refactored code

* Adding server tests

* changelog added

* Proto gen added

* Adding codegen changes

* changing url, response object

* Fixing lint error by having namespace and partition directly

* Tests changes

* refactoring tests

* Simplified uniqueness logic for exported services, sorted the response in order of service name

* Fix lint errors, refactored code
2024-01-23 10:06:59 +05:30
Nathan Coleman 995ba32cc0
Use null route cluster for default router when no matches on v2 mesh gateway (#20270)
* Use black hole cluster for default router when no matches

* Update test assertions

* Use null route cluster instead of black hole cluster concept

* Update test assertions
2024-01-22 10:50:04 -08:00
Melissa Kam a9dd6f5c02
Add a separate test for initializer retries (#20298)
Separate test for initializer retries
2024-01-19 16:59:44 -06:00
Melissa Kam 98c9702ba3
[CC-7031] Add initialization support to resource controllers (#20138)
* Add Initializer to the controller

The Initializer adds support for running any required initialization
steps when the controller is first started.

* Implement HCP Link initializer

The link initializer will create a Link resource if the
cloud configuration has been set.

* Simplify retry logic and testing

* Remove internal retry, replace with logging logic
2024-01-19 11:47:48 -06:00
Matt Keeler 0a261682cd
Migrate the node health controller to use the cache (#20248)
Some edge case error testing had to be removed because it was no longer possible to force errors when going through the cache layer as opposed to the resource service itself.
2024-01-19 12:22:45 -05:00
Matt Keeler cee9df574d
Deflake the catalog v2beta1 integration tests (#20278) 2024-01-19 10:49:47 -05:00
Nick Cellino fe678e9da1
Sync cluster attributes from GNM to Link resource (#20158)
* Add 'GetCluster' function to HCP client

* Sync cluster data inside Link controller

* Add access mode to HCP Link

* Sync AccessLevel property

* Fix imports and remove outdated comments

* Switch accessMode to access level

* Add comment around HCPClientFn

* Fix spacing in link.proto

* Add helper for writing status. Fix reconciliation loop
2024-01-19 10:02:55 -05:00
Matt Keeler f9c04881f9
Failover policy cache (#20244)
* Migrate the Failover controller to use the controller cache
* Remove the Catalog FailoverMapper and its usage in the mesh routes controller.
2024-01-19 09:35:34 -05:00
Nathan Coleman c40b59823a
[NET-6431] Remove explicit endpoints function from PST builder (#20262)
This isn't needed since we just populate RequiredEndpoints, which is already done for the base case
2024-01-18 19:13:37 -05:00
Dan Stough 0edfa74d15
feat(v2dns): recursor support (#20249)
* feat(v2dns): recursor support

* test: fix leaking test agent in dns svc test
2024-01-18 18:30:04 -05:00
Matt Keeler 59cb12c798
Migrate the Endpoints controller to use the controller cache (#20241)
* Add cache resource decoding helpers

* Implement a common package for workload selection facilities. This includes:

   * Controller cache Index
   * ACL hooks
   * Dependency Mapper to go from workload to list of resources which select it
   * Dependency Mapper to go from a resource which selects workloads to all the workloads it selects.

* Update the endpoints controller to use the cache instead of custom mappers.

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2024-01-18 17:52:52 -05:00
Matt Keeler 9897be76ad
Update workload health controller to use the controller cache (#20240) 2024-01-18 16:30:11 -05:00
John Maguire 7888d00e49
[NET6429] Add listeners for mesh-gateway v2 (#20253)
Add listeners for mesh-gateway v2
2024-01-18 17:52:06 +00:00
Dan Stough 15ab80c832
feat(v2dns): move DNSPolicy to workload/endpoints (#20246) 2024-01-18 11:37:42 -05:00