362 Commits (30c5f0490c68bddaa73a409a6f297acffac6e8bc)

Author	SHA1	Message	Date
John Murret	029ac10acc	update serf links (#21797) ... * update serf links * add .markdown file extension * update serf links to use /blob/master/ * fix broken links --------- Co-authored-by: github-team-consul-core <github-team-consul-core@hashicorp.com>	2 months ago
R.B. Boyer	3e6f1c1fe1	remove v2 tenancy, catalog, and mesh (#21592) ... * remove v2 tenancy, catalog, and mesh - Inline the v2tenancy experiment to false - Inline the resource-apis experiment to false - Inline the hcp-v2-resource-apis experiment to false - Remove ACL policy templates and rule language changes related to workload identities (a v2-only concept) (e.g. identity and identity_prefix) - Update the gRPC endpoint used by consul-dataplane to no longer respond specially for v2 - Remove stray v2 references scattered throughout the DNS v1.5 newer implementation. * changelog * go mod tidy on consul containers * lint fixes from ENT --------- Co-authored-by: John Murret <john.murret@hashicorp.com>	3 months ago
R.B. Boyer	1535844c62	gossip: refactor some gossip related libraries into a central place (#21036) ... This refactors and relocates the following packages to live under internal/gossip instead of either in the toplevel lib or agent/consul: - librtt : related to serf coordinates - libserf : random serf stuff	7 months ago
Michael Zalimeni	86b0818c1f	[NET-8601] security: upgrade vault/api to remove go-jose.v2 (#20910) ... security: upgrade vault/api to remove go-jose.v2 This dependency has an open vulnerability (GO-2024-2631), and is no longer needed by the latest `vault/api`. This is a follow-up to the upgrade of `go-jose/v3` in this repository to make all our dependencies consolidate on v3. Also remove the recently added security scan triage block for GO-2024-2631, which was added due to incorrect reports that `go-jose/v3@3.0.3` was impacted; in reality, is was this indirect client dependency (not impacted by CVE) that the scanner was flagging. A bug report has been filed to address the incorrect reporting.	7 months ago
sarahalsmiller	be8d572eb2	NET-8524 Remove registation of api gateway controller (#20950) ... remove registation of api gateway controller	8 months ago
George Ma	44facc2ea3	chore: remove repetitive words (#20890) ... Signed-off-by: availhang <mayangang@outlook.com>	8 months ago
Chris S. Kim	f3f2175edd	Update go-jose library (#20888)	8 months ago
Derek Menteer	0ac8ae6c3b	Fix xDS deadlock due to syncLoop termination. (#20867) ... * Fix xDS deadlock due to syncLoop termination. This fixes an issue where agentless xDS streams can deadlock permanently until a server is restarted. When this issue occurs, no new proxies are able to successfully connect to the server. Effectively, the trigger for this deadlock stems from the following return statement: https://github.com/hashicorp/consul/blob/v1.18.0/agent/proxycfg-sources/catalog/config_source.go#L199-L202 When this happens, the entire `syncLoop()` terminates and stops consuming from the following channel: https://github.com/hashicorp/consul/blob/v1.18.0/agent/proxycfg-sources/catalog/config_source.go#L182-L192 Which results in the `ConfigSource.cleanup()` function never receiving a response and holding a mutex indefinitely: https://github.com/hashicorp/consul/blob/v1.18.0/agent/proxycfg-sources/catalog/config_source.go#L241-L247 Because this mutex is shared, it effectively deadlocks the server's ability to process new xDS streams. ---- The fix to this issue involves removing the `chan chan struct{}` used like an RPC-over-channels pattern and replacing it with two distinct channels: + `stopSyncLoopCh` - indicates that the `syncLoop()` should terminate soon. + `syncLoopDoneCh` - indicates that the `syncLoop()` has terminated. Splitting these two concepts out and deferring a `close(syncLoopDoneCh)` in the `syncLoop()` function ensures that the deadlock above should no longer occur. We also now evict xDS connections of all proxies for the corresponding `syncLoop()` whenever it encounters an irrecoverable error. This is done by hoisting the new `syncLoopDoneCh` upwards so that it's visible to the xDS delta processing. Prior to this fix, the behavior was to simply orphan them so they would never receive catalog-registration or service-defaults updates. * Add changelog.	8 months ago
Matt Keeler	8fcafb139c	Add `consul snapshot decode` command (#20824) ... Add snapshot decoding command	8 months ago
Michael Zalimeni	d4761c0ccd	security: upgrade google.golang.org/protobuf to 1.33.0 (#20801) ... Resolves CVE-2024-24786.	9 months ago
Matt Keeler	1a454b7b47	Make routes controller tests more robust when testing with multiple tenancies (#20743) ... When executing the test with multiple tenancies the various mesh controllers remained running across multiple sub-tests. While the resourcetest.Client cleans up resources created with it, resources created by the controllers were not being cleaned up. This could result in subsequent sub tests encountering unexpected values and failing when they should pass.	9 months ago
Semir Patel	8ba919f913	v2tenancy: make CE specific version of `resource.Registration` (#20681)	9 months ago
Semir Patel	943426bc79	v2tenancy: add optional LicenseFeature to type Registration struct (#20673)	9 months ago
Chris S. Kim	53afd8f4c5	[CE] Test tenancies for exported-services config manager (#20678) ... Sync controller tests from ENT	9 months ago
John Maguire	2ed67ba9ae	[NET-7375] Add Skeleton for APIGW ProxyStateTemplateBuilder (#20628) ... * WIP * got empty state working and cleaned up additional code * Moved api gateway mapper to it's own file, removed mesh port usage for api gateway, allow gateway to reconcile without routes * Update internal/mesh/internal/controllers/gatewayproxy/controller.go Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com> * Feedback from PR Review: - rename referencedTCPRoutes variable to be more descriptive - fetchers are in alphabetical order - move log statements to be more indicative of internal state --------- Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>	9 months ago
Chris S. Kim	248969c2a7	Update ComputedTrafficPermissions ACL hooks (#20622)	9 months ago
Chris S. Kim	0c2c7ca9c7	Add samenessgroup registration for enterprise (#20624)	9 months ago
Chris S. Kim	ab3c6cf1e5	Add BoundReferences to ComputedTrafficPermissions (#20593)	9 months ago
Chris S. Kim	7575b53737	Refactor xTP tests (#20597)	9 months ago
Semir Patel	b716a9ef6b	resource: reconcile managed types every ~8hrs (#20606)	9 months ago
R.B. Boyer	671c436415	mesh: use ComputedImplicitDestinations resource in the sidecar controller (#20553) ... Wire the ComputedImplicitDestinations resource into the sidecar controller, replacing the inline version already present. Also: - Rewrite the controller to use the controller cache - Rewrite it to no longer depend on ServiceEndpoints - Remove the fetcher and (local) cache abstraction	10 months ago
John Murret	7e8f2e5f08	NET-7644/NET-7634 - Implement query lookup for tagged addresses on nodes and services including WAN translation. (#20583) ... NET-7644 - Implement tagged addresses and wan translation	10 months ago
John Maguire	ec76090be9	[NET-7450] Fix listenerToProtocol function for input (#20536) ... * make listenerProtocolToCatalogProtocol function more forgiving for different cased input * update tests	10 months ago
Nick Cellino	5fb6ab6a3a	Move HCP Manager lifecycle management out of Link controller (#20401) ... * Add function to get update channel for watching HCP Link * Add MonitorHCPLink function This function can be called in a goroutine to manage the lifecycle of the HCP manager. * Update HCP Manager config in link monitor before starting This updates HCPMonitorLink so it updates the HCP manager with an HCP client and management token when a Link is upserted. * Let MonitorHCPManager handle lifecycle instead of link controller * Remove cleanup from Link controller and move it to MonitorHCPLink Previously, the Link Controller was responsible for cleaning up the HCP-related files on the file system. This change makes it so MonitorHCPLink handles this cleanup. As a result, we are able to remove the PlacementEachServer placement strategy for the Link controller because it no longer needs to do this per-node cleanup. * Remove HCP Manager dependency from Link Controller The Link controller does not need to have HCP Manager as a dependency anymore, so this removes that dependency in order to simplify the design. * Add Linked prefix to Linked status variables This is in preparation for adding a new status type to the Link resource. * Add new "validated" status type to link resource The link resource controller will now set a "validated" status in addition to the "linked" status. This is needed so that other components (eg the HCP manager) know when the Link is ready to link with HCP. * Fix tests * Handle new 'EndOfSnapshot' WatchList event * Fix watch test * Remove unnecessary config from TestAgent_scadaProvider Since the Scada provider is now started on agent startup regardless of whether a cloud config is provided, this removes the cloud config override from the relevant test. This change is not exactly related to the changes from this PR, but rather is something small and sort of related that was noticed while working on this PR. * Simplify link watch test and remove sleep from link watch This updates the link watch test so that it uses more mocks and does not require setting up the infrastructure for the HCP Link controller. This also removes the time.Sleep delay in the link watcher loop in favor of an error counter. When we receive 10 consecutive errors, we shut down the link watcher loop. * Add better logging for link validation. Remove EndOfSnapshot test. * Refactor link monitor test into a table test * Add some clarifying comments to link monitor * Simplify link watch test * Test a bunch more errors cases in link monitor test * Use exponential backoff instead of errorCounter in LinkWatch * Move link watch and link monitor into a single goroutine called from server.go * Refactor HCP link watcher to use single go-routine. Previously, if the WatchClient errored, we would've never recovered because we never retry to create the stream. With this change, we have a single goroutine that runs for the life of the server agent and if the WatchClient stream ever errors, we retry the creation of the stream with an exponential backoff.	10 months ago
Tauhid Anjum	9d8f9a5470	NET-7783: Fix sameness group expansion to 0 sources error CE (#20584)	10 months ago
R.B. Boyer	6742340878	mesh: add ComputedImplicitDestinations resource for future use (#20547) ... Creates a new controller to create ComputedImplicitDestinations resources by composing ComputedRoutes, Services, and ComputedTrafficPermissions to infer all ParentRef services that could possibly send some portion of traffic to a Service that has at least one accessible Workload Identity. A followup PR will rewire the sidecar controller to make use of this new resource. As this is a performance optimization, rather than a security feature the following aspects of traffic permissions have been ignored: - DENY rules - port rules (all ports are allowed) Also: - Add some v2 TestController machinery to help test complex dependency mappers.	10 months ago
Matt Keeler	6c4b83c119	Allow reuse of cache indexes (#20562) ... Previously calling `index.New` would return an object with the index information such as the Indexer, whether it was required, and the name of the index as well as a radix tree to store indexed data. Now the main `Index` type doesn’t contain the radix tree for indexed data. Instead the `IndexedData` method can be used to combine the main `Index` with a radix tree in the `IndexedData` structure. The cache still only allows configuring the `Index` type and will invoke the `IndexedData` method on the provided indexes to get the structure that the cache can use for actual data management. All of this makes it now safe to reuse the `index.Index` types.	10 months ago
John Maguire	7c3a379e48	Fixes gatewayproxy controller tests for ent (#20543) ... fix tests for ent	10 months ago
Eric Haberkorn	b26282568f	Move sameness groups to v2beta1 version (#20531)	10 months ago
John Maguire	2ee32b1980	Add tests for gw proxy controller (#20510) ... * Added basic tests for gatewayproxy controller * add copyright header * Clean up tests	10 months ago
Nathan Coleman	45d645471b	[NET-7414] Reconcile PST for mesh gateway workloads on change to ComputedExportedServices (#20271) ... * Reconcile ProxyStateTemplate on change to ComputedExportedServices * gofmt changeset --------- Co-authored-by: NiniOak <anita.akaeze@hashicorp.com>	10 months ago
skpratt	57bad0df85	add traffic permissions excludes and tests (#20453) ... * add traffic permissions tests * review fixes * Update internal/mesh/internal/controllers/sidecarproxy/builder/local_app.go Co-authored-by: John Landa <jonathanlanda@gmail.com> --------- Co-authored-by: John Landa <jonathanlanda@gmail.com>	10 months ago
Eric Haberkorn	1bd253021b	V1 Compat Exported Services Controller Optimizations (#20517) ... V1 compat exported services controller optimizations * Don't start the v2 exported services controller in v1 mode. * Use the controller cache.	10 months ago
Matt Keeler	3ca4f39fa1	Register the multicluster types for the catalogtest integration tests (#20516) ... In particular the failover controller needs these in Consul Enterprise	10 months ago
John Maguire	24e9603d9b	Fix Gatewayproxy Controller and Re-Enable APIGW v2 Controller (#20508) ... re-enable apigw controller, fix typo in key name for metadata for gatewayproxy	10 months ago
Matt Keeler	49e6c0232d	Panic for unregistered types (#20476) ... * Panic when controllers attempt to make invalid requests to the resource service This will help to catch bugs in tests that could cause infinite errors to be emitted. * Disable the API GW v2 controller With the previous commit, this would cause a server to panic due to watching a type which has not yet been created/registered. * Ensure that a test server gets the full type registry instead of constructing its own * Skip TestServer_ControllerDependencies * Fix peering tests so that they use the full resource registry.	10 months ago
John Maguire	54c974748e	[NET-7280] Add APIGW support to the gatewayproxy controller (#20484) ... * Add APIGW support to the gatewayproxy controller * update copywrite headers	10 months ago
Tauhid Anjum	88b8a1cc36	NET-6776 - Update Routes controller to use ComputedFailoverPolicy CE (#20496) ... Update Routes controller to use ComputedFailoverPolicy	10 months ago
R.B. Boyer	deca6a49bd	catalog: improve the bound workload identity encoding on services (#20458) ... The endpoints controller currently encodes the list of unique workload identities referenced by all workload matched by a Service into a special data-bearing status condition on that Service. This allows a downstream controller to avoid an expensive watch on the ServiceEndpoints type just to get this data. The current encoding does not lend itself well to machine parsing, which is what the field is meant for, so this PR simplifies the encoding from: "blah blah: " + strings.Join(ids, ",") + "." to strings.Join(ids, ",") It also provides an exported utility function to easily extract this data.	10 months ago
Nick Ethier	9d4ad74a63	internal/hcp: prevent write loop on telemetrystate resource updates (#20435) ... * internal/hcp: prevent write loop on telemetrystate resource updates * Update controller.go Co-authored-by: Nick Cellino <nick.cellino@hashicorp.com> * internal/hcp: add assertion for looping controller --------- Co-authored-by: Nick Cellino <nick.cellino@hashicorp.com>	10 months ago
R.B. Boyer	c029b20615	v2: ensure the controller caches are fully populated before first use (#20421) ... The new controller caches are initialized before the DependencyMappers or the Reconciler run, but importantly they are not populated. The expectation is that when the WatchList call is made to the resource service it will send an initial snapshot of all resources matching a single type, and then perpetually send UPSERT/DELETE events afterward. This initial snapshot will cycle through the caching layer and will catch it up to reflect the stored data. Critically the dependency mappers and reconcilers will race against the restoration of the caches on server startup or leader election. During this time it is possible a mapper or reconciler will use the cache to lookup a specific relationship and not find it. That very same reconciler may choose to then recompute some persisted resource and in effect rewind it to a prior computed state. Change - Since we are updating the behavior of the WatchList RPC, it was aligned to match that of pbsubscribe and pbpeerstream using a protobuf oneof instead of the enum+fields option. - The WatchList rpc now has 3 alternating response events: Upsert, Delete, EndOfSnapshot. When set the initial batch of "snapshot" Upserts sent on a new watch, those operations will be followed by an EndOfSnapshot event before beginning the never-ending sequence of Upsert/Delete events. - Within the Controller startup code we will launch N+1 goroutines to execute WatchList queries for the watched types. The UPSERTs will be applied to the nascent cache only (no mappers will execute). - Upon witnessing the END operation, those goroutines will terminate. - When all cache priming routines complete, then the normal set of N+1 long lived watch routines will launch to officially witness all events in the system using the primed cached.	10 months ago
Eric Haberkorn	543c6a30af	Trigger the V1 Compat exported-services Controller when V1 Config Entries are Updated (#20456) ... * Trigger the v1 compat exported-services controller when the v1 config entry is modified. * Hook up exported-services config entries to the event publisher. * Add tests to the v2 exported services shim. * Use the local materializer trigger updates on the v1 compat exported services controller when exported-services config entries are modified. * stop sleeping when context is cancelled	10 months ago
Eric Haberkorn	d0243b618d	Change the multicluster group to v2 (#20430)	10 months ago
Melisa Griffin	7c00d396cf	[NET-6417] Add validation of MeshGateway name + listeners (#20425) ... * Add validation of MeshGateway name + listeners * Adds test for ValidateMeshGateway * Fixes data fetcher test for gatewayproxy --------- Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>	10 months ago
Nick Ethier	383d92e9ab	hcp.v2.TelemetryState resource and controller implementation (#20257) ... * pbhcp: add TelemetryState resource * agent/hcp: add GetObservabilitySecrets to client * internal/hcp: add TelemetryState controller logic * hcp/telemetry-state: added config options for hcp sdk and debug key to skip deletion during reconcile * pbhcp: update proto documentation * hcp: address PR feedback, additional validations and code cleanup * internal/hcp: fix type sig change in test * update testdata/v2-resource-dependencies	10 months ago
Nathan Coleman	74e4200d07	[NET-6429] Program ProxyStateTemplate to route cross-partition traffi… (#20410) ... [NET-6429] Program ProxyStateTemplate to route cross-partition traffic to the correct destination mesh gateway * Program mesh port to route wildcarded gateway SNI to the appropriate remote partition's mesh gateway * Update target + route ports in service endpoint refs when building PST * Use proper name of local datacenter when constructing SNI for gateway target * Use destination identities for TLS when routing L4 traffic through the mesh gateway * Use new constants, move comment to correct location * Use new constants for port names * Update test assertions * Undo debug logging change	10 months ago
Ronald	8799c36410	[NET-6231] Handle Partition traffic permissions when reconciling traffic permissions (#20408) ... [NET-6231] Partition traffic permissions Co-authored-by: Chris S. Kim <ckim@hashicorp.com>	10 months ago
Chris S. Kim	7cc88a1577	Handle NamespaceTrafficPermissions when reconciling TrafficPermissions (#20407)	10 months ago
Nathan Coleman	21b3c18d5d	Use a full EndpointRef on ComputedRoutes targets instead of just the ID (#20400) ... * Use a full EndpointRef on ComputedRoutes targets instead of just the ID Today, the `ComputedRoutes` targets have the appropriate ID set for their `ServiceEndpoints` reference; however, the `MeshPort` and `RoutePort` are assumed to be that of the target when adding the endpoints reference in the sidecar's `ProxyStateTemplate`. This is problematic when the target lives behind a `MeshGateway` and the `Mesh/RoutePort` used in the sidecar's `ProxyStateTemplate` should be that of the `MeshGateway` instead of the target. Instead of assuming the `MeshPort` and `RoutePort` when building the `ProxyStateTemplate` for the sidecar, let's just add the full `EndpointRef` -- including the ID and the ports -- when hydrating the computed destinations. * Make sure the UID from the existing ServiceEndpoints makes it onto ComputedRoutes * Update test assertions * Undo confusing whitespace change * Remove one-line function wrapper * Use plural name for endpoints ref * Add constants for gateway name, kind and port names	10 months ago
Ronald	783f33db3b	[NET-7074] Exported Services typo fix (#20402)	10 months ago