This add a fix to properly verify the gateway mode before creating a watch specific to mesh gateways. This watch have a high performance cost and when mesh gateways are not used is not used.
This also adds an optimization to only return the nodes when watching the Internal.ServiceDump RPC to avoid unnecessary disco chain compilation. As watches in proxy config only need the nodes.
* Option to set HCP client at runtime
Allows us to initially set a nil HCP client for the
telemetry provider and update it later.
* Set telemetry provider HCP client in HCP manager
Set the telemetry provider as a dependency and pass it to
the manager. Update the telemetry provider's HCP client
when the HCP manager starts.
* Add a provider interface for the metrics client
This provider will allow us to configure and reconfigure the
retryable HTTP client and the headers for the metrics client.
* Move HTTP retryable client to separate file
Copied directly from the metrics client.
* Abstract HCP specific values in HTTP client
Remove HCP specific references and instead initiate with
a generic TLS configuration and authentication source.
* Set up HTTP client and headers in the provider
Move setup from the metrics client to the HCP telemetry
provider.
* Update the telemetry provider in the HCP manager
Initialize the provider without the HCP configs and then update
it in the HCP manager to enable it.
* Improve test assertion, fix method comment
* Move client provider to metrics client
* Stop the manager on setup error
* Add separate lock for http configuration
* Start telemetry provider in HCP manager
* Update HCP client and config as part of Run
* Remove option to set config at initialization
* Simplify and clean up setting HCP configs
* Add test for telemetry provider Run method
* Fix race condition
* Use clone of HTTP headers
* Only allow initial update and run once
* Increase timeouts for flakey peering test.
* Various test fixes.
* Fix race condition in reconcilePeering.
This resolves an issue where a peering object in the state store was
incorrectly mutated by a function, resulting in the test being flagged as
failing when the -race flag was used.
To fix an issue displaying the current reloaded config in the
v1/agent/self endpoint #18681 caused the agent's internal
config struct member to be deepcopied and replaced on reload.
This is not safe because the field is not protected by a lock, nor
should it be due to how it is accessed by the rest of the system.
This PR does the same deepcopy, but into a new field solely for
the point of capturing the current reloaded values for display
purposes. If there has been no reload then the original config is used.
* Implement In-Process gRPC for use by controller caching/indexing
This replaces the pipe base listener implementation we were previously using. The new style CAN avoid cloning resources which our controller caching/indexing is taking advantage of to not duplicate resource objects in memory.
To maintain safety for controllers and for them to be able to modify data they get back from the cache and the resource service, the client they are presented in their runtime will be wrapped with an autogenerated client which clones request and response messages as they pass through the client.
Another sizable change in this PR is to consolidate how server specific gRPC services get registered and managed. Before this was in a bunch of different methods and it was difficult to track down how gRPC services were registered. Now its all in one place.
* Fix race in tests
* Ensure the resource service is registered to the multiplexed handler for forwarding from client agents
* Expose peer streaming on the internal handler
* Add HCCLink resource type
* Register HCCLink resource type with basic validation
* Add validation for required fields
* Add test for default ACLs
* Add no-op controller for HCCLink
* Add resource-apis semantic validation check in hcclink controller
* Add copyright headers
* Rename HCCLink to Link
* Add hcp_cluster_url to link proto
* Update 'disabled' reason with more detail
* Update link status name to consul.io/hcp/link
* Change link version from v1 to v2
* Use feature flag/experiment to enable v2 resources with HCP
* Update SCADA provider version
Also update mocks for SCADA provider.
* Create SCADA provider w/o HCP config, then update
Adds a placeholder config option to allow us to initialize a SCADA provider
without the HCP configuration. Also adds an update method to then add the
HCP configuration. We need this to be able to eventually always register a
SCADA listener at startup before the HCP config values are known.
* Pass cloud configuration to HCP manager
Save the entire cloud configuration and pass it to the HCP
manager.
* Update and start SCADA provider in HCP manager
Move config updating and starting to the HCP manager. The HCP manager
will eventually be responsible for all processes that contribute
to linking to HCP.
* NET-6945 - Replace usage of deprecated Envoy field envoy.config.core.v3.HeaderValueOption.append
* update proto for v2 and then update xds v2 logic
* add changelog
* Update 20078.txt to be consistent with existing changelog entries
* swap enum values tomatch envoy.
* NET-6946 - Replace usage of deprecated Envoy field envoy.config.route.v3.HeaderMatcher.safe_regex_match
* removing unrelated changes
* update golden files
* do not set engine type
* NET-6426 Create ProxyStateTemplate when reconciling MeshGateway resource
* Add TODO for switching fetch method based on gateway type
* Use gateway-kind in workload metadata instead of owner reference
* Create ProxyStateTemplate builder for gatewayproxy controller
* Update to use new controller interface
* Add copyright headers
* Set correct name for ProxyStateTemplate identity reference
* Generate empty ProxyStateTemplate by fetching MeshGateway
This cheats and looks up the MeshGateway directly. In the future, we will need a Workload => xGateway mapper
* Specify owner reference when writing ProxyStateTemplate
* Update dependency mapper to account for multiple controllers per resource type
* Regenerate v2 resource dependencies map
* Add helpful trace logs, tag TODOs with ticket identifiers
This commit fixes an issue where the partition was not properly set
on the peering query failover target created from sameness-groups.
Before this change, it was always empty, meaning that the data
would be queried with respect to the default partition always. This
resulted in a situation where a PQ that was attempting to use a
sameness-group for failover would select peers from the default
partition, rather than the partition of the sameness-group itself.
* add a hash to config entries when normalizing
* add GetHash and implement comparing hashes
* only update if the Hash is different
* only update if the Hash is different and not 0
* fix proto to include the Hash
* fix proto gen
* buf format
* add SetHash and fix tests
* fix config load tests
* fix state test and config test
* recalculate hash when restoring config entries
* fix snapshot restore test
* add changelog
* fix missing normalize, fix proto indexes and add normalize test
When a large number of upstreams are configured on a single envoy
proxy, there was a chance that it would timeout when waiting for
ClusterLoadAssignments. While this doesn't always immediately cause
issues, consul-dataplane instances appear to consistently drop
endpoints from their configurations after an xDS connection is
re-established (the server dies, random disconnect, etc).
This commit adds an `xds_fetch_timeout_ms` config to service registrations
so that users can set the value higher for large instances that have
many upstreams. The timeout can be disabled by setting a value of `0`.
This configuration was introduced to reduce the risk of causing a
breaking change for users if there is ever a scenario where endpoints
would never be received. Rather than just always blocking indefinitely
or for a significantly longer period of time, this config will affect
only the service instance associated with it.
This fixes the following race condition:
- Send update endpoints
- Send update cluster
- Recv ACK endpoints
- Recv ACK cluster
Prior to this fix, it would have resulted in the endpoints NOT existing in
Envoy. This occurred because the cluster update implicitly clears the endpoints
in Envoy, but we would never re-send the endpoint data to compensate for the
loss, because we would incorrectly ACK the invalid old endpoint hash. Since the
endpoint's hash did not actually change, they would not be resent.
The fix for this is to effectively clear out the invalid pending ACKs for child
resources whenever the parent changes. This ensures that we do not store the
child's hash as accepted when the race occurs.
An escape-hatch environment variable `XDS_PROTOCOL_LEGACY_CHILD_RESEND` was
added so that users can revert back to the old legacy behavior in the event
that this produces unknown side-effects. Visit the following thread for some
extra context on why certainty around these race conditions is difficult:
https://github.com/envoyproxy/envoy/issues/13009
This bug report and fix was mostly implemented by @ksmiley with some minor
tweaks.
Co-authored-by: Keith Smiley <ksmiley@salesforce.com>
* Add CE version of gateway-upstream-disambiguation
* Use NamespaceOrDefault and PartitionOrDefault
* Add Changelog entry
* Remove the unneeded reassignment
* Use c.ID()
The client.rpc metric now excludes internal retries for consistency
with client.rpc.exceeded and client.rpc.failed. All of these metrics
now increment at most once per RPC method call, allowing for
accurate calculation of failure / rate limit application occurrence.
Additionally, if an RPC fails because no servers are present,
client.rpc.failed is now incremented.
* Add a make target to run lint-consul-retry on all the modules
* Cleanup sdk/testutil/retry
* Fix a bunch of retry.Run* usage to not use the outer testing.T
* Fix some more recent retry lint issues and pin to v1.4.0 of lint-consul-retry
* Fix codegen copywrite lint issues
* Don’t perform cleanup after each retry attempt by default.
* Use the common testutil.TestingTB interface in test-integ/tenancy
* Fix retry tests
* Update otel access logging extension test to perform requests within the retry block
* Upgrade hcp-sdk-go to latest version v0.73
Changes:
- go get github.com/hashicorp/hcp-sdk-go
- go mod tidy
* From upgrade: regenerate protobufs for upgrade from 1.30 to 1.31
Ran: `make proto`
Slack: https://hashicorp.slack.com/archives/C0253EQ5B40/p1701105418579429
* From upgrade: fix mock interface implementation
After upgrading, there is the following compile error:
cannot use &mockHCPCfg{} (value of type *mockHCPCfg) as "github.com/hashicorp/hcp-sdk-go/config".HCPConfig value in return statement: *mockHCPCfg does not implement "github.com/hashicorp/hcp-sdk-go/config".HCPConfig (missing method Logout)
Solution: update the mock to have the missing Logout method
* From upgrade: Lint: remove usage of deprecated req.ServerState.TLS
Due to upgrade, linting is erroring due to usage of a newly deprecated field
22:47:56 [consul]: make lint
--> Running golangci-lint (.)
agent/hcp/testing.go:157:24: SA1019: req.ServerState.TLS is deprecated: use server_tls.internal_rpc instead. (staticcheck)
time.Until(time.Time(req.ServerState.TLS.CertExpiry)).Hours()/24,
^
* From upgrade: adjust oidc error message
From the upgrade, this test started failing:
=== FAIL: internal/go-sso/oidcauth TestOIDC_ClaimsFromAuthCode/failed_code_exchange (re-run 2) (0.01s)
oidc_test.go:393: unexpected error: Provider login failed: Error exchanging oidc code: oauth2: "invalid_grant" "unexpected auth code"
Prior to the upgrade, the error returned was:
```
Provider login failed: Error exchanging oidc code: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_grant\",\"error_description\":\"unexpected auth code\"}\n
```
Now the error returned is as below and does not contain "cannot fetch token"
```
Provider login failed: Error exchanging oidc code: oauth2: "invalid_grant" "unexpected auth code"
```
* Update AgentPushServerState structs with new fields
HCP-side changes for the new fields are in:
https://github.com/hashicorp/cloud-global-network-manager-service/pull/1195/files
* Minor refactor for hcpServerStatus to abstract tlsInfo into struct
This will make it easier to set the same tls-info information to both
- status.TLS (deprecated field)
- status.ServerTLSMetadata (new field to use instead)
* Update hcpServerStatus to parse out information for new fields
Changes:
- Improve error message and handling (encountered some issues and was confused)
- Set new field TLSInfo.CertIssuer
- Collect certificate authority metadata and set on TLSInfo.CertificateAuthorities
- Set TLSInfo on both server.TLS and server.ServerTLSMetadata.InternalRPC
* Update serverStatusToHCP to convert new fields to GNM rpc
* Add changelog
* Feedback: connect.ParseCert, caCerts
* Feedback: refactor and unit test server status
* Feedback: test to use expected struct
* Feedback: certificate with intermediate
* Feedback: catch no leaf, remove expectedErr
* Feedback: update todos with jira ticket
* Feedback: mock tlsConfigurator
* Update catalog and ui endpoints to show APIGateway in gateway service
topology view
* Added initial implementation for service view
* updated ui
* Fix topology view for gateways
* Adding tests for gw controller
* remove unused args
* Undo formatting changes
* Fix call sites for upstream/downstream gw changes
* Add config entry tests
* Fix function calls again
* Move from ServiceKey to ServiceName, cleanup from PR review
* Add additional check for length of services in bound apigateway for
IsSame comparison
* fix formatting for proto
* gofmt
* Add DeepCopy for retrieved BoundAPIGateway
* gofmt
* gofmt
* Rename function to be more consistent
* Add meshconfiguration/controller
* Add MeshConfiguration Registration function
* Fix the TODOs on the RegisterMeshGateway function
* Call RegisterMeshConfiguration
* Add comment to MeshConfigurationRegistration
* Add a test for Reconcile and some comments
Matt Keeler