113 Commits (b64674623e22e0873e88811830ad9c218f146d98)

Author	SHA1	Message	Date
Ronald	b64674623e	Copyright headers for missing files/folders (#16708) ... * copyright headers for agent folder	2 years ago
Ronald	94ec4eb2f4	copyright headers for agent folder (#16704) ... * copyright headers for agent folder * Ignore test data files * fix proto files and remove headers in agent/uiserver folder * ignore deep-copy files	2 years ago
Semir Patel	440f11203f	Resource service List(..) endpoint (#16753)	2 years ago
Semir Patel	032aba3175	WatchList(..) endpoint for the resource service (#16726)	2 years ago
Semir Patel	3415689eb6	Read(...) endpoint for the resource service (#16655)	2 years ago
Semir Patel	176945aa86	GRPC stub for the ResourceService (#16528)	2 years ago
Chris S. Kim	a518893685	Fix various flaky tests (#16396)	2 years ago
Derek Menteer	ad865f549b	Fix issue with peer services incorrectly appearing as connect-enabled. (#16339) ... Prior to this commit, all peer services were transmitted as connect-enabled as long as a one or more mesh-gateways were healthy. With this change, there is now a difference between typical services and connect services transmitted via peering. A service will be reported as "connect-enabled" as long as any of these conditions are met: 1. a connect-proxy sidecar is registered for the service name. 2. a connect-native instance of the service is registered. 3. a service resolver / splitter / router is registered for the service name. 4. a terminating gateway has registered the service.	2 years ago
Matt Keeler	085c0addc0	Protobuf Refactoring for Multi-Module Cleanliness (#16302) ... Protobuf Refactoring for Multi-Module Cleanliness This commit includes the following: Moves all packages that were within proto/ to proto/private Rewrites imports to account for the packages being moved Adds in buf.work.yaml to enable buf workspaces Names the proto-public buf module so that we can override the Go package imports within proto/buf.yaml Bumps the buf version dependency to 1.14.0 (I was trying out the version to see if it would get around an issue - it didn't but it also doesn't break things and it seemed best to keep up with the toolchain changes) Why: In the future we will need to consume other protobuf dependencies such as the Google HTTP annotations for openapi generation or grpc-gateway usage. There were some recent changes to have our own ratelimiting annotations. The two combined were not working when I was trying to use them together (attempting to rebase another branch) Buf workspaces should be the solution to the problem Buf workspaces means that each module will have generated Go code that embeds proto file names relative to the proto dir and not the top level repo root. This resulted in proto file name conflicts in the Go global protobuf type registry. The solution to that was to add in a private/ directory into the path within the proto/ directory. That then required rewriting all the imports. Is this safe? AFAICT yes The gRPC wire protocol doesn't seem to care about the proto file names (although the Go grpc code does tack on the proto file name as Metadata in the ServiceDesc) Other than imports, there were no changes to any generated code as a result of this.	2 years ago
Dan Stough	f1436109ea	[OSS] security: update go to 1.20.1 (#16263) ... * security: update go to 1.20.1	2 years ago
Nathan Coleman	72a73661c9	Implement APIGateway proxycfg snapshot (#16194) ... * Stub proxycfg handler for API gateway * Add Service Kind constants/handling for API Gateway * Begin stubbing for SDS * Add new Secret type to xDS order of operations * Continue stubbing of SDS * Iterate on proxycfg handler for API gateway * Handle BoundAPIGateway config entry subscription in proxycfg-glue * Add API gateway to config snapshot validation * Add API gateway to config snapshot clone, leaf, etc. * Subscribe to bound route + cert config entries on bound-api-gateway * Track routes + certs on API gateway config snapshot * Generate DeepCopy() for types used in watch.Map * Watch all active references on api-gateway, unwatch inactive * Track loading of initial bound-api-gateway config entry * Use proper proto package for SDS mapping * Use ResourceReference instead of ServiceName, collect resources * Fix typo, add + remove TODOs * Watch discovery chains for TCPRoute * Add TODO for updating gateway services for api-gateway * make proto * Regenerate deep-copy for proxycfg * Set datacenter on upstream ID from query source * Watch discovery chains for http-route service backends * Add ServiceName getter to HTTP+TCP Service structs * Clean up unwatched discovery chains on API Gateway * Implement watch for ingress leaf certificate * Collect upstreams on http-route + tcp-route updates * Remove unused GatewayServices update handler * Remove unnecessary gateway services logic for API Gateway * Remove outdate TODO * Use .ToIngress where appropriate, including TODO for cleaning up * Cancel before returning error * Remove GatewayServices subscription * Add godoc for handlerAPIGateway functions * Update terminology from Connect => Consul Service Mesh Consistent with terminology changes in https://github.com/hashicorp/consul/pull/12690 * Add missing TODO * Remove duplicate switch case * Rerun deep-copy generator * Use correct property on config snapshot * Remove unnecessary leaf cert watch * Clean up based on code review feedback * Note handler properties that are initialized but set elsewhere * Add TODO for moving helper func into structs pkg * Update generated DeepCopy code * gofmt * Generate DeepCopy() for API gateway listener types * Improve variable name * Regenerate DeepCopy() code * Fix linting issue * Temporarily remove the secret type from resource generation	2 years ago
skpratt	a010902978	Remove legacy acl policies (#15922) ... * remove legacy tokens * remove legacy acl policies * flatten test policies to *_prefix * address oss feedback re: phrasing and tests	2 years ago
Dan Upton	7a55de375c	xds: don't attempt to load-balance sessions for local proxies (#15789) ... Previously, we'd begin a session with the xDS concurrency limiter regardless of whether the proxy was registered in the catalog or in the server's local agent state. This caused problems for users who run `consul connect envoy` directly against a server rather than a client agent, as the server's locally registered proxies wouldn't be included in the limiter's capacity. Now, the `ConfigSource` is responsible for beginning the session and we only do so for services in the catalog. Fixes: https://github.com/hashicorp/consul/issues/15753	2 years ago
Chris S. Kim	e4a268e33e	Warn if ACL is enabled but no token is provided to Envoy (#15967)	2 years ago
Dan Stough	6d2880e894	feat: add access logs to dataplane bootstrap rpc (#15951)	2 years ago
Matt Keeler	5afd4657ec	Protobuf Modernization (#15949) ... * Protobuf Modernization Remove direct usage of golang/protobuf in favor of google.golang.org/protobuf Marshallers (protobuf and json) needed some changes to account for different APIs. Moved to using the google.golang.org/protobuf/types/known/* for the well known types including replacing some custom Struct manipulation with whats available in the structpb well known type package. This also updates our devtools script to install protoc-gen-go from the right location so that files it generates conform to the correct interfaces. * Fix go-mod-tidy make target to work on all modules	2 years ago
Chris S. Kim	a7b34d50fc	Output user-friendly name for anonymous token (#15884)	2 years ago
Dan Upton	b78de5a7a2	grpc/acl: fix bug where ACL token was required even if disabled (#15904) ... Fixes a bug introduced by #15346 where we'd always require an ACL token even if ACLs were disabled because we were erroneously treating `nil` identity as anonymous.	2 years ago
Dan Upton	d4c435856b	grpc: `protoc` plugin for generating gRPC rate limit specifications (#15564) ... Adds automation for generating the map of `gRPC Method Name → Rate Limit Type` used by the middleware introduced in #15550, and will ensure we don't forget to add new endpoints. Engineers must annotate their RPCs in the proto file like so: ``` rpc Foo(FooRequest) returns (FooResponse) { option (consul.internal.ratelimit.spec) = { operation_type: READ, }; } ``` When they run `make proto` a protoc plugin `protoc-gen-consul-rate-limit` will be installed that writes rate-limit specs as a JSON array to a file called `.ratelimit.tmp` (one per protobuf package/directory). After running Buf, `make proto` will execute a post-process script that will ingest all of the `.ratelimit.tmp` files and generate a Go file containing the mappings in the `agent/grpc-middleware` package. In the enterprise repository, it will write an additional file with the enterprise-only endpoints. If an engineer forgets to add the annotation to a new RPC, the plugin will return an error like so: ``` RPC Foo is missing rate-limit specification, fix it with: import "proto-public/annotations/ratelimit/ratelimit.proto"; service Bar { rpc Foo(...) returns (...) { option (hashicorp.consul.internal.ratelimit.spec) = { operation_type: OPERATION_READ | OPERATION_WRITE | OPERATION_EXEMPT, }; } } ``` In the future, this annotation can be extended to support rate-limit category (e.g. KV vs Catalog) and to determine the retry policy.	2 years ago
Dan Upton	7c7503c849	grpc/acl: relax permissions required for "core" endpoints (#15346) ... Previously, these endpoints required `service:write` permission on _any_ service as a sort of proxy for "is the caller allowed to participate in the mesh?". Now, they're called as part of the process of establishing a server connection by any consumer of the consul-server-connection-manager library, which will include non-mesh workloads (e.g. Consul KV as a storage backend for Vault) as well as ancillary components such as consul-k8s' acl-init process, which likely won't have `service:write` permission. So this commit relaxes those requirements to accept *any* valid ACL token on the following gRPC endpoints: - `hashicorp.consul.dataplane.DataplaneService/GetSupportedDataplaneFeatures` - `hashicorp.consul.serverdiscovery.ServerDiscoveryService/WatchServers` - `hashicorp.consul.connectca.ConnectCAService/WatchRoots`	2 years ago
John Murret	f5e01f8c6b	Rate Limit Handler - ensure rate limiting is not in the code path when not configured (#15819) ... * Rate limiting handler - ensure configuration has changed before modifying limiters * Updating test to validate arguments to UpdateConfig * Removing duplicate test. Updating mock. * Renaming NullRateLimiter to NullRequestLimitsHandler * Rate Limit Handler - ensure rate limiting is not in the code path when not configured * Update agent/consul/rate/handler.go Co-authored-by: Dhia Ayachi <dhia@hashicorp.com> * formatting handler.go * Rate limiting handler - ensure configuration has changed before modifying limiters * Updating test to validate arguments to UpdateConfig * Removing duplicate test. Updating mock. * adding logging for when UpdateConfig is called but the config has not changed. * Update agent/consul/rate/handler.go Co-authored-by: Dhia Ayachi <dhia@hashicorp.com> * Update agent/consul/rate/handler_test.go Co-authored-by: Dan Upton <daniel@floppy.co> * modifying existing variable name based on pr feedback * updating a broken merge conflict; Co-authored-by: Dhia Ayachi <dhia@hashicorp.com> Co-authored-by: Dan Upton <daniel@floppy.co>	2 years ago
John Murret	e027c94b52	adding config for request_limits (#15531) ... * server: add placeholder glue for rate limit handler This commit adds a no-op implementation of the rate-limit handler and adds it to the `consul.Server` struct and setup code. This allows us to start working on the net/rpc and gRPC interceptors and config logic. * Add handler errors * Set the global read and write limits * fixing multilimiter moving packages * Fix typo * Simplify globalLimit usage * add multilimiter and tests * exporting LimitedEntity * Apply suggestions from code review Co-authored-by: John Murret <john.murret@hashicorp.com> * add config update and rename config params * add doc string and split config * Apply suggestions from code review Co-authored-by: Dan Upton <daniel@floppy.co> * use timer to avoid go routine leak and change the interface * add comments to tests * fix failing test * add prefix with config edge, refactor tests * Apply suggestions from code review Co-authored-by: Dan Upton <daniel@floppy.co> * refactor to apply configs for limiters under a prefix * add fuzz tests and fix bugs found. Refactor reconcile loop to have a simpler logic * make KeyType an exported type * split the config and limiter trees to fix race conditions in config update * rename variables * fix race in test and remove dead code * fix reconcile loop to not create a timer on each loop * add extra benchmark tests and fix tests * fix benchmark test to pass value to func * server: add placeholder glue for rate limit handler This commit adds a no-op implementation of the rate-limit handler and adds it to the `consul.Server` struct and setup code. This allows us to start working on the net/rpc and gRPC interceptors and config logic. * Set the global read and write limits * fixing multilimiter moving packages * add server configuration for global rate limiting. * remove agent test * remove added stuff from handler * remove added stuff from multilimiter * removing unnecessary TODOs * Removing TODO comment from handler * adding in defaulting to infinite * add disabled status in there * adding in documentation for disabled mode. * make disabled the default. * Add mock and agent test * addig documentation and missing mock file. * Fixing test TestLoad_IntegrationWithFlags * updating docs based on PR feedback. * Updating Request Limits mode to use int based on PR feedback. * Adding RequestLimits struct so we have a nested struct in ReloadableConfig. * fixing linting references * Update agent/consul/rate/handler.go Co-authored-by: Dan Upton <daniel@floppy.co> * Update agent/consul/config.go Co-authored-by: Dan Upton <daniel@floppy.co> * removing the ignore of the request limits in JSON. addingbuilder logic to convert any read rate or write rate less than 0 to rate.Inf * added conversion function to convert request limits object to handler config. * Updating docs to reflect gRPC and RPC are rate limit and as a result, HTTP requests are as well. * Updating values for TestLoad_FullConfig() so that they were different and discernable. * Updating TestRuntimeConfig_Sanitize * Fixing TestLoad_IntegrationWithFlags test * putting nil check in place * fixing rebase * removing change for missing error checks. will put in another PR * Rebasing after default multilimiter config change * resolving rebase issues * updating reference for incomingRPCLimiter to use interface * updating interface * Updating interfaces * Fixing mock reference Co-authored-by: Daniel Upton <daniel@floppy.co> Co-authored-by: Dhia Ayachi <dhia@hashicorp.com>	2 years ago
Dan Upton	c692802dec	grpc: add rate-limiting middleware (#15550) ... Implements the gRPC middleware for rate-limiting as a tap.ServerInHandle function (executed before the request is unmarshaled). Mappings between gRPC methods and their operation type are generated by a protoc plugin introduced by #15564.	2 years ago
R.B. Boyer	11a277f372	peering: better represent non-passing states during peer check flattening (#15615) ... During peer stream replication we flatten checks from the source cluster and build one thin overall check to hide the irrelevant details from the consuming cluster. This flattening logic did correctly flip to non-passing if there were any non-passing checks, but WHICH status it got during that was random (warn/error). Also it didn't represent "maintenance" operations. There is an api package call AggregatedStatus which more correctly flattened check statuses. This PR replicated the more complete logic into the peer stream package.	2 years ago
Dan Upton	7b2d08d461	chore: remove unused argument from MergeNodeServiceWithCentralConfig (#15024) ... Previously, the MergeNodeServiceWithCentralConfig method accepted a ServiceSpecificRequest argument, of which only the Datacenter and QueryOptions fields were used. Digging a little deeper, it turns out these fields were only passed down to the ComputeResolvedServiceConfig method (through the ServiceConfigRequest struct) which didn't actually use them. As such, not all call-sites passed a valid ServiceSpecificRequest so it's safer to remove the argument altogether to prevent future changes from depending on it.	2 years ago
Derek Menteer	b64972d486	Bring back parameter ServerExternalAddresses in GenerateToken endpoint (#15267) ... Re-add ServerExternalAddresses parameter in GenerateToken endpoint This reverts commit 5e156772f6 and adds extra functionality to support newer peering behaviors.	2 years ago
Chris S. Kim	985a4ee1b1	Update hcp-scada-provider to fix diamond dependency problem with go-msgpack (#15185)	2 years ago
Derek Menteer	18d6c338f4	Backport tests from ent. (#15260) ... * Backport agent tests. Original commit: 0710b2d12fb51a29cedd1119b5fb086e5c71f632 Original commit: aaedb3c28bfe247266f21013d500147d8decb7cd (partial) * Backport test fix and reduce flaky failures.	2 years ago
Derek Menteer	f4cb2f82bf	Backport various fixes from ENT. (#15254) ... * Regenerate golden files. * Backport from ENT: "Avoid race" Original commit: 5006c8c858b0e332be95271ef9ba35122453315b Original author: freddygv * Backport from ENT: "chore: fix flake peerstream test" Original commit: b74097e7135eca48cc289798c5739f9ef72c0cc8 Original author: DanStough	2 years ago
malizz	617a5f2dc2	convert stream status time fields to pointers (#15252)	2 years ago
Derek Menteer	bd1019fadb	Prevent peering acceptor from subscribing to addr updates. (#15214)	2 years ago
Derek Menteer	693c8a4706	Allow peering endpoints to bypass verify_incoming.	2 years ago
cskh	a9427e1310	fix(peering): nil pointer in calling handleUpdateService (#15160) ... * fix(peering): nil pointer in calling handleUpdateService * changelog	2 years ago
Chris S. Kim	bde57c0dd0	Regenerate files according to 1.19.2 formatter	2 years ago
cskh	db82ffe503	fix(peering): replicating wan address (#15108) ... * fix(peering): replicating wan address * add changelog * unit test	2 years ago
R.B. Boyer	300860412c	chore: update golangci-lint to v1.50.1 (#15022)	2 years ago
cskh	d562d363fc	peering: skip registering duplicate node and check from the peer (#14994) ... * peering: skip register duplicate node and check from the peer * Prebuilt the nodes map and checks map to avoid repeated for loop * use key type to struct: node id, service id, and check id	2 years ago
Freddy	24d0c8801a	Merge pull request #14981 from hashicorp/peering/dial-through-gateways	2 years ago
Derek Menteer	0f424e3cdf	Reset wait on ensureServerAddrSubscription	2 years ago
freddygv	96fdd3728a	Fix CA init error code	2 years ago
malizz	b0b0cbb8ee	increase protobuf size limit for cluster peering (#14976)	2 years ago
Derek Menteer	8742fbe14f	Prevent consul peer-exports by discovery chain.	2 years ago
Derek Menteer	f366edcb8d	Prevent the "consul" service from being exported.	2 years ago
Derek Menteer	caa1396255	Add remote peer partition and datacenter info.	2 years ago
Freddy	9ca8bb8ec4	Merge pull request #14958 from hashicorp/peering/nonce	2 years ago
freddygv	1b46b35041	Actually track nonce in test	2 years ago
Derek Menteer	f330438a45	Fix incorrect backoff-wait logic.	2 years ago
freddygv	7f9a5d0f58	Add basic nonce management ... This commit adds a monotonically increasing nonce to include in peering replication response messages. Every ack/nack from the peer handling a response will include this nonce, allowing to correlate the ack/nack with a specific resource. At the moment nothing is done with the nonce when it is received. In the future we may want to add functionality such as retries on NACKs, depending on the class of error.	2 years ago
Paul Glass	d17af23641	gRPC server metrics (#14922) ... * Move stats.go from grpc-internal to grpc-middleware * Update grpc server metrics with server type label * Add stats test to grpc-external * Remove global metrics instance from grpc server tests	2 years ago
freddygv	bf72df7b0e	Fixup test	2 years ago