consul

Commit Graph

Author	SHA1	Message	Date
Dhia Ayachi	2d902b26ac	add log-drop package (#15670 ) * add log-drop package * refactor to extract level * extract metrics * Apply suggestions from code review Co-authored-by: Dan Upton <daniel@floppy.co> * fix compile errors * change to implement a log sink * fix tests to remove sleep * rename and add go docs * fix expending variadic Co-authored-by: Dan Upton <daniel@floppy.co>	2 years ago
Paul Glass	619032cfcd	Deprecate -join and -join-wan (#15598 )	2 years ago
Dhia Ayachi	6468e3e09c	Server side rate limiter: handle the race condition for limiters tree write in multilimiter (#15767 ) * change to perform all tree writes in the same go routine to avoid race condition. * rename runStoreOnce to reconcile * Apply suggestions from code review Co-authored-by: Dan Upton <daniel@floppy.co> * reduce nesting Co-authored-by: Dan Upton <daniel@floppy.co>	2 years ago
Semir Patel	bafa5c7156	Pass remote addr of incoming HTTP requests through to RPC(..) calls (#15700 )	2 years ago
John Murret	e027c94b52	adding config for request_limits (#15531 ) * server: add placeholder glue for rate limit handler This commit adds a no-op implementation of the rate-limit handler and adds it to the `consul.Server` struct and setup code. This allows us to start working on the net/rpc and gRPC interceptors and config logic. * Add handler errors * Set the global read and write limits * fixing multilimiter moving packages * Fix typo * Simplify globalLimit usage * add multilimiter and tests * exporting LimitedEntity * Apply suggestions from code review Co-authored-by: John Murret <john.murret@hashicorp.com> * add config update and rename config params * add doc string and split config * Apply suggestions from code review Co-authored-by: Dan Upton <daniel@floppy.co> * use timer to avoid go routine leak and change the interface * add comments to tests * fix failing test * add prefix with config edge, refactor tests * Apply suggestions from code review Co-authored-by: Dan Upton <daniel@floppy.co> * refactor to apply configs for limiters under a prefix * add fuzz tests and fix bugs found. Refactor reconcile loop to have a simpler logic * make KeyType an exported type * split the config and limiter trees to fix race conditions in config update * rename variables * fix race in test and remove dead code * fix reconcile loop to not create a timer on each loop * add extra benchmark tests and fix tests * fix benchmark test to pass value to func * server: add placeholder glue for rate limit handler This commit adds a no-op implementation of the rate-limit handler and adds it to the `consul.Server` struct and setup code. This allows us to start working on the net/rpc and gRPC interceptors and config logic. * Set the global read and write limits * fixing multilimiter moving packages * add server configuration for global rate limiting. * remove agent test * remove added stuff from handler * remove added stuff from multilimiter * removing unnecessary TODOs * Removing TODO comment from handler * adding in defaulting to infinite * add disabled status in there * adding in documentation for disabled mode. * make disabled the default. * Add mock and agent test * addig documentation and missing mock file. * Fixing test TestLoad_IntegrationWithFlags * updating docs based on PR feedback. * Updating Request Limits mode to use int based on PR feedback. * Adding RequestLimits struct so we have a nested struct in ReloadableConfig. * fixing linting references * Update agent/consul/rate/handler.go Co-authored-by: Dan Upton <daniel@floppy.co> * Update agent/consul/config.go Co-authored-by: Dan Upton <daniel@floppy.co> * removing the ignore of the request limits in JSON. addingbuilder logic to convert any read rate or write rate less than 0 to rate.Inf * added conversion function to convert request limits object to handler config. * Updating docs to reflect gRPC and RPC are rate limit and as a result, HTTP requests are as well. * Updating values for TestLoad_FullConfig() so that they were different and discernable. * Updating TestRuntimeConfig_Sanitize * Fixing TestLoad_IntegrationWithFlags test * putting nil check in place * fixing rebase * removing change for missing error checks. will put in another PR * Rebasing after default multilimiter config change * resolving rebase issues * updating reference for incomingRPCLimiter to use interface * updating interface * Updating interfaces * Fixing mock reference Co-authored-by: Daniel Upton <daniel@floppy.co> Co-authored-by: Dhia Ayachi <dhia@hashicorp.com>	2 years ago
Dan Stough	233dbcb67f	feat: add access logging API to proxy defaults (#15780 )	2 years ago
cskh	04bf24c8c1	feat(ingress-gateway): support outlier detection of upstream service for ingress gateway (#15614 ) * feat(ingress-gateway): support outlier detection of upstream service for ingress gateway * changelog Co-authored-by: Eric Haberkorn <erichaberkorn@gmail.com>	2 years ago
Derek Menteer	e87d35e313	Fix DialedDirectly configuration for Consul dataplane. (#15760 ) Fix DialedDirectly configuration for Consul dataplane.	2 years ago
Dan Upton	c692802dec	grpc: add rate-limiting middleware (#15550 ) Implements the gRPC middleware for rate-limiting as a tap.ServerInHandle function (executed before the request is unmarshaled). Mappings between gRPC methods and their operation type are generated by a protoc plugin introduced by #15564.	2 years ago
Dan Upton	eef38c2199	server: add placeholder glue for rate limit handler (#15539 ) Adds a no-op implementation of the rate-limit handler and exposes it on the consul.Server struct. It allows us to start working on the net/rpc and gRPC interceptors and config (re)loading logic, without having to implement the full handler up-front. Co-authored-by: John Murret <john.murret@hashicorp.com> Co-authored-by: Dhia Ayachi <dhia@hashicorp.com>	2 years ago
John Murret	cd53120cd7	agent: Fix assignment of error when auto-reloading cert and key file changes. (#15769 ) * Adding the setting of errors missing in config file watcher code in agent. * add changelog	2 years ago
R.B. Boyer	4a32070210	test: remove variable shadowing in TestDNS_ServiceLookup_ARecordLimits (#15740 )	2 years ago
Eric Haberkorn	4268c1c25c	Remove the `connect.enable_serverless_plugin` agent configuration option (#15710 )	2 years ago
Dhia Ayachi	81e40c1fac	add multilimiter and tests (#15467 ) * add multilimiter and tests * exporting LimitedEntity * go mod tidy * Apply suggestions from code review Co-authored-by: John Murret <john.murret@hashicorp.com> * add config update and rename config params * add doc string and split config * Apply suggestions from code review Co-authored-by: Dan Upton <daniel@floppy.co> * use timer to avoid go routine leak and change the interface * add comments to tests * fix failing test * add prefix with config edge, refactor tests * Apply suggestions from code review Co-authored-by: Dan Upton <daniel@floppy.co> * refactor to apply configs for limiters under a prefix * add fuzz tests and fix bugs found. Refactor reconcile loop to have a simpler logic * make KeyType an exported type * split the config and limiter trees to fix race conditions in config update * rename variables * fix race in test and remove dead code * fix reconcile loop to not create a timer on each loop * add extra benchmark tests and fix tests * fix benchmark test to pass value to func * use a separate go routine to write limiters (#15643) * use a separate go routine to write limiters * Add updating limiter when another limiter is created * fix waiter to be a ticker, so we commit more than once. * fix tests and add tests for coverage * unexport members and add tests * make UpdateConfig thread safe and multi call to Run safe * replace swith with if * fix review comments * replace time.sleep with retries * fix flaky test and remove unnecessary init * fix test races * remove unnecessary negative test case * remove fixed todo Co-authored-by: John Murret <john.murret@hashicorp.com> Co-authored-by: Dan Upton <daniel@floppy.co>	2 years ago
cskh	3df68751f5	Flakiness test: case-cfg-splitter-peering-ingress-gateways (#15707 ) * integ-test: fix flaky test - case-cfg-splitter-peering-ingress-gateways * add retry peering to all peering cases Co-authored-by: Dan Stough <dan.stough@hashicorp.com>	2 years ago
Derek Menteer	97ec5279aa	Fix local mesh gateway with peering discovery chains. (#15690 ) Fix local mesh gateway with peering discovery chains. Prior to this patch, discovery chains with peers would not properly honor the mesh gateway mode for two reasons. 1. An incorrect target upstream ID was used to lookup the mesh gateway mode. To fix this, the parent upstream uid is now used instead of the discovery-chain-target-uid to find the intended mesh gateway mode. 2. The watch for local mesh gateways was never initialized for discovery chains. To fix this, the discovery chains are now scanned, and a local GW watch is spawned if: the mesh gateway mode is local and the target is a peering connection.	2 years ago
R.B. Boyer	5af94fb2a0	connect: use -dev-no-store-token for test vaults to reduce source of flakes (#15691 ) It turns out that by default the dev mode vault server will attempt to interact with the filesystem to store the provided root token. If multiple vault instances are running they'll all awkwardly share the filesystem and if timing results in one server stopping while another one is starting then the starting one will error with: Error initializing Dev mode: rename /home/circleci/.vault-token.tmp /home/circleci/.vault-token: no such file or directory This change uses `-dev-no-store-token` to bypass that source of flakes. Also the stdout/stderr from the vault process is included if the test fails. The introduction of more `t.Parallel` use in https://github.com/hashicorp/consul/pull/15669 increased the likelihood of this failure, but any of the tests with multiple vaults in use (or running multiple package tests in parallel that all use vault) were eventually going to flake on this.	2 years ago
R.B. Boyer	900584ca82	connect: ensure all vault connect CA tests use limited privilege tokens (#15669 ) All of the current integration tests where Vault is the Connect CA now use non-root tokens for the test. This helps us detect privilege changes in the vault model so we can keep our guides up to date. One larger change was that the RenewIntermediate function got refactored slightly so it could be used from a test, rather than the large duplicated function we were testing in a test which seemed error prone.	2 years ago
R.B. Boyer	4940a728ab	Detect Vault 1.11+ import in secondary datacenters and update default issuer (#15661 ) The fix outlined and merged in #15253 fixed the issue as it occurs in the primary DC. There is a similar issue that arises when vault is used as the Connect CA in a secondary datacenter that is fixed by this PR. Additionally: this PR adds support to run the existing suite of vault related integration tests against the last 4 versions of vault (1.9, 1.10, 1.11, 1.12)	2 years ago
Chris S. Kim	c046d1a4d8	Add warn log when all ACL policies are filtered out (#15632 )	2 years ago
cskh	36f05bc8fb	integ-test: test consul upgrade from the snapshot of a running cluster (#15595 ) * integ-test: test consul upgrade from the snapshot of a running cluster * use Target version as default Co-authored-by: Dan Stough <dan.stough@hashicorp.com>	2 years ago
R.B. Boyer	11a277f372	peering: better represent non-passing states during peer check flattening (#15615 ) During peer stream replication we flatten checks from the source cluster and build one thin overall check to hide the irrelevant details from the consuming cluster. This flattening logic did correctly flip to non-passing if there were any non-passing checks, but WHICH status it got during that was random (warn/error). Also it didn't represent "maintenance" operations. There is an api package call AggregatedStatus which more correctly flattened check statuses. This PR replicated the more complete logic into the peer stream package.	2 years ago
Freddy	941f6da202	Remove log line about server mgmt token init (#15610 ) * Remove log line about server mgmt token init Currently the server management token is only being bootstrapped in the primary datacenter. That means that servers on the secondary datacenter will never have this token available, and would log this line any time a token is resolved. Bootstrapping the token in secondary datacenters will be done in a follow-up. * Add changelog entry	2 years ago
James Oulman	7e78fb7818	Add support for configuring Envoys route idle_timeout (#14340 ) * Add idleTimeout Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> Co-authored-by: Dhia Ayachi <dhia@hashicorp.com>	2 years ago
Derek Menteer	95dc0c7b30	Add peering `.service` and `.node` DNS lookups. (#15596 ) Add peering `.service` and `.node` DNS lookups.	2 years ago
cskh	97c9432843	fix(peering): increase the gRPC limit to 8MB (#15503 ) * fix(peering): increase the gRPC limit to 50MB * changelog * update gRPC limit to 8MB	2 years ago
Chris S. Kim	c9ec9fa320	Fix Vault managed intermediate PKI bug (#15525 )	2 years ago
Chris S. Kim	27c53f6c82	Use backport-compatible assertion (#15546 ) * Use backport-compatible assertion * Add workaround for broken apt-get	2 years ago
Chris S. Kim	386da5439a	Use rpcHoldTimeout to calculate blocking timeout (#15541 ) Adds buffer to clients so that servers have time to respond to blocking queries.	2 years ago
Jared Kirschner	3e7e8ae9c5	Support RFC 2782 for prepared query DNS lookups (#14465 ) Format: _<query id or name>._tcp.query[.<datacenter>].<domain>	2 years ago
Alexander Scheel	2b90307f6d	Detect Vault 1.11+ import, update default issuer (#15253 ) Consul used to rely on implicit issuer selection when calling Vault endpoints to issue new CSRs. Vault 1.11+ changed that behavior, which caused Consul to check the wrong (previous) issuer when renewing its Intermediate CA. This patch allows Consul to explicitly set a default issuer when it detects that the response from Vault is 1.11+. Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com> Co-authored-by: Chris S. Kim <ckim@hashicorp.com>	2 years ago
cskh	435e16ecda	fix: clarifying error message when acquiring a lock in remote dc (#15394 ) * fix: clarifying error message when acquiring a lock in remote dc * Update website/content/commands/lock.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>	2 years ago
Kyle Havlovitz	f4c3e54b11	auto-config: relax node name validation for JWT authorization (#15370 ) * auto-config: relax node name validation for JWT authorization This changes the JWT authorization logic to allow all non-whitespace, non-quote characters when validating node names. Consul had previously allowed these characters in node names, until this validation was added to fix a security vulnerability with whitespace/quotes being passed to the `bexpr` library. This unintentionally broke node names with characters like `.` which aren't related to this vulnerability. * Update website/content/docs/agent/config/cli-flags.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>	2 years ago
Dhia Ayachi	225ae55e83	Leadership transfer cmd (#14132 ) * add leadership transfer command * add RPC call test (flaky) * add missing import * add changelog * add command registration * Apply suggestions from code review Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com> * add the possibility of providing an id to raft leadership transfer. Add few tests. * delete old file from cherry pick * rename changelog filename to PR # * rename changelog and fix import * fix failing test * check for OperatorWrite Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com> * rename from leader-transfer to transfer-leader * remove version check and add test for operator read * move struct to operator.go * first pass * add code for leader transfer in the grpc backend and tests * wire the http endpoint to the new grpc endpoint * remove the RPC endpoint * remove non needed struct * fix naming * add mog glue to API * fix comment * remove dead code * fix linter error * change package name for proto file * remove error wrapping * fix failing test * add command registration * add grpc service mock tests * fix receiver to be pointer * use defined values Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com> * reuse MockAclAuthorizer * add documentation * remove usage of external.TokenFromContext * fix failing tests * fix proto generation * Apply suggestions from code review Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com> * Apply suggestions from code review * add more context in doc for the reason * Apply suggestions from docs code review Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> * regenerate proto * fix linter errors Co-authored-by: github-team-consul-core <github-team-consul-core@hashicorp.com> Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com> Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com> Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com>	2 years ago
Freddy	706866fa00	Ensure that NodeDump imported nodes are filtered (#15356 )	2 years ago
Freddy	c58f86a00f	Fixup authz for data imported from peers (#15347 ) There are a few changes that needed to be made to to handle authorizing reads for imported data: - If the data was imported from a peer we should not attempt to read the data using the traditional authz rules. This is because the name of services/nodes in a peer cluster are not equivalent to those of the importing cluster. - If the data was imported from a peer we need to check whether the token corresponds to a service, meaning that it has service:write permissions, or to a local read only token that can read all nodes/services in a namespace. This required changes at the policyAuthorizer level, since that is the only view available to OSS Consul, and at the enterprise partition/namespace level.	2 years ago
Kyle Havlovitz	dde5c524ad	connect: strip port from DNS SANs for ingress gateway leaf cert (#15320 ) * connect: strip port from DNS SANs for ingress gateway leaf cert * connect: format DNS SANs in CreateCSR * connect: Test wildcard case when formatting SANs	2 years ago
Derek Menteer	931cec42b3	Prevent serving TLS via ports.grpc (#15339 ) Prevent serving TLS via ports.grpc We remove the ability to run the ports.grpc in TLS mode to avoid confusion and to simplify configuration. This breaking change ensures that any user currently using ports.grpc in an encrypted mode will receive an error message indicating that ports.grpc_tls must be explicitly used. The suggested action for these users is to simply swap their ports.grpc to ports.grpc_tls in the configuration file. If both ports are defined, or if the user has not configured TLS for grpc, then the error message will not be printed.	2 years ago
Dan Stough	626249fbf5	[OSS] fix: wait and try longer to peer through mesh gw (#15328 )	2 years ago
Kyle Schochenmaier	bf0f61a878	removes ioutil usage everywhere which was deprecated in go1.16 (#15297 ) * update go version to 1.18 for api and sdk, go mod tidy * removes ioutil usage everywhere which was deprecated in go1.16 in favour of io and os packages. Also introduces a lint rule which forbids use of ioutil going forward. Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>	2 years ago
malizz	b51f0e25e9	update ACLs for cluster peering (#15317 ) * update ACLs for cluster peering * add changelog * Update .changelog/15317.txt Co-authored-by: Eric Haberkorn <erichaberkorn@gmail.com> Co-authored-by: Eric Haberkorn <erichaberkorn@gmail.com>	2 years ago
malizz	b9a9e1219c	update config defaults, add docs (#15302 ) * update config defaults, add docs * update grpc tls port for non-default values * add changelog * Update website/content/docs/upgrading/upgrade-specific.mdx Co-authored-by: Derek Menteer <105233703+hashi-derek@users.noreply.github.com> * Update website/content/docs/agent/config/config-files.mdx Co-authored-by: Derek Menteer <105233703+hashi-derek@users.noreply.github.com> * update logic for setting grpc tls port value * move default config to default.go, update changelog * update docs * Fix config tests. * Fix linter error. * Fix ConnectCA tests. * Cleanup markdown on upgrade notes. Co-authored-by: Derek Menteer <105233703+hashi-derek@users.noreply.github.com> Co-authored-by: Derek Menteer <derek.menteer@hashicorp.com>	2 years ago
Eric Haberkorn	c340922991	Log Warnings When Peering With Mesh Gateway Mode None (#15304 ) warn when mesh gateway mode is set to none for peering	2 years ago
Derek Menteer	418bd62c44	Fix mesh gateway configuration with proxy-defaults (#15186 ) * Fix mesh gateway proxy-defaults not affecting upstreams. * Clarify distinction with upstream settings Top-level mesh gateway mode in proxy-defaults and service-defaults gets merged into NodeService.Proxy.MeshGateway, and only gets merged with the mode attached to an an upstream in proxycfg/xds. * Fix mgw mode usage for peered upstreams There were a couple issues with how mgw mode was being handled for peered upstreams. For starters, mesh gateway mode from proxy-defaults and the top-level of service-defaults gets stored in NodeService.Proxy.MeshGateway, but the upstream watch for peered data was only considering the mesh gateway config attached in NodeService.Proxy.Upstreams[i]. This means that applying a mesh gateway mode via global proxy-defaults or service-defaults on the downstream would not have an effect. Separately, transparent proxy watches for peered upstreams didn't consider mesh gateway mode at all. This commit addresses the first issue by ensuring that we overlay the upstream config for peered upstreams as we do for non-peered. The second issue is addressed by re-using setupWatchesForPeeredUpstream when handling transparent proxy updates. Note that for transparent proxies we do not yet support mesh gateway mode per upstream, so the NodeService.Proxy.MeshGateway mode is used. * Fix upstream mesh gateway mode handling in xds This commit ensures that when determining the mesh gateway mode for peered upstreams we consider the NodeService.Proxy.MeshGateway config as a baseline. In absense of this change, setting a mesh gateway mode via proxy-defaults or the top-level of service-defaults will not have an effect for peered upstreams. * Merge service/proxy defaults in cfg resolver Previously the mesh gateway mode for connect proxies would be merged at three points: 1. On servers, in ComputeResolvedServiceConfig. 2. On clients, in MergeServiceConfig. 3. On clients, in proxycfg/xds. The first merge returns a ServiceConfigResponse where there is a top-level MeshGateway config from proxy/service-defaults, along with per-upstream config. The second merge combines per-upstream config specified at the service instance with per-upstream config specified centrally. The third merge combines the NodeService.Proxy.MeshGateway config containing proxy/service-defaults data with the per-upstream mode. This third merge is easy to miss, which led to peered upstreams not considering the mesh gateway mode from proxy-defaults. This commit removes the third merge, and ensures that all mesh gateway config is available at the upstream. This way proxycfg/xds do not need to do additional overlays. * Ensure that proxy-defaults is considered in wc Upstream defaults become a synthetic Upstream definition under a wildcard key "". Now that proxycfg/xds expect Upstream definitions to have the final MeshGateway values, this commit ensures that values from proxy-defaults/service-defaults are the default for this synthetic upstream. Add changelog. Co-authored-by: freddygv <freddy@hashicorp.com>	2 years ago
Dan Upton	7b2d08d461	chore: remove unused argument from MergeNodeServiceWithCentralConfig (#15024 ) Previously, the MergeNodeServiceWithCentralConfig method accepted a ServiceSpecificRequest argument, of which only the Datacenter and QueryOptions fields were used. Digging a little deeper, it turns out these fields were only passed down to the ComputeResolvedServiceConfig method (through the ServiceConfigRequest struct) which didn't actually use them. As such, not all call-sites passed a valid ServiceSpecificRequest so it's safer to remove the argument altogether to prevent future changes from depending on it.	2 years ago
Derek Menteer	b64972d486	Bring back parameter ServerExternalAddresses in GenerateToken endpoint (#15267 ) Re-add ServerExternalAddresses parameter in GenerateToken endpoint This reverts commit `5e156772f6` and adds extra functionality to support newer peering behaviors.	2 years ago
cskh	a3f57cc5e8	fix(mesh-gateway): remove deregistered service from mesh gateway (#15272 ) * fix(mesh-gateway): remove deregistered service from mesh gateway * changelog Co-authored-by: Derek Menteer <105233703+hashi-derek@users.noreply.github.com> Co-authored-by: Evan Culver <eculver@users.noreply.github.com>	2 years ago
Freddy	7f5f7e9cf9	Avoid blocking child type updates on parent ack (#15083 )	2 years ago
Derek Menteer	c064ddf606	Backport test fix from ent. (#15279 )	2 years ago
Chris S. Kim	985a4ee1b1	Update hcp-scada-provider to fix diamond dependency problem with go-msgpack (#15185 )	2 years ago

... 3 4 5 6 7 ...

4994 Commits (bdc3dd14c2644b56d2dbfd460290a0e47ad90c11)