66 Commits (4034c6f753a5023fdaacd28b67f86ffb52dc1206)

Author	SHA1	Message	Date
Dan Upton	73b9b407ba	grpc: fix data race in balancer registration (#16229) ... Registering gRPC balancers is thread-unsafe because they are stored in a global map variable that is accessed without holding a lock. Therefore, it's expected that balancers are registered _once_ at the beginning of your program (e.g. in a package `init` function) and certainly not after you've started dialing connections, etc. > NOTE: this function must only be called during initialization time > (i.e. in an init() function), and is not thread-safe. While this is fine for us in production, it's challenging for tests that spin up multiple agents in-memory. We currently register a balancer per- agent which holds agent-specific state that cannot safely be shared. This commit introduces our own registry that _is_ thread-safe, and implements the Builder interface such that we can call gRPC's `Register` method once, on start-up. It uses the same pattern as our resolver registry where we use the dial target's host (aka "authority"), which is unique per-agent, to determine which builder to use.	2 years ago
Paul Banks	8ac211b427	Correct WAL metrics registrations (#16388)	2 years ago
Dhia Ayachi	233eacf0a4	inject logger and create logdrop sink (#15822) ... * inject logger and create logdrop sink * init sink with an empty struct instead of nil * wrap a logger instead of a sink and add a discard logger to avoid double logging * fix compile errors * fix linter errors * Fix bug where log arguments aren't properly formatted * Move log sink construction outside of handler * Add prometheus definition and docs for log drop counter Co-authored-by: Daniel Upton <daniel@floppy.co>	2 years ago
Dan Upton	d53ce39c32	grpc: switch servers and retry on error (#15892) ... This is the OSS portion of enterprise PR 3822. Adds a custom gRPC balancer that replicates the router's server cycling behavior. Also enables automatic retries for RESOURCE_EXHAUSTED errors, which we now get for free.	2 years ago
Chris S. Kim	386da5439a	Use rpcHoldTimeout to calculate blocking timeout (#15541) ... Adds buffer to clients so that servers have time to respond to blocking queries.	2 years ago
R.B. Boyer	da70daba43	test: ensure that all dependencies in a test agent use the test logger (#14996)	2 years ago
Chris S. Kim	29a297d3e9	Refactor client RPC timeouts (#14965) ... Fix an issue where rpc_hold_timeout was being used as the timeout for non-blocking queries. Users should be able to tune read timeouts without fiddling with rpc_hold_timeout. A new configuration `rpc_read_timeout` is created. Refactor some implementation from the original PR 11500 to remove the misleading linkage between RPCInfo's timeout (used to retry in case of certain modes of failures) and the client RPC timeouts.	2 years ago
Paul Glass	bcda205f88	Add consul.xds.server.streamStart metric (#14957) ... This adds a new consul.xds.server.streamStart metric to measure the time taken to first generate xDS resources after an xDS stream is opened.	2 years ago
Paul Glass	d17af23641	gRPC server metrics (#14922) ... * Move stats.go from grpc-internal to grpc-middleware * Update grpc server metrics with server type label * Add stats test to grpc-external * Remove global metrics instance from grpc server tests	2 years ago
Nick Ethier	1c1b0994b8	add HCP integration component (#14723) ... * add HCP integration * lint: use non-deprecated logging interface	2 years ago
Dan Upton	1c2c975b0b	xDS Load Balancing (#14397) ... Prior to #13244, connect proxies and gateways could only be configured by an xDS session served by the local client agent. In an upcoming release, it will be possible to deploy a Consul service mesh without client agents. In this model, xDS sessions will be handled by the servers themselves, which necessitates load-balancing to prevent a single server from receiving a disproportionate amount of load and becoming overwhelmed. This introduces a simple form of load-balancing where Consul will attempt to achieve an even spread of load (xDS sessions) between all healthy servers. It does so by implementing a concurrent session limiter (limiter.SessionLimiter) and adjusting the limit according to autopilot state and proxy service registrations in the catalog. If a server is already over capacity (i.e. the session limit is lowered), Consul will begin draining sessions to rebalance the load. This will result in the client receiving a `RESOURCE_EXHAUSTED` status code. It is the client's responsibility to observe this response and reconnect to a different server. Users of the gRPC client connection brokered by the consul-server-connection-manager library will get this for free. The rate at which Consul will drain sessions to rebalance load is scaled dynamically based on the number of proxies in the catalog.	2 years ago
alex	927cee692b	peering: emit exported services count metric (#13811) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
Dan Upton	b9e525d689	grpc: rename public/private directories to external/internal (#13721) ... Previously, public referred to gRPC services that are both exposed on the dedicated gRPC port and have their definitions in the proto-public directory (so were considered usable by 3rd parties). Whereas private referred to services on the multiplexed server port that are only usable by agents and other servers. Now, we're splitting these definitions, such that external/internal refers to the port and public/private refers to whether they can be used by 3rd parties. This is necessary because the peering replication API needs to be exposed on the dedicated port, but is not (yet) suitable for use by 3rd parties.	2 years ago
Daniel Upton	653b8c4f9d	proxycfg: server-local config entry data sources ... This is the OSS portion of enterprise PR 2056. This commit provides server-local implementations of the proxycfg.ConfigEntry and proxycfg.ConfigEntryList interfaces, that source data from streaming events. It makes use of the LocalMaterializer type introduced for peering replication, adding the necessary support for authorization. It also adds support for "wildcard" subscriptions (within a topic) to the event publisher, as this is needed to fetch service-resolvers for all services when configuring mesh gateways. Currently, events will be emitted for just the ingress-gateway, service-resolver, and mesh config entry types, as these are the only entries required by proxycfg — the events will be emitted on topics named IngressGateway, ServiceResolver, and MeshConfig topics respectively. Though these events will only be consumed "locally" for now, they can also be consumed via the gRPC endpoint (confirmed using grpcurl) so using them from client agents should be a case of swapping the LocalMaterializer for an RPCMaterializer.	2 years ago
cskh	74158a8aa2	Add isLeader metric to track if a server is a leader (#13304) ... CTIA-21: sdd is_leader metric to track if a server is a leader Co-authored-by: alex <8968914+acpana@users.noreply.github.com>	3 years ago
cskh	364d4f5efe	Retry on bad dogstatsd connection (#13091) ... - Introduce a new telemetry configurable parameter retry_failed_connection. User can set the value to true to let consul agent continue its start process on failed connection to datadog server. When set to false, agent will stop on failed start. The default behavior is true. Co-authored-by: Dan Upton <daniel@floppy.co> Co-authored-by: Evan Culver <eculver@users.noreply.github.com>	3 years ago
Will Jordan	c48120d005	Add timeout to Client RPC calls (#11500) ... Adds a timeout (deadline) to client RPC calls, so that streams will no longer hang indefinitely in unstable network conditions. Co-authored-by: kisunji <ckim@hashicorp.com>	3 years ago
FFMMM	5245251bbf	[rpc/middleware][consul] plumb intercept off, add server level happy test (#12692)	3 years ago
Dhia Ayachi	16b19dd82d	auto-reload configuration when config files change (#12329) ... * add config watcher to the config package * add logging to watcher * add test and refactor to add WatcherEvent. * add all API calls and fix a bug with recreated files * add tests for watcher * remove the unnecessary use of context * Add debug log and a test for file rename * use inode to detect if the file is recreated/replaced and only listen to create events. * tidy ups (#1535) * tidy ups * Add tests for inode reconcile * fix linux vs windows syscall * fix linux vs windows syscall * fix windows compile error * increase timeout * use ctime ID * remove remove/creation test as it's a use case that fail in linux * fix linux/windows to use Ino/CreationTime * fix the watcher to only overwrite current file id * fix linter error * fix remove/create test * set reconcile loop to 200 Milliseconds * fix watcher to not trigger event on remove, add more tests * on a remove event try to add the file back to the watcher and trigger the handler if success * fix race condition * fix flaky test * fix race conditions * set level to info * fix when file is removed and get an event for it after * fix to trigger handler when we get a remove but re-add fail * fix error message * add tests for directory watch and fixes * detect if a file is a symlink and return an error on Add * rename Watcher to FileWatcher and remove symlink deref * add fsnotify@v1.5.1 * fix go mod * do not reset timer on errors, rename OS specific files * rename New func * events trigger on write and rename * add missing test * fix flaking tests * fix flaky test * check reconcile when removed * delete invalid file * fix test to create files with different mod time. * back date file instead of sleeping * add watching file in agent command. * fix watcher call to use new API * add configuration and stop watcher when server stop * add certs as watched files * move FileWatcher to the agent start instead of the command code * stop watcher before replacing it * save watched files in agent * add add and remove interfaces to the file watcher * fix remove to not return an error * use `Add` and `Remove` to update certs files * fix tests * close events channel on the file watcher even when the context is done * extract `NotAutoReloadableRuntimeConfig` is a separate struct * fix linter errors * add Ca configs and outgoing verify to the not auto reloadable config * add some logs and fix to use background context * add tests to auto-config reload * remove stale test * add tests to changes to config files * add check to see if old cert files still trigger updates * rename `NotAutoReloadableRuntimeConfig` to `StaticRuntimeConfig` * fix to re add both key and cert file. Add test to cover this case. * review suggestion Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com> * add check to static runtime config changes * fix test * add changelog file * fix review comments * Apply suggestions from code review Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com> * update flag description Co-authored-by: FFMMM <FFMMM@users.noreply.github.com> * fix compilation error * add static runtime config support * fix test * fix review comments * fix log test * Update .changelog/12329.txt Co-authored-by: Dan Upton <daniel@floppy.co> * transfer tests to runtime_test.go * fix filewatcher Replace to not deadlock. * avoid having lingering locks Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com> * split ReloadConfig func * fix warning message Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com> * convert `FileWatcher` into an interface * fix compilation errors * fix tests * extract func for adding and removing files Co-authored-by: Ashwin Venkatesh <ashwin@hashicorp.com> Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com> Co-authored-by: FFMMM <FFMMM@users.noreply.github.com> Co-authored-by: Daniel Upton <daniel@floppy.co>	3 years ago
FFMMM	1adfd7b94c	polish rpc.service.call metric behavior (#12624)	3 years ago
FFMMM	c39854de78	fix bad oss sync, use gauges not counters (#12611)	3 years ago
Dan Upton	7298967070	Restructure gRPC server setup (#12586) ... OSS sync of enterprise changes at 0b44395e	3 years ago
FFMMM	e5ebc47a94	pre register new rpc metric, rename metric (#12582)	3 years ago
Dan Upton	b36d4e16b6	Support per-listener TLS configuration ⚙️ (#12504) ... Introduces the capability to configure TLS differently for Consul's listeners/ports (i.e. HTTPS, gRPC, and the internal multiplexed RPC port) which is useful in scenarios where you may want the HTTPS or gRPC interfaces to present a certificate signed by a well-known/public CA, rather than the certificate used for internal communication which must have a SAN in the form `server.<dc>.consul`.	3 years ago
Daniel Nephin	a8e2e1c365	agent: move agent tls metric monitor to a more appropriate place ... And add a test for it	3 years ago
Daniel Nephin	7948720bbb	telemetry: only emit leader cert expiry metrics on the servers	3 years ago
FFMMM	62980ffaa2	fix: only add prom autopilot gauges to servers (#11241) ... Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>	3 years ago
Daniel Nephin	c84867feda	acl: remove ACL.Apply ... As part of removing the legacy ACL system.	3 years ago
R.B. Boyer	5b6d96d27d	grpc: ensure that streaming gRPC requests work over mesh gateway based wan federation (#10838) ... Fixes #10796	3 years ago
Daniel Nephin	c3149ec0fd	debug: use the new metrics stream in debug command	3 years ago
R.B. Boyer	fc9b1a277d	sync changes to oss files made in enterprise (#10670)	3 years ago
Daniel Nephin	1c980e4700	connect: emit a metric for the number of seconds until root CA expiration	4 years ago
Daniel Nephin	29e93f6338	grpc: fix a data race by using a static resolver ... We have seen test flakes caused by 'concurrent map read and map write', and the race detector reports the problem as well (prevent us from running some tests with -race). The root of the problem is the grpc expects resolvers to be registered at init time before any requests are made, but we were using a separate resolver for each test. This commit introduces a resolver registry. The registry is registered as the single resolver for the consul scheme. Each test uses the Authority section of the target (instead of the scheme) to identify the resolver that should be used for the test. The scheme is used for lookup, which is why it can no longer be used as the unique key. This allows us to use a lock around the map of resolvers, preventing the data race.	4 years ago
Matt Keeler	da31e0449e	Move some things around to allow for license updating via config reload ... The bulk of this commit is moving the LeaderRoutineManager from the agent/consul package into its own package: lib/gort. It also got a renaming and its Start method now requires a context. Requiring that context required updating a whole bunch of other places in the code.	4 years ago
Matt Keeler	caafc02449	hcs-1936: Prepare for adding license auto-retrieval to auto-config in enterprise	4 years ago
Matt Keeler	234d0a3c2a	Preparation for changing where license management is done.	4 years ago
R.B. Boyer	ede14b7c54	xds: emit a labeled gauge of connected xDS streams by version (#10243) ... Fixes #10099	4 years ago
Paul Banks	3ad754ca7b	Make Raft trailing logs and snapshot timing reloadable (#10129) ... * WIP reloadable raft config * Pre-define new raft gauges * Update go-metrics to change gauge reset behaviour * Update raft to pull in new metric and reloadable config * Add snapshot persistance timing and installSnapshot to our 'protected' list as they can be infrequent but are important * Update telemetry docs * Update config and telemetry docs * Add note to oldestLogAge on when it is visible * Add changelog entry * Update website/content/docs/agent/options.mdx Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com> Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com>	4 years ago
Daniel Nephin	79a54f208e	agent: fix data race in tests caused by grpc log init ... grpclog.SetLoggerV2 is meant to be called only once before any gRPC requests are received, but each test that uses TestAgent will call NewBaseDeps again. Use a sync.Once to prevent the grpc logging from being re-initialized by each test. This will mean that a test can't use a fake logger to capture logs from the gRPC server.	4 years ago
Daniel Nephin	55a677b7d1	rpcclient/health: integrate submatview.Store into rpcclient/health	4 years ago
Matt Keeler	65d73771a5	Add prometheus guage definitions for replication metrics. (#10109)	4 years ago
Daniel Nephin	d9b8d30cad	cache: log a warning when Cache.Notify handles an error ... Without these warnings, errors are silently ignored, which can make debugging problems more challenging.	4 years ago
Daniel Nephin	97a577502d	config: improve the interface of Load ... This commit reduces the interface to Load() a bit, in preparation for unexporting NewBuilder and having everything call Load. The three arguments are reduced to a single argument by moving the other two into the options struct. The three return values are reduced to two by moving the RuntimeConfig and Warnings into a LoadResult struct.	4 years ago
Pierre Souchay	e2f2d4b0d7	[Streaming][bugfix] handle TLS signalisation when TLS is disabled on client side ... Tnis is an alternative to https://github.com/hashicorp/consul/pull/9494	4 years ago
Kenia	27f6899ec8	Create consul version metric with version label (#9350) ... * create consul version metric with version label * agent/agent.go: add pre-release Version as well as label Co-Authored-By: Radha13 <kumari.radha3@gmail.com> * verion and pre-release version labels. * hyphen/- breaks prometheus * Add Prometheus gauge defintion for version metric * Add new metric to telemetry docs Co-authored-by: Radha Kumari <kumari.radha3@gmail.com> Co-authored-by: Aestek <thib.gilles@gmail.com> Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>	4 years ago
Kit Patella	b81edac7bb	use the MetricsPrefix to set the service name and provide as slice literal to avoid bugs from append modifying its first arg	4 years ago
Kit Patella	5e0e4098c9	push prometheus sink definiitons into prometheus.PrometheusOpts	4 years ago
Kit Patella	5da2f1efa8	finish adding static server metrics	4 years ago
Kit Patella	06d59c03b9	add the service name in the agent rather than in the definitions themselves	4 years ago
Kit Patella	24a2471029	first pass on agent-configured prometheusDefs and adding defs for every consul metric	4 years ago