109 Commits (bdf6ed9d6d988061540e5e91f71bd715faa9c241)

Author	SHA1	Message	Date
R.B. Boyer	d07aac8d7e	Revert "cache: refactor agent cache fetching to prevent unnecessary f… (#16818) (#17046) ... Revert "cache: refactor agent cache fetching to prevent unnecessary fetches on error (#14956)" Co-authored-by: Derek Menteer <105233703+hashi-derek@users.noreply.github.com>	2 years ago
Ronald	94ec4eb2f4	copyright headers for agent folder (#16704) ... * copyright headers for agent folder * Ignore test data files * fix proto files and remove headers in agent/uiserver folder * ignore deep-copy files	2 years ago
Dan Upton	7a55de375c	xds: don't attempt to load-balance sessions for local proxies (#15789) ... Previously, we'd begin a session with the xDS concurrency limiter regardless of whether the proxy was registered in the catalog or in the server's local agent state. This caused problems for users who run `consul connect envoy` directly against a server rather than a client agent, as the server's locally registered proxies wouldn't be included in the limiter's capacity. Now, the `ConfigSource` is responsible for beginning the session and we only do so for services in the catalog. Fixes: https://github.com/hashicorp/consul/issues/15753	2 years ago
Matt Keeler	5afd4657ec	Protobuf Modernization (#15949) ... * Protobuf Modernization Remove direct usage of golang/protobuf in favor of google.golang.org/protobuf Marshallers (protobuf and json) needed some changes to account for different APIs. Moved to using the google.golang.org/protobuf/types/known/* for the well known types including replacing some custom Struct manipulation with whats available in the structpb well known type package. This also updates our devtools script to install protoc-gen-go from the right location so that files it generates conform to the correct interfaces. * Fix go-mod-tidy make target to work on all modules	2 years ago
Derek Menteer	f3776894bf	Fix agent cache incorrectly notifying unchanged protobufs. (#15866) ... Fix agent cache incorrectly notifying unchanged protobufs. This change fixes a situation where the protobuf private fields would be read by reflect.DeepEqual() and indicate data was modified. This resulted in change notifications being fired every time, which could cause performance problems in proxycfg.	2 years ago
R.B. Boyer	3c44116a8f	cache: refactor agent cache fetching to prevent unnecessary fetches on error (#14956) ... This continues the work done in #14908 where a crude solution to prevent a goroutine leak was implemented. The former code would launch a perpetual goroutine family every iteration (+1 +1) and the fixed code simply caused a new goroutine family to first cancel the prior one to prevent the leak (-1 +1 == 0). This PR refactors this code completely to: - make it more understandable - remove the recursion-via-goroutine strangeness - prevent unnecessary RPC fetches when the prior one has errored. The core issue arose from a conflation of the entry.Fetching field to mean: - there is an RPC (blocking query) in flight right now - there is a goroutine running to manage the RPC fetch retry loop The problem is that the goroutine-leak-avoidance check would treat Fetching like (2), but within the body of a goroutine it would flip that boolean back to false before the retry sleep. This would cause a new chain of goroutines to launch which #14908 would correct crudely. The refactored code uses a plain for-loop and changes the semantics to track state for "is there a goroutine associated with this cache entry" instead of the former. We use a uint64 unique identity per goroutine instead of a boolean so that any orphaned goroutines can tell when they've been replaced when the expiry loop deletes a cache entry while the goroutine is still running and is later replaced.	2 years ago
Chris S. Kim	bde57c0dd0	Regenerate files according to 1.19.2 formatter	2 years ago
R.B. Boyer	fe2d41ddad	cache: prevent goroutine leak in agent cache (#14908) ... There is a bug in the error handling code for the Agent cache subsystem discovered: 1. NotifyCallback calls notifyBlockingQuery which calls getWithIndex in a loop (which backs off on-error up to 1 minute) 2. getWithIndex calls fetch if there’s no valid entry in the cache 3. fetch starts a goroutine which calls Fetch on the cache-type, waits for a while (again with backoff up to 1 minute for errors) and then calls fetch to trigger a refresh The end result being that every 1 minute notifyBlockingQuery spawns an ancestry of goroutines that essentially lives forever. This PR ensures that the goroutine started by `fetch` cancels any prior goroutine spawned by the same line for the same key. In isolated testing where a cache type was tweaked to indefinitely error, this patch prevented goroutine counts from skyrocketing.	2 years ago
skpratt	b761589340	add non-double-prefixed metrics (#14193)	2 years ago
R.B. Boyer	2a945facec	test: update mockery use to put mocks into test files (#13656) ... --testonly doesn't do anything anymore so switch to --filename instead	2 years ago
Dan Upton	d7f8a8e4ef	proxycfg: remove dependency on `cache.UpdateEvent` (#13144) ... OSS portion of enterprise PR 1857. This removes (most) references to the `cache.UpdateEvent` type in the `proxycfg` package. As we're going to be direct usage of the agent cache with interfaces that can be satisfied by alternative server-local datasources, it doesn't make sense to depend on this type everywhere anymore (particularly on the `state.ch` channel). We also plan to extract `proxycfg` out of Consul into a shared library in the future, which would require removing this dependency. Aside from a fairly rote find-and-replace, the main change is that the `cache.Cache` and `health.Client` types now accept a callback function parameter, rather than a `chan<- cache.UpdateEvents`. This allows us to do the type conversion without running another goroutine.	3 years ago
R.B. Boyer	f507f62f3c	peering: initial sync (#12842) ... - Add endpoints related to peering: read, list, generate token, initiate peering - Update node/service/check table indexing to account for peers - Foundational changes for pushing service updates to a peer - Plumb peer name through Health.ServiceNodes path see: ENT-1765, ENT-1280, ENT-1283, ENT-1283, ENT-1756, ENT-1739, ENT-1750, ENT-1679, ENT-1709, ENT-1704, ENT-1690, ENT-1689, ENT-1702, ENT-1701, ENT-1683, ENT-1663, ENT-1650, ENT-1678, ENT-1628, ENT-1658, ENT-1640, ENT-1637, ENT-1597, ENT-1634, ENT-1613, ENT-1616, ENT-1617, ENT-1591, ENT-1588, ENT-1596, ENT-1572, ENT-1555 Co-authored-by: R.B. Boyer <rb@hashicorp.com> Co-authored-by: freddygv <freddy@hashicorp.com> Co-authored-by: Chris S. Kim <ckim@hashicorp.com> Co-authored-by: Evan Culver <eculver@hashicorp.com> Co-authored-by: Nitya Dhanushkodi <nitya@hashicorp.com>	3 years ago
R.B. Boyer	4274e67b47	chore: upgrade mockery to v2 and regenerate (#12836)	3 years ago
R.B. Boyer	f3ce353a87	ca: fix a bug that caused a non blocking leaf cert query after a blocking leaf cert query to block (#12820) ... Fixes #12048 Fixes #12319 Regression introduced in #11693 Local reproduction steps: 1. `consul agent -dev` 2. `curl -sLiv 'localhost:8500/v1/agent/connect/ca/leaf/web'` 3. make note of the `X-Consul-Index` header returned 4. `curl -sLi 'localhost:8500/v1/agent/connect/ca/leaf/web?index=<VALUE_FROM_STEP_3>'` 5. Kill the above curl when it hangs with Ctrl-C 6. Repeat (2) and it should not hang.	3 years ago
R.B. Boyer	b60d89e7ef	bulk rewrite using this script ... set -euo pipefail unset CDPATH cd "$(dirname "$0")" for f in $(git grep '\brequire := require\.New(' | cut -d':' -f1 | sort -u); do echo "=== require: $f ===" sed -i '/require := require.New(t)/d' $f # require.XXX(blah) but not require.XXX(tblah) or require.XXX(rblah) sed -i 's/\brequire\.\([a-zA-Z0-9_]*\)(\([^tr]\)/require.\1(t,\2/g' $f # require.XXX(tblah) but not require.XXX(t, blah) sed -i 's/\brequire\.\([a-zA-Z0-9_]*\)(\(t[^,]\)/require.\1(t,\2/g' $f # require.XXX(rblah) but not require.XXX(r, blah) sed -i 's/\brequire\.\([a-zA-Z0-9_]*\)(\(r[^,]\)/require.\1(t,\2/g' $f gofmt -s -w $f done for f in $(git grep '\bassert := assert\.New(' | cut -d':' -f1 | sort -u); do echo "=== assert: $f ===" sed -i '/assert := assert.New(t)/d' $f # assert.XXX(blah) but not assert.XXX(tblah) or assert.XXX(rblah) sed -i 's/\bassert\.\([a-zA-Z0-9_]*\)(\([^tr]\)/assert.\1(t,\2/g' $f # assert.XXX(tblah) but not assert.XXX(t, blah) sed -i 's/\bassert\.\([a-zA-Z0-9_]*\)(\(t[^,]\)/assert.\1(t,\2/g' $f # assert.XXX(rblah) but not assert.XXX(r, blah) sed -i 's/\bassert\.\([a-zA-Z0-9_]*\)(\(r[^,]\)/assert.\1(t,\2/g' $f gofmt -s -w $f done	3 years ago
Paul Banks	1406671290	cache: Fix bug where connection errors can cause early cache expiry (#9979) ... Fixes a cache bug where TTL is not updated while a value isn't changing or cache entry is returning fetch errors.	4 years ago
Paul Banks	ee04d452be	cache: fix bug where TTLs were ignored leading to leaked memory in client agents (#9978) ... * Fix bug in cache where TTLs are effectively ignored This mostly affects streaming since streaming will immediately return from Fetch calls when the state is Closed on eviction which causes the race condition every time. However this also affects all other cache types if the fetch call happens to return between the eviction and then next time around the Get loop by any client. There is a separate bug that allows cache items to be evicted even when there are active clients which is the trigger here. * Add changelog entry * Update .changelog/9978.txt	4 years ago
Daniel Nephin	d9b8d30cad	cache: log a warning when Cache.Notify handles an error ... Without these warnings, errors are silently ignored, which can make debugging problems more challenging.	4 years ago
Matt Keeler	8dbe342ec9	Stop background refresh of cached data for requests that result in ACL not found errors (#9738)	4 years ago
Daniel Nephin	b9e60c0775	testing: skip slow tests with -short ... Add a skip condition to all tests slower than 100ms. This change was made using `gotestsum tool slowest` with data from the last 3 CI runs of master. See https://github.com/gotestyourself/gotestsum#finding-and-skipping-slow-tests With this change: ``` $ time go test -count=1 -short ./agent ok github.com/hashicorp/consul/agent 0.743s real 0m4.791s $ time go test -count=1 -short ./agent/consul ok github.com/hashicorp/consul/agent/consul 4.229s real 0m8.769s ```	4 years ago
Hans Hasselberg	57701695c3	add missing descriptions for metrics	4 years ago
Kit Patella	5da2f1efa8	finish adding static server metrics	4 years ago
Daniel Nephin	0d4fa882b3	lib/ttlcache: unexport key and additional godoc	4 years ago
Daniel Nephin	c17baadbf8	lib/ttlcache: add a constant for NotIndexed	4 years ago
Daniel Nephin	6c09ab3dd8	cache: fix a bug with Prepopulate ... Prepopulate was setting entry.Expiry.HeapIndex to 0. Previously this would result in a call to heap.Fix(0) which wasn't correct, but was also not really a problem because at worse it would re-notify. With the recent change to extract cachettl it was changed to call Update(idx), which would have updated the wrong entry. A previous commit removed the setting of entry.Expiry so that the HeapIndex would be reported as -1, and this commit adds a test and handles the -1 heap index.	4 years ago
Daniel Nephin	bbb816aa8a	lib/ttlcache: extract package from agent/cache	4 years ago
Daniel Nephin	c4122edd22	cache: export ExpiryHeap ... and hide internal methods on an unexported type, so that when it is extrated those methods are not exported.	4 years ago
Daniel Nephin	343d133183	cache: Refactor heap.notify to make it more explicit. ... And remove duplicate notifications. Instead of performing the check in the heap implementation, check the index in the higher level interface (Add,Remove,Update) and notify if one of the relevant indexes is 0.	4 years ago
Daniel Nephin	499f2822cf	cache: Move more of the expiryLoop into the Heap	4 years ago
Daniel Nephin	2cdc90e01b	cache: extract cache eviction heap ... Start creating an interface that doesn't require using heap and hides more of the entry internals.	4 years ago
Daniel Nephin	3fa08beecf	submatview: add a test for handling of NewSnapshotToFollow ... Also add some godoc Rename some vars and functions Fix a data race in the new cache test for entry closing.	4 years ago
Daniel Nephin	b576a2d3c7	cache-types: Update Streaming health cache-type ... To use latest protobuf types	4 years ago
Daniel Nephin	132b76acef	agent/cache: Add cache-type and materialized view for streaming health ... Extracted from d97412ce4c Co-authored-by: Paul Banks <banks@banksco.de>	4 years ago
Daniel Nephin	6956477be5	Merge pull request #8548 from edevil/fix_flake ... Fix flaky TestACLResolver_Client/Concurrent-Token-Resolve	4 years ago
Pierre Souchay	d5974b1d17	Added Unit test for cache reloading	4 years ago
Pierre Souchay	879d087f65	Added `options.Equals()` and minor fixes indentation fixes	4 years ago
Pierre Souchay	d2be9d38da	Ensure that Cache options are reloaded when `consul reload` is performed. ... This will apply cache throttling parameters are properly applied: * cache.EntryFetchMaxBurst * cache.EntryFetchRate When values are updated, a log is displayed in info.	4 years ago
André Cruz	9a0792139c	Decrease test flakiness ... Fix flaky TestACLResolver_Client/Concurrent-Token-Resolve and TestCacheNotifyPolling	4 years ago
Hans Hasselberg	054595b1f8	agent/cache test for cache throttling. (#8396)	4 years ago
Matt Keeler	be01c4241d	Default Cache rate limiting options in New ... Also get rid of the TestCache helper which was where these defaults were happening previously.	4 years ago
Pierre Souchay	505de6dc29	Added ratelimit to handle throtling cache (#8226) ... This implements a solution for #7863 It does: Add a new config cache.entry_fetch_rate to limit the number of calls/s for a given cache entry, default value = rate.Inf Add cache.entry_fetch_max_burst size of rate limit (default value = 2) The new configuration now supports the following syntax for instance to allow 1 query every 3s: command line HCL: -hcl 'cache = { entry_fetch_rate = 0.333}' in JSON { "cache": { "entry_fetch_rate": 0.333 } }	4 years ago
Matt Keeler	12acdd7481	Disable background cache refresh for Connect Leaf Certs ... The rationale behind removing them is that all of our own code (xDS, builtin connect proxy) use the cache notification mechanism. This ensures that the blocking fetch behind the scenes is always executing. Therefore the only way you might go to get a certificate and have to wait is when 1) the request has never been made for that cert before or 2) you are using the v1/agent/connect/ca/leaf API for retrieving the cert yourself. In the first case, the refresh change doesn’t alter the behavior. In the second case, it can be mitigated by using blocking queries with that API which just like normal cache notification mechanism will cause the blocking fetch to be initiated and to get leaf certs as soon as needed. If you are not using blocking queries, or Envoy/xDS, or the builtin connect proxy but are retrieving the certs yourself then the HTTP endpoint might take a little longer to respond. This also renames the RefreshTimeout field on the register options to QueryTimeout to more accurately reflect that it is used for any type that supports blocking queries.	4 years ago
Daniel Nephin	2ec3760b70	agent/cache: Use AllowNotModifiedResponse in CatalogListServices ... Co-authored-by: Pierre Souchay <pierresouchay@users.noreply.github.com>	4 years ago
Daniel Nephin	dbb8d14728	agent/cache: Update some docstrings	4 years ago
Matt Keeler	8837907de4	Make the Agent Cache more Context aware (#8092) ... Blocking queries issues will still be uncancellable (that cannot be helped until we get rid of net/rpc). However this makes it so that if calling getWithIndex (like during a cache Notify go routine) we can cancell the outer routine. Previously it would keep issuing more blocking queries until the result state actually changed.	5 years ago
Daniel Nephin	81755c860a	agent/cache: remove error return from fetch ... A previous change removed the only error, so the return value can be removed now.	5 years ago
Daniel Nephin	4ef9fc9f27	agent/cache: reduce function arguments by removing duplicates ... A few of the unexported functions in agent/cache took a large number of arguments. These arguments were effectively overrides for values that were provided in RequestInfo. By using a struct we can not only reduce the number of arguments, but also simplify the logic by removing the need for overrides.	5 years ago
Daniel Nephin	5fe7043439	agent/cache: Make all cache options RegisterOptions ... Previously the SupportsBlocking option was specified by a method on the type, and all the other options were specified from RegisterOptions. This change moves RegisterOptions to a method on the type, and moves SupportsBlocking into the options struct. Currently there are only 2 cache-types. So all cache-types can implement this method by embedding a struct with those predefined values. In the future if a cache type needs to be registered more than once with different options it can remove the embedded type and implement the method in a way that allows for paramaterization.	5 years ago
Daniel Nephin	89f41bddfe	Remove TTL from cacheEntryExpiry ... This should very slightly reduce the amount of memory required to store each item in the cache. It will also enable setting different TTLs based on the type of result. For example we may want to use a shorter TTL when the result indicates the resource does not exist, as storing these types of records could easily lead to a DOS caused by OOM.	5 years ago
Daniel Nephin	7246d8b6cb	agent/cache: Reduce differences between notify implementations ... These two notify functions are very similar. There appear to be just enough differences that trying to parameterize the differences may not improve things. For now, reduce some of the cosmetic differences so that the material differences are more obvious.	5 years ago