52 Commits (f16c5d87abc5a0c81e9cfc918e15c0b7b6da5f06)

Author	SHA1	Message	Date
Michael Wilkerson	0dd4ea2033	* added Sameness Group to proto files (#16998) ... - added Sameness Group to config entries - added Sameness Group to subscriptions * generated proto files * added Sameness Group events to the state store - added test cases * Refactored health RPC Client - moved code that is common to rpcclient under rpcclient common.go. This will help set us up to support future RPC clients * Refactored proxycfg glue views - Moved views to rpcclient config entry. This will allow us to reuse this code for a config entry client * added config entry RPC Client - Copied most of the testing code from rpcclient/health * hooked up new rpcclient in agent * fixed documentation and comments for clarity	2 years ago
Ronald	94ec4eb2f4	copyright headers for agent folder (#16704) ... * copyright headers for agent folder * Ignore test data files * fix proto files and remove headers in agent/uiserver folder * ignore deep-copy files	2 years ago
Matt Keeler	085c0addc0	Protobuf Refactoring for Multi-Module Cleanliness (#16302) ... Protobuf Refactoring for Multi-Module Cleanliness This commit includes the following: Moves all packages that were within proto/ to proto/private Rewrites imports to account for the packages being moved Adds in buf.work.yaml to enable buf workspaces Names the proto-public buf module so that we can override the Go package imports within proto/buf.yaml Bumps the buf version dependency to 1.14.0 (I was trying out the version to see if it would get around an issue - it didn't but it also doesn't break things and it seemed best to keep up with the toolchain changes) Why: In the future we will need to consume other protobuf dependencies such as the Google HTTP annotations for openapi generation or grpc-gateway usage. There were some recent changes to have our own ratelimiting annotations. The two combined were not working when I was trying to use them together (attempting to rebase another branch) Buf workspaces should be the solution to the problem Buf workspaces means that each module will have generated Go code that embeds proto file names relative to the proto dir and not the top level repo root. This resulted in proto file name conflicts in the Go global protobuf type registry. The solution to that was to add in a private/ directory into the path within the proto/ directory. That then required rewriting all the imports. Is this safe? AFAICT yes The gRPC wire protocol doesn't seem to care about the proto file names (although the Go grpc code does tack on the proto file name as Metadata in the ServiceDesc) Other than imports, there were no changes to any generated code as a result of this.	2 years ago
Dan Upton	7a55de375c	xds: don't attempt to load-balance sessions for local proxies (#15789) ... Previously, we'd begin a session with the xDS concurrency limiter regardless of whether the proxy was registered in the catalog or in the server's local agent state. This caused problems for users who run `consul connect envoy` directly against a server rather than a client agent, as the server's locally registered proxies wouldn't be included in the limiter's capacity. Now, the `ConfigSource` is responsible for beginning the session and we only do so for services in the catalog. Fixes: https://github.com/hashicorp/consul/issues/15753	2 years ago
Chris S. Kim	985a4ee1b1	Update hcp-scada-provider to fix diamond dependency problem with go-msgpack (#15185)	2 years ago
Daniel Upton	13c04a13af	proxycfg: terminate stream on irrecoverable errors ... This is the OSS portion of enterprise PR 2339. It improves our handling of "irrecoverable" errors in proxycfg data sources. The canonical example of this is what happens when the ACL token presented by Envoy is deleted/revoked. Previously, the stream would get "stuck" until the xDS server re-checked the token (after 5 minutes) and terminated the stream. Materializers would also sit burning resources retrying something that could never succeed. Now, it is possible for data sources to mark errors as "terminal" which causes the xDS stream to be closed immediately. Similarly, the submatview.Store will evict materializers when it observes they have encountered such an error.	2 years ago
Dan Upton	b9e525d689	grpc: rename public/private directories to external/internal (#13721) ... Previously, public referred to gRPC services that are both exposed on the dedicated gRPC port and have their definitions in the proto-public directory (so were considered usable by 3rd parties). Whereas private referred to services on the multiplexed server port that are only usable by agents and other servers. Now, we're splitting these definitions, such that external/internal refers to the port and public/private refers to whether they can be used by 3rd parties. This is necessary because the peering replication API needs to be exposed on the dedicated port, but is not (yet) suitable for use by 3rd parties.	2 years ago
Daniel Upton	653b8c4f9d	proxycfg: server-local config entry data sources ... This is the OSS portion of enterprise PR 2056. This commit provides server-local implementations of the proxycfg.ConfigEntry and proxycfg.ConfigEntryList interfaces, that source data from streaming events. It makes use of the LocalMaterializer type introduced for peering replication, adding the necessary support for authorization. It also adds support for "wildcard" subscriptions (within a topic) to the event publisher, as this is needed to fetch service-resolvers for all services when configuring mesh gateways. Currently, events will be emitted for just the ingress-gateway, service-resolver, and mesh config entry types, as these are the only entries required by proxycfg — the events will be emitted on topics named IngressGateway, ServiceResolver, and MeshConfig topics respectively. Though these events will only be consumed "locally" for now, they can also be consumed via the gRPC endpoint (confirmed using grpcurl) so using them from client agents should be a case of swapping the LocalMaterializer for an RPCMaterializer.	2 years ago
Dan Upton	d7f8a8e4ef	proxycfg: remove dependency on `cache.UpdateEvent` (#13144) ... OSS portion of enterprise PR 1857. This removes (most) references to the `cache.UpdateEvent` type in the `proxycfg` package. As we're going to be direct usage of the agent cache with interfaces that can be satisfied by alternative server-local datasources, it doesn't make sense to depend on this type everywhere anymore (particularly on the `state.ch` channel). We also plan to extract `proxycfg` out of Consul into a shared library in the future, which would require removing this dependency. Aside from a fairly rote find-and-replace, the main change is that the `cache.Cache` and `health.Client` types now accept a callback function parameter, rather than a `chan<- cache.UpdateEvents`. This allows us to do the type conversion without running another goroutine.	3 years ago
R.B. Boyer	901fd4dd68	remove remaining shim runStep functions (#13015) ... Wraps up the refactor from #13013	3 years ago
R.B. Boyer	0d6d16ddfb	add general runstep test helper instead of copying it all over the place (#13013)	3 years ago
R.B. Boyer	f507f62f3c	peering: initial sync (#12842) ... - Add endpoints related to peering: read, list, generate token, initiate peering - Update node/service/check table indexing to account for peers - Foundational changes for pushing service updates to a peer - Plumb peer name through Health.ServiceNodes path see: ENT-1765, ENT-1280, ENT-1283, ENT-1283, ENT-1756, ENT-1739, ENT-1750, ENT-1679, ENT-1709, ENT-1704, ENT-1690, ENT-1689, ENT-1702, ENT-1701, ENT-1683, ENT-1663, ENT-1650, ENT-1678, ENT-1628, ENT-1658, ENT-1640, ENT-1637, ENT-1597, ENT-1634, ENT-1613, ENT-1616, ENT-1617, ENT-1591, ENT-1588, ENT-1596, ENT-1572, ENT-1555 Co-authored-by: R.B. Boyer <rb@hashicorp.com> Co-authored-by: freddygv <freddy@hashicorp.com> Co-authored-by: Chris S. Kim <ckim@hashicorp.com> Co-authored-by: Evan Culver <eculver@hashicorp.com> Co-authored-by: Nitya Dhanushkodi <nitya@hashicorp.com>	3 years ago
Matt Keeler	8bad5105b7	Move to using a shared EventPublisher (#12673) ... Previously we had 1 EventPublisher per state.Store. When a state store was closed/abandoned such as during a consul snapshot restore, this had the behavior of force closing subscriptions for that topic and evicting event snapshots from the cache. The intention of this commit is to keep all that behavior. To that end, the shared EventPublisher now supports the ability to refresh a topic. That will perform the force close + eviction. The FSM upon abandoning the previous state.Store will call RefreshTopic for all the topics with events generated by the state.Store.	3 years ago
Mark Anderson	98a2e282be	Fixup acl.EnterpriseMeta ... Signed-off-by: Mark Anderson <manderson@hashicorp.com>	3 years ago
Dan Upton	a70e1886c9	WatchRoots gRPC endpoint (#12678) ... Adds a new gRPC streaming endpoint (WatchRoots) that dataplane clients will use to fetch the current list of active Connect CA roots and receive new lists whenever the roots are rotated.	3 years ago
Eric	e4b4f175ed	Bump go-control-plane ... * `go get cloud.google.com/go@v0.59.0` * `go get github.com/envoyproxy/go-control-plane@v0.9.9` * `make envoy-library` * Bumpprotoc to 3.15.8	3 years ago
Eric	5f050614e5	remove the rest of gogo	3 years ago
Eric	776f5843d0	remove gogo from pbservice	3 years ago
Dhia Ayachi	72a997242b	split `pbcommon` to `pbcommon` and `pbcommongogo` (#12587) ... * mogify needed pbcommon structs * mogify needed pbconnect structs * fix compilation errors and make config_translate_test pass * add missing file * remove redundant oss func declaration * fix EnterpriseMeta to copy the right data for enterprise * rename pbcommon package to pbcommongogo * regenerate proto and mog files * add missing mog files * add pbcommon package * pbcommon no mog * fix enterprise meta code generation * fix enterprise meta code generation (pbcommongogo) * fix mog generation for gogo * use `protoc-go-inject-tag` to inject tags * rename proto package * pbcommon no mog * use `protoc-go-inject-tag` to inject tags * add non gogo proto to make file * fix proto get	3 years ago
Dan Upton	7298967070	Restructure gRPC server setup (#12586) ... OSS sync of enterprise changes at 0b44395e	3 years ago
R.B. Boyer	706fc8bcd0	grpc: strip local ACL tokens from RPCs during forwarding if crossing datacenters (#11099) ... Fixes #11086	3 years ago
Daniel Nephin	bc4d349ccf	streaming: support X-Cache-Hit header ... If a value was already available in the local view the request is considered a cache hit. If the materialized had to wait for a value, it is considered a cache miss.	3 years ago
Daniel Nephin	3d5ff8b5db	submatview: add test cases for store.Get with timeout and no index ... Also set a more unique name for the serviceRequest.Type to prevent potential name conflicts in the future.	4 years ago
Daniel Nephin	c5f0cf9456	submatview: and Store integration test with stream backend	4 years ago
Daniel Nephin	59d201e148	submatview: fix a bug with Store.Get ... When info.Timeout is 0, it should have no timeout. Previously it was using a 0 duration timeout which caused it to return without waiting. This bug was masked by using a timeout in the tests. Removing the timeout caused the tests to fail.	4 years ago
Daniel Nephin	c94eaa4957	submatview: improve a couple comments	4 years ago
Daniel Nephin	3a27fce2ad	submatview: fix godoc and comment typos ... Co-authored-by: Paul Banks <banks@banksco.de>	4 years ago
Daniel Nephin	11398892b3	submatview: avoid sorting results unnecessarily ... Previous getFromView would call view.Result when the result may not have been returned (because the index is updated past the minIndex. This would allocate a slice and sort it for no reason, because the values would never be returned. Fix this by re-ordering the operations in getFromView. The test changes in this commit were an attempt to cover the case where an update is received but the index does not exceed the minIndex.	4 years ago
Daniel Nephin	23e1cd6f93	submatview: only return materializer from getEntry ... Also rename it to readEntry now that it doesn't return the entire entry. Based on feedback in PR review, the full entry is not used by the caller, and accessing the fields wouldn't be safe outside the lock, so it is safer to return only the Materializer	4 years ago
Daniel Nephin	46b7d0b573	submatview: godoc	4 years ago
Daniel Nephin	d9537411b8	rpcclient/health: convert tests to the new submatview.Store interface ... Also fixes a minor data race in Materializer. Capture the error before releasing the lock.	4 years ago
Daniel Nephin	dacf50038d	submatview: fix two flaky tests ... These tests can flake when we get a notification for an earlier event. Retry the read from update channel a few times to make sure we get the event we expect.	4 years ago
Daniel Nephin	c932833acb	rpcclient:health: fix a data race and flake in tests ... Split the TestStreamingClient into the two logical components the real client uses. This allows us to test multiple clients properly. Previously writing of ctx from multiple Subscribe calls was showing a data race. Once this was fixed a test started to fail because the request had to be made with a greater index, so that the store.Get call did not return immediately.	4 years ago
Daniel Nephin	5fa0dea63a	rpcclient/health: fix data race in a test ... The idleTTL was being written and read concurrently. Instead move the idleTTL to a struct field so that when one test patches the TTL it does not impact others. The background goroutines for the store can outlive a test because context cancellation is async.	4 years ago
Daniel Nephin	4fb2ba9de7	submatview: move error return to NewMaterializer ... So that we don't have to create views ahead of time, when we will never use that view.	4 years ago
Daniel Nephin	6ebb0f8b74	submatview: test Store.Run	4 years ago
Daniel Nephin	9854c424eb	submatview: test store with Get and Notify calls together	4 years ago
Daniel Nephin	c1063999fc	submatview: more test cases for Store.Get ... And simplify the return value by using a Result type.	4 years ago
Daniel Nephin	832f1a2847	submatview: reduce the getFromView implementation ... Remove View.Result error return value, it was always nil, and seems like it will likely always remain nill since it is simply reading a stored value. Also replace some cache types with local types.	4 years ago
Daniel Nephin	1d9d7d0aa5	submatview: track requests instead of notifiers ... And only start expiration time when the last request ends. This makes tracking expiry simpler, and ensures that no entry can be expired while there are active requests.	4 years ago
Daniel Nephin	7d13465c7b	submatview: set up expiry of materializers	4 years ago
Daniel Nephin	c574108354	submatview: setup testing structure	4 years ago
Daniel Nephin	ddddbdb990	submatview: rough outline of the Get and Notify methods.	4 years ago
Daniel Nephin	3d52abb5e2	submatview: setup the interface for the store	4 years ago
Daniel Nephin	9423c887c4	submatview: do not reset retry waiter when materializer is reset ... The materializer is often reset when an error is received. By resetting the retryWaiter we effectively never wait. The retryWaiter should only be reset when we get an event without error. This is done in Materializer.updateView().	4 years ago
Daniel Nephin	5972bdc87c	streaming: improve godoc for cache-type ... And fix a bug where any error that implemented the temporary interface was considered a temporary error, even when the method would return false.	4 years ago
Daniel Nephin	d0b87cd5d0	submatview: remove notifyUpdateLocked from reset ... This call appears to only be necessary because reset() was called from NewMaterializer. This commit has the constructor set a default value for updateCh, and removes both the call to reset() from New(), and the call to notifyUpdateLocked() from reset(). This should ensure that we do not notify the Fetch() call before we have new values to report.	4 years ago
Daniel Nephin	5eab1d8cef	submatview: remove method receiver from handlers	4 years ago
Daniel Nephin	3fa08beecf	submatview: add a test for handling of NewSnapshotToFollow ... Also add some godoc Rename some vars and functions Fix a data race in the new cache test for entry closing.	4 years ago
Daniel Nephin	534d8b45bb	submatview: refactor Materializer ... Refactor of Materializer.Run Use handlers to manage state in Materializer Rename Materializer receiver rename m.l to m.lock, and flip some conditionals to remove the negative. Improve godoc, rename Deps, move resetErr, and pass err into notifyUpdate Update for NewSnapshotToFollow events Refactor to move context cancel out of Materializer	4 years ago