50 Commits (9f949ead34012d6f3e8c776f7efb272e1871ffd3)

Author	SHA1	Message	Date
John Murret	9b2c1be053	NET-5879 - expose sameness group param on service health endpoint and move sameness group health fallback logic into HealthService RPC layer (#21096) ... * NET-5879 - move the filter for non-passing to occur in the health RPC layer rather than the callers of the RPC * fix import of slices * NET-5879 - expose sameness group param on service health endpoint and move sameness group health fallback logic into HealthService RPC layer * fixing deepcopy * fix license headers	6 months ago
John Murret	a975b04302	NET-5879 - move the filter for non-passing to occur in the health RPC layer rather than the callers of the RPC (#21098) ... * NET-5879 - move the filter for non-passing to occur in the health RPC layer rather than the callers of the RPC * fix import of slices * fix test	6 months ago
Derek Menteer	ac83ac1343	Fix streaming RPCs for agentless. (#20868) ... * Fix streaming RPCs for agentless. This PR fixes an issue where cross-dc RPCs were unable to utilize the streaming backend due to having the node name set. The result of this was the agent-cache being utilized, which would cause high cpu utilization and memory consumption due to the fact that it keeps queries alive for 72 hours before purging inactive entries. This resource consumption is compounded by the fact that each pod in consul-k8s gets a unique token. Since the agent-cache uses the token as a component of the key, the same query is duplicated for each pod that is deployed. * Add changelog.	8 months ago
Semir Patel	53e28a4963	OSS -> CE (community edition) changes (#18517)	1 year ago
hashicorp-copywrite[bot]	5fb9df1640	[COMPLIANCE] License changes (#18443) ... * Adding explicit MPL license for sub-package This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository. * Adding explicit MPL license for sub-package This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository. * Updating the license from MPL to Business Source License Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at <Blog URL>, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl. * add missing license headers * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 --------- Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>	1 year ago
Michael Wilkerson	0dd4ea2033	* added Sameness Group to proto files (#16998) ... - added Sameness Group to config entries - added Sameness Group to subscriptions * generated proto files * added Sameness Group events to the state store - added test cases * Refactored health RPC Client - moved code that is common to rpcclient under rpcclient common.go. This will help set us up to support future RPC clients * Refactored proxycfg glue views - Moved views to rpcclient config entry. This will allow us to reuse this code for a config entry client * added config entry RPC Client - Copied most of the testing code from rpcclient/health * hooked up new rpcclient in agent * fixed documentation and comments for clarity	2 years ago
Ronald	94ec4eb2f4	copyright headers for agent folder (#16704) ... * copyright headers for agent folder * Ignore test data files * fix proto files and remove headers in agent/uiserver folder * ignore deep-copy files	2 years ago
Andrew Stucki	cc0765b87d	Fix resolution of service resolvers with subsets for external upstreams (#16499) ... * Fix resolution of service resolvers with subsets for external upstreams * Add tests * Add changelog entry * Update view filter logic	2 years ago
Matt Keeler	085c0addc0	Protobuf Refactoring for Multi-Module Cleanliness (#16302) ... Protobuf Refactoring for Multi-Module Cleanliness This commit includes the following: Moves all packages that were within proto/ to proto/private Rewrites imports to account for the packages being moved Adds in buf.work.yaml to enable buf workspaces Names the proto-public buf module so that we can override the Go package imports within proto/buf.yaml Bumps the buf version dependency to 1.14.0 (I was trying out the version to see if it would get around an issue - it didn't but it also doesn't break things and it seemed best to keep up with the toolchain changes) Why: In the future we will need to consume other protobuf dependencies such as the Google HTTP annotations for openapi generation or grpc-gateway usage. There were some recent changes to have our own ratelimiting annotations. The two combined were not working when I was trying to use them together (attempting to rebase another branch) Buf workspaces should be the solution to the problem Buf workspaces means that each module will have generated Go code that embeds proto file names relative to the proto dir and not the top level repo root. This resulted in proto file name conflicts in the Go global protobuf type registry. The solution to that was to add in a private/ directory into the path within the proto/ directory. That then required rewriting all the imports. Is this safe? AFAICT yes The gRPC wire protocol doesn't seem to care about the proto file names (although the Go grpc code does tack on the proto file name as Metadata in the ServiceDesc) Other than imports, there were no changes to any generated code as a result of this.	2 years ago
Semir Patel	bafa5c7156	Pass remote addr of incoming HTTP requests through to RPC(..) calls (#15700)	2 years ago
R.B. Boyer	d04cf25fa8	test: fix flaky TestHealthServiceNodes_NodeMetaFilter by waiting until the streaming subsystem has a valid grpc connection (#15019) ... Also potentially unflakes TestHealthIngressServiceNodes for similar reasons.	2 years ago
Daniel Upton	673d02d30f	proxycfg-glue: server-local implementation of the `Health` interface ... This is the OSS portion of enterprise PR 2249. This PR introduces an implementation of the proxycfg.Health interface based on a local materialized view of the health events. It reuses the view and request machinery from agent/rpcclient/health, which made it super straightforward.	2 years ago
Daniel Upton	653b8c4f9d	proxycfg: server-local config entry data sources ... This is the OSS portion of enterprise PR 2056. This commit provides server-local implementations of the proxycfg.ConfigEntry and proxycfg.ConfigEntryList interfaces, that source data from streaming events. It makes use of the LocalMaterializer type introduced for peering replication, adding the necessary support for authorization. It also adds support for "wildcard" subscriptions (within a topic) to the event publisher, as this is needed to fetch service-resolvers for all services when configuring mesh gateways. Currently, events will be emitted for just the ingress-gateway, service-resolver, and mesh config entry types, as these are the only entries required by proxycfg — the events will be emitted on topics named IngressGateway, ServiceResolver, and MeshConfig topics respectively. Though these events will only be consumed "locally" for now, they can also be consumed via the gRPC endpoint (confirmed using grpcurl) so using them from client agents should be a case of swapping the LocalMaterializer for an RPCMaterializer.	2 years ago
Riddhi Shah	d8d8c8603e	Add support for merge-central-config query param (#13001) ... Adds a new query param merge-central-config for use with the below endpoints: /catalog/service/:service /catalog/connect/:service /health/service/:service /health/connect/:service If set on the request, the response will include a fully resolved service definition which is merged with the proxy-defaults/global and service-defaults/:service config entries (on-demand style). This is useful to view the full service definition for a mesh service (connect-proxy kind or gateway kind) which might not be merged before being written into the catalog (example: in case of services in the agentless model).	3 years ago
R.B. Boyer	bbcb1fa805	agent: allow for service discovery queries involving peer name to use streaming (#13168)	3 years ago
Dan Upton	d7f8a8e4ef	proxycfg: remove dependency on `cache.UpdateEvent` (#13144) ... OSS portion of enterprise PR 1857. This removes (most) references to the `cache.UpdateEvent` type in the `proxycfg` package. As we're going to be direct usage of the agent cache with interfaces that can be satisfied by alternative server-local datasources, it doesn't make sense to depend on this type everywhere anymore (particularly on the `state.ch` channel). We also plan to extract `proxycfg` out of Consul into a shared library in the future, which would require removing this dependency. Aside from a fairly rote find-and-replace, the main change is that the `cache.Cache` and `health.Client` types now accept a callback function parameter, rather than a `chan<- cache.UpdateEvents`. This allows us to do the type conversion without running another goroutine.	3 years ago
R.B. Boyer	901fd4dd68	remove remaining shim runStep functions (#13015) ... Wraps up the refactor from #13013	3 years ago
R.B. Boyer	0d6d16ddfb	add general runstep test helper instead of copying it all over the place (#13013)	3 years ago
R.B. Boyer	11213ae180	health: ensure /v1/health/service/:service endpoint returns the most recent results when a filter is used with streaming (#12640) ... The primary bug here is in the streaming subsystem that makes the overall v1/health/service/:service request behave incorrectly when servicing a blocking request with a filter provided. There is a secondary non-streaming bug being fixed here that is much less obvious related to when to update the `reply` variable in a `blockingQuery` evaluation. It is unlikely that it is triggerable in practical environments and I could not actually get the bug to manifest, but I fixed it anyway while investigating the original issue. Simple reproduction (streaming): 1. Register a service with a tag. curl -sL --request PUT 'http://localhost:8500/v1/agent/service/register' \ --header 'Content-Type: application/json' \ --data-raw '{ "ID": "ID1", "Name": "test", "Tags":[ "a" ], "EnableTagOverride": true }' 2. Do an initial filter query that matches on the tag. curl -sLi --get 'http://localhost:8500/v1/health/service/test' --data-urlencode 'filter=a in Service.Tags' 3. Note you get one result. Use the `X-Consul-Index` header to establish a blocking query in another terminal, this should not return yet. curl -sLi --get 'http://localhost:8500/v1/health/service/test?index=$INDEX' --data-urlencode 'filter=a in Service.Tags' 4. Re-register that service with a different tag. curl -sL --request PUT 'http://localhost:8500/v1/agent/service/register' \ --header 'Content-Type: application/json' \ --data-raw '{ "ID": "ID1", "Name": "test", "Tags":[ "b" ], "EnableTagOverride": true }' 5. Your blocking query from (3) should return with a header `X-Consul-Query-Backend: streaming` and empty results if it works correctly `[]`. Attempts to reproduce with non-streaming failed (where you add `&near=_agent` to the read queries and ensure `X-Consul-Query-Backend: blocking-query` shows up in the results).	3 years ago
R.B. Boyer	f507f62f3c	peering: initial sync (#12842) ... - Add endpoints related to peering: read, list, generate token, initiate peering - Update node/service/check table indexing to account for peers - Foundational changes for pushing service updates to a peer - Plumb peer name through Health.ServiceNodes path see: ENT-1765, ENT-1280, ENT-1283, ENT-1283, ENT-1756, ENT-1739, ENT-1750, ENT-1679, ENT-1709, ENT-1704, ENT-1690, ENT-1689, ENT-1702, ENT-1701, ENT-1683, ENT-1663, ENT-1650, ENT-1678, ENT-1628, ENT-1658, ENT-1640, ENT-1637, ENT-1597, ENT-1634, ENT-1613, ENT-1616, ENT-1617, ENT-1591, ENT-1588, ENT-1596, ENT-1572, ENT-1555 Co-authored-by: R.B. Boyer <rb@hashicorp.com> Co-authored-by: freddygv <freddy@hashicorp.com> Co-authored-by: Chris S. Kim <ckim@hashicorp.com> Co-authored-by: Evan Culver <eculver@hashicorp.com> Co-authored-by: Nitya Dhanushkodi <nitya@hashicorp.com>	3 years ago
Eric	e4b4f175ed	Bump go-control-plane ... * `go get cloud.google.com/go@v0.59.0` * `go get github.com/envoyproxy/go-control-plane@v0.9.9` * `make envoy-library` * Bumpprotoc to 3.15.8	3 years ago
Eric	5f050614e5	remove the rest of gogo	3 years ago
Eric	776f5843d0	remove gogo from pbservice	3 years ago
Dhia Ayachi	72a997242b	split `pbcommon` to `pbcommon` and `pbcommongogo` (#12587) ... * mogify needed pbcommon structs * mogify needed pbconnect structs * fix compilation errors and make config_translate_test pass * add missing file * remove redundant oss func declaration * fix EnterpriseMeta to copy the right data for enterprise * rename pbcommon package to pbcommongogo * regenerate proto and mog files * add missing mog files * add pbcommon package * pbcommon no mog * fix enterprise meta code generation * fix enterprise meta code generation (pbcommongogo) * fix mog generation for gogo * use `protoc-go-inject-tag` to inject tags * rename proto package * pbcommon no mog * use `protoc-go-inject-tag` to inject tags * add non gogo proto to make file * fix proto get	3 years ago
Eric	4e6b34725d	Remove gogo stdduration from the healthcheck protobufs	3 years ago
R.B. Boyer	097e1645e3	agent: ensure that most agent behavior correctly respects partition configuration (#10880)	3 years ago
Daniel Nephin	242b3a2dc5	streaming: set a default timeout ... The blocking query backend sets the default value on the server side. The streaming backend does not using blocking queries, so we must set the timeout on the client.	3 years ago
R.B. Boyer	188e8dc51f	agent/structs: add a bunch more EnterpriseMeta helper functions to help with partitioning (#10669)	3 years ago
Daniel Nephin	bc4d349ccf	streaming: support X-Cache-Hit header ... If a value was already available in the local view the request is considered a cache hit. If the materialized had to wait for a value, it is considered a cache miss.	3 years ago
Daniel Nephin	c78391797d	streaming: fix enable of streaming in the client ... And add checks to all the tests that explicitly use streaming.	3 years ago
Daniel Nephin	3d5ff8b5db	submatview: add test cases for store.Get with timeout and no index ... Also set a more unique name for the serviceRequest.Type to prevent potential name conflicts in the future.	4 years ago
Daniel Nephin	23e1cd6f93	submatview: only return materializer from getEntry ... Also rename it to readEntry now that it doesn't return the entire entry. Based on feedback in PR review, the full entry is not used by the caller, and accessing the fields wouldn't be safe outside the lock, so it is safer to return only the Materializer	4 years ago
Daniel Nephin	46b7d0b573	submatview: godoc	4 years ago
Daniel Nephin	d9537411b8	rpcclient/health: convert tests to the new submatview.Store interface ... Also fixes a minor data race in Materializer. Capture the error before releasing the lock.	4 years ago
Daniel Nephin	10ec9c2be3	rpcclient: close the grpc.ClientConn on shutdown	4 years ago
Daniel Nephin	a16c377ca7	rpcclient/health: move all backend routing logic to client	4 years ago
Daniel Nephin	0558586dbd	health: use blocking queries for near query parameter	4 years ago
Daniel Nephin	4fb2ba9de7	submatview: move error return to NewMaterializer ... So that we don't have to create views ahead of time, when we will never use that view.	4 years ago
Daniel Nephin	55a677b7d1	rpcclient/health: integrate submatview.Store into rpcclient/health	4 years ago
Daniel Nephin	6f29fa0de8	rpcclient: move streaming cache tests	4 years ago
R.B. Boyer	e494313e7b	api: ensure v1/health/ingress/:service endpoint works properly when streaming is enabled (#9967) ... The streaming cache type for service health has no way to handle v1/health/ingress/:service queries as there is no equivalent topic that would return the appropriate data. Ensure that attempts to use this endpoint will use the old cache-type for now so that they return appropriate data when streaming is enabled.	4 years ago
Daniel Nephin	f40b76af2d	proxycfg: use rpcclient/health.Client instead of passing around cache name ... This should allow us to swap out the implementation with something other than `agent/cache` without making further code changes.	4 years ago
Daniel Nephin	1a764553c0	rpcclient: use streaming for connect health	4 years ago
Daniel Nephin	d5cc206e39	streaming: move ServiceTag and NodeMetaFiltering to the cache-entry ... So that all the client side filtering is in the same place. Previously only the bexpr filter was in the cache-entry. Also makes a small change to the filtering so that instead of rebuilding slices of items, the filtering can return a bool to determine if the event payload is saved or not.	4 years ago
Pierre Souchay	5bee19ee16	Do not filter tags unless req.TagFilter is set ... Send empty array [] instead of [""] in DNS requests when TagFilter is not set Do not change case sensitivity of services anymore in `getServiceNodes()` since cache keys are now case insensitive	4 years ago
Pierre Souchay	2fe3ab7db0	[Streaming] Properly filters node-meta queries on health ... This wil fix https://github.com/hashicorp/consul/issues/9730	4 years ago
Pierre Souchay	7a024ed074	Streaming filter tags + case insensitive lookups for Service Names ... Will fix: * https://github.com/hashicorp/consul/issues/9695 * https://github.com/hashicorp/consul/issues/9702	4 years ago
Daniel Nephin	bd44952c2e	streaming: disable streaming when requesting connect events ... Until the correct events are created for terminating gateways.	4 years ago
Daniel Nephin	e7d505dc33	config: add field for enabling streaming in the client ... agent: register the new streaming cache-type	4 years ago
Daniel Nephin	529f252d5c	rpcclient: Add health.Client and use it in http and dns ... This new package provides a client agent implementation of an interface for fetching the health of services. This approach has a number of benefits: 1. It provides a much more explicit interface. Instead of everything dependency on `RPC()` and `Cache.Get()` for many unrelated things they can depend on a type that are named according to the behaviour it provides. 2. It gives us a single place to vary the behaviour and migrate to a new form of RPC (gRPC). The current implementation has two options (cache, or direct RPC), and in the future we will have more. It is also a great opporunity to start adding `context.Context` args to these operations, which in the future will allow us to cancel the operations. 3. As a concequence of the first, in the Server agent where we make these calls we can replace the current in-memory RPC calls with a thin adapter for the real method. This removes the `net/rpc` machinery from the call in places where it is not needed. This new package is quite small right now, but I think we can expect it to grow to a more reasonable size as other RPC calls are replaced. This change also happens to replace two very similar implementations with a single implementation.	4 years ago