80 Commits (94791f76b523a964e3e664d443356144333d7968)

Author	SHA1	Message	Date
Derek Menteer	b8b8ad46fc	Various race condition and test fixes. (#20212) ... * Increase timeouts for flakey peering test. * Various test fixes. * Fix race condition in reconcilePeering. This resolves an issue where a peering object in the state store was incorrectly mutated by a function, resulting in the test being flagged as failing when the -race flag was used.	10 months ago
Matt Keeler	326c0ecfbe	In-Memory gRPC (#19942) ... * Implement In-Process gRPC for use by controller caching/indexing This replaces the pipe base listener implementation we were previously using. The new style CAN avoid cloning resources which our controller caching/indexing is taking advantage of to not duplicate resource objects in memory. To maintain safety for controllers and for them to be able to modify data they get back from the cache and the resource service, the client they are presented in their runtime will be wrapped with an autogenerated client which clones request and response messages as they pass through the client. Another sizable change in this PR is to consolidate how server specific gRPC services get registered and managed. Before this was in a bunch of different methods and it was difficult to track down how gRPC services were registered. Now its all in one place. * Fix race in tests * Ensure the resource service is registered to the multiplexed handler for forwarding from client agents * Expose peer streaming on the internal handler	11 months ago
Semir Patel	53e28a4963	OSS -> CE (community edition) changes (#18517)	1 year ago
hashicorp-copywrite[bot]	5fb9df1640	[COMPLIANCE] License changes (#18443) ... * Adding explicit MPL license for sub-package This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository. * Adding explicit MPL license for sub-package This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository. * Updating the license from MPL to Business Source License Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at <Blog URL>, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl. * add missing license headers * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 * Update copyright file headers to BUSL-1.1 --------- Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>	1 year ago
Dan Stough	d935c7b466	[OSS] gRPC Blocking Queries (#17426) ... * feat: initial grpc blocking queries * changelog and docs update	2 years ago
Ronald	94ec4eb2f4	copyright headers for agent folder (#16704) ... * copyright headers for agent folder * Ignore test data files * fix proto files and remove headers in agent/uiserver folder * ignore deep-copy files	2 years ago
Derek Menteer	8f75d99299	Fix issue with trust bundle read ACL check. (#16630) ... This commit fixes an issue where trust bundles could not be read by services in a non-default namespace, unless they had excessive ACL permissions given to them. Prior to this change, `service:write` was required in the default namespace in order to read the trust bundle. Now, `service:write` to a service in any namespace is sufficient.	2 years ago
Eric Haberkorn	57e2493415	allow setting locality on services and nodes (#16581)	2 years ago
Eric Haberkorn	dbaf8bf49c	add agent locality and replicate it across peer streams (#16522)	2 years ago
Matt Keeler	085c0addc0	Protobuf Refactoring for Multi-Module Cleanliness (#16302) ... Protobuf Refactoring for Multi-Module Cleanliness This commit includes the following: Moves all packages that were within proto/ to proto/private Rewrites imports to account for the packages being moved Adds in buf.work.yaml to enable buf workspaces Names the proto-public buf module so that we can override the Go package imports within proto/buf.yaml Bumps the buf version dependency to 1.14.0 (I was trying out the version to see if it would get around an issue - it didn't but it also doesn't break things and it seemed best to keep up with the toolchain changes) Why: In the future we will need to consume other protobuf dependencies such as the Google HTTP annotations for openapi generation or grpc-gateway usage. There were some recent changes to have our own ratelimiting annotations. The two combined were not working when I was trying to use them together (attempting to rebase another branch) Buf workspaces should be the solution to the problem Buf workspaces means that each module will have generated Go code that embeds proto file names relative to the proto dir and not the top level repo root. This resulted in proto file name conflicts in the Go global protobuf type registry. The solution to that was to add in a private/ directory into the path within the proto/ directory. That then required rewriting all the imports. Is this safe? AFAICT yes The gRPC wire protocol doesn't seem to care about the proto file names (although the Go grpc code does tack on the proto file name as Metadata in the ServiceDesc) Other than imports, there were no changes to any generated code as a result of this.	2 years ago
Derek Menteer	19a46d6ca4	Enforce lowercase peer names. (#15697) ... Enforce lowercase peer names. Prior to this change peer names could be mixed case. This can cause issues, as peer names are used as DNS labels in various locations. It also caused issues with envoy configuration.	2 years ago
Matt Keeler	5afd4657ec	Protobuf Modernization (#15949) ... * Protobuf Modernization Remove direct usage of golang/protobuf in favor of google.golang.org/protobuf Marshallers (protobuf and json) needed some changes to account for different APIs. Moved to using the google.golang.org/protobuf/types/known/* for the well known types including replacing some custom Struct manipulation with whats available in the structpb well known type package. This also updates our devtools script to install protoc-gen-go from the right location so that files it generates conform to the correct interfaces. * Fix go-mod-tidy make target to work on all modules	2 years ago
Dan Stough	626249fbf5	[OSS] fix: wait and try longer to peer through mesh gw (#15328)	2 years ago
Derek Menteer	b64972d486	Bring back parameter ServerExternalAddresses in GenerateToken endpoint (#15267) ... Re-add ServerExternalAddresses parameter in GenerateToken endpoint This reverts commit 5e156772f6 and adds extra functionality to support newer peering behaviors.	2 years ago
malizz	617a5f2dc2	convert stream status time fields to pointers (#15252)	2 years ago
freddygv	d65e60de86	Return forbidden on permission denied ... This commit updates the establish endpoint to bubble up a 403 status code to callers when the establishment secret from the token is invalid. This is a signal that a new peering token must be generated.	2 years ago
Nitya Dhanushkodi	5e156772f6	Remove ability to specify external addresses in GenerateToken endpoint (#14930) ... * Reverts "update generate token endpoint to take external addresses (#13844)" This reverts commit f47319b7c6.	2 years ago
freddygv	e69bc727ec	Update peering establishment to maybe use gateways ... When peering through mesh gateways we expect outbound dials to peer servers to flow through the local mesh gateway addresses. Now when establishing a peering we get a list of dial addresses as a ring buffer that includes local mesh gateway addresses if the local DC is configured to peer through mesh gateways. The ring buffer includes the mesh gateway addresses first, but also includes the remote server addresses as a fallback. This fallback is present because it's possible that direct egress from the servers may be allowed. If not allowed then the leader will cycle back to a mesh gateway address through the ring. When attempting to dial the remote servers we retry up to a fixed timeout. If using mesh gateways we also have an initial wait in order to allow for the mesh gateways to configure themselves. Note that if we encounter a permission denied error we do not retry since that error indicates that the secret in the peering token is invalid.	2 years ago
Derek Menteer	4e140c98bc	Address PR comments.	2 years ago
Derek Menteer	1e394da400	Disallow peering to the same cluster.	2 years ago
Derek Menteer	caa1396255	Add remote peer partition and datacenter info.	2 years ago
Chris S. Kim	4f4112662e	Fix nil pointer	2 years ago
Chris S. Kim	b0a4c5c563	Include stream-related information in peering endpoints	2 years ago
Freddy	4abad02abd	Merge pull request #14796 from hashicorp/peering/use-connect-ca	2 years ago
freddygv	3034df6a5c	Require Connect and TLS to generate peering tokens ... By requiring Connect and a gRPC TLS listener we can automatically configure TLS for all peering control-plane traffic.	2 years ago
freddygv	fac3ddc857	Use internal server certificate for peering TLS ... A previous commit introduced an internally-managed server certificate to use for peering-related purposes. Now the peering token has been updated to match that behavior: - The server name matches the structure of the server cert - The CA PEMs correspond to the Connect CA Note that if Conect is disabled, and by extension the Connect CA, we fall back to the previous behavior of returning the manually configured certs and local server SNI. Several tests were updated to use the gRPC TLS port since they enable Connect by default. This means that the peering token will embed the Connect CA, and the dialer will expect a TLS listener.	2 years ago
DanStough	77ab28c5c7	feat: xDS updates for peerings control plane through mesh gw	2 years ago
freddygv	4ff9d475b0	Return mesh gateway addrs if peering through mgw	2 years ago
Eric Haberkorn	80e51ff907	Add exported services event to cluster peering replication. (#14797)	2 years ago
malizz	84b0f408fa	Support Stale Queries for Trust Bundle Lookups (#14724) ... * initial commit * add tags, add conversations * add test for query options utility functions * update previous tests * fix test * don't error out on empty context * add changelog * update decode config	2 years ago
DanStough	485e1b5d4e	fix(peering): generate token metrics only for leader	2 years ago
DanStough	2a2debee64	feat(peering): validate server name conflicts on establish	2 years ago
freddygv	650e48624d	Allow terminated peerings to be deleted ... Peerings are terminated when a peer decides to delete the peering from their end. Deleting a peering sends a termination message to the peer and triggers them to mark the peering as terminated but does NOT delete the peering itself. This is to prevent peerings from disappearing from both sides just because one side deleted them. Previously the Delete endpoint was skipping the deletion if the peering was not marked as active. However, terminated peerings are also inactive. This PR makes some updates so that peerings marked as terminated can be deleted by users.	2 years ago
Chris S. Kim	6ddcc04613	Replace ring buffer with async version (#14314) ... We need to watch for changes to peerings and update the server addresses which get served by the ring buffer. Also, if there is an active connection for a peer, we are getting up-to-date server addresses from the replication stream and can safely ignore the token's addresses which may be stale.	2 years ago
freddygv	c04515a844	Use proto message for each secrets write op ... Previously there was a field indicating the operation that triggered a secrets write. Now there is a message for each operation and it contains the secret ID being persisted.	2 years ago
freddygv	60d6e28c97	Pass explicit signal with op for secrets write ... Previously the updates to the peering secrets UUID table relied on inferring what action triggered the update based on a reconciliation against the existing secrets. Instead we now explicitly require the operation to be given so that the inference isn't necessary. This makes the UUID table logic easier to reason about and fixes some related bugs. There is also an update so that the peering secrets get handled on snapshots/restores.	2 years ago
freddygv	9ca687bc7c	Avoid deleting peering secret UUIDs at dialers ... Dialers do not keep track of peering secret UUIDs, so they should not attempt to clean up data from that table when their peering is deleted. We also now keep peer server addresses when marking peerings for deletion. Peer server addresses are used by the ShouldDial() helper when determining whether the peering is for a dialer or an acceptor. We need to keep this data so that peering secrets can be cleaned up accordingly.	2 years ago
Matt Keeler	f74d0cef7a	Implement/Utilize secrets for Peering Replication Stream (#13977)	2 years ago
acpana	6033584349	use EqualPartitions ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
acpana	0351ca5136	better fix ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
acpana	8b2ef80336	sync w ent ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
alex	437a28d18a	peering: prevent peering in same partition (#13851) ... Co-authored-by: Chris S. Kim <ckim@hashicorp.com>	2 years ago
Luke Kysow	3530d3782d	peering: read endpoints can now return failing status (#13849) ... Track streams that have been disconnected due to an error and set their statuses to failing.	2 years ago
Chris S. Kim	73a84f256f	Preserve PeeringState on upsert (#13666) ... Fixes a bug where if the generate token is called twice, the second call upserts the zero-value (undefined) of PeeringState.	2 years ago
freddygv	b544ce6485	Add ACL enforcement to peering endpoints	2 years ago
alex	279d458e6e	peering: use ShouldDial to validate peer role (#13823) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
Luke Kysow	a1e6d69454	peering: add config to enable/disable peering (#13867) ... * peering: add config to enable/disable peering Add config: ``` peering { enabled = true } ``` Defaults to true. When disabled: 1. All peering RPC endpoints will return an error 2. Leader won't start its peering establishment goroutines 3. Leader won't start its peering deletion goroutines	2 years ago
Nitya Dhanushkodi	f47319b7c6	update generate token endpoint to take external addresses (#13844) ... Update generate token endpoint (rpc, http, and api module) If ServerExternalAddresses are set, it will override any addresses gotten from the "consul" service, and be used in the token instead, and dialed by the dialer. This allows for setting up a load balancer for example, in front of the consul servers.	2 years ago
alex	de5a991d8c	peering: refactor reconcile, cleanup (#13795) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago
alex	a9ae2ff4fa	peering: track exported services (#13784) ... Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>	2 years ago