Commit Graph

67 Commits (a152e0ee7c440db8459c79e83af0a16f2cb1b007)

Author SHA1 Message Date
R.B. Boyer 21c6e0e8e6
fix two typos (#17389) 2023-05-17 08:50:26 -07:00
Matt Keeler d37572bd44
Add a Node health controller (#17214)
This will aggregate all HealthStatus objects owned by the Node and update the status of the Node with an overall health.
2023-05-15 09:55:03 -04:00
Dan Upton 879b775459
docs: initial documentation for the new State Store (#17315) 2023-05-15 12:34:36 +01:00
Matt Keeler 456156ebec
Add type validations for the catalog resources (#17211)
Also adding some common resource validation error types to the internal/resource package.
2023-05-12 09:24:55 -04:00
Dan Upton 6c24a66f73
resource: optionally compare timestamps in `EqualStatus` (#17275) 2023-05-10 10:37:54 +01:00
Semir Patel 40eefaba18
Reaper controller for cascading deletes of owner resources (#17256) 2023-05-09 13:57:40 -05:00
Dan Upton d53a1d4a27
resource: add helpers for more efficiently comparing IDs etc (#17224) 2023-05-09 19:02:24 +01:00
Dan Upton 972998203e
controller: deduplicate items in queue (#17168) 2023-05-09 18:14:20 +01:00
Dan Upton 6e1bc57469
Controller Runtime 2023-05-09 15:25:55 +01:00
Matt Keeler 34915670f2
Register new catalog & mesh protobuf types with the resource registry (#17225) 2023-05-08 15:36:35 -04:00
Semir Patel 991a002fcc
resource: List resources by owner (#17190) 2023-05-08 12:26:19 -05:00
Semir Patel 9fef1c7f17
Create tombstone on resource `Delete` (#17108) 2023-04-28 10:49:08 -05:00
Dan Upton eeaa636164
Cleanup from unblocking the pipeline 🧹 (#17121) 2023-04-26 13:59:58 +01:00
Semir Patel e7bb8fdf15
Fix or disable pipeline breaking changes that made it into main in last day or so (#17130)
* Fix straggler from renaming Register->RegisterTypes

* somehow a lint failure got through previously

* Fix lint-consul-retry errors

* adding in fix for success jobs getting skipped. (#17132)

* Temporarily disable inmem backend conformance test to get green pipeline

* Another test needs disabling

---------

Co-authored-by: John Murret <john.murret@hashicorp.com>
2023-04-25 15:17:48 -05:00
Dan Upton b9c485dcb8
Controller Supervision (#17016) 2023-04-25 12:52:35 +01:00
Dan Upton ba4a314772
storage: fix bug where WatchList would (rarely) return duplicate events (#17067) 2023-04-25 11:48:13 +01:00
hashicorp-copywrite[bot] 9f81fc01e9
[COMPLIANCE] Add Copyright and License Headers (#16854)
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Ronald <roncodingenthusiast@users.noreply.github.com>
2023-04-20 12:40:22 +00:00
Semir Patel 2f7d591702
Tenancy wildcard validaton for `Write`, `Read`, and `Delete` endpoints (#17004) 2023-04-17 16:33:20 -05:00
Semir Patel b8c9e133be
Add mutate hook to `Write` endpoint (#16958) 2023-04-12 16:50:07 -05:00
Semir Patel 3b83c7ee9a
Enforce ACLs on resource `Write` and `Delete` endpoints (#16956) 2023-04-12 16:22:44 -05:00
Dan Upton d595e6ade9
resource: `WriteStatus` endpoint (#16886) 2023-04-11 19:23:14 +01:00
Semir Patel 317240fca7
Resource validation hook for `Write` endpoint (#16950) 2023-04-11 06:55:32 -05:00
Semir Patel 686f49346c
Check acls on resource `Read`, `List`, and `WatchList` (#16842) 2023-04-11 06:10:14 -05:00
Dan Upton 4fa2537b3b
Resource `Write` endpoint (#16786) 2023-04-06 10:40:04 +01:00
Dan Upton 671d5825ca
Raft storage backend (#16619) 2023-04-04 17:30:06 +01:00
Dan Upton 651549c97d
storage: fix resource leak in Watch (#16817) 2023-03-31 13:24:19 +01:00
Ronald b64674623e
Copyright headers for missing files/folders (#16708)
* copyright headers for agent folder
2023-03-28 18:48:58 -04:00
Dan Upton 81df781e5f
Add storage backend interface and in-memory implementation (#16538)
Introduces `storage.Backend`, which will serve as the interface between the
Resource Service and the underlying storage system (Raft today, but in the
future, who knows!).

The primary design goal of this interface is to keep its surface area small,
and push as much functionality as possible into the layers above, so that new
implementations can be added with little effort, and easily proven to be
correct. To that end, we also provide a suite of "conformance" tests that can
be run against a backend implementation to check it behaves correctly.

In this commit, we introduce an initial in-memory storage backend, which is
suitable for tests and when running Consul in development mode. This backend is
a thin wrapper around the `Store` type, which implements a resource database
using go-memdb and our internal pub/sub system. `Store` will also be used to
handle reads in our Raft backend, and in the future, used as a local cache for
external storage systems.
2023-03-27 10:30:53 +01:00
Poonam Jadhav 9c64731a56
feat: add category annotation to RPC and gRPC methods (#16646) 2023-03-20 11:24:29 -04:00
Semir Patel 5a3fec6238
Basic resource type registry (#16622) 2023-03-14 13:30:25 -05:00
Matt Keeler 84156afe87
Remove private prefix from proto-gen-rpc-glue e2e test (#16433) 2023-03-03 14:05:14 -05:00
Matt Keeler 085c0addc0
Protobuf Refactoring for Multi-Module Cleanliness (#16302)
Protobuf Refactoring for Multi-Module Cleanliness

This commit includes the following:

Moves all packages that were within proto/ to proto/private
Rewrites imports to account for the packages being moved
Adds in buf.work.yaml to enable buf workspaces
Names the proto-public buf module so that we can override the Go package imports within proto/buf.yaml
Bumps the buf version dependency to 1.14.0 (I was trying out the version to see if it would get around an issue - it didn't but it also doesn't break things and it seemed best to keep up with the toolchain changes)

Why:

In the future we will need to consume other protobuf dependencies such as the Google HTTP annotations for openapi generation or grpc-gateway usage.
There were some recent changes to have our own ratelimiting annotations.
The two combined were not working when I was trying to use them together (attempting to rebase another branch)
Buf workspaces should be the solution to the problem
Buf workspaces means that each module will have generated Go code that embeds proto file names relative to the proto dir and not the top level repo root.
This resulted in proto file name conflicts in the Go global protobuf type registry.
The solution to that was to add in a private/ directory into the path within the proto/ directory.
That then required rewriting all the imports.

Is this safe?

AFAICT yes
The gRPC wire protocol doesn't seem to care about the proto file names (although the Go grpc code does tack on the proto file name as Metadata in the ServiceDesc)
Other than imports, there were no changes to any generated code as a result of this.
2023-02-17 16:14:46 -05:00
Matt Keeler 5afd4657ec
Protobuf Modernization (#15949)
* Protobuf Modernization

Remove direct usage of golang/protobuf in favor of google.golang.org/protobuf

Marshallers (protobuf and json) needed some changes to account for different APIs.

Moved to using the google.golang.org/protobuf/types/known/* for the well known types including replacing some custom Struct manipulation with whats available in the structpb well known type package.

This also updates our devtools script to install protoc-gen-go from the right location so that files it generates conform to the correct interfaces.

* Fix go-mod-tidy make target to work on all modules
2023-01-11 09:39:10 -05:00
Dan Upton ee2d47da83
Fix missing comment in generated enterprise rate-limit file (#15895)
Fixes bug in #15564 where gofmt would strip out the generated code
warning comment because it was on the same line as the build tag.
2023-01-05 10:31:14 +00:00
Dan Upton d4c435856b
grpc: `protoc` plugin for generating gRPC rate limit specifications (#15564)
Adds automation for generating the map of `gRPC Method Name → Rate Limit Type`
used by the middleware introduced in #15550, and will ensure we don't forget
to add new endpoints.

Engineers must annotate their RPCs in the proto file like so:

```
rpc Foo(FooRequest) returns (FooResponse) {
  option (consul.internal.ratelimit.spec) = {
    operation_type: READ,
  };
}
```

When they run `make proto` a protoc plugin `protoc-gen-consul-rate-limit` will
be installed that writes rate-limit specs as a JSON array to a file called
`.ratelimit.tmp` (one per protobuf package/directory).

After running Buf, `make proto` will execute a post-process script that will
ingest all of the `.ratelimit.tmp` files and generate a Go file containing the
mappings in the `agent/grpc-middleware` package. In the enterprise repository,
it will write an additional file with the enterprise-only endpoints.

If an engineer forgets to add the annotation to a new RPC, the plugin will
return an error like so:

```
RPC Foo is missing rate-limit specification, fix it with:

	import "proto-public/annotations/ratelimit/ratelimit.proto";

	service Bar {
	  rpc Foo(...) returns (...) {
	    option (hashicorp.consul.internal.ratelimit.spec) = {
	      operation_type: OPERATION_READ | OPERATION_WRITE | OPERATION_EXEMPT,
	    };
	  }
	}
```

In the future, this annotation can be extended to support rate-limit
category (e.g. KV vs Catalog) and to determine the retry policy.
2023-01-04 16:07:02 +00:00
Chris S. Kim cc819ad83b
[OSS] Add boilerplate for proto files implementing BlockableQuery (#15554) 2022-11-25 15:46:56 -05:00
Kyle Schochenmaier bf0f61a878
removes ioutil usage everywhere which was deprecated in go1.16 (#15297)
* update go version to 1.18 for api and sdk, go mod tidy
* removes ioutil usage everywhere which was deprecated in go1.16 in favour of io and os packages. Also introduces a lint rule which forbids use of ioutil going forward.
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2022-11-10 10:26:01 -06:00
Chris S. Kim bde57c0dd0 Regenerate files according to 1.19.2 formatter 2022-10-24 16:12:08 -04:00
Chris S. Kim 3d05a8e9af Remove unused methods from template 2022-10-18 16:35:16 -04:00
Chris S. Kim 29a297d3e9
Refactor client RPC timeouts (#14965)
Fix an issue where rpc_hold_timeout was being used as the timeout for non-blocking queries. Users should be able to tune read timeouts without fiddling with rpc_hold_timeout. A new configuration `rpc_read_timeout` is created.

Refactor some implementation from the original PR 11500 to remove the misleading linkage between RPCInfo's timeout (used to retry in case of certain modes of failures) and the client RPC timeouts.
2022-10-18 15:05:09 -04:00
Paul Glass 77afe0e76e
Extract AWS auth implementation out of Consul (#13760) 2022-07-19 16:26:44 -05:00
alex 7c0daeade8
fix leader annotation (#13786)
Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>
2022-07-18 10:34:59 -07:00
alex b7043f7150
peering: add warning about AllowStaleRead (#13768) 2022-07-15 09:56:33 -07:00
Evan Culver 88449b1f1b
internal: port RPC glue changes from Enterprise (#13034)
Co-authored-by: R.B. Boyer <rb@hashicorp.com>
2022-05-11 15:03:07 -07:00
Will Jordan c48120d005
Add timeout to Client RPC calls (#11500)
Adds a timeout (deadline) to client RPC calls, so that streams will no longer hang indefinitely in unstable network conditions.

Co-authored-by: kisunji <ckim@hashicorp.com>
2022-04-21 16:21:35 -04:00
Paul Glass d79ca2f024
acl: Fix tag parsing for IAM users and roles in IAM auth method (#12797)
* acl: Fix tag parsing on IAM users and roles in IAM auth method

* Add changelog
2022-04-14 16:45:35 -05:00
Paul Glass 99f373dde4
acl: Adjust region handling in AWS IAM auth method (#12774)
* acl: Adjust region handling in AWS IAM auth method
2022-04-13 14:31:37 -05:00
Paul Glass 706c844423
Add IAM Auth Method (#12583)
This adds an aws-iam auth method type which supports authenticating to Consul using AWS IAM identities.

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2022-03-31 10:18:48 -05:00
R.B. Boyer 4676960b93
proto-gen-rpc-glue: support QueryMeta and QueryOptions (#12637) 2022-03-28 13:12:51 -05:00
R.B. Boyer dc023cb0dd
proto-gen-rpc-glue: use a shallow copy of proto/pbcommon instead of a consul dependency (#12634) 2022-03-28 10:08:41 -05:00