consul

Commit Graph

Author	SHA1	Message	Date
Matt Keeler	085c0addc0	Protobuf Refactoring for Multi-Module Cleanliness (#16302 ) Protobuf Refactoring for Multi-Module Cleanliness This commit includes the following: Moves all packages that were within proto/ to proto/private Rewrites imports to account for the packages being moved Adds in buf.work.yaml to enable buf workspaces Names the proto-public buf module so that we can override the Go package imports within proto/buf.yaml Bumps the buf version dependency to 1.14.0 (I was trying out the version to see if it would get around an issue - it didn't but it also doesn't break things and it seemed best to keep up with the toolchain changes) Why: In the future we will need to consume other protobuf dependencies such as the Google HTTP annotations for openapi generation or grpc-gateway usage. There were some recent changes to have our own ratelimiting annotations. The two combined were not working when I was trying to use them together (attempting to rebase another branch) Buf workspaces should be the solution to the problem Buf workspaces means that each module will have generated Go code that embeds proto file names relative to the proto dir and not the top level repo root. This resulted in proto file name conflicts in the Go global protobuf type registry. The solution to that was to add in a private/ directory into the path within the proto/ directory. That then required rewriting all the imports. Is this safe? AFAICT yes The gRPC wire protocol doesn't seem to care about the proto file names (although the Go grpc code does tack on the proto file name as Metadata in the ServiceDesc) Other than imports, there were no changes to any generated code as a result of this.	2 years ago
Dan Stough	f1436109ea	[OSS] security: update go to 1.20.1 (#16263 ) * security: update go to 1.20.1	2 years ago
Nitya Dhanushkodi	b8b37c2357	refactor: remove troubleshoot module dependency on consul top level module (#16162 ) Ensure nothing in the troubleshoot go module depends on consul's top level module. This is so we can import troubleshoot into consul-k8s and not import all of consul. * turns troubleshoot into a go module [authored by @curtbushko] * gets the envoy protos into the troubleshoot module [authored by @curtbushko] * adds a new go module `envoyextensions` which has xdscommon and extensioncommon folders that both the xds package and the troubleshoot package can import * adds testing and linting for the new go modules * moves the unit tests in `troubleshoot/validateupstream` that depend on proxycfg/xds into the xds package, with a comment describing why those tests cannot be in the troubleshoot package * fixes all the imports everywhere as a result of these changes Co-authored-by: Curt Bushko <cbushko@gmail.com>	2 years ago
Matt Keeler	5afd4657ec	Protobuf Modernization (#15949 ) * Protobuf Modernization Remove direct usage of golang/protobuf in favor of google.golang.org/protobuf Marshallers (protobuf and json) needed some changes to account for different APIs. Moved to using the google.golang.org/protobuf/types/known/* for the well known types including replacing some custom Struct manipulation with whats available in the structpb well known type package. This also updates our devtools script to install protoc-gen-go from the right location so that files it generates conform to the correct interfaces. * Fix go-mod-tidy make target to work on all modules	2 years ago
Matt Keeler	8b4eb0c8c1	Fix usage of protoc-gen-go-binary (#15920 ) * Fix usage of protoc-gen-go-binary * Pin protoc-gen-go-binary version	2 years ago
Dan Upton	d4c435856b	grpc: `protoc` plugin for generating gRPC rate limit specifications (#15564 ) Adds automation for generating the map of `gRPC Method Name → Rate Limit Type` used by the middleware introduced in #15550, and will ensure we don't forget to add new endpoints. Engineers must annotate their RPCs in the proto file like so: ``` rpc Foo(FooRequest) returns (FooResponse) { option (consul.internal.ratelimit.spec) = { operation_type: READ, }; } ``` When they run `make proto` a protoc plugin `protoc-gen-consul-rate-limit` will be installed that writes rate-limit specs as a JSON array to a file called `.ratelimit.tmp` (one per protobuf package/directory). After running Buf, `make proto` will execute a post-process script that will ingest all of the `.ratelimit.tmp` files and generate a Go file containing the mappings in the `agent/grpc-middleware` package. In the enterprise repository, it will write an additional file with the enterprise-only endpoints. If an engineer forgets to add the annotation to a new RPC, the plugin will return an error like so: ``` RPC Foo is missing rate-limit specification, fix it with: import "proto-public/annotations/ratelimit/ratelimit.proto"; service Bar { rpc Foo(...) returns (...) { option (hashicorp.consul.internal.ratelimit.spec) = { operation_type: OPERATION_READ \| OPERATION_WRITE \| OPERATION_EXEMPT, }; } } ``` In the future, this annotation can be extended to support rate-limit category (e.g. KV vs Catalog) and to determine the retry policy.	2 years ago
Chris S. Kim	41019adf56	Update go version to 1.19	2 years ago
Dan Upton	e6b55d1d81	perf: remove expensive reflection from xDS hot path (#14934 ) Replaces the reflection-based implementation of proxycfg's ConfigSnapshot.Clone with code generated by deep-copy. While load testing server-based xDS (for consul-dataplane) we discovered this method is extremely expensive. The ConfigSnapshot struct, directly or indirectly, contains a copy of many of the structs in the agent/structs package, which creates a large graph for copystructure.Copy to traverse at runtime, on every proxy reconfiguration.	2 years ago
Chris Chapman	2c6ef41893	Add mocks for probuf generation	2 years ago
DanStough	d8cb7731dd	chore: add multi-arch docker build for testing	2 years ago
Luke Kysow	bb943bc77c	makefile: give better error for tool installed by wrong package (#13797 ) I had protoc-gen-go installed through `google.golang.org/protobuf` instead of `github.com/golang/protobuf` and `make proto` was failing silently. This change will ensure you get an error: ``` protoc-gen-go is already installed by module "google.golang.org/protobuf" but should be installed by module "github.com/golang/protobuf". Delete it and re-run to re-install. ```	2 years ago
Mark Anderson	175728b292	Minor cleanup for build-date script Signed-off-by: Mark Anderson <manderson@hashicorp.com>	3 years ago
Mark Anderson	9343e05498	Fix build-date to use UTC Signed-off-by: Mark Anderson <manderson@hashicorp.com>	3 years ago
Mark Anderson	8945b68a9d	Cleanup and extend basic build date Signed-off-by: Mark Anderson <manderson@hashicorp.com>	3 years ago
Mark Anderson	d8663a741d	Fixup build scripts Signed-off-by: Mark Anderson <manderson@hashicorp.com>	3 years ago
Fulvio	d457d8b6ce	UDP check for service stanza #12221 (#12722 ) * UDP check for service stanza #12221 * add pass status on timeout condition * delete useless files * Update check_test.go improve comment in test * fix test * fix requested changes and update TestRuntimeConfig_Sanitize.golden * add freeport to TestCheckUDPCritical * improve comment for CheckUDP struct * fix requested changes * fix requested changes * fix requested changes * add UDP to proto * add UDP to proto and add a changelog * add requested test on agent_endpoint_test.go * add test for given endpoints * fix failing tests * add documentation for udp healthcheck * regenerate proto using buf * Update website/content/api-docs/agent/check.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * Update website/content/api-docs/agent/check.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * Update website/content/docs/discovery/checks.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * Update website/content/docs/ecs/configuration-reference.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * Update website/content/docs/ecs/configuration-reference.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * add debug echo * add debug circle-ci * add debug circle-ci bash * use echo instead of status_stage * remove debug and status from devtools script and use echo instead * Update website/content/api-docs/agent/check.mdx Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com> * fix test * replace status_stage with status * replace functions with echo Co-authored-by: Dhia Ayachi <dhia@hashicorp.com> Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com>	3 years ago
R.B. Boyer	1b69366a39	build: ensure tools match go toolchain version (#13338 )	3 years ago
Chris S. Kim	f0a9b30174	Update repo to use go:embed (#10996 ) Replace bindata packages with stdlib go:embed. Modernize some uiserver code with newer interfaces introduced in go 1.16 (mainly working with fs.File instead of http.File. Remove steps that are no longer used from our build files. Add Github Action to detect differences in agent/uiserver/dist and verify that the files are correct (by compiling UI assets and comparing contents).	3 years ago
R.B. Boyer	a85b8a4705	api: ensure peering API endpoints do not use protobufs (#13204 ) I noticed that the JSON api endpoints for peerings json encodes protobufs directly, rather than converting them into their `api` package equivalents before marshal/unmarshaling them. I updated this and used `mog` to do the annoying part in the middle. Other changes: - the status enum was converted into the friendlier string form of the enum for readability with tools like `curl` - some of the `api` library functions were slightly modified to match other similar endpoints in UX (cc: @ndhanushkodi ) - peeringRead returns `nil` if not found - partitions are NOT inferred from the agent's partition (matching 1.11-style logic)	3 years ago
Kyle Havlovitz	4bc6c23357	Add connection limit setting to service defaults	3 years ago
R.B. Boyer	9e1f362499	ci: github actions run with TERM=dumb and can't do tput commands (#13187 )	3 years ago
R.B. Boyer	7b72ddfb60	build: wire up remaining 5 helper tools into the same auto-install logic used for protobuf tools (#13184 )	3 years ago
Matt Keeler	26f4ea3f01	Migrate from `protoc` to `buf` (#12841 ) * Install `buf` instead of `protoc` * Created `buf.yaml` and `buf.gen.yaml` files in the two proto directories to control how `buf` generates/lints proto code. * Invoke `buf` instead of `protoc` * Added a `proto-format` make target. * Committed the reformatted proto files. * Added a `proto-lint` make target. * Integrated proto linting with CI * Fixed tons of proto linter warnings. * Got rid of deprecated builtin protoc-gen-go grpc plugin usage. Moved to direct usage of protoc-gen-go-grpc. * Unified all proto directories / go packages around using pb prefixes but ensuring all proto packages do not have the prefix.	3 years ago
DanStough	95250e7915	Update go version to 1.18.1	3 years ago
John Murret	1006c8a94b	Correcting an uncapitalized word setup at the beginning of titles to be capitalized in vault section. (#12759 )	3 years ago
R.B. Boyer	96748197fc	build: remove unused tools from being installed (#12671 )	3 years ago
R.B. Boyer	a85a4b3bbd	build: conditionally install or reinstall protobuf supporting tools as needed (#12674 )	3 years ago
R.B. Boyer	dc62293978	build: run mog separately after the protobufs are generated (#12665 ) Also ensure that we run mog serially on each package in dependency order.	3 years ago
Eric	e4b4f175ed	Bump go-control-plane * `go get cloud.google.com/go@v0.59.0` * `go get github.com/envoyproxy/go-control-plane@v0.9.9` * `make envoy-library` * Bumpprotoc to 3.15.8	3 years ago
R.B. Boyer	82bfc04f87	build: auto install correct version of protoc locally (#12651 )	3 years ago
R.B. Boyer	cef336a0a3	build: install mog and execute it during protobuf compilation (#12647 ) - also import replace isn't needed anymore	3 years ago
Eric	76c65e873c	code review changes	3 years ago
Eric	5f050614e5	remove the rest of gogo	3 years ago
R.B. Boyer	3d725a1762	regenerate rpc glue stubs in protobuf files using comments (#12625 )	3 years ago
Eric	5cab213e81	assorted changes required to remove gogo	3 years ago
Eric	776f5843d0	remove gogo from pbservice	3 years ago
Eric	14530c7caa	remove gogo pbconnect, pbconfig and pbautoconf	3 years ago
Dhia Ayachi	72a997242b	split `pbcommon` to `pbcommon` and `pbcommongogo` (#12587 ) * mogify needed pbcommon structs * mogify needed pbconnect structs * fix compilation errors and make config_translate_test pass * add missing file * remove redundant oss func declaration * fix EnterpriseMeta to copy the right data for enterprise * rename pbcommon package to pbcommongogo * regenerate proto and mog files * add missing mog files * add pbcommon package * pbcommon no mog * fix enterprise meta code generation * fix enterprise meta code generation (pbcommongogo) * fix mog generation for gogo * use `protoc-go-inject-tag` to inject tags * rename proto package * pbcommon no mog * use `protoc-go-inject-tag` to inject tags * add non gogo proto to make file * fix proto get	3 years ago
Evan Culver	c17aa3720f	Fix build script (#12367 )	3 years ago
Daniel Nephin	f504a02776	Replace build script with 'go build'	3 years ago
Chris S. Kim	8bdbe7c0fb	Update CI and release go versions to 1.17.5 (#11799 )	3 years ago
Daniel Nephin	d46f2429ec	build-support: remove test-flake machinery This machinery was not used, and does not appear to be maintained. In practice we really don't need anything to detect flaky tests. Our CI system identifies flaky tests at https://app.circleci.com/insights/github/hashicorp/consul/workflows/go-tests/tests?branch=main Mostly what we need is a way to reproduce flakes, which can be done directly with the Go CLI, using the -race, -count, and (new in Go 1.17) -shuffle flags.	3 years ago
Daniel Nephin	846ff9a2c7	Merge pull request #11364 from hashicorp/use-go1.17 ci: test against go1.17	3 years ago
Daniel Nephin	66798ca42d	ci: test against go1.17	3 years ago
R.B. Boyer	bc912192af	build: fix ui make target to use the same frontend stack as the regular release (#11368 )	3 years ago
Daniel Nephin	1f9479603c	Add failures_before_warning to checks (#10969 ) Signed-off-by: Jakub Sokołowski <jakub@status.im> * agent: add failures_before_warning setting The new setting allows users to specify the number of check failures that have to happen before a service status us updated to be `warning`. This allows for more visibility for detected issues without creating alerts and pinging administrators. Unlike the previous behavior, which caused the service status to not update until it reached the configured `failures_before_critical` setting, now Consul updates the Web UI view with the `warning` state and the output of the service check when `failures_before_warning` is breached. The default value of `FailuresBeforeWarning` is the same as the value of `FailuresBeforeCritical`, which allows for retaining the previous default behavior of not triggering a warning. When `FailuresBeforeWarning` is set to a value higher than that of `FailuresBeforeCritical it has no effect as `FailuresBeforeCritical` takes precedence. Resolves: https://github.com/hashicorp/consul/issues/10680 Signed-off-by: Jakub Sokołowski <jakub@status.im> Co-authored-by: Jakub Sokołowski <jakub@status.im>	3 years ago
R.B. Boyer	e0d1e2689c	ci: upgrade to use Go 1.16.7 (#10856 )	3 years ago
Daniel Nephin	a639d657ae	Merge pull request #10353 from hashicorp/dnephin/prune-build-scripts-1 Remove a few unused things from build-support	3 years ago
jkirschner-hashicorp	5f73de6fbc	Merge pull request #10560 from jkirschner-hashicorp/change-sane-to-reasonable Replace use of 'sane' where appropriate	3 years ago
Daniel Nephin	6117ba3b2e	Remove a few unused things from build-support	3 years ago

1 2 3 4

156 Commits (6b5e48b2d93e9dd04a2f2f925f601ef8361d49d1)