65 Commits (d34bde0e4eefad6088df2f01c4f89f6289d1e93c)

Author	SHA1	Message	Date
R.B. Boyer	0ee95df4e0	proto: clear out old ratelimit.tmp files before making new ones (#17292)	2 years ago
Ronald	b64674623e	Copyright headers for missing files/folders (#16708) ... * copyright headers for agent folder	2 years ago
Matt Keeler	085c0addc0	Protobuf Refactoring for Multi-Module Cleanliness (#16302) ... Protobuf Refactoring for Multi-Module Cleanliness This commit includes the following: Moves all packages that were within proto/ to proto/private Rewrites imports to account for the packages being moved Adds in buf.work.yaml to enable buf workspaces Names the proto-public buf module so that we can override the Go package imports within proto/buf.yaml Bumps the buf version dependency to 1.14.0 (I was trying out the version to see if it would get around an issue - it didn't but it also doesn't break things and it seemed best to keep up with the toolchain changes) Why: In the future we will need to consume other protobuf dependencies such as the Google HTTP annotations for openapi generation or grpc-gateway usage. There were some recent changes to have our own ratelimiting annotations. The two combined were not working when I was trying to use them together (attempting to rebase another branch) Buf workspaces should be the solution to the problem Buf workspaces means that each module will have generated Go code that embeds proto file names relative to the proto dir and not the top level repo root. This resulted in proto file name conflicts in the Go global protobuf type registry. The solution to that was to add in a private/ directory into the path within the proto/ directory. That then required rewriting all the imports. Is this safe? AFAICT yes The gRPC wire protocol doesn't seem to care about the proto file names (although the Go grpc code does tack on the proto file name as Metadata in the ServiceDesc) Other than imports, there were no changes to any generated code as a result of this.	2 years ago
Nitya Dhanushkodi	b8b37c2357	refactor: remove troubleshoot module dependency on consul top level module (#16162) ... Ensure nothing in the troubleshoot go module depends on consul's top level module. This is so we can import troubleshoot into consul-k8s and not import all of consul. * turns troubleshoot into a go module [authored by @curtbushko] * gets the envoy protos into the troubleshoot module [authored by @curtbushko] * adds a new go module `envoyextensions` which has xdscommon and extensioncommon folders that both the xds package and the troubleshoot package can import * adds testing and linting for the new go modules * moves the unit tests in `troubleshoot/validateupstream` that depend on proxycfg/xds into the xds package, with a comment describing why those tests cannot be in the troubleshoot package * fixes all the imports everywhere as a result of these changes Co-authored-by: Curt Bushko <cbushko@gmail.com>	2 years ago
Matt Keeler	5afd4657ec	Protobuf Modernization (#15949) ... * Protobuf Modernization Remove direct usage of golang/protobuf in favor of google.golang.org/protobuf Marshallers (protobuf and json) needed some changes to account for different APIs. Moved to using the google.golang.org/protobuf/types/known/* for the well known types including replacing some custom Struct manipulation with whats available in the structpb well known type package. This also updates our devtools script to install protoc-gen-go from the right location so that files it generates conform to the correct interfaces. * Fix go-mod-tidy make target to work on all modules	2 years ago
Matt Keeler	8b4eb0c8c1	Fix usage of protoc-gen-go-binary (#15920) ... * Fix usage of protoc-gen-go-binary * Pin protoc-gen-go-binary version	2 years ago
Dan Upton	d4c435856b	grpc: `protoc` plugin for generating gRPC rate limit specifications (#15564) ... Adds automation for generating the map of `gRPC Method Name → Rate Limit Type` used by the middleware introduced in #15550, and will ensure we don't forget to add new endpoints. Engineers must annotate their RPCs in the proto file like so: ``` rpc Foo(FooRequest) returns (FooResponse) { option (consul.internal.ratelimit.spec) = { operation_type: READ, }; } ``` When they run `make proto` a protoc plugin `protoc-gen-consul-rate-limit` will be installed that writes rate-limit specs as a JSON array to a file called `.ratelimit.tmp` (one per protobuf package/directory). After running Buf, `make proto` will execute a post-process script that will ingest all of the `.ratelimit.tmp` files and generate a Go file containing the mappings in the `agent/grpc-middleware` package. In the enterprise repository, it will write an additional file with the enterprise-only endpoints. If an engineer forgets to add the annotation to a new RPC, the plugin will return an error like so: ``` RPC Foo is missing rate-limit specification, fix it with: import "proto-public/annotations/ratelimit/ratelimit.proto"; service Bar { rpc Foo(...) returns (...) { option (hashicorp.consul.internal.ratelimit.spec) = { operation_type: OPERATION_READ | OPERATION_WRITE | OPERATION_EXEMPT, }; } } ``` In the future, this annotation can be extended to support rate-limit category (e.g. KV vs Catalog) and to determine the retry policy.	2 years ago
Dan Upton	e6b55d1d81	perf: remove expensive reflection from xDS hot path (#14934) ... Replaces the reflection-based implementation of proxycfg's ConfigSnapshot.Clone with code generated by deep-copy. While load testing server-based xDS (for consul-dataplane) we discovered this method is extremely expensive. The ConfigSnapshot struct, directly or indirectly, contains a copy of many of the structs in the agent/structs package, which creates a large graph for copystructure.Copy to traverse at runtime, on every proxy reconfiguration.	2 years ago
Chris Chapman	2c6ef41893	Add mocks for probuf generation	2 years ago
Luke Kysow	bb943bc77c	makefile: give better error for tool installed by wrong package (#13797) ... I had protoc-gen-go installed through `google.golang.org/protobuf` instead of `github.com/golang/protobuf` and `make proto` was failing silently. This change will ensure you get an error: ``` protoc-gen-go is already installed by module "google.golang.org/protobuf" but should be installed by module "github.com/golang/protobuf". Delete it and re-run to re-install. ```	2 years ago
Mark Anderson	175728b292	Minor cleanup for build-date script ... Signed-off-by: Mark Anderson <manderson@hashicorp.com>	3 years ago
Mark Anderson	d8663a741d	Fixup build scripts ... Signed-off-by: Mark Anderson <manderson@hashicorp.com>	3 years ago
Fulvio	d457d8b6ce	UDP check for service stanza #12221 (#12722) ... * UDP check for service stanza #12221 * add pass status on timeout condition * delete useless files * Update check_test.go improve comment in test * fix test * fix requested changes and update TestRuntimeConfig_Sanitize.golden * add freeport to TestCheckUDPCritical * improve comment for CheckUDP struct * fix requested changes * fix requested changes * fix requested changes * add UDP to proto * add UDP to proto and add a changelog * add requested test on agent_endpoint_test.go * add test for given endpoints * fix failing tests * add documentation for udp healthcheck * regenerate proto using buf * Update website/content/api-docs/agent/check.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * Update website/content/api-docs/agent/check.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * Update website/content/docs/discovery/checks.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * Update website/content/docs/ecs/configuration-reference.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * Update website/content/docs/ecs/configuration-reference.mdx Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> * add debug echo * add debug circle-ci * add debug circle-ci bash * use echo instead of status_stage * remove debug and status from devtools script and use echo instead * Update website/content/api-docs/agent/check.mdx Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com> * fix test * replace status_stage with status * replace functions with echo Co-authored-by: Dhia Ayachi <dhia@hashicorp.com> Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com>	3 years ago
R.B. Boyer	1b69366a39	build: ensure tools match go toolchain version (#13338)	3 years ago
Chris S. Kim	f0a9b30174	Update repo to use go:embed (#10996) ... Replace bindata packages with stdlib go:embed. Modernize some uiserver code with newer interfaces introduced in go 1.16 (mainly working with fs.File instead of http.File. Remove steps that are no longer used from our build files. Add Github Action to detect differences in agent/uiserver/dist and verify that the files are correct (by compiling UI assets and comparing contents).	3 years ago
R.B. Boyer	a85b8a4705	api: ensure peering API endpoints do not use protobufs (#13204) ... I noticed that the JSON api endpoints for peerings json encodes protobufs directly, rather than converting them into their `api` package equivalents before marshal/unmarshaling them. I updated this and used `mog` to do the annoying part in the middle. Other changes: - the status enum was converted into the friendlier string form of the enum for readability with tools like `curl` - some of the `api` library functions were slightly modified to match other similar endpoints in UX (cc: @ndhanushkodi ) - peeringRead returns `nil` if not found - partitions are NOT inferred from the agent's partition (matching 1.11-style logic)	3 years ago
Kyle Havlovitz	4bc6c23357	Add connection limit setting to service defaults	3 years ago
R.B. Boyer	7b72ddfb60	build: wire up remaining 5 helper tools into the same auto-install logic used for protobuf tools (#13184)	3 years ago
Matt Keeler	26f4ea3f01	Migrate from `protoc` to `buf` (#12841) ... * Install `buf` instead of `protoc` * Created `buf.yaml` and `buf.gen.yaml` files in the two proto directories to control how `buf` generates/lints proto code. * Invoke `buf` instead of `protoc` * Added a `proto-format` make target. * Committed the reformatted proto files. * Added a `proto-lint` make target. * Integrated proto linting with CI * Fixed tons of proto linter warnings. * Got rid of deprecated builtin protoc-gen-go grpc plugin usage. Moved to direct usage of protoc-gen-go-grpc. * Unified all proto directories / go packages around using pb prefixes but ensuring all proto packages do not have the prefix.	3 years ago
R.B. Boyer	a85a4b3bbd	build: conditionally install or reinstall protobuf supporting tools as needed (#12674)	3 years ago
R.B. Boyer	dc62293978	build: run mog separately after the protobufs are generated (#12665) ... Also ensure that we run mog serially on each package in dependency order.	3 years ago
Eric	e4b4f175ed	Bump go-control-plane ... * `go get cloud.google.com/go@v0.59.0` * `go get github.com/envoyproxy/go-control-plane@v0.9.9` * `make envoy-library` * Bumpprotoc to 3.15.8	3 years ago
R.B. Boyer	82bfc04f87	build: auto install correct version of protoc locally (#12651)	3 years ago
R.B. Boyer	cef336a0a3	build: install mog and execute it during protobuf compilation (#12647) ... - also import replace isn't needed anymore	3 years ago
Eric	76c65e873c	code review changes	3 years ago
Eric	5f050614e5	remove the rest of gogo	3 years ago
R.B. Boyer	3d725a1762	regenerate rpc glue stubs in protobuf files using comments (#12625)	3 years ago
Eric	5cab213e81	assorted changes required to remove gogo	3 years ago
Eric	776f5843d0	remove gogo from pbservice	3 years ago
Eric	14530c7caa	remove gogo pbconnect, pbconfig and pbautoconf	3 years ago
Dhia Ayachi	72a997242b	split `pbcommon` to `pbcommon` and `pbcommongogo` (#12587) ... * mogify needed pbcommon structs * mogify needed pbconnect structs * fix compilation errors and make config_translate_test pass * add missing file * remove redundant oss func declaration * fix EnterpriseMeta to copy the right data for enterprise * rename pbcommon package to pbcommongogo * regenerate proto and mog files * add missing mog files * add pbcommon package * pbcommon no mog * fix enterprise meta code generation * fix enterprise meta code generation (pbcommongogo) * fix mog generation for gogo * use `protoc-go-inject-tag` to inject tags * rename proto package * pbcommon no mog * use `protoc-go-inject-tag` to inject tags * add non gogo proto to make file * fix proto get	3 years ago
Daniel Nephin	f504a02776	Replace build script with 'go build'	3 years ago
Daniel Nephin	d46f2429ec	build-support: remove test-flake machinery ... This machinery was not used, and does not appear to be maintained. In practice we really don't need anything to detect flaky tests. Our CI system identifies flaky tests at https://app.circleci.com/insights/github/hashicorp/consul/workflows/go-tests/tests?branch=main Mostly what we need is a way to reproduce flakes, which can be done directly with the Go CLI, using the -race, -count, and (new in Go 1.17) -shuffle flags.	3 years ago
Daniel Nephin	1f9479603c	Add failures_before_warning to checks (#10969) ... Signed-off-by: Jakub Sokołowski <jakub@status.im> * agent: add failures_before_warning setting The new setting allows users to specify the number of check failures that have to happen before a service status us updated to be `warning`. This allows for more visibility for detected issues without creating alerts and pinging administrators. Unlike the previous behavior, which caused the service status to not update until it reached the configured `failures_before_critical` setting, now Consul updates the Web UI view with the `warning` state and the output of the service check when `failures_before_warning` is breached. The default value of `FailuresBeforeWarning` is the same as the value of `FailuresBeforeCritical`, which allows for retaining the previous default behavior of not triggering a warning. When `FailuresBeforeWarning` is set to a value higher than that of `FailuresBeforeCritical it has no effect as `FailuresBeforeCritical` takes precedence. Resolves: https://github.com/hashicorp/consul/issues/10680 Signed-off-by: Jakub Sokołowski <jakub@status.im> Co-authored-by: Jakub Sokołowski <jakub@status.im>	3 years ago
Daniel Nephin	a639d657ae	Merge pull request #10353 from hashicorp/dnephin/prune-build-scripts-1 ... Remove a few unused things from build-support	3 years ago
Daniel Nephin	6117ba3b2e	Remove a few unused things from build-support	3 years ago
Jared Kirschner	bd536151e1	Replace use of 'sane' where appropriate ... HashiCorp voice, style, and language guidelines recommend avoiding ableist language unless its reference to ability is accurate in a particular use.	3 years ago
R.B. Boyer	85a718da63	xds: ensure that all envoyproxy/go-control-plane protobuf symbols are linked into the final binary (#10131) ... This ensures that if someone does include some extension Consul does not currently make use of, that extension is actually usable. Without linking these envoy protobufs into the main binary it can't round trip the escape hatches to send them down to envoy. Whenenver the go-control-plane library is upgraded next we just have to re-run 'make envoy-library'.	4 years ago
R.B. Boyer	4db8b78854	connect: update centralized upstreams representation in service-defaults (#10015)	4 years ago
R.B. Boyer	e113dc0fe2	upstream some differences from enterprise (#8902)	4 years ago
Daniel Nephin	24198cfc7f	Update build-support/scripts/proto-gen.sh	4 years ago
Daniel Nephin	f4791f0e82	ci: Add check-generated ... And remove makefile targets that can be accomplished with --always-make	4 years ago
Matt Keeler	9b01f9423c	Implement the insecure version of the Cluster.AutoConfig RPC endpoint ... Right now this is only hooked into the insecure RPC server and requires JWT authorization. If no JWT authorizer is setup in the configuration then we inject a disabled “authorizer” to always report that JWT authorization is disabled.	5 years ago
Matt Keeler	28221f66f2	Use encoding/json instead of jsonpb even for protobuf types (#6572) ... This only works so long as we use simplistic protobuf types. Constructs such as oneof or Any types that require type annotations for decoding properly will fail hard but that is by design. If/when we want to use any of that we will probably need to consider a v2 API.	5 years ago
Matt Keeler	abed91d069	Generate JSON and Binary Marshalers for Protobuf Types (#6564) ... * Add JSON and Binary Marshaler Generators for Protobuf Types * Generate files with the correct version of gogo/protobuf I have pinned the version in the makefile so when you run make tools you get the right version. This pulls the version out of go.mod so it should remain up to date. The version at the time of this commit we are using is v1.2.1 * Fixup some shell output * Update how we determine the version of gogo This just greps the go.mod file instead of expecting the go mod cache to already be present * Fixup vendoring and remove no longer needed json encoder functions	5 years ago
Matt Keeler	51dcd126b7	Add support for implementing new requests with protobufs instea… (#6502) ... * Add build system support for protobuf generation This is done generically so that we don’t have to keep updating the makefile to add another proto generation. Note: anything not in the vendor directory and with a .proto extension will be run through protoc if the corresponding namespace.pb.go file is not up to date. If you want to rebuild just a single proto file you can do so with: make proto-rebuild PROTOFILES=<list of proto files to rebuild> Providing the PROTOFILES var will override the default behavior of finding all the .proto files. * Start adding types to the agent/proto package These will be needed for some other work and are by no means comprehensive. * Add ability to resolve/fixup the agentpb.ACLLinks structure in the state store. * Use protobuf marshalling of raft requests instead of msgpack for protoc generated types. This does not change any encoding of existing types. * Removed structs package automatically encoding with protobuf marshalling Instead the caller of raftApply that wants to opt-in to protobuf encoding will have to call `raftApplyProtobuf` * Run update-vendor to fixup modules.txt Nothing changed as far as dependencies go but the ordering of modules in that file depends on the time they are first seen and its not alphabetical. * Rename some things and implement the structs.RPCInfo interface bits agentpb.QueryOptions and agentpb.WriteRequest implement 3 of the 4 RPCInfo funcs and the new TargetDatacenter message type implements the fourth. * Use the right encoding function. * Renamed agent/proto package to agent/agentpb to prevent package name conflicts * Update modules.txt to fix ordering * Change blockingQuery to take in interfaces for the query options and meta * Add %T to error output. * Add/Update some comments	5 years ago
Freddy	b2c12550d7	Remove old UI, option to use it, and its build processes	6 years ago
Freddy	b2032b05be	Add script and makefile goal to help debug flaky tests	6 years ago
John Cowen	80a307cb9f	UI: Add conditional enterprise logo (#4432) ... Adds additional 'enterprise' text underneath the 'startup' logo if the ui is built with a CONSUL_BINARY_TYPE environment variable that doesn't equal `oss`.	6 years ago
Matt Keeler	8f2a19a747	Update verify.sh script	7 years ago