You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
consul/Makefile

637 lines
23 KiB

# For documentation on building consul from source, refer to:
# https://www.consul.io/docs/install#compiling-from-source
SHELL = bash
GO_MODULES := $(shell find . -name go.mod -exec dirname {} \; | grep -v "proto-gen-rpc-glue/e2e" | sort)
###
# These version variables can either be a valid string for "go install <module>@<version>"
# or the string @DEV to imply use what is currently installed locally.
###
GOLANGCI_LINT_VERSION='v1.55.2'
MOCKERY_VERSION='v2.37.1'
BUF_VERSION='v1.26.0'
Protobuf Refactoring for Multi-Module Cleanliness (#16302) Protobuf Refactoring for Multi-Module Cleanliness This commit includes the following: Moves all packages that were within proto/ to proto/private Rewrites imports to account for the packages being moved Adds in buf.work.yaml to enable buf workspaces Names the proto-public buf module so that we can override the Go package imports within proto/buf.yaml Bumps the buf version dependency to 1.14.0 (I was trying out the version to see if it would get around an issue - it didn't but it also doesn't break things and it seemed best to keep up with the toolchain changes) Why: In the future we will need to consume other protobuf dependencies such as the Google HTTP annotations for openapi generation or grpc-gateway usage. There were some recent changes to have our own ratelimiting annotations. The two combined were not working when I was trying to use them together (attempting to rebase another branch) Buf workspaces should be the solution to the problem Buf workspaces means that each module will have generated Go code that embeds proto file names relative to the proto dir and not the top level repo root. This resulted in proto file name conflicts in the Go global protobuf type registry. The solution to that was to add in a private/ directory into the path within the proto/ directory. That then required rewriting all the imports. Is this safe? AFAICT yes The gRPC wire protocol doesn't seem to care about the proto file names (although the Go grpc code does tack on the proto file name as Metadata in the ServiceDesc) Other than imports, there were no changes to any generated code as a result of this.
2 years ago
PROTOC_GEN_GO_GRPC_VERSION='v1.2.0'
MOG_VERSION='v0.4.1'
PROTOC_GO_INJECT_TAG_VERSION='v1.3.0'
PROTOC_GEN_GO_BINARY_VERSION='v0.1.0'
DEEP_COPY_VERSION='bc3f5aa5735d8a54961580a3a24422c308c831c2'
COPYWRITE_TOOL_VERSION='v0.16.4'
LINT_CONSUL_RETRY_VERSION='v1.4.0'
# Go imports formatter
GCI_VERSION='v0.11.2'
MOCKED_PB_DIRS= pbdns
GOTAGS ?=
GOPATH=$(shell go env GOPATH)
GOARCH?=$(shell go env GOARCH)
MAIN_GOPATH=$(shell go env GOPATH | cut -d: -f1)
export PATH := $(PWD)/bin:$(GOPATH)/bin:$(PATH)
# Get the git commit
GIT_COMMIT?=$(shell git rev-parse --short HEAD)
GIT_COMMIT_YEAR?=$(shell git show -s --format=%cd --date=format:%Y HEAD)
GIT_DIRTY?=$(shell test -n "`git status --porcelain`" && echo "+CHANGES" || true)
GIT_IMPORT=github.com/hashicorp/consul/version
DATE_FORMAT="%Y-%m-%dT%H:%M:%SZ" # it's tricky to do an RFC3339 format in a cross platform way, so we hardcode UTC
GIT_DATE=$(shell $(CURDIR)/build-support/scripts/build-date.sh) # we're using this for build date because it's stable across platform builds
GOLDFLAGS=-X $(GIT_IMPORT).GitCommit=$(GIT_COMMIT)$(GIT_DIRTY) -X $(GIT_IMPORT).BuildDate=$(GIT_DATE)
GOTESTSUM_PATH?=$(shell command -v gotestsum)
ifeq ($(FORCE_REBUILD),1)
NOCACHE=--no-cache
else
NOCACHE=
endif
DOCKER_BUILD_QUIET?=1
ifeq (${DOCKER_BUILD_QUIET},1)
QUIET=-q
else
QUIET=
endif
ifeq ("$(GOTAGS)","")
CONSUL_COMPAT_TEST_IMAGE=hashicorp/consul
else
CONSUL_COMPAT_TEST_IMAGE=hashicorp/consul-enterprise
endif
CONSUL_DEV_IMAGE?=consul-dev
GO_BUILD_TAG?=consul-build-go
UI_BUILD_TAG?=consul-build-ui
BUILD_CONTAINER_NAME?=consul-builder
CONSUL_IMAGE_VERSION?=latest
# When changing the method of Go version detection, also update
# version detection in CI workflows (reusable-get-go-version.yml).
GOLANG_VERSION?=$(shell head -n 1 .go-version)
ENVOY_VERSION?='1.28.0'
CONSUL_DATAPLANE_IMAGE := $(or $(CONSUL_DATAPLANE_IMAGE),"docker.io/hashicorppreview/consul-dataplane:1.3-dev-ubi")
DEPLOYER_CONSUL_DATAPLANE_IMAGE := $(or $(DEPLOYER_CONSUL_DATAPLANE_IMAGE), "docker.io/hashicorppreview/consul-dataplane:1.3-dev")
CONSUL_VERSION?=$(shell cat version/VERSION)
TEST_MODCACHE?=1
TEST_BUILDCACHE?=1
# You can only use as many CPUs as you have allocated to docker
ifdef TEST_DOCKER_CPUS
TEST_DOCKER_RESOURCE_CONSTRAINTS=--cpus $(TEST_DOCKER_CPUS)
TEST_PARALLELIZATION=-e GOMAXPROCS=$(TEST_DOCKER_CPUS)
else
TEST_DOCKER_RESOURCE_CONSTRAINTS=
TEST_PARALLELIZATION=
endif
ifeq ($(TEST_MODCACHE), 1)
TEST_MODCACHE_VOL=-v $(MAIN_GOPATH)/pkg/mod:/go/pkg/mod
else
TEST_MODCACHE_VOL=
endif
ifeq ($(TEST_BUILDCACHE), 1)
TEST_BUILDCACHE_VOL=-v $(shell go env GOCACHE):/root/.caches/go-build
else
TEST_BUILDCACHE_VOL=
endif
DIST_TAG?=1
DIST_BUILD?=1
DIST_SIGN?=1
ifdef DIST_VERSION
DIST_VERSION_ARG=-v "$(DIST_VERSION)"
else
DIST_VERSION_ARG=
endif
ifdef DIST_RELEASE_DATE
DIST_DATE_ARG=-d "$(DIST_RELEASE_DATE)"
else
DIST_DATE_ARG=
endif
ifdef DIST_PRERELEASE
DIST_REL_ARG=-r "$(DIST_PRERELEASE)"
else
DIST_REL_ARG=
endif
PUB_GIT?=1
PUB_WEBSITE?=1
ifeq ($(PUB_GIT),1)
PUB_GIT_ARG=-g
else
PUB_GIT_ARG=
endif
ifeq ($(PUB_WEBSITE),1)
PUB_WEBSITE_ARG=-w
else
PUB_WEBSITE_ARG=
endif
export GO_BUILD_TAG
export UI_BUILD_TAG
export BUILD_CONTAINER_NAME
export GIT_COMMIT
export GIT_COMMIT_YEAR
export GIT_DIRTY
export GOTAGS
export GOLDFLAGS
# Allow skipping docker build during integration tests in CI since we already
# have a built binary
ENVOY_INTEG_DEPS?=docker-envoy-integ
ifdef SKIP_DOCKER_BUILD
ENVOY_INTEG_DEPS=noop
endif
##@ Build
.PHONY: all
all: dev-build ## Command running by default
# used to make integration dependencies conditional
noop: ;
.PHONY: dev
dev: dev-build ## Dev creates binaries for testing locally - these are put into ./bin
.PHONY: dev-build
dev-build: ## Same as dev
mkdir -p bin
CGO_ENABLED=0 go install -ldflags "$(GOLDFLAGS)" -tags "$(GOTAGS)"
# rm needed due to signature caching (https://apple.stackexchange.com/a/428388)
rm -f ./bin/consul
cp ${MAIN_GOPATH}/bin/consul ./bin/consul
.PHONY: dev-docker-dbg
dev-docker-dbg: dev-docker ## Build containers for debug mode
@echo "Pulling consul container image - $(CONSUL_IMAGE_VERSION)"
@docker pull hashicorp/consul:$(CONSUL_IMAGE_VERSION) >/dev/null
@echo "Building Consul Development container - $(CONSUL_DEV_IMAGE)"
@# 'consul-dbg:local' tag is needed to run the integration tests
@# 'consul-dev:latest' is needed by older workflows
@docker buildx use default && docker buildx build -t $(CONSUL_COMPAT_TEST_IMAGE)-dbg:local \
--platform linux/$(GOARCH) \
--build-arg CONSUL_IMAGE_VERSION=$(CONSUL_IMAGE_VERSION) \
--load \
-f $(CURDIR)/build-support/docker/Consul-Dev-Dbg.dockerfile $(CURDIR)/pkg/bin/
.PHONY: dev-docker
dev-docker: linux dev-build ## Build and tag docker images in dev env
@echo "Pulling consul container image - $(CONSUL_IMAGE_VERSION)"
@docker pull hashicorp/consul:$(CONSUL_IMAGE_VERSION) >/dev/null
@echo "Building Consul Development container - $(CONSUL_DEV_IMAGE)"
@# 'consul:local' tag is needed to run the integration tests
@# 'consul-dev:latest' is needed by older workflows
@docker buildx use default && docker buildx build -t 'consul:local' -t '$(CONSUL_DEV_IMAGE)' \
--platform linux/$(GOARCH) \
--build-arg CONSUL_IMAGE_VERSION=$(CONSUL_IMAGE_VERSION) \
--label org.opencontainers.image.version=$(CONSUL_VERSION) \
--label version=$(CONSUL_VERSION) \
--load \
-f $(CURDIR)/build-support/docker/Consul-Dev-Multiarch.dockerfile $(CURDIR)/pkg/bin/
docker tag 'consul:local' '$(CONSUL_COMPAT_TEST_IMAGE):local'
.PHONY: check-remote-dev-image-env
check-remote-dev-image-env: ## Check remote dev image env
ifndef REMOTE_DEV_IMAGE
$(error REMOTE_DEV_IMAGE is undefined: set this image to <your_docker_repo>/<your_docker_image>:<image_tag>, e.g. hashicorp/consul-k8s-dev:latest)
endif
.PHONY: remote-docker
remote-docker: check-remote-dev-image-env ## Remote docker
$(MAKE) GOARCH=amd64 linux
$(MAKE) GOARCH=arm64 linux
@echo "Pulling consul container image - $(CONSUL_IMAGE_VERSION)"
@docker pull hashicorp/consul:$(CONSUL_IMAGE_VERSION) >/dev/null
@echo "Building and Pushing Consul Development container - $(REMOTE_DEV_IMAGE)"
@if ! docker buildx inspect consul-builder; then \
docker buildx create --name consul-builder --driver docker-container --bootstrap; \
fi;
@docker buildx use consul-builder && docker buildx build -t '$(REMOTE_DEV_IMAGE)' \
--platform linux/amd64,linux/arm64 \
--build-arg CONSUL_IMAGE_VERSION=$(CONSUL_IMAGE_VERSION) \
--label org.opencontainers.image.version=$(CONSUL_VERSION) \
--label version=$(CONSUL_VERSION) \
--push \
-f $(CURDIR)/build-support/docker/Consul-Dev-Multiarch.dockerfile $(CURDIR)/pkg/bin/
linux: ## Linux builds a linux binary compatible with the source platform
@mkdir -p ./pkg/bin/linux_$(GOARCH)
CGO_ENABLED=0 GOOS=linux GOARCH=$(GOARCH) go build -o ./pkg/bin/linux_$(GOARCH) -ldflags "$(GOLDFLAGS)" -tags "$(GOTAGS)"
.PHONY: go-mod-tidy
go-mod-tidy: $(foreach mod,$(GO_MODULES),go-mod-tidy/$(mod)) ## Run go mod tidy in every module
.PHONY: mod-tidy/%
go-mod-tidy/%:
@echo "--> Running go mod tidy ($*)"
@cd $* && go mod tidy
.PHONY: go-mod-get
go-mod-get: $(foreach mod,$(GO_MODULES),go-mod-get/$(mod)) ## Run go get and go mod tidy in every module for the given dependency
.PHONY: go-mod-get/%
go-mod-get/%:
ifndef DEP_VERSION
$(error DEP_VERSION is undefined: set this to <dependency>@<version>, e.g. github.com/hashicorp/go-hclog@v1.5.0)
endif
@echo "--> Running go get ${DEP_VERSION} ($*)"
@cd $* && go get $(DEP_VERSION)
@echo "--> Running go mod tidy ($*)"
@cd $* && go mod tidy
##@ Checks
.PHONY: fmt
fmt: $(foreach mod,$(GO_MODULES),fmt/$(mod)) ## Format go modules
.PHONY: fmt/%
fmt/%:
@echo "--> Running go fmt ($*)"
@cd $* && gofmt -s -l -w .
.PHONY: lint
lint: $(foreach mod,$(GO_MODULES),lint/$(mod)) lint-container-test-deps ## Lint go modules and test deps
.PHONY: lint/%
lint/%:
@echo "--> Running golangci-lint ($*)"
@cd $* && GOWORK=off golangci-lint run --build-tags '$(GOTAGS)'
@echo "--> Running lint-consul-retry ($*)"
@cd $* && GOWORK=off lint-consul-retry
@echo "--> Running enumcover ($*)"
@cd $* && GOWORK=off enumcover ./...
.PHONY: lint-consul-retry
lint-consul-retry: $(foreach mod,$(GO_MODULES),lint-consul-retry/$(mod))
.PHONY: lint-consul-retry/%
lint-consul-retry/%: lint-tools
@echo "--> Running lint-consul-retry ($*)"
@cd $* && GOWORK=off lint-consul-retry
# check that the test-container module only imports allowlisted packages
# from the root consul module. Generally we don't want to allow these imports.
# In a few specific instances though it is okay to import test definitions and
# helpers from some of the packages in the root module.
.PHONY: lint-container-test-deps
lint-container-test-deps: ## Check that the test-container module only imports allowlisted packages from the root consul module.
@echo "--> Checking container tests for bad dependencies"
@cd test/integration/consul-container && \
$(CURDIR)/build-support/scripts/check-allowed-imports.sh \
github.com/hashicorp/consul \
"internal/catalog/catalogtest" \
"internal/resource/resourcetest"
##@ Testing
.PHONY: cover
cover: cov ## Run tests and generate coverage report
.PHONY: cov
cov: other-consul dev-build
go test -tags '$(GOTAGS)' ./... -coverprofile=coverage.out
cd sdk && go test -tags '$(GOTAGS)' ./... -coverprofile=../coverage.sdk.part
cd api && go test -tags '$(GOTAGS)' ./... -coverprofile=../coverage.api.part
grep -h -v "mode: set" coverage.{sdk,api}.part >> coverage.out
rm -f coverage.{sdk,api}.part
go tool cover -html=coverage.out
.PHONY: test
test: other-consul dev-build lint test-internal
.PHONY: test-internal
test-internal: ## Test internal
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
7 years ago
@echo "--> Running go test"
@rm -f test.log exit-code
@# Dump verbose output to test.log so we can surface test names on failure but
@# hide it from travis as it exceeds their log limits and causes job to be
@# terminated (over 4MB and over 10k lines in the UI). We need to output
@# _something_ to stop them terminating us due to inactivity...
@echo "===================== submodule: sdk =====================" | tee -a test.log
cd sdk && { go test -v $(GOTEST_FLAGS) -tags '$(GOTAGS)' ./... 2>&1 ; echo $$? >> ../exit-code ; } | tee -a ../test.log | egrep '^(ok|FAIL|panic:|--- FAIL|--- PASS)'
@echo "===================== submodule: api =====================" | tee -a test.log
cd api && { go test -v $(GOTEST_FLAGS) -tags '$(GOTAGS)' ./... 2>&1 ; echo $$? >> ../exit-code ; } | tee -a ../test.log | egrep '^(ok|FAIL|panic:|--- FAIL|--- PASS)'
@echo "===================== submodule: root =====================" | tee -a test.log
{ go test -v $(GOTEST_FLAGS) -tags '$(GOTAGS)' ./... 2>&1 ; echo $$? >> exit-code ; } | tee -a test.log | egrep '^(ok|FAIL|panic:|--- FAIL|--- PASS)'
@# if everything worked fine then all 3 zeroes will be collapsed to a single zero here
@exit_codes="$$(sort -u exit-code)" ; echo "$$exit_codes" > exit-code
@echo "Exit code: $$(cat exit-code)"
@# This prints all the race report between ====== lines
@awk '/^WARNING: DATA RACE/ {do_print=1; print "=================="} do_print==1 {print} /^={10,}/ {do_print=0}' test.log || true
@grep -A10 'panic: ' test.log || true
@# Prints all the failure output until the next non-indented line - testify
@# helpers often output multiple lines for readability but useless if we can't
@# see them. Un-intuitive order of matches is necessary. No || true because
@# awk always returns true even if there is no match and it breaks non-bash
@# shells locally.
@awk '/^[^[:space:]]/ {do_print=0} /--- SKIP/ {do_print=1} do_print==1 {print}' test.log
@awk '/^[^[:space:]]/ {do_print=0} /--- FAIL/ {do_print=1} do_print==1 {print}' test.log
@grep '^FAIL' test.log || true
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
7 years ago
@if [ "$$(cat exit-code)" == "0" ] ; then echo "PASS" ; exit 0 ; else exit 1 ; fi
.PHONY: test-all
test-all: other-consul dev-build lint $(foreach mod,$(GO_MODULES),test-module/$(mod)) ## Test all
.PHONY: test-module/%
test-module/%:
@echo "--> Running go test ($*)"
cd $* && go test $(GOTEST_FLAGS) -tags '$(GOTAGS)' ./...
.PHONY: test-race
test-race: ## Test race
$(MAKE) GOTEST_FLAGS=-race
.PHONY: other-consul
other-consul: ## Checking for other consul instances
New config parser, HCL support, multiple bind addrs (#3480) * new config parser for agent This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed. * Upgrade github.com/hashicorp/hcl to support int64 * improve error messages * fix directory permission test * Fix rtt test * Fix ForceLeave test * Skip performance test for now until we know what to do * Update github.com/hashicorp/memberlist to update log prefix * Make memberlist use the default logger * improve config error handling * do not fail on non-existing data-dir * experiment with non-uniform timeouts to get a handle on stalled leader elections * Run tests for packages separately to eliminate the spurious port conflicts * refactor private address detection and unify approach for ipv4 and ipv6. Fixes #2825 * do not allow unix sockets for DNS * improve bind and advertise addr error handling * go through builder using test coverage * minimal update to the docs * more coverage tests fixed * more tests * fix makefile * cleanup * fix port conflicts with external port server 'porter' * stop test server on error * do not run api test that change global ENV concurrently with the other tests * Run remaining api tests concurrently * no need for retry with the port number service * monkey patch race condition in go-sockaddr until we understand why that fails * monkey patch hcl decoder race condidtion until we understand why that fails * monkey patch spurious errors in strings.EqualFold from here * add test for hcl decoder race condition. Run with go test -parallel 128 * Increase timeout again * cleanup * don't log port allocations by default * use base command arg parsing to format help output properly * handle -dc deprecation case in Build * switch autopilot.max_trailing_logs to int * remove duplicate test case * remove unused methods * remove comments about flag/config value inconsistencies * switch got and want around since the error message was misleading. * Removes a stray debug log. * Removes a stray newline in imports. * Fixes TestACL_Version8. * Runs go fmt. * Adds a default case for unknown address types. * Reoders and reformats some imports. * Adds some comments and fixes typos. * Reorders imports. * add unix socket support for dns later * drop all deprecated flags and arguments * fix wrong field name * remove stray node-id file * drop unnecessary patch section in test * drop duplicate test * add test for LeaveOnTerm and SkipLeaveOnInt in client mode * drop "bla" and add clarifying comment for the test * split up tests to support enterprise/non-enterprise tests * drop raft multiplier and derive values during build phase * sanitize runtime config reflectively and add test * detect invalid config fields * fix tests with invalid config fields * use different values for wan sanitiziation test * drop recursor in favor of recursors * allow dns_config.udp_answer_limit to be zero * make sure tests run on machines with multiple ips * Fix failing tests in a few more places by providing a bind address in the test * Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case for builder. * Add porter to server_test.go to make tests there less flaky * go fmt
7 years ago
@echo "--> Checking for other consul instances"
@if ps -ef | grep 'consul agent' | grep -v grep ; then \
echo "Found other running consul agents. This may affect your tests." ; \
exit 1 ; \
fi
# Use GO_TEST_FLAGS to run specific tests:
# make test-envoy-integ GO_TEST_FLAGS="-run TestEnvoy/case-basic"
# NOTE: Always uses amd64 images, even when running on M1 macs, to match CI/CD environment.
# You can also specify the envoy version (example: 1.27.0) setting the environment variable: ENVOY_VERSION=1.27.0
.PHONY: test-envoy-integ
test-envoy-integ: $(ENVOY_INTEG_DEPS) ## Run envoy integration tests.
@go test -v -timeout=30m -tags integration $(GO_TEST_FLAGS) ./test/integration/connect/envoy
# NOTE: Use DOCKER_BUILDKIT=0, if docker build fails to resolve consul:local base image
.PHONY: test-compat-integ-setup
test-compat-integ-setup: test-deployer-setup
@# 'consul-envoy:target-version' is needed by compatibility integ test
@docker build -t consul-envoy:target-version --build-arg CONSUL_IMAGE=$(CONSUL_COMPAT_TEST_IMAGE):local --build-arg ENVOY_VERSION=${ENVOY_VERSION} -f ./test/integration/consul-container/assets/Dockerfile-consul-envoy ./test/integration/consul-container/assets
@docker build -t consul-dataplane:local --build-arg CONSUL_IMAGE=$(CONSUL_COMPAT_TEST_IMAGE):local --build-arg CONSUL_DATAPLANE_IMAGE=${CONSUL_DATAPLANE_IMAGE} -f ./test/integration/consul-container/assets/Dockerfile-consul-dataplane ./test/integration/consul-container/assets
# NOTE: Use DOCKER_BUILDKIT=0, if docker build fails to resolve consul:local base image
.PHONY: test-deployer-setup
test-deployer-setup: dev-docker
@docker tag consul-dev:latest $(CONSUL_COMPAT_TEST_IMAGE):local
@docker run --rm -t $(CONSUL_COMPAT_TEST_IMAGE):local consul version
.PHONY: test-deployer
test-deployer: test-deployer-setup ## Run deployer-based integration tests (skipping peering_commontopo).
@cd ./test-integ && \
NOLOGBUFFER=1 \
TEST_LOG_LEVEL=debug \
DEPLOYER_CONSUL_DATAPLANE_IMAGE=$(DEPLOYER_CONSUL_DATAPLANE_IMAGE) \
gotestsum \
--raw-command \
--format=standard-verbose \
--debug \
-- \
go test \
-tags "$(GOTAGS)" \
-timeout=20m \
-json \
$(shell sh -c "cd test-integ ; go list -tags \"$(GOTAGS)\" ./... | grep -v peering_commontopo") \
--target-image $(CONSUL_COMPAT_TEST_IMAGE) \
--target-version local \
--latest-image $(CONSUL_COMPAT_TEST_IMAGE) \
--latest-version latest
.PHONY: test-deployer-peering
test-deployer-peering: test-deployer-setup ## Run deployer-based integration tests (just peering_commontopo).
@cd ./test-integ/peering_commontopo && \
NOLOGBUFFER=1 \
TEST_LOG_LEVEL=debug \
DEPLOYER_CONSUL_DATAPLANE_IMAGE=$(DEPLOYER_CONSUL_DATAPLANE_IMAGE) \
gotestsum \
--raw-command \
--format=standard-verbose \
--debug \
-- \
go test \
-tags "$(GOTAGS)" \
-timeout=20m \
-json \
. \
--target-image $(CONSUL_COMPAT_TEST_IMAGE) \
--target-version local \
--latest-image $(CONSUL_COMPAT_TEST_IMAGE) \
--latest-version latest
.PHONY: test-compat-integ
test-compat-integ: test-compat-integ-setup ## Run consul-container based integration tests.
ifeq ("$(GOTESTSUM_PATH)","")
@cd ./test/integration/consul-container && \
go test \
-v \
-timeout=30m \
./... \
--tags $(GOTAGS) \
--target-image $(CONSUL_COMPAT_TEST_IMAGE) \
--target-version local \
--latest-image $(CONSUL_COMPAT_TEST_IMAGE) \
--latest-version latest
else
@cd ./test/integration/consul-container && \
gotestsum \
--format=short-verbose \
--debug \
--rerun-fails=3 \
--packages="./..." \
-- \
--tags $(GOTAGS) \
-timeout=30m \
./... \
--target-image $(CONSUL_COMPAT_TEST_IMAGE) \
--target-version local \
--latest-image $(CONSUL_COMPAT_TEST_IMAGE) \
--latest-version latest
endif
Makefile: add vet target Add a vet target in order to catch suspicious constructs reported by go vet. Vet has successfully detected problems in the past, for example, see c9333b1b9b472feb5cad80e2c8276d41b64bde88 Some vet flags are noisy. In particular, the following flags reports a large amount of generally unharmful constructs: ``` -assign: check for useless assignments -composites: check that composite literals used field-keyed elements -shadow: check for shadowed variables -shadowstrict: whether to be strict about shadowing -unreachable: check for unreachable code ``` In order to skip running the flags mentioned above, vet is invoked on a directory basis with `go tool vet .` since package- level type-checking with `go vet` doesn't accept flags. Hence, each file is vetted in isolation, which is weaker than package-level type-checking. But nevertheless, it might catch suspicious constructs that pose a real issue. The vet target runs the following flags on the entire repo: ``` -asmdecl: check assembly against Go declarations -atomic: check for common mistaken usages of the sync/atomic package -bool: check for mistakes involving boolean operators -buildtags: check that +build tags are valid -copylocks: check that locks are not passed by value -methods: check that canonically named methods are canonically defined -nilfunc: check for comparisons between functions and nil -printf: check printf-like invocations -rangeloops: check that range loop variables are used correctly -shift: check for useless shifts -structtags: check that struct field tags have canonical format and apply to exported fields as needed -unsafeptr: check for misuse of unsafe.Pointer ``` Now and then, it might make sense to check the output of the disabled flags manually. For example, `VETARGS=-unreachable make vet` can detect several lines of dead code that can be deleted, etc.
10 years ago
.PHONY: test-metrics-integ
test-metrics-integ: test-compat-integ-setup ## Test metrics integ
@cd ./test/integration/consul-container && \
go test -v -timeout=7m ./test/metrics \
--target-image $(CONSUL_COMPAT_TEST_IMAGE) \
--target-version local \
--latest-image $(CONSUL_COMPAT_TEST_IMAGE) \
--latest-version latest
.PHONY: test-connect-ca-providers
test-connect-ca-providers: ## Running /agent/connect/ca tests in verbose mode
@echo "Running /agent/connect/ca tests in verbose mode"
@go test -v ./agent/connect/ca
@go test -v ./agent/consul -run Vault
@go test -v ./agent -run Vault
##@ UI
.PHONY: ui
ui: ui-docker ## Build the static web ui inside a Docker container. For local testing only; do not commit these assets.
.PHONY: ui-regen
ui-regen: ## Build the static web ui with yarn. This is the version to commit.
cd $(CURDIR)/ui && make && cd ..
rm -rf $(CURDIR)/agent/uiserver/dist
mv $(CURDIR)/ui/packages/consul-ui/dist $(CURDIR)/agent/uiserver/
.PHONY: ui-build-image
ui-build-image: ## Building UI build container
@echo "Building UI build container"
@docker build $(NOCACHE) $(QUIET) -t $(UI_BUILD_TAG) - < build-support/docker/Build-UI.dockerfile
.PHONY: ui-docker
ui-docker: ui-build-image ## Builds ui within docker container and copy all the relevant artifacts out of the containers back to the source
@$(SHELL) $(CURDIR)/build-support/scripts/build-docker.sh ui
##@ Tools
.PHONY: tools
tools: ## Installs various supporting Go tools.
@$(SHELL) $(CURDIR)/build-support/scripts/devtools.sh
.PHONY: lint-tools
lint-tools: ## Install tools for linting
@$(SHELL) $(CURDIR)/build-support/scripts/devtools.sh -lint
.PHONY: codegen-tools
codegen-tools: ## Install tools for codegen
@$(SHELL) $(CURDIR)/build-support/scripts/devtools.sh -codegen
.PHONY: codegen
codegen: codegen-tools ## Deep copy
@$(SHELL) $(CURDIR)/agent/structs/deep-copy.sh
@$(SHELL) $(CURDIR)/agent/proxycfg/deep-copy.sh
@$(SHELL) $(CURDIR)/agent/consul/state/deep-copy.sh
@$(SHELL) $(CURDIR)/agent/config/deep-copy.sh
copywrite headers
# Special case for MPL headers in /api and /sdk
cd api && $(CURDIR)/build-support/scripts/copywrite-exceptions.sh
cd sdk && $(CURDIR)/build-support/scripts/copywrite-exceptions.sh
print-% : ; @echo $($*) ## utility to echo a makefile variable (i.e. 'make print-GOPATH')
.PHONY: module-versions
module-versions: ## Print a list of modules which can be updated. Columns are: module current_version date_of_current_version latest_version
@go list -m -u -f '{{if .Update}} {{printf "%-50v %-40s" .Path .Version}} {{with .Time}} {{ .Format "2006-01-02" -}} {{else}} {{printf "%9s" ""}} {{end}} {{ .Update.Version}} {{end}}' all
.PHONY: docs
docs: ## Point your web browser to http://localhost:3000/consul to live render docs from ./website/
make -C website
##@ Release
.PHONY: version
version: ## Current Consul version
@echo -n "Version: "
@$(SHELL) $(CURDIR)/build-support/scripts/version.sh
@echo -n "Version + release: "
@$(SHELL) $(CURDIR)/build-support/scripts/version.sh -r
@echo -n "Version + git: "
@$(SHELL) $(CURDIR)/build-support/scripts/version.sh -g
@echo -n "Version + release + git: "
@$(SHELL) $(CURDIR)/build-support/scripts/version.sh -r -g
New ACLs (#4791) This PR is almost a complete rewrite of the ACL system within Consul. It brings the features more in line with other HashiCorp products. Obviously there is quite a bit left to do here but most of it is related docs, testing and finishing the last few commands in the CLI. I will update the PR description and check off the todos as I finish them over the next few days/week. Description At a high level this PR is mainly to split ACL tokens from Policies and to split the concepts of Authorization from Identities. A lot of this PR is mostly just to support CRUD operations on ACLTokens and ACLPolicies. These in and of themselves are not particularly interesting. The bigger conceptual changes are in how tokens get resolved, how backwards compatibility is handled and the separation of policy from identity which could lead the way to allowing for alternative identity providers. On the surface and with a new cluster the ACL system will look very similar to that of Nomads. Both have tokens and policies. Both have local tokens. The ACL management APIs for both are very similar. I even ripped off Nomad's ACL bootstrap resetting procedure. There are a few key differences though. Nomad requires token and policy replication where Consul only requires policy replication with token replication being opt-in. In Consul local tokens only work with token replication being enabled though. All policies in Nomad are globally applicable. In Consul all policies are stored and replicated globally but can be scoped to a subset of the datacenters. This allows for more granular access management. Unlike Nomad, Consul has legacy baggage in the form of the original ACL system. The ramifications of this are: A server running the new system must still support other clients using the legacy system. A client running the new system must be able to use the legacy RPCs when the servers in its datacenter are running the legacy system. The primary ACL DC's servers running in legacy mode needs to be a gate that keeps everything else in the entire multi-DC cluster running in legacy mode. So not only does this PR implement the new ACL system but has a legacy mode built in for when the cluster isn't ready for new ACLs. Also detecting that new ACLs can be used is automatic and requires no configuration on the part of administrators. This process is detailed more in the "Transitioning from Legacy to New ACL Mode" section below.
6 years ago
.PHONY: docker-images
docker-images: go-build-image ui-build-image
.PHONY: go-build-image
go-build-image: ## Building Golang build container
@echo "Building Golang $(GOLANG_VERSION) build container"
@docker build $(NOCACHE) $(QUIET) -t $(GO_BUILD_TAG) --build-arg GOLANG_VERSION=$(GOLANG_VERSION) - < build-support/docker/Build-Go.dockerfile
New ACLs (#4791) This PR is almost a complete rewrite of the ACL system within Consul. It brings the features more in line with other HashiCorp products. Obviously there is quite a bit left to do here but most of it is related docs, testing and finishing the last few commands in the CLI. I will update the PR description and check off the todos as I finish them over the next few days/week. Description At a high level this PR is mainly to split ACL tokens from Policies and to split the concepts of Authorization from Identities. A lot of this PR is mostly just to support CRUD operations on ACLTokens and ACLPolicies. These in and of themselves are not particularly interesting. The bigger conceptual changes are in how tokens get resolved, how backwards compatibility is handled and the separation of policy from identity which could lead the way to allowing for alternative identity providers. On the surface and with a new cluster the ACL system will look very similar to that of Nomads. Both have tokens and policies. Both have local tokens. The ACL management APIs for both are very similar. I even ripped off Nomad's ACL bootstrap resetting procedure. There are a few key differences though. Nomad requires token and policy replication where Consul only requires policy replication with token replication being opt-in. In Consul local tokens only work with token replication being enabled though. All policies in Nomad are globally applicable. In Consul all policies are stored and replicated globally but can be scoped to a subset of the datacenters. This allows for more granular access management. Unlike Nomad, Consul has legacy baggage in the form of the original ACL system. The ramifications of this are: A server running the new system must still support other clients using the legacy system. A client running the new system must be able to use the legacy RPCs when the servers in its datacenter are running the legacy system. The primary ACL DC's servers running in legacy mode needs to be a gate that keeps everything else in the entire multi-DC cluster running in legacy mode. So not only does this PR implement the new ACL system but has a legacy mode built in for when the cluster isn't ready for new ACLs. Also detecting that new ACLs can be used is automatic and requires no configuration on the part of administrators. This process is detailed more in the "Transitioning from Legacy to New ACL Mode" section below.
6 years ago
.PHONY: consul-docker
consul-docker: go-build-image ## Builds consul in a docker container and then dumps executable into ./pkg/bin/...
@$(SHELL) $(CURDIR)/build-support/scripts/build-docker.sh consul
New ACLs (#4791) This PR is almost a complete rewrite of the ACL system within Consul. It brings the features more in line with other HashiCorp products. Obviously there is quite a bit left to do here but most of it is related docs, testing and finishing the last few commands in the CLI. I will update the PR description and check off the todos as I finish them over the next few days/week. Description At a high level this PR is mainly to split ACL tokens from Policies and to split the concepts of Authorization from Identities. A lot of this PR is mostly just to support CRUD operations on ACLTokens and ACLPolicies. These in and of themselves are not particularly interesting. The bigger conceptual changes are in how tokens get resolved, how backwards compatibility is handled and the separation of policy from identity which could lead the way to allowing for alternative identity providers. On the surface and with a new cluster the ACL system will look very similar to that of Nomads. Both have tokens and policies. Both have local tokens. The ACL management APIs for both are very similar. I even ripped off Nomad's ACL bootstrap resetting procedure. There are a few key differences though. Nomad requires token and policy replication where Consul only requires policy replication with token replication being opt-in. In Consul local tokens only work with token replication being enabled though. All policies in Nomad are globally applicable. In Consul all policies are stored and replicated globally but can be scoped to a subset of the datacenters. This allows for more granular access management. Unlike Nomad, Consul has legacy baggage in the form of the original ACL system. The ramifications of this are: A server running the new system must still support other clients using the legacy system. A client running the new system must be able to use the legacy RPCs when the servers in its datacenter are running the legacy system. The primary ACL DC's servers running in legacy mode needs to be a gate that keeps everything else in the entire multi-DC cluster running in legacy mode. So not only does this PR implement the new ACL system but has a legacy mode built in for when the cluster isn't ready for new ACLs. Also detecting that new ACLs can be used is automatic and requires no configuration on the part of administrators. This process is detailed more in the "Transitioning from Legacy to New ACL Mode" section below.
6 years ago
.PHONY: docker-envoy-integ
docker-envoy-integ: ## Build image used to run integration tests locally.
$(MAKE) GOARCH=amd64 linux
docker build \
--platform linux/amd64 $(NOCACHE) $(QUIET) \
-t 'consul:local' \
--build-arg CONSUL_IMAGE_VERSION=$(CONSUL_IMAGE_VERSION) \
$(CURDIR)/pkg/bin/linux_amd64 \
-f $(CURDIR)/build-support/docker/Consul-Dev.dockerfile
##@ Proto
.PHONY: proto
proto: proto-tools proto-gen proto-mocks ## Protobuf setup command
.PHONY: proto-tools
proto-tools: ## Install tools for protobuf
@$(SHELL) $(CURDIR)/build-support/scripts/devtools.sh -protobuf
.PHONY: proto-gen
proto-gen: proto-tools ## Regenerates all Go files from protobuf definitions
@$(SHELL) $(CURDIR)/build-support/scripts/protobuf.sh
.PHONY: proto-mocks
proto-mocks: ## Proto mocks
@rm -rf grpcmocks/*
@mockery --config .grpcmocks.yaml
.PHONY: proto-format
proto-format: proto-tools ## Proto format
@buf format -w
.PHONY: proto-lint
proto-lint: proto-tools ## Proto lint
Protobuf Refactoring for Multi-Module Cleanliness (#16302) Protobuf Refactoring for Multi-Module Cleanliness This commit includes the following: Moves all packages that were within proto/ to proto/private Rewrites imports to account for the packages being moved Adds in buf.work.yaml to enable buf workspaces Names the proto-public buf module so that we can override the Go package imports within proto/buf.yaml Bumps the buf version dependency to 1.14.0 (I was trying out the version to see if it would get around an issue - it didn't but it also doesn't break things and it seemed best to keep up with the toolchain changes) Why: In the future we will need to consume other protobuf dependencies such as the Google HTTP annotations for openapi generation or grpc-gateway usage. There were some recent changes to have our own ratelimiting annotations. The two combined were not working when I was trying to use them together (attempting to rebase another branch) Buf workspaces should be the solution to the problem Buf workspaces means that each module will have generated Go code that embeds proto file names relative to the proto dir and not the top level repo root. This resulted in proto file name conflicts in the Go global protobuf type registry. The solution to that was to add in a private/ directory into the path within the proto/ directory. That then required rewriting all the imports. Is this safe? AFAICT yes The gRPC wire protocol doesn't seem to care about the proto file names (although the Go grpc code does tack on the proto file name as Metadata in the ServiceDesc) Other than imports, there were no changes to any generated code as a result of this.
2 years ago
@buf lint
@for fn in $$(find proto -name '*.proto'); do \
Protobuf Refactoring for Multi-Module Cleanliness (#16302) Protobuf Refactoring for Multi-Module Cleanliness This commit includes the following: Moves all packages that were within proto/ to proto/private Rewrites imports to account for the packages being moved Adds in buf.work.yaml to enable buf workspaces Names the proto-public buf module so that we can override the Go package imports within proto/buf.yaml Bumps the buf version dependency to 1.14.0 (I was trying out the version to see if it would get around an issue - it didn't but it also doesn't break things and it seemed best to keep up with the toolchain changes) Why: In the future we will need to consume other protobuf dependencies such as the Google HTTP annotations for openapi generation or grpc-gateway usage. There were some recent changes to have our own ratelimiting annotations. The two combined were not working when I was trying to use them together (attempting to rebase another branch) Buf workspaces should be the solution to the problem Buf workspaces means that each module will have generated Go code that embeds proto file names relative to the proto dir and not the top level repo root. This resulted in proto file name conflicts in the Go global protobuf type registry. The solution to that was to add in a private/ directory into the path within the proto/ directory. That then required rewriting all the imports. Is this safe? AFAICT yes The gRPC wire protocol doesn't seem to care about the proto file names (although the Go grpc code does tack on the proto file name as Metadata in the ServiceDesc) Other than imports, there were no changes to any generated code as a result of this.
2 years ago
if [[ "$$fn" = "proto/private/pbsubscribe/subscribe.proto" ]]; then \
continue ; \
Protobuf Refactoring for Multi-Module Cleanliness (#16302) Protobuf Refactoring for Multi-Module Cleanliness This commit includes the following: Moves all packages that were within proto/ to proto/private Rewrites imports to account for the packages being moved Adds in buf.work.yaml to enable buf workspaces Names the proto-public buf module so that we can override the Go package imports within proto/buf.yaml Bumps the buf version dependency to 1.14.0 (I was trying out the version to see if it would get around an issue - it didn't but it also doesn't break things and it seemed best to keep up with the toolchain changes) Why: In the future we will need to consume other protobuf dependencies such as the Google HTTP annotations for openapi generation or grpc-gateway usage. There were some recent changes to have our own ratelimiting annotations. The two combined were not working when I was trying to use them together (attempting to rebase another branch) Buf workspaces should be the solution to the problem Buf workspaces means that each module will have generated Go code that embeds proto file names relative to the proto dir and not the top level repo root. This resulted in proto file name conflicts in the Go global protobuf type registry. The solution to that was to add in a private/ directory into the path within the proto/ directory. That then required rewriting all the imports. Is this safe? AFAICT yes The gRPC wire protocol doesn't seem to care about the proto file names (although the Go grpc code does tack on the proto file name as Metadata in the ServiceDesc) Other than imports, there were no changes to any generated code as a result of this.
2 years ago
elif [[ "$$fn" = "proto/private/pbpartition/partition.proto" ]]; then \
continue ; \
fi ; \
pkg=$$(grep "^package " "$$fn" | sed 's/^package \(.*\);/\1/'); \
if [[ "$$pkg" != hashicorp.consul.internal.* ]]; then \
echo "ERROR: $$fn: is missing 'hashicorp.consul.internal' package prefix: $$pkg" >&2; \
exit 1; \
fi \
done
Add support for implementing new requests with protobufs instea… (#6502) * Add build system support for protobuf generation This is done generically so that we don’t have to keep updating the makefile to add another proto generation. Note: anything not in the vendor directory and with a .proto extension will be run through protoc if the corresponding namespace.pb.go file is not up to date. If you want to rebuild just a single proto file you can do so with: make proto-rebuild PROTOFILES=<list of proto files to rebuild> Providing the PROTOFILES var will override the default behavior of finding all the .proto files. * Start adding types to the agent/proto package These will be needed for some other work and are by no means comprehensive. * Add ability to resolve/fixup the agentpb.ACLLinks structure in the state store. * Use protobuf marshalling of raft requests instead of msgpack for protoc generated types. This does not change any encoding of existing types. * Removed structs package automatically encoding with protobuf marshalling Instead the caller of raftApply that wants to opt-in to protobuf encoding will have to call `raftApplyProtobuf` * Run update-vendor to fixup modules.txt Nothing changed as far as dependencies go but the ordering of modules in that file depends on the time they are first seen and its not alphabetical. * Rename some things and implement the structs.RPCInfo interface bits agentpb.QueryOptions and agentpb.WriteRequest implement 3 of the 4 RPCInfo funcs and the new TargetDatacenter message type implements the fourth. * Use the right encoding function. * Renamed agent/proto package to agent/agentpb to prevent package name conflicts * Update modules.txt to fix ordering * Change blockingQuery to take in interfaces for the query options and meta * Add %T to error output. * Add/Update some comments
5 years ago
##@ Envoy
.PHONY: envoy-library
envoy-library: ## Ensures that all of the protobuf packages present in the github.com/envoyproxy/go-control-plane library are referenced in the consul codebase
@$(SHELL) $(CURDIR)/build-support/scripts/envoy-library-references.sh
New ACLs (#4791) This PR is almost a complete rewrite of the ACL system within Consul. It brings the features more in line with other HashiCorp products. Obviously there is quite a bit left to do here but most of it is related docs, testing and finishing the last few commands in the CLI. I will update the PR description and check off the todos as I finish them over the next few days/week. Description At a high level this PR is mainly to split ACL tokens from Policies and to split the concepts of Authorization from Identities. A lot of this PR is mostly just to support CRUD operations on ACLTokens and ACLPolicies. These in and of themselves are not particularly interesting. The bigger conceptual changes are in how tokens get resolved, how backwards compatibility is handled and the separation of policy from identity which could lead the way to allowing for alternative identity providers. On the surface and with a new cluster the ACL system will look very similar to that of Nomads. Both have tokens and policies. Both have local tokens. The ACL management APIs for both are very similar. I even ripped off Nomad's ACL bootstrap resetting procedure. There are a few key differences though. Nomad requires token and policy replication where Consul only requires policy replication with token replication being opt-in. In Consul local tokens only work with token replication being enabled though. All policies in Nomad are globally applicable. In Consul all policies are stored and replicated globally but can be scoped to a subset of the datacenters. This allows for more granular access management. Unlike Nomad, Consul has legacy baggage in the form of the original ACL system. The ramifications of this are: A server running the new system must still support other clients using the legacy system. A client running the new system must be able to use the legacy RPCs when the servers in its datacenter are running the legacy system. The primary ACL DC's servers running in legacy mode needs to be a gate that keeps everything else in the entire multi-DC cluster running in legacy mode. So not only does this PR implement the new ACL system but has a legacy mode built in for when the cluster isn't ready for new ACLs. Also detecting that new ACLs can be used is automatic and requires no configuration on the part of administrators. This process is detailed more in the "Transitioning from Legacy to New ACL Mode" section below.
6 years ago
.PHONY: envoy-regen
envoy-regen: ## Regenerating envoy golden files
$(info regenerating envoy golden files)
@for d in endpoints listeners routes clusters rbac; do \
if [[ -d "agent/xds/testdata/$${d}" ]]; then \
find "agent/xds/testdata/$${d}" -name '*.golden' -delete ; \
fi \
done
@go test -tags '$(GOTAGS)' ./agent/xds -update
@find "command/connect/envoy/testdata" -name '*.golden' -delete
@go test -tags '$(GOTAGS)' ./command/connect/envoy -update
##@ Help
# The help target prints out all targets with their descriptions organized
# beneath their categories. The categories are represented by '##@' and the
# target descriptions by '##'. The awk commands is responsible for reading the
# entire set of makefiles included in this invocation, looking for lines of the
# file as xyz: ## something, and then pretty-format the target and help. Then,
# if there's a line with ##@ something, that gets pretty-printed as a category.
# More info on the usage of ANSI control characters for terminal formatting:
# https://en.wikipedia.org/wiki/ANSI_escape_code#SGR_parameters
# More info on the awk command:
# http://linuxcommand.org/lc3_adv_awk.php
.PHONY: help
help: ## Display this help.
@awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n make \033[36m<target>\033[0m\n"} /^[a-zA-Z_0-9-]+:.*?##/ { printf " \033[36m%-15s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' $(MAKEFILE_LIST)