42 Commits (593051848969fe7a69aa5e0815b0fc0b61a33268)

Author	SHA1	Message	Date
Derek Menteer	1bcaeabfc3	Remove deprecated service-defaults upstream behavior. (#16957) ... Prior to this change, peer services would be targeted by service-default overrides as long as the new `peer` field was not found in the config entry. This commit removes that deprecated backwards-compatibility behavior. Now it is necessary to specify the `peer` field in order for upstream overrides to apply to a peer upstream.	2 years ago
Ronald	94ec4eb2f4	copyright headers for agent folder (#16704) ... * copyright headers for agent folder * Ignore test data files * fix proto files and remove headers in agent/uiserver folder * ignore deep-copy files	2 years ago
Derek Menteer	2f149d60cc	[OSS] Add Peer field to service-defaults upstream overrides (#15956) ... * Add Peer field to service-defaults upstream overrides. * add api changes, compat mode for service default overrides * Fixes based on testing --------- Co-authored-by: DanStough <dan.stough@hashicorp.com>	2 years ago
Derek Menteer	5f5e6864ca	Fix proxy-defaults incorrectly merging config on upstreams. (#16021)	2 years ago
Derek Menteer	2facf50923	Fix configuration merging for implicit tproxy upstreams. (#16000) ... Fix configuration merging for implicit tproxy upstreams. Change the merging logic so that the wildcard upstream has correct proxy-defaults and service-defaults values combined into it. It did not previously merge all fields, and the wildcard upstream did not exist unless service-defaults existed (it ignored proxy-defaults, essentially). Change the way we fetch upstream configuration in the xDS layer so that it falls back to the wildcard when no matching upstream is found. This is what allows implicit peer upstreams to have the correct "merged" config. Change proxycfg to always watch local mesh gateway endpoints whenever a peer upstream is found. This simplifies the logic so that we do not have to inspect the "merged" configuration on peer upstreams to extract the mesh gateway mode.	2 years ago
Semir Patel	bafa5c7156	Pass remote addr of incoming HTTP requests through to RPC(..) calls (#15700)	2 years ago
Kyle Schochenmaier	bf0f61a878	removes ioutil usage everywhere which was deprecated in go1.16 (#15297) ... * update go version to 1.18 for api and sdk, go mod tidy * removes ioutil usage everywhere which was deprecated in go1.16 in favour of io and os packages. Also introduces a lint rule which forbids use of ioutil going forward. Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>	2 years ago
Riddhi Shah	d8d8c8603e	Add support for merge-central-config query param (#13001) ... Adds a new query param merge-central-config for use with the below endpoints: /catalog/service/:service /catalog/connect/:service /health/service/:service /health/connect/:service If set on the request, the response will include a fully resolved service definition which is merged with the proxy-defaults/global and service-defaults/:service config entries (on-demand style). This is useful to view the full service definition for a mesh service (connect-proxy kind or gateway kind) which might not be merged before being written into the catalog (example: in case of services in the agentless model).	3 years ago
R.B. Boyer	b60d89e7ef	bulk rewrite using this script ... set -euo pipefail unset CDPATH cd "$(dirname "$0")" for f in $(git grep '\brequire := require\.New(' | cut -d':' -f1 | sort -u); do echo "=== require: $f ===" sed -i '/require := require.New(t)/d' $f # require.XXX(blah) but not require.XXX(tblah) or require.XXX(rblah) sed -i 's/\brequire\.\([a-zA-Z0-9_]*\)(\([^tr]\)/require.\1(t,\2/g' $f # require.XXX(tblah) but not require.XXX(t, blah) sed -i 's/\brequire\.\([a-zA-Z0-9_]*\)(\(t[^,]\)/require.\1(t,\2/g' $f # require.XXX(rblah) but not require.XXX(r, blah) sed -i 's/\brequire\.\([a-zA-Z0-9_]*\)(\(r[^,]\)/require.\1(t,\2/g' $f gofmt -s -w $f done for f in $(git grep '\bassert := assert\.New(' | cut -d':' -f1 | sort -u); do echo "=== assert: $f ===" sed -i '/assert := assert.New(t)/d' $f # assert.XXX(blah) but not assert.XXX(tblah) or assert.XXX(rblah) sed -i 's/\bassert\.\([a-zA-Z0-9_]*\)(\([^tr]\)/assert.\1(t,\2/g' $f # assert.XXX(tblah) but not assert.XXX(t, blah) sed -i 's/\bassert\.\([a-zA-Z0-9_]*\)(\(t[^,]\)/assert.\1(t,\2/g' $f # assert.XXX(rblah) but not assert.XXX(r, blah) sed -i 's/\bassert\.\([a-zA-Z0-9_]*\)(\(r[^,]\)/assert.\1(t,\2/g' $f gofmt -s -w $f done	3 years ago
Mark Anderson	7e8228a20b	Remove some usage of md5 from the system (#11491) ... * Remove some usage of md5 from the system OSS side of https://github.com/hashicorp/consul-enterprise/pull/1253 This is a potential security issue because an attacker could conceivably manipulate inputs to cause persistence files to collide, effectively deleting the persistence file for one of the colliding elements. Signed-off-by: Mark Anderson <manderson@hashicorp.com>	3 years ago
R.B. Boyer	ef559dfdd4	agent: refactor the agent delegate interface to be partition friendly (#11429)	3 years ago
Dhia Ayachi	1950ebbe1f	oss portion of ent #1069 (#10883)	3 years ago
R.B. Boyer	188e8dc51f	agent/structs: add a bunch more EnterpriseMeta helper functions to help with partitioning (#10669)	3 years ago
Freddy	cf4821885d	Avoid panic on concurrent writes to cached service config map (#10647) ... If multiple instances of a service are co-located on the same node then their proxies will all share a cache entry for their resolved service configuration. This is because the cache key contains the name of the watched service but does not take into account the ID of the watching proxies. This means that there will be multiple agent service manager watches that can wake up on the same cache update. These watchers then concurrently modify the value in the cache when merging the resolved config into the local proxy definitions. To avoid this concurrent map write we will only delete the key from opaque config in the local proxy definition after the merge, rather than from the cached value before the merge.	3 years ago
Daniel Nephin	a0ca381037	agent: remove deprecated call in a test	3 years ago
Daniel Nephin	678014de1d	agent: fix a data race in a test ... The test was modifying a pointer to a struct that had been passed to another goroutine. Instead create a new struct to modify. ``` WARNING: DATA RACE Write at 0x00c01407c3c0 by goroutine 832: github.com/hashicorp/consul/agent.TestServiceManager_PersistService_API() /home/daniel/pers/code/consul/agent/service_manager_test.go:446 +0x1d86 testing.tRunner() /usr/lib/go/src/testing/testing.go:1193 +0x202 Previous read at 0x00c01407c3c0 by goroutine 938: reflect.typedmemmove() /usr/lib/go/src/runtime/mbarrier.go:177 +0x0 reflect.Value.Set() /usr/lib/go/src/reflect/value.go:1569 +0x13b github.com/mitchellh/copystructure.(*walker).Primitive() /home/daniel/go/pkg/mod/github.com/mitchellh/copystructure@v1.0.0/copystructure.go:289 +0x190 github.com/mitchellh/reflectwalk.walkPrimitive() /home/daniel/go/pkg/mod/github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:252 +0x31b github.com/mitchellh/reflectwalk.walk() /home/daniel/go/pkg/mod/github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:179 +0x24d github.com/mitchellh/reflectwalk.walkStruct() /home/daniel/go/pkg/mod/github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:386 +0x4ec github.com/mitchellh/reflectwalk.walk() /home/daniel/go/pkg/mod/github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:188 +0x656 github.com/mitchellh/reflectwalk.walkStruct() /home/daniel/go/pkg/mod/github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:386 +0x4ec github.com/mitchellh/reflectwalk.walk() /home/daniel/go/pkg/mod/github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:188 +0x656 github.com/mitchellh/reflectwalk.Walk() /home/daniel/go/pkg/mod/github.com/mitchellh/reflectwalk@v1.0.1/reflectwalk.go:92 +0x164 github.com/mitchellh/copystructure.Config.Copy() /home/daniel/go/pkg/mod/github.com/mitchellh/copystructure@v1.0.0/copystructure.go:69 +0xe7 github.com/mitchellh/copystructure.Copy() /home/daniel/go/pkg/mod/github.com/mitchellh/copystructure@v1.0.0/copystructure.go:13 +0x84 github.com/hashicorp/consul/agent.mergeServiceConfig() /home/daniel/pers/code/consul/agent/service_manager.go:362 +0x56 github.com/hashicorp/consul/agent.(*serviceConfigWatch).handleUpdate() /home/daniel/pers/code/consul/agent/service_manager.go:279 +0x250 github.com/hashicorp/consul/agent.(*serviceConfigWatch).runWatch() /home/daniel/pers/code/consul/agent/service_manager.go:246 +0x2d4 Goroutine 832 (running) created at: testing.(*T).Run() /usr/lib/go/src/testing/testing.go:1238 +0x5d7 testing.runTests.func1() /usr/lib/go/src/testing/testing.go:1511 +0xa6 testing.tRunner() /usr/lib/go/src/testing/testing.go:1193 +0x202 testing.runTests() /usr/lib/go/src/testing/testing.go:1509 +0x612 testing.(*M).Run() /usr/lib/go/src/testing/testing.go:1417 +0x3b3 main.main() _testmain.go:1181 +0x236 Goroutine 938 (running) created at: github.com/hashicorp/consul/agent.(*serviceConfigWatch).start() /home/daniel/pers/code/consul/agent/service_manager.go:223 +0x4e4 github.com/hashicorp/consul/agent.(*ServiceManager).AddService() /home/daniel/pers/code/consul/agent/service_manager.go:98 +0x344 github.com/hashicorp/consul/agent.(*Agent).addServiceLocked() /home/daniel/pers/code/consul/agent/agent.go:1942 +0x2e4 github.com/hashicorp/consul/agent.(*Agent).AddService() /home/daniel/pers/code/consul/agent/agent.go:1929 +0x337 github.com/hashicorp/consul/agent.TestServiceManager_PersistService_API() /home/daniel/pers/code/consul/agent/service_manager_test.go:400 +0x17c4 testing.tRunner() /usr/lib/go/src/testing/testing.go:1193 +0x202 ```	3 years ago
Jared Kirschner	bd536151e1	Replace use of 'sane' where appropriate ... HashiCorp voice, style, and language guidelines recommend avoiding ableist language unless its reference to ability is accurate in a particular use.	3 years ago
Freddy	429f9d8bb8	Add flag for transparent proxies to dial individual instances (#10329)	4 years ago
freddygv	7bd51ff536	Replace TransparentProxy bool with ProxyMode ... This PR replaces the original boolean used to configure transparent proxy mode. It was replaced with a string mode that can be set to: - "": Empty string is the default for when the setting should be defaulted from other configuration like config entries. - "direct": Direct mode is how applications originally opted into the mesh. Proxy listeners need to be dialed directly. - "transparent": Transparent mode enables configuring Envoy as a transparent proxy. Traffic must be captured and redirected to the inbound and outbound listeners. This PR also adds a struct for transparent proxy specific configuration. Initially this is not stored as a pointer. Will revisit that decision before GA.	4 years ago
freddygv	098b9af901	Fixup enterprise tests from tproxy changes	4 years ago
freddygv	df1f3995f8	Update service manager to store centrally configured upstreams	4 years ago
freddygv	acec711a6a	Update server-side config resolution and client-side merging	4 years ago
freddygv	87cde19b4c	Create new types for service-defaults upstream cfg	4 years ago
Daniel Nephin	45b315060a	agent: Minor cosmetic changes in ServiceManager ... Also use the non-deprecated func in a test	4 years ago
Daniel Nephin	3d39359bcb	agent: move deprecated AddServiceFromSource to a test file ... The method is only used in tests, and only exists for legacy calls. There was one other package which used this method in tests. Export the AddServiceRequest and a couple of its fields so the new function can be used in those tests.	4 years ago
Daniel Nephin	6757231b82	agent: rename AddService->AddServiceFromSource ... In preparation for extracting a single AddService func that accepts a request struct.	4 years ago
Daniel Nephin	b9e60c0775	testing: skip slow tests with -short ... Add a skip condition to all tests slower than 100ms. This change was made using `gotestsum tool slowest` with data from the last 3 CI runs of master. See https://github.com/gotestyourself/gotestsum#finding-and-skipping-slow-tests With this change: ``` $ time go test -count=1 -short ./agent ok github.com/hashicorp/consul/agent 0.743s real 0m4.791s $ time go test -count=1 -short ./agent/consul ok github.com/hashicorp/consul/agent/consul 4.229s real 0m8.769s ```	4 years ago
R.B. Boyer	9801ef8eb1	agent: enable enable_central_service_config by default (#8746)	4 years ago
Daniel Nephin	3a4e62836b	testing: Remove TestAgent.Key and change TestAgent.DataDir ... TestAgent.Key was only used by 3 tests. Extracting it from the common helper that is used in hundreds of tests helps keep the shared part small and more focused. This required a second change (which I was planning on making anyway), which was to change the behaviour of DataDir. Now in all cases the TestAgent will use the DataDir, and clean it up once the test is complete.	4 years ago
Daniel Nephin	e759daafdd	Rename NewTestAgentWithFields to StartTestAgent ... This function now only starts the agent. Using: git grep -l 'StartTestAgent(t, true,' | \ xargs sed -i -e 's/StartTestAgent(t, true,/StartTestAgent(t,/g'	5 years ago
Daniel Nephin	475659a132	Remove name from NewTestAgent ... Using: git grep -l 'NewTestAgent(t, t.Name(),' | \ xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'	5 years ago
Freddy	18d356899c	Enable CLI to register terminating gateways (#7500) ... * Enable CLI to register terminating gateways * Centralize gateway proxy configuration	5 years ago
Matt Keeler	228da48f5d	Minor Non-Functional Updates (#7215) ... * Cleanup the discovery chain compilation route handling Nothing functionally should be different here. The real difference is that when creating new targets or handling route destinations we use the router config entries name and namespace instead of that of the top level request. Today they SHOULD always be the same but that may not always be the case. This hopefully also makes it easier to understand how the router entries are handled. * Refactor a small bit of the service manager tests in oss We used to use the stringHash function to compute part of the filename where things would get persisted to. This has been changed in the core code to calling the StringHash method on the ServiceID type. It just so happens that the new method will output the same value for anything in the default namespace (by design actually). However, logically this filename computation in the test should do the same thing as the core code itself so I updated it here. Also of note is that newer enterprise-only tests for the service manager cannot use the old stringHash function at all because it will produce incorrect results for non-default namespaces.	5 years ago
Matt Keeler	c09693e545	Updates to Config Entries and Connect for Namespaces (#7116)	5 years ago
Aestek	ba8fd8296f	Add support for dual stack IPv4/IPv6 network (#6640) ... * Use consts for well known tagged adress keys * Add ipv4 and ipv6 tagged addresses for node lan and wan * Add ipv4 and ipv6 tagged addresses for service lan and wan * Use IPv4 and IPv6 address in DNS	5 years ago
Matt Keeler	5934f803bf	Sync of OSS changes to support namespaces (#6909)	5 years ago
R.B. Boyer	5882e21b2b	agent: tolerate more failure scenarios during service registration with central config enabled (#6472) ... Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.	5 years ago
R.B. Boyer	913d85ea5b	connect: allow mesh gateways to use central config (#6302)	5 years ago
Paul Banks	0cfb6051ea	Add integration test for central config; fix central config WIP (#5752) ... * Add integration test for central config; fix central config WIP * Add integration test for central config; fix central config WIP * Set proxy protocol correctly and begin adding upstream support * Add upstreams to service config cache key and start new notify watcher if they change. This doesn't update the tests to pass though. * Fix some merging logic get things working manually with a hack (TODO fix properly) * Simplification to not allow enabling sidecars centrally - it makes no sense without upstreams anyway * Test compile again and obvious ones pass. Lots of failures locally not debugged yet but may be flakes. Pushing up to see what CI does * Fix up service manageer and API test failures * Remove the enable command since it no longer makes much sense without being able to turn on sidecar proxies centrally * Remove version.go hack - will make integration test fail until release * Remove unused code from commands and upstream merge * Re-bump version to 1.5.0	6 years ago
Matt Keeler	3145bf5230	Centralized Config CLI (#5731) ... * Add HTTP endpoints for config entry management * Finish implementing decoding in the HTTP Config entry apply endpoint * Add CAS operation to the config entry apply endpoint Also use this for the bootstrapping and move the config entry decoding function into the structs package. * First pass at the API client for the config entries * Fixup some of the ConfigEntry APIs Return a singular response object instead of a list for the ConfigEntry.Get RPC. This gets plumbed through the HTTP API as well. Dont return QueryMeta in the JSON response for the config entry listing HTTP API. Instead just return a list of config entries. * Minor API client fixes * Attempt at some ConfigEntry api client tests These don’t currently work due to weak typing in JSON * Get some of the api client tests passing * Implement reflectwalk magic to correct JSON encoding a ProxyConfigEntry Also added a test for the HTTP endpoint that exposes the problem. However, since the test doesn’t actually do the JSON encode/decode its still failing. * Move MapWalk magic into a binary marshaller instead of JSON. * Add a MapWalk test * Get rid of unused func * Get rid of unused imports * Fixup some tests now that the decoding from msgpack coerces things into json compat types * Stub out most of the central config cli Fully implement the config read command. * Basic config delete command implementation * Implement config write command * Implement config list subcommand Not entirely sure about the output here. Its basically the read output indented with a line specifying the kind/name of each type which is also duplicated in the indented output. * Update command usage * Update some help usage formatting * Add the connect enable helper cli command * Update list command output * Rename the config entry API client methods. * Use renamed apis * Implement config write tests Stub the others with the noTabs tests. * Change list output format Now just simply output 1 line per named config * Add config read tests * Add invalid args write test. * Add config delete tests * Add config list tests * Add connect enable tests * Update some CLI commands to use CAS ops This also modifies the HTTP API for a write op to return a boolean indicating whether the value was written or not. * Fix up the HTTP API CAS tests as I realized they weren’t testing what they should. * Update config entry rpc tests to properly test CAS * Fix up a few more tests * Fix some tests that using ConfigEntries.Apply * Update config_write_test.go * Get rid of unused import	6 years ago
Kyle Havlovitz	c269369760	Make central service config opt-in and rework the initial registration	6 years ago
Kyle Havlovitz	88e1d8ce03	Fill out the service manager functionality and fix tests	6 years ago