consul

Commit Graph

Author	SHA1	Message	Date
Hans Hasselberg	b6499fe6b8	Do not surface left servers (#6420 ) * do not surface left servers in catalog	5 years ago
R.B. Boyer	6439af86eb	agent: clients should only attempt to remove pruned nodes once per call (#6591 )	5 years ago
Sarah Christoff	5e26971864	Prune Unhealthy Agents (#6571 ) * Add -prune flag to ForceLeave	5 years ago
R.B. Boyer	06ca8cd2d7	agent: updates to the agent token trigger anti-entropy full syncs (#6577 )	5 years ago
Matt Keeler	d65bbbfd4e	Implement Leader Routine Management (#6580 ) * Implement leader routine manager Switch over the following to use it for go routine management: • Config entry Replication • ACL replication - tokens, policies, roles and legacy tokens • ACL legacy token upgrade • ACL token reaping • Intention Replication • Secondary CA Roots Watching • CA Root Pruning Also added the StopAll call into the Server Shutdown method to ensure all leader routines get killed off when shutting down. This should be mostly unnecessary as `revokeLeadership` should manually stop each one but just in case we really want these to go away (eventually).	5 years ago
Matt Keeler	28221f66f2	Use encoding/json instead of jsonpb even for protobuf types (#6572 ) This only works so long as we use simplistic protobuf types. Constructs such as oneof or Any types that require type annotations for decoding properly will fail hard but that is by design. If/when we want to use any of that we will probably need to consider a v2 API.	5 years ago
Matt Keeler	fc4bcfd81f	Add EnterpriseConfig stubs (#6566 )	5 years ago
Matt Keeler	abed91d069	Generate JSON and Binary Marshalers for Protobuf Types (#6564 ) * Add JSON and Binary Marshaler Generators for Protobuf Types * Generate files with the correct version of gogo/protobuf I have pinned the version in the makefile so when you run make tools you get the right version. This pulls the version out of go.mod so it should remain up to date. The version at the time of this commit we are using is v1.2.1 * Fixup some shell output * Update how we determine the version of gogo This just greps the go.mod file instead of expecting the go mod cache to already be present * Fixup vendoring and remove no longer needed json encoder functions	5 years ago
John Cowen	b3b32dc0f6	ui: UI Release Merge (ui-staging merge) (#6527 ) ## HTTPAdapter (#5637) ## Ember upgrade 2.18 > 3.12 (#6448) ### Proxies can no longer get away with not calling _super This means that we can't use create anymore to define dynamic methods. Therefore we dynamically make 2 extended Proxies on demand, and then create from those. Therefore we can call _super in the init method of the extended Proxies. ### We aren't allowed to reset a service anymore We never actually need to now anyway, this is a remnant of the refactor from browser based confirmations. We fix it as simply as possible here but will revisit and remove the old browser confirm functionality at a later date ### Revert classes to use ES5 style to workaround babel transp. probs Using a mixture of ES6 classes (and hence super) and arrow functions means that when babel transpiles the arrow functions down to ES5, a reference to this is moved before the call to super, hence causing a js error. Furthermore, we the testing environment no longer lets use use apply/call on the constructor. These errors only manifests during testing (only in the testing environment), the application itself runs fine with no problems without this change. Using ES5 style class definitions give us freedom to do all of the above without causing any errors, so we reverted these classes back to ES5 class definitions ### Skip test that seems to have changed due to a change in RSVP timing This test tests a usecase/area of the API that will probably never ever be used, it was more testing out the API. We've skipped the test for now as this doesn't affect the application itself, but left a note to come back here later to investigate further ### Remove enumerableContentDidChange Initial testing looks like we don't need to call this function anymore, the function no longer exists ### Rework Changeset.isSaving to take into account new ember APIs Setting/hanging a computedProperty of an instantiated object no longer works. Move to setting it on the prototype/class definition instead ### Change how we detect whether something requires listening New ember API's have changed how you can detect whether something is a computedProperty or not. It's not immediately clear if its even possible now. Therefore we change how we detect whether something should be listened to or not by just looking for presence of `addEventListener` ### Potentially temporary change of ci test scripts to ensure deps exist All our tooling scripts run through a Makefile (for people familiar with only using those), which then call yarn scripts which can be called independently (for people familar with only using yarn). The Makefile targets always check to make sure all the dependencies are installed before running anything that requires them (building, testing etc). The CI scripts/targets didn't follow this same route and called the yarn scripts directly (usually CI builds a cache of the dependencies first). For some reason this cache isn't doing what it usually does, and it looks as though, in CI, ember isn't installed. This commit makes the CI scripts consistently use the same method as all of the other tooling scripts (Makefile target > Install Deps if required > call yarn script). This should install the dependencies if for some reason the CI cache building doesn't complete/isn't successful. Potentially this commit may be reverted if, the root of the problem is elsewhere, although consistency is always good, so it might be a good idea to leave this commit as is even if we need to debug and fix things elsewhere. ### Make test-parallel consistent with the rest of the tooling scripts As we are here making changes for CI purposes (making test-ci consistent), we spotted that test-parallel is also inconsistent and also the README manual instructions won't work without `ember` installed globally. This commit makes everything consistent and changes the manual instructions to use the local ember instance that gets installed via yarn ### Re-wrangle catchable to fit with new ember 3.12 APIs In the upgrade from ember 3.8 > 3.12 the public interfaces for ComputedProperties have changed slightly. `meta` is no longer a public property of ComputedProperty but of a ComputedDecoratorImpl mixin instead. `7e4ba1096e/packages/%40ember/-internals/metal/lib/computed.ts (L725)` There seems to be no way, by just using publically available methods, to replicate this behaviour so that we can create our own 'ComputedProperty` factory via injecting the ComputedProperty class as we did previously. `3f333bada1/ui-v2/app/utils/computed/factory.js (L1-L18)` Instead we dynamically hang our `Catchable` `catch` method off the instantiated ComputedProperty. In doing it like this `ComputedProperty` has already has its `meta` method mixed in so we don't have to manually mix it in ourselves (which doesn't seem possible) This functionality is only used during our work in trying to ensure our EventSource/BlockingQuery work was as 'ember-like' as possible (i.e. using the traditional Route.model hooks and ember-like Controller properties). Our ongoing/upcoming work on a componentized approach to data a.k.a `<DataSource />` means we will be able to remove the majority of the code involved here now that it seems to be under an amount of flux in ember. ### Build bindata_assetfs.go with new UI changes	5 years ago
Matt Keeler	923d8671a4	Add support for parameterizing the ACL config used with a TestA… (#6559 ) * Add support for parameterizing the ACL config used with a TestAgent Using tokens that are UUIDs will get rid of some warnings * Refactor to allow setting all tokens and change the template to ignore unset values.	5 years ago
R.B. Boyer	c4b92d5534	connect: connect CA Roots in secondary datacenters should use a SigningKeyID derived from their local intermediate (#6513 ) This fixes an issue where leaf certificates issued in secondary datacenters would be reissued very frequently (every ~20 seconds) because the logic meant to detect root rotation was errantly triggering because a hash of the ultimate root (in the primary) was being compared against a hash of the local intermediate root (in the secondary) and always failing.	5 years ago
R.B. Boyer	9566df524e	agent: cache notifications work after error if the underlying RPC returns index=1 (#6547 ) Fixes #6521 Ensure that initial failures to fetch an agent cache entry using the notify API where the underlying RPC returns a synthetic index of 1 correctly recovers when those RPCs resume working. The bug in the Cache.notifyBlockingQuery used to incorrectly "fix" the index for the next query from 0 to 1 for all queries, when it should have not done so for queries that errored. Also fixed some things that made debugging difficult: - config entry read/list endpoints send back QueryMeta headers - xds event loops don't swallow the cache notification errors	5 years ago
Matt Keeler	76cf54068b	Expand the QueryOptions and QueryMeta interfaces (#6545 ) In a previous PR I made it so that we had interfaces that would work enough to allow blockingQueries to work. However to complete this we need all fields to be settable and gettable. Notes: • If Go ever gets contracts/generics then we could get rid of all the Getters/Setters • protoc / protoc-gen-gogo are going to generate all the getters for us. • I copied all the getters/setters from the protobuf funcs into agent/structs/protobuf_compat.go • Also added JSON marshaling funcs that use jsonpb for protobuf types.	5 years ago
Freddy	fdd10dd8b8	Expose HTTP-based paths through Connect proxy (#6446 ) Fixes: #5396 This PR adds a proxy configuration stanza called expose. These flags register listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only listening on the loopback interface, while still accepting traffic from non Connect-enabled services. Under expose there is a boolean checks flag that would automatically expose all registered HTTP and gRPC check paths. This stanza also accepts a paths list to expose individual paths. The primary use case for this functionality would be to expose paths for third parties like Prometheus or the kubelet. Listeners for requests to exposed paths are be configured dynamically at run time. Any time a proxy, or check can be registered, a listener can also be created. In this initial implementation requests to these paths are not authenticated/encrypted.	5 years ago
R.B. Boyer	5882e21b2b	agent: tolerate more failure scenarios during service registration with central config enabled (#6472 ) Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.	5 years ago
Matt Keeler	100ebd63f9	Allow for enterprise only leader routines (#6533 ) Eventually I am thinking we may need a way to register these at different priority levels but for now sticking this here is fine	5 years ago
R.B. Boyer	af01d397a5	connect: don't colon-hex-encode the AuthorityKeyId and SubjectKeyId fields in connect certs (#6492 ) The fields in the certs are meant to hold the original binary representation of this data, not some ascii-encoded version. The only time we should be colon-hex-encoding fields is for display purposes or marshaling through non-TLS mediums (like RPC).	5 years ago
R.B. Boyer	796de297c8	connect: intermediate CA certs generated with the vault provider lack URI SANs (#6491 ) This only affects vault versions >=1.1.1 because the prior code accidentally relied upon a bug that was fixed in https://github.com/hashicorp/vault/pull/6505 The existing tests should have caught this, but they were using a vendored copy of vault version 0.10.3. This fixes the tests by running an actual copy of vault instead of an in-process copy. This has the added benefit of changing the dependency on vault to just vault/api. Also update VaultProvider to use similar SetIntermediate validation code as the ConsulProvider implementation.	5 years ago
Matt Keeler	51dcd126b7	Add support for implementing new requests with protobufs instea… (#6502 ) * Add build system support for protobuf generation This is done generically so that we don’t have to keep updating the makefile to add another proto generation. Note: anything not in the vendor directory and with a .proto extension will be run through protoc if the corresponding namespace.pb.go file is not up to date. If you want to rebuild just a single proto file you can do so with: make proto-rebuild PROTOFILES=<list of proto files to rebuild> Providing the PROTOFILES var will override the default behavior of finding all the .proto files. * Start adding types to the agent/proto package These will be needed for some other work and are by no means comprehensive. * Add ability to resolve/fixup the agentpb.ACLLinks structure in the state store. * Use protobuf marshalling of raft requests instead of msgpack for protoc generated types. This does not change any encoding of existing types. * Removed structs package automatically encoding with protobuf marshalling Instead the caller of raftApply that wants to opt-in to protobuf encoding will have to call `raftApplyProtobuf` * Run update-vendor to fixup modules.txt Nothing changed as far as dependencies go but the ordering of modules in that file depends on the time they are first seen and its not alphabetical. * Rename some things and implement the structs.RPCInfo interface bits agentpb.QueryOptions and agentpb.WriteRequest implement 3 of the 4 RPCInfo funcs and the new TargetDatacenter message type implements the fourth. * Use the right encoding function. * Renamed agent/proto package to agent/agentpb to prevent package name conflicts * Update modules.txt to fix ordering * Change blockingQuery to take in interfaces for the query options and meta * Add %T to error output. * Add/Update some comments	5 years ago
R.B. Boyer	f9496dc627	sdk: add freelist tracking and ephemeral port range skipping to freeport This should cut down on test flakiness. Problems handled: - If you had enough parallel test cases running, the former circular approach to handling the port block could hand out the same port to multiple cases before they each had a chance to bind them, leading to one of the two tests to fail. - The freeport library would allocate out of the ephemeral port range. This has been corrected for Linux (which should cover CI). - The library now waits until a formerly-in-use port is verified to be free before putting it back into circulation.	5 years ago
R.B. Boyer	7ccaa13514	fix typo of 'unknown' in log messages	5 years ago
R.B. Boyer	20eb0d3e94	cache: remove data race in agent cache In normal operations there is a read/write race related to request QueryOptions fields. An example race: WARNING: DATA RACE Read at 0x00c000836950 by goroutine 30: github.com/hashicorp/consul/agent/structs.(ServiceConfigRequest).CacheInfo() /go/src/github.com/hashicorp/consul/agent/structs/config_entry.go:506 +0x109 github.com/hashicorp/consul/agent/cache.(Cache).getWithIndex() /go/src/github.com/hashicorp/consul/agent/cache/cache.go:262 +0x5c github.com/hashicorp/consul/agent/cache.(Cache).notifyBlockingQuery() /go/src/github.com/hashicorp/consul/agent/cache/watch.go:89 +0xd7 Previous write at 0x00c000836950 by goroutine 147: github.com/hashicorp/consul/agent/cache-types.(ResolvedServiceConfig).Fetch() /go/src/github.com/hashicorp/consul/agent/cache-types/resolved_service_config.go:31 +0x219 github.com/hashicorp/consul/agent/cache.(*Cache).fetch.func1() /go/src/github.com/hashicorp/consul/agent/cache/cache.go:495 +0x112 This patch does a lightweight copy of the request struct so that the embedded QueryOptions fields that are mutated during Fetch() are scoped to just that one RPC.	5 years ago
hashicorp-ci	4f499a8044	update bindata_assetfs.go	5 years ago
Hans Hasselberg	4a20efda9b	agent: handleEnterpriseLeave (#6453 )	5 years ago
R.B. Boyer	a86e63f81e	test: actually wait for the TestAgent to be fully shutdown (#6441 )	5 years ago
Sarah Adams	001137e5e5	test: ensure all TestAgent constructions use a constructor (#6443 ) ensure all TestAgent constructions use a constructor to get start retries + test logs going to the right place Fixes #6435	5 years ago
Sarah Adams	74461406e0	remove funky panic/recover in agent tests (#6442 )	5 years ago
Sarah Adams	4ed5515fca	refactor & add better retry logic to NewTestAgent (#6363 ) Fixes #6361	5 years ago
Pierre Souchay	be50400c62	Distinguish between DC not existing and not being available (#6399 )	5 years ago
Aestek	7c7b7f24fd	Add option to register services and their checks idempotently (#4905 )	5 years ago
Sarah Adams	9f4b329b6d	txn: don't try to decode request bodies > raft.SuggestedMaxDataSize (#6422 ) txn: don't try to decode request bodies > raft.SuggestedMaxDataSize	5 years ago
Matt Keeler	42d608587f	Store primaries root in secondary after intermediate signature (#6333 ) * Store primaries root in secondary after intermediate signature This ensures that the intermediate exists within the CA root stored in raft and not just in the CA provider state. This has the very nice benefit of actually outputting the intermediate cert within the ca roots HTTP/RPC endpoints. This change means that if signing the intermediate fails it will not set the root within raft. So far I have not come up with a reason why that is bad. The secondary CA roots watch will pull the root again and go through all the motions. So as soon as getting an intermediate CA works the root will get set. * Make TestAgentAntiEntropy_Check_DeferSync less flaky I am not sure this is the full fix but it seems to help for me.	5 years ago
R.B. Boyer	7deaba63e1	test: ensure the node name is a valid dns name (#6424 ) The space in the node name was making every test emit a useless warning.	5 years ago
R.B. Boyer	3be88df207	test: explicitly run the pprof tests for 1s instead of the 30s default (#6421 )	5 years ago
R.B. Boyer	d979d8c239	test: add additional http status code assertions in coordinate HTTP API tests (#6410 ) When this test flakes sometimes this happens: --- FAIL: TestCoordinate_Node (1.69s) panic: interface conversion: interface {} is nil, not structs.Coordinates [recovered] FAIL github.com/hashicorp/consul/agent 19.999s Exit code: 1 panic: interface conversion: interface {} is nil, not structs.Coordinates [recovered] panic: interface conversion: interface {} is nil, not structs.Coordinates There is definitely a bug lurking, but the code seems to imply this can only return nil on 404. The tests previously were not checking the status code. The underlying cause of the flake is unknown, but this should turn the failure into a more normal test failure.	5 years ago
Pierre Souchay	58f04815d5	Display IPs of machines when node names conflict to ease troubleshooting When there is an node name conflicts, such messages are displayed within Consul: `consul.fsm: EnsureRegistration failed: failed inserting node: Error while renaming Node ID: "e1d456bc-f72d-98e5-ebb3-26ae80d785cf": Node name node001 is reserved by node 05f10209-1b9c-b90c-e3e2-059e64556d4a with name node001` While it is easy to find the node that has reserved the name, it is hard to find the node trying to aquire the name since it is not registered, because it is not part of `consul members` output This PR will display the IP of the offender and solve far more easily those issues.	5 years ago
Alvin Huang	c516fabfac	revert commits on master (#6413 )	5 years ago
tradel	5a22b77340	update tests to match new method signatures	5 years ago
tradel	1ff46f3f0a	confi\gure providers with DC and domain	5 years ago
tradel	5ba28a6a7b	create a common name for autoTLS agent certs	5 years ago
tradel	9b1ac4e7ef	add subject names to issued certs	5 years ago
tradel	7f36a5b676	construct a common name for each CSR	5 years ago
tradel	672e181399	add serviceID to leaf cert request	5 years ago
tradel	a4312d2e6e	add domain and nodeName to agent cert request	5 years ago
tradel	82ae7caf3e	Added DC and domain args to Configure method	5 years ago
R.B. Boyer	b962fe38cd	test: send testagent logs through testing.Logf (#6411 )	5 years ago
R.B. Boyer	91da908d2f	test: fix TestAgent.Start() to not segfault if the DNSServer cannot ListenAndServe (#6409 ) The embedded `Server` field on a `DNSServer` is only set inside of the `ListenAndServe` method. If that method fails for reasons like the address being in use and is not bindable, then the `Server` field will not be set and the overall `Agent.Start()` will fail. This will trigger the inner loop of `TestAgent.Start()` to invoke `ShutdownEndpoints` which will attempt to pretty print the DNS servers using fields on that inner `Server` field. Because it was never set, this causes a nil pointer dereference and crashes the test.	5 years ago
Alvin Huang	0be1531d80	add nil pointer check for pointer to ACLToken struct (#6407 )	5 years ago
Hans Hasselberg	faa54ab989	auto_encrypt: verify_incoming_rpc is good enough for auto_encrypt.allow_tls (#6376 ) Previously `verify_incoming` was required when turning on `auto_encrypt.allow_tls`, but that doesn't work together with HTTPS UI in some scenarios. Adding `verify_incoming_rpc` to the allowed configurations.	5 years ago
R.B. Boyer	7bc941575c	test: don't leak agent goroutines in TestAgent_sidecarServiceFromNodeService (#6396 ) A goroutine dump using runtime.Stack() before/after shows a drop from 121 => 4.	5 years ago

... 2 3 4 5 6 ...

1824 Commits (5f625666b42ed684fa4f2f2f20febcabb4c2b95d)