consul

Commit Graph

Author	SHA1	Message	Date
Hans Hasselberg	e7134a0dab	agent: only use TestAgent when appropriate (#5502 )	6 years ago
Paul Banks	0b5a078b95	Optimize health watching to single chan/goroutine. (#5449 ) Refs #4984. Watching chans for every node we touch in a health query is wasteful. In #4984 it shows that if there are more than 682 service instances we always fallback to watching all services which kills performance. We already have a record in MemDB that is reliably update whenever the service health result should change thanks to per-service watch indexes. So in general, provided there is at least one service instances and we actually have a service index for it (we always do now) we only ever need to watch a single channel. This saves us from ever falling back to the general index and causing the performance cliff in #4984, but it also means fewer goroutines and work done for every blocking health query. It also saves some allocations made during the query because we no longer have to populate a WatchSet with 3 chans per service instance which saves the internal map allocation. This passes all state store tests except the one that explicitly checked for the fallback behaviour we've now optimized away and in general seems safe.	6 years ago
Pierre Souchay	88d4383410	Ensure we remove Connect proxy before deregistering the service itself (#5482 ) This will fix https://github.com/hashicorp/consul/issues/5351	6 years ago
Valentin Fritz	21f149de8b	Fix checks removal when removing service (#5457 ) Fix my recently discovered issue described here: #5456	6 years ago
R.B. Boyer	cd96af4fc0	acl: reduce complexity of token resolution process with alternative singleflighting (#5480 ) acl: reduce complexity of token resolution process with alternative singleflighting Switches acl resolution to use golang.org/x/sync/singleflight. For the identity/legacy lookups this is a drop-in replacement with the same overall approach to request coalescing. For policies this is technically a change in behavior, but when considered holistically is approximately performance neutral (with the benefit of less code). There are two goals with this blob of code (speaking specifically of policy resolution here): 1) Minimize cross-DC requests. 2) Minimize client-to-server LAN requests. The previous iteration of this code was optimizing for the case of many possibly different tokens being resolved concurrently that have a significant overlap in linked policies such that deduplication would be worth the complexity. While this is laudable there are some things to consider that can help to adjust expectations: 1) For v1.4+ policies are always replicated, and once a single policy shows up in a secondary DC the replicated data is considered authoritative for requests made in that DC. This means that our earlier concerns about minimizing cross-DC requests are irrelevant because there will be no cross-DC policy reads that occur. 2) For Server nodes the in-memory ACL policy cache is capped at zero, meaning it has no caching. Only Client nodes run with a cache. This means that instead of having an entire DC's worth of tokens (what a Server might see) that can have policy resolutions coalesced these nodes will only ever be seeing node-local token resolutions. In a reasonable worst-case scenario where a scheduler like Kubernetes has "filled" a node with Connect services, even that will only schedule ~100 connect services per node. If every service has a unique token there will only be 100 tokens to coalesce and even then those requests have to occur concurrently AND be hitting an empty consul cache. Instead of seeing a great coalescing opportunity for cutting down on redundant Policy resolutions, in practice it's far more likely given node densities that you'd see requests for the same token concurrently than you would for two tokens sharing a policy concurrently (to a degree that would warrant the overhead of the current variation of singleflighting. Given that, this patch switches the Policy resolution process to only singleflight by requesting token (but keeps the cache as by-policy).	6 years ago
Kyle Havlovitz	8ae6547934	Update state store test for changing node ID	6 years ago
Kyle Havlovitz	6932b06f97	Add a test for changing a failed node's ID	6 years ago
Hans Hasselberg	7e11dd82aa	agent: enable reloading of tls config (#5419 ) This PR introduces reloading tls configuration. Consul will now be able to reload the TLS configuration which previously required a restart. It is not yet possible to turn TLS ON or OFF with these changes. Only when TLS is already turned on, the configuration can be reloaded. Most importantly the certificates and CAs.	6 years ago
R.B. Boyer	2e175be41b	acl: correctly extend the cache for acl identities during resolution (#5475 )	6 years ago
Aestek	4bea29f15a	[catalog] Update the node's services indexes on update (#5458 ) Node updates were not updating the service indexes, which are used for service related queries. This caused the X-Consul-Index to stay the same after a node update as seen from a service query even though the node data is returned in heath queries. If that happened in between queries the client would miss this change. We now update the indexes of the services on the node when it is updated. Fixes: #5450	6 years ago
Kyle Havlovitz	1e4523f55e	Add logic to allow changing a failed node's ID	6 years ago
Alvin Huang	8cb8108b1b	fix typos	6 years ago
R.B. Boyer	f4a3b9d518	fix typos reported by golangci-lint:misspell (#5434 )	6 years ago
R.B. Boyer	2ffbea41c8	improve flaky LANReap tests by expliciting configuring the tombstone timeout In TestServer_LANReap autopilot is running, so the alternate flow through the serf reaping function is possible. In that situation the ReconnectTimeout is not relevant so for parity also override the TombstoneTimeout value as well. For additional parity update the TestServer_WANReap and TestClient_LANReap versions of this test in the same way even though autopilot is irrelevant here .	6 years ago
R.B. Boyer	5bea49ecb0	tests: avoid leaking child processes from agent/proxyprocess package	6 years ago
Matt Keeler	567e41ff6b	Release v1.4.3	6 years ago
Matt Keeler	90040f8bff	Fixes for CVE-2019-8336 Fix error in detecting raft replication errors. Detect redacted token secrets and prevent attempting to insert. Add a Redacted field to the TokenBatchRead and TokenRead RPC endpoints This will indicate whether token secrets have been redacted. Ensure any token with a redacted secret in secondary datacenters is removed. Test that redacted tokens cannot be replicated.	6 years ago
Hans Hasselberg	d35824b1fa	default to tls 1.2 as promised. (#5340 )	6 years ago
Aestek	2aac4d5168	Register and deregisters services and their checks atomically in the local state (#5012 ) Prevent race between register and deregister requests by saving them together in the local state on registration. Also adds more cleaning in case of failure when registering services / checks.	6 years ago
Matt Keeler	6e6910ea11	Dont modify memdb owned token data for get/list requests of tokens (#5412 ) Previously we were fixing up the token links directly on the *ACLToken returned by memdb. This invalidated some assumptions that a snapshot is immutable as well as potentially being able to cause a crash. The fix here is to give the policy link fixing function copy on write semantics. When no fixes are necessary we can return the memdb object directly, otherwise we copy it and create a new list of links. Eventually we might find a better way to keep those policy links in sync but for now this fixes the issue.	6 years ago
Aestek	02f991843f	Fix race condition in DNS when using cache (#5398 ) * Fix race condition in DNS when using cache The healty node filtering was modifying the result from the cache, which caused a crash when multiple queries were made to the same service simultaneously. We now copy the node slice before filtering to ensure we do not modify the data stored in the cache. * Fix wording in dns cache config doc s/dns_max_age/cache_max_age/	6 years ago
Matt Keeler	200c0fb3e9	Call RemoveServer for reap events (#5317 ) This ensures that servers are removed from RPC routing when they are reaped.	6 years ago
R.B. Boyer	409c901f8e	test: fix concurrent map access when setting up test vault	6 years ago
R.B. Boyer	6955186239	fix ignored errors in state store internals as reported by errcheck	6 years ago
R.B. Boyer	c7067645dd	fix a few leap-year related clock math inaccuracies and failing tests	6 years ago
Matt Keeler	118adbb123	ACL Token Persistence and Reloading (#5328 ) This PR adds two features which will be useful for operators when ACLs are in use. 1. Tokens set in configuration files are now reloadable. 2. If `acl.enable_token_persistence` is set to `true` in the configuration, tokens set via the `v1/agent/token` endpoint are now persisted to disk and loaded when the agent starts (or during configuration reload) Note that token persistence is opt-in so our users who do not want tokens on the local disk will see no change. Some other secondary changes: * Refactored a bunch of places where the replication token is retrieved from the token store. This token isn't just for replicating ACLs and now it is named accordingly. * Allowed better paths in the `v1/agent/token/` API. Instead of paths like: `v1/agent/token/acl_replication_token` the path can now be just `v1/agent/token/replication`. The old paths remain to be valid. * Added a couple new API functions to set tokens via the new paths. Deprecated the old ones and pointed to the new names. The names are also generally better and don't imply that what you are setting is for ACLs but rather are setting ACL tokens. There is a minor semantic difference there especially for the replication token as again, its no longer used only for ACL token/policy replication. The new functions will detect 404s and fallback to using the older token paths when talking to pre-1.4.3 agents. * Docs updated to reflect the API additions and to show using the new endpoints. * Updated the ACL CLI set-agent-tokens command to use the non-deprecated APIs.	6 years ago
Kyle Havlovitz	f07e928afc	Merge pull request #5325 from hashicorp/consul-ca-panic connect/ca: fix a potential panic in the Consul provider	6 years ago
Hans Hasselberg	80e7d63fc2	Centralise tls configuration part 2 (#5374 ) This PR is based on #5366 and continues to centralise the tls configuration in order to be reloadable eventually! This PR is another refactoring. No tests are changed, beyond calling other functions or cosmetic stuff. I added a bunch of tests, even though they might be redundant.	6 years ago
Hans Hasselberg	786b3b1095	Centralise tls configuration part 1 (#5366 ) In order to be able to reload the TLS configuration, we need one way to generate the different configurations. This PR introduces a `tlsutil.Configurator` which holds a `tlsutil.Config`. Afterwards it is responsible for rendering every `tls.Config`. In this particular PR I moved `IncomingHTTPSConfig`, `IncomingTLSConfig`, and `OutgoingTLSWrapper` into `tlsutil.Configurator`. This PR is a pure refactoring - not a single feature added. And not a single test added. I only slightly modified existing tests as necessary.	6 years ago
Aestek	f1cdfbe40e	Allow DNS interface to use agent cache (#5300 ) Adds two new configuration parameters "dns_config.use_cache" and "dns_config.cache_max_age" controlling how DNS requests use the agent cache when querying servers.	6 years ago
Alvin Huang	77eecf1046	add wait to TestClient_JoinLAN	6 years ago
Alvin Huang	136df63e2c	add retry to TestResetSessionTimerLocked	6 years ago
Alvin Huang	a7180f715a	add serf check to testDNSServiceLookupResponseLimits, checkDNSService	6 years ago
Alvin Huang	d10b5a396b	add wait to TestOperator_AutopilotCASConfiguration	6 years ago
Alvin Huang	dc200daf21	add wait to TestSnapshot	6 years ago
Alvin Huang	c2a19e5090	add wait to TestAgent_RPCPing	6 years ago
Alvin Huang	c23eb91262	fix TestAgent_CheckCriticalTime and better error output	6 years ago
Alvin Huang	6c9b516a29	skip TestCheckTCPPassing on CircleCI	6 years ago
R.B. Boyer	c2a30c5fdd	fix incorrect body of TestACLEndpoint_PolicyBatchRead Lifted from PR #5307 as it was an unrelated drive-by fix on that PR anyway. s/token/policy/	6 years ago
R.B. Boyer	b569f222f9	update agent/agent_endpoint_test.go to use V2 tokens with attached policies	6 years ago
Nicholas Jackson	99fe9dabce	Envoy config cluster (#5308 ) * Start adding tests for cluster override * Refactor tests for clusters * Passing tests for custom upstream cluster override * Added capability to customise local app cluster * Rename config for local cluster override	6 years ago
Kainoa Seto	b2af8862c7	Deferred updating response meta with consul headers (#5355 )	6 years ago
R.B. Boyer	ef8258cd4e	test: switch test file from assert -> require for consistency Also in acl_endpoint_test.go: * convert logical blocks in some token tests to subtests * remove use of require.New This removes a lot of noise in a later PR.	6 years ago
Matt Keeler	766d771017	Pass a testing.T into NewTestAgent and TestAgent.Start (#5342 ) This way we can avoid unnecessary panics which cause other tests not to run. This doesn't remove all the possibilities for panics causing other tests not to run, it just fixes the TestAgent	6 years ago
R.B. Boyer	adbe8ed370	correct some typos	6 years ago
R.B. Boyer	88bb53d001	ensure that we plumb our configured logger into all parts of the raft library	6 years ago
R.B. Boyer	2c983902be	reduce the local scope of variable	6 years ago
R.B. Boyer	de0f585583	agent: only enable TLS on gRPC if the HTTPS API port is enabled (#5287 ) Currently the gRPC server assumes that if you have configured TLS certs on the agent (for RPC) that you want gRPC to be encrypted. If gRPC is bound to localhost this can be overkill. For the API we let the user choose to offer HTTP or HTTPS API endpoints independently of the TLS cert configuration for a similar reason. This setting will let someone encrypt RPC traffic with TLS but avoid encrypting local gRPC traffic if that is what they want to do by only enabling TLS on gRPC if the HTTPS API port is enabled.	6 years ago
R.B. Boyer	f2ed3a3777	clarify the ACL.PolicyDelete endpoint (#5337 ) There was an errant early-return in PolicyDelete() that bypassed the rest of the function. This was ok because the only caller of this function ignores the results. This removes the early-return making it structurally behave like TokenDelete() and for both PolicyDelete and TokenDelete clarify the lone callers to indicate that the return values are ignored. We may wish to avoid the entire return value as well, but this patch doesn't go that far.	6 years ago
R.B. Boyer	324ba5df17	update TestStateStore_ACLBootstrap to not rely upon request mutation (#5335 )	6 years ago
Matt Keeler	7073ba4ed2	Move autopilot initialization to prevent race (#5322 ) `establishLeadership` invoked during leadership monitoring may use autopilot to do promotions etc. There was a race with doing that and having autopilot initialized and this fixes it.	6 years ago
Kyle Havlovitz	29e4c17b07	connect/ca: fix a potential panic in the Consul provider	6 years ago
Matt Keeler	acfd87c673	Improve Connect with Prepared Queries (#5291 ) Given a query like: ``` { "Name": "tagged-connect-query", "Service": { "Service": "foo", "Tags": ["tag"], "Connect": true } } ``` And a Consul configuration like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": {} }, "tags": ["tag"] ] } ``` If you executed the query it would always turn up with 0 results. This was because the sidecar service was being created without any tags. You could instead make your config look like: ``` { "services": [ "name": "foo", "port": 8080, "connect": { "sidecar_service": { "tags": ["tag"] } }, "tags": ["tag"] ] } ``` However that is a bit redundant for most cases. This PR ensures that the tags and service meta of the parent service get copied to the sidecar service. If there are any tags or service meta set in the sidecar service definition then this copying does not take place. After the changes, the query will now return the expected results. A second change was made to prepared queries in this PR which is to allow filtering on ServiceMeta just like we allow for filtering on NodeMeta.	6 years ago
R.B. Boyer	e1e4249e90	testutil: redirect some test agent logs to testing.T.Logf (#5304 ) When tests fail, only the logs for the failing run are dumped to the console which helps in diagnosis. This is easily added to other test scenarios as they come up.	6 years ago
R.B. Boyer	db8a871309	Merge pull request #5237 from hashicorp/term-grpc-stream-on-token-failure Check ACLs more often for xDS endpoints.	6 years ago
mkeeler	c97c712e96	Release v1.4.2	6 years ago
Kyle Havlovitz	7118f42950	Fix failing TestAgent_PurgeCheckOnDuplicate after merge	6 years ago
Matt Keeler	1736e24fb3	Don't generate TXT records just to discard them (#5272 ) * Don't generate TXT records just to discard them * Add unit test for formatNodeRecord to ensure it prevents returning TXT records	6 years ago
Kyle Havlovitz	928b7ec60d	Merge branch 'healthcheck-duration-fix'	6 years ago
Kyle Havlovitz	1a4978fb94	Re-add ReadableDuration types to health check definition This is to fix the backwards-incompatible change made in 1.4.1 by changing these fields to time.Duration.	6 years ago
R.B. Boyer	e9a2eab316	speed up TestHTTPAPI_MethodNotAllowed_OSS from 11s -> 0.5s (#5268 )	6 years ago
Hans Hasselberg	552e150536	correct name	6 years ago
Hans Hasselberg	aebb50d47d	simpler fix	6 years ago
Hans Hasselberg	5db185a7e4	do not export that type	6 years ago
Hans Hasselberg	7f44100101	fix marshalling	6 years ago
Hans Hasselberg	d4790b2827	demo nomad problem	6 years ago
banks	65d2c9b51d	Release v1.4.1	6 years ago
Matt Keeler	d5a3ba6cda	Disregard rules when set on a management token (#5261 ) * Disregard rules when set on a management token * Add unit test for legacy mgmt token with rules	6 years ago
Kyle Havlovitz	88c044759f	connect: Forward intention RPCs if this isn't the primary	6 years ago
Kyle Havlovitz	6b28434f8a	Merge pull request #5249 from hashicorp/ca-fixes-oss Minor CA fixes	6 years ago
Kyle Havlovitz	5bdf130767	Merge pull request #4869 from hashicorp/txn-checks Add node/service/check operations to transaction api	6 years ago
Kyle Havlovitz	a28ba4687d	connect/ca: return a better error message if the CA isn't fully initialized when signing	6 years ago
Matt Keeler	579a8b32ed	Fix several ACL token/policy resolution issues. (#5246 ) * Fix 2 remote ACL policy resolution issues 1 - Use the right method to fire async not found errors when the ACL.PolicyResolve RPC returns that error. This was previously accidentally firing a token result instead of a policy result which would have effectively done nothing (unless there happened to be a token with a secret id == the policy id being resolved. 2. When concurrent policy resolution is being done we single flight the requests. The bug before was that for the policy resolution that was going to piggy back on anothers RPC results it wasn’t waiting long enough for the results to come back due to looping with the wrong variable. * Fix a handful of other edge case ACL scenarios The main issue was that token specific issues (not able to access a particular policy or the token being deleted after initial fetching) were poisoning the policy cache. A second issue was that for concurrent token resolutions, the first resolution to get started would go fetch all the policies. If before the policies were retrieved a second resolution request came in, the new request would register watchers for those policies but then never block waiting for them to complete. This resulted in using the default policy when it shouldn't have.	6 years ago
Paul Banks	ef9f27cbc8	connect: tame thundering herd of CSRs on CA rotation (#5228 ) * Support rate limiting and concurrency limiting CSR requests on servers; handle CA rotations gracefully with jitter and backoff-on-rate-limit in client * Add CSR rate limiting docs * Fix config naming and add tests for new CA configs	6 years ago
R.B. Boyer	d3eb781384	Check ACLs more often for xDS endpoints. For established xDS gRPC streams recheck ACLs for each DiscoveryRequest or DiscoveryResponse. If more than 5 minutes has elapsed since the last ACL check, recheck even without an incoming DiscoveryRequest or DiscoveryResponse. ACL failures will terminate the stream.	6 years ago
Kyle Havlovitz	ddc4a8d848	oss: add the enterprise server stub for intention replication check	6 years ago
R.B. Boyer	2dea3e2bd7	Fix some test typos.	6 years ago
Matt Keeler	7e6b3e6a0c	Implement prepared query upstreams watching for envoy (#5224 ) Fixes #4969 This implements non-blocking request polling at the cache layer which is currently only used for prepared queries. Additionally this enables the proxycfg manager to poll prepared queries for use in envoy proxy upstreams.	6 years ago
Kyle Havlovitz	21380021af	txn: update existing txn api docs with new operations	6 years ago
Matt Keeler	1ec5f2a27f	Store leaf cert indexes in raft and use for the ModifyIndex on the returned certs (#5211 ) * Store leaf cert indexes in raft and use for the ModifyIndex on the returned certs This ensures that future certificate signings will have a strictly greater ModifyIndex than any previous certs signed.	6 years ago
Aestek	4afbe792df	Improve blocking queries on services that do not exist (#4810 ) ## Background When making a blocking query on a missing service (was never registered, or is not registered anymore) the query returns as soon as any service is updated. On clusters with frequent updates (5~10 updates/s in our DCs) these queries virtually do not block, and clients with no protections againt this waste ressources on the agent and server side. Clients that do protect against this get updates later than they should because of the backoff time they implement between requests. ## Implementation While reducing the number of unnecessary updates we still want : * Clients to be notified as soon as when the last instance of a service disapears. * Clients to be notified whenever there's there is an update for the service. * Clients to be notified as soon as the first instance of the requested service is added. To reduce the number of unnecessary updates we need to block when a request to a missing service is made. However in the following case : 1. Client `client1` makes a query for service `foo`, gets back a node and X-Consul-Index 42 2. `foo` is unregistered 3. `client1` makes a query for `foo` with `index=42` -> `foo` does not exist, the query blocks and `client1` is not notified of the change on `foo` We could store the last raft index when each service was last alive to know wether we should block on the incoming query or not, but that list could grow indefinetly. We instead store the last raft index when a service was unregistered and use it when a query targets a service that does not exist. When a service `srv` is unregistered this "missing service index" is always greater than any X-Consul-Index held by the clients while `srv` was up, allowing us to immediatly notify them. 1. Client `client1` makes a query for service `foo`, gets back a node and `X-Consul-Index: 42` 2. `foo` is unregistered, we set the "missing service index" to 43 3. `client1` makes a blocking query for `foo` with `index=42` -> `foo` does not exist, we check against the "missing service index" and return immediatly with `X-Consul-Index: 43` 4. `client1` makes a blocking query for `foo` with `index=43` -> we block 5. Other changes happen in the cluster, but foo still doesn't exist and "missing service index" hasn't changed, the query is still blocked 6. `foo` is registered again on index 62 -> `foo` exists and its index is greater than 43, we unblock the query	6 years ago
Matt Keeler	baa8946ea6	cache: Pass through wait query param to the cache.Get (#5203 ) This adds a MaxQueryTime field to the connect ca leaf cache request type and populates it via the wait query param. The cache will then do the right thing and timeout the operation as expected if no new leaf cert is available within that time. Fixes #4462 The reproduction scenario in the original issue now times out appropriately.	6 years ago
Aestek	c043de5381	[Security] Allow blocking Write endpoints on Agent using Network Addresses (#4719 ) * Add -write-allowed-nets option * Add documentation for the new write_allowed_nets option	6 years ago
Matt Keeler	1048f3d5e7	acl: Prevent tokens from deleting themselves (#5210 ) Fixes #4897 Also apparently token deletion could segfault in secondary DCs when attempting to delete non-existant tokens. For that reason both checks are wrapped within the non-nil check.	6 years ago
Paul Banks	0638e09b6e	connect: agent leaf cert caching improvements (#5091 ) * Add State storage and LastResult argument into Cache so that cache.Types can safely store additional data that is eventually expired. * New Leaf cache type working and basic tests passing. TODO: more extensive testing for the Root change jitter across blocking requests, test concurrent fetches for different leaves interact nicely with rootsWatcher. * Add multi-client and delayed rotation tests. * Typos and cleanup error handling in roots watch * Add comment about how the FetchResult can be used and change ca leaf state to use a non-pointer state. * Plumb test override of root CA jitter through TestAgent so that tests are deterministic again! * Fix failing config test	6 years ago
Kyle Havlovitz	c07c5446a8	txn: clean up some state store/acl code	6 years ago
Hans Hasselberg	067027230b	connect: add tls config for vault connect ca provider (#5125 ) * add tlsconfig for vault connect ca provider. * add options to the docs * add tests for new configuration	6 years ago
Alejandro Guirao Rodríguez	9f33353c14	agent/config: Fix typo in comment (#5202 )	6 years ago
Paul Banks	bb7145f27d	agent: add default weights to service in local state to prevent AE churn (#5126 ) * Add default weights when adding a service with no weights to local state to prevent constant AE re-sync. This fix was contributed by @42wim in https://github.com/hashicorp/consul/pull/5096 but was merged against the wrong base. This adds it to master and adds a test to cover the behaviour. * Fix tests that broke due to comparing internal state which now has default weights	6 years ago
Paul Banks	0589525ae9	agent: Don't leave old errors around in cache (#5094 ) * Fixes #4480. Don't leave old errors around in cache that can be hit in specific circumstances. * Move error reset to cover extreme edge case of nil Value, nil err Fetch	6 years ago
Pierre Souchay	ae7f88f995	Avoid to have infinite recursion in DNS lookups when resolving CNAMEs (#4918 ) * Avoid to have infinite recursion in DNS lookups when resolving CNAMEs This will avoid killing Consul when a Service.Address is using CNAME to a Consul CNAME that creates an infinite recursion. This will fix https://github.com/hashicorp/consul/issues/4907 * Use maxRecursionLevel = 3 to allow several recursions	6 years ago
Paul Banks	b29bc906ee	bugfix: use ServiceTags to generate cache key hash (#4987 ) * bugfix: use ServiceTags to generate cahce key hash * update unit test * update * remote print log * Update .gitignore * Completely deprecate ServiceTag field internally for clarity * Add explicit test for CacheInfo cases	6 years ago
Aestek	8709213d6e	Prevent status flap when re-registering a check (#4904 ) Fixes point `#2` of: https://github.com/hashicorp/consul/issues/4903 When registering a service each healthcheck status is saved and restored (https://github.com/hashicorp/consul/blob/master/agent/agent.go#L1914) to avoid unnecessary flaps in health state. This change extends this feature to single check registration by moving this protection in `AddCheck()` so that both `PUT /v1/agent/service/register` and `PUT /v1/agent/check/register` behave in the same idempotent way. #### Steps to reproduce 1. Register a check : ``` curl -X PUT \ http://127.0.0.1:8500/v1/agent/check/register \ -H 'Content-Type: application/json' \ -d '{ "Name": "my_check", "ServiceID": "srv", "Interval": "10s", "Args": ["true"] }' ``` 2. The check will initialize and change to `passing` 3. Run the same request again 4. The check status will quickly go from `critical` to `passing` (the delay for this transission is determined by https://github.com/hashicorp/consul/blob/master/agent/checks/check.go#L95)	6 years ago
Mitchell Hashimoto	f76022fa63	CA Provider Plugins (#4751 ) This adds the `agent/connect/ca/plugin` library for consuming/serving Connect CA providers as [go-plugin](https://github.com/hashicorp/go-plugin) plugins. This does not wire this up in any way to Consul itself, so this will not enable using these plugins yet. ## Why? We want to enable CA providers to be pluggable without modifying Consul so that any CA or PKI system can potentially back the Connect certificates. This CA system may also be used in the future for easier bootstrapping and internal cluster security. ### go-plugin The benefit of `go-plugin` is that for the plugin consumer, the fact that the interface implementation is communicating over multi-process RPC is invisible. Internals of Consul will continue to just use `ca.Provider` interface implementations as if they're local. For plugin _authors_, they simply have to implement the interface. The network/transport/process management issues are handled by go-plugin itself. The CA provider plugins support both `net/rpc` and gRPC transports. This enables easy authoring in any language. go-plugin handles the actual protocol handshake and connection. This is just a feature of go-plugin. `go-plugin` is already in production use for years by Packer, Terraform, Nomad, Vault, and Sentinel. We've shown stability for both desktop and server-side software. It is very mature. ## Implementation Details ### `map[string]interface{}` The `Configure` method passes a `map[string]interface{}`. This map contains only Go primitives and containers of primitives (no funcs, chans, etc.). For `net/rpc` we encode as-is using Gob. For gRPC we marshal to JSON and transmit as a `bytes` type. This is the same approach we take with Vault and other software. Note that this is just the transport protocol, the end software views it fully decoded. ### `x509.Certificate` and `CertificateRequest` We transmit the raw ASN.1 bytes and decode on the other side. Unit tests are verifying we get the same cert/csrs across the wire. ### Testing `go-plugin` exposes test helpers that enable testing the full plugin RPC over real loopback network connections. We test all endpoints for success and error for both `net/rpc` and gRPC. ### Vendoring This PR doesn't introduce vendoring for two reasons: 1. @banks's `f-envoy` branch introduces a lot of these and I didn't want conflict. 2. The library isn't actually used yet so it doesn't introduce compile-time errors (it does introduce test errors). ## Next Steps With this in place, we need to figure out the proper way to actually hook these up to Consul, load them, etc. This discussion can happen elsewhere, since regardless of approach this plugin library implementation is the exact same.	6 years ago
Grégoire Seux	4f62a3b528	Implement /v1/agent/health/service/<service name> endpoint (#3551 ) This endpoint aggregates all checks related to <service id> on the agent and return an appropriate http code + the string describing the worst check. This allows to cleanly expose service status to other component, hiding complexity of multiple checks. This is especially useful to use consul to feed a load balancer which would delegate health checking to consul agent. Exposing this endpoint on the agent is necessary to avoid a hit on consul servers and avoid decreasing resiliency (this endpoint will work even if there is no consul leader in the cluster).	6 years ago
Aestek	5960974db1	[Fix] Services sometimes not being synced with acl_enforce_version_8 = false (#4771 ) Fixes: https://github.com/hashicorp/consul/issues/3676 This fixes a bug were registering an agent with a non-existent ACL token can prevent other services registered with a good token from being synced to the server when using `acl_enforce_version_8 = false`. ## Background When `acl_enforce_version_8` is off the agent does not check the ACL token validity before storing the service in its state. When syncing a service registered with a missing ACL token we fall into the default error handling case (https://github.com/hashicorp/consul/blob/master/agent/local/state.go#L1255) and stop the sync (https://github.com/hashicorp/consul/blob/master/agent/local/state.go#L1082) without setting its Synced property to true like in the permission denied case. This means that the sync will always stop at the faulty service(s). The order in which the services are synced is random since we iterate on a map. So eventually all services with good ACL tokens will be synced, this can however take some time and is influenced by the cluster size, the bigger the slower because retries are less frequent. Having a service in this state also prevent all further sync of checks as they are done after the services. ## Changes This change modify the sync process to continue even if there is an error. This fixes the issue described above as well as making the sync more error tolerant: if the server repeatedly refuses a service (the ACL token could have been deleted by the time the service is synced, the servers were upgraded to a newer version that has more strict checks on the service definition...). Then all services and check that can be synced will, and those that don't will be marked as errors in the logs instead of blocking the whole process.	6 years ago
Hans Hasselberg	0b4a879203	ui: serve /robots.txt when UI is enabled. (#5089 ) * serve /robots.txt * robots.txt: disallow everything	6 years ago
Kyle Havlovitz	995e728ea0	txn: fix an issue with querying nodes by name instead of ID	6 years ago
Pierre Souchay	f4dc8b42e0	[Travis][UnstableTests] Fixed unstable tests in travis (#5013 ) * [Travis][UnstableTests] Fixed unstable tests in travis as seen in https://travis-ci.org/hashicorp/consul/jobs/460824602 * Fixed unstable tests in https://travis-ci.org/hashicorp/consul/jobs/460857687	6 years ago
Kyle Havlovitz	67bac7a815	api: add support for new txn operations	6 years ago
Kyle Havlovitz	de4dbf583e	txn: add tests for RPC endpoint	6 years ago
Kyle Havlovitz	6a512e5c0f	txn: add ACL enforcement/validation to new txn ops	6 years ago
Kyle Havlovitz	9467067432	state: add tests for new txn ops	6 years ago
Kyle Havlovitz	7759e9ea8b	txn: add service operations	6 years ago
Kyle Havlovitz	ab58986ac3	txn: add node operations	6 years ago
Kyle Havlovitz	01e1b5b1df	txn: add pre-check operations to txn endpoint	6 years ago
Kyle Havlovitz	b371ea8783	Add check operations to transaction api	6 years ago
Kyle Havlovitz	c7e0d3b919	Merge pull request #5061 from hashicorp/blank-ca-fix connect/ca: prevent blank CA config in snapshot	6 years ago
Kyle Havlovitz	4f2715d4e2	connect/ca: prevent blank CA config in snapshot This PR both prevents a blank CA config from being written out to a snapshot and allows Consul to gracefully recover from a snapshot with an invalid CA config. Fixes #4954.	6 years ago
Jack Pearkes	b64e8b262f	Documentation and changes for `verify_server_hostname` (#5069 ) * verify_server_hostname implies verify_outgoing * mention CVE in the docs.	6 years ago
R.B. Boyer	c1eccfd1db	agent: remove some stray fmt.Print* calls (#5015 )	6 years ago
Pierre Souchay	c5ae9caa28	Fixed another list of unstable unit tests in travis (#4915 ) * Fixed another list of unstable unit tests in travis Fixed failing tests in https://travis-ci.org/hashicorp/consul/jobs/451357061 * Fixed another list of unstable unit tests in travis. Fixed failing tests in https://travis-ci.org/hashicorp/consul/jobs/451357061	6 years ago
banks	0bddfa23a2	Release v1.4.0	6 years ago
Kyle Havlovitz	76f102a1e0	Merge pull request #4952 from hashicorp/test-version tests: Bump test server version to 1.4.0	6 years ago
R.B. Boyer	934fae659f	acl: add stub hooks to support some plumbing in enterprise (#4951 )	6 years ago
Kyle Havlovitz	269354c61d	oss: bump test server version to 1.4.0	6 years ago
Aestek	4942e66440	Fix catalog tag filter backward compat (#4944 ) Fix catalog service node filtering (ex /v1/catalog/service/srv?tag=tag1) between agent version <=v1.2.3 and server >=v1.3.0. New server version did not account for the old field when filtering hence request made from old agent were not tag-filtered.	6 years ago
Jack Pearkes	a90c29e60d	Doc changes for 1.4 Final (#4870 ) * website: add multi-dc enterprise landing page * website: switch all 1.4.0 alerts/RC warnings * website: connect product wording Co-Authored-By: pearkes <jackpearkes@gmail.com> * website: remove RC notification * commmand/acl: fix usage docs for ACL tokens * agent: remove comment, OperatorRead * website: improve multi-dc docs Still not happy with this but tried to make it slightly more informative. * website: put back acl guide warning for 1.4.0 * website: simplify multi-dc page and respond to feedback * Fix Multi-DC typos on connect index page. * Improve Multi-DC overview. A full guide is a WIP and will be added post-release. * Fixes typo avaiable > available	6 years ago
Paul Banks	54c2ff6aca	connect: remove additional trust-domain validation (#4934 ) * connct: Remove additional trust-domain validation * Comment typos * Update connect_ca.go	6 years ago
Kyle Havlovitz	4a73a59d70	Merge pull request #4917 from hashicorp/replication-token-cleanup Use acl replication_token for connect	6 years ago
Kyle Havlovitz	972177071d	update non-voting server test to fix enterprise diff	6 years ago
Kyle Havlovitz	643bd13aed	oss: do a proper check-and-set on the CA roots/config fsm operation	6 years ago
R.B. Boyer	e30cc73b1d	Update agent tests to wait a bit longer for the /v1/agent/self endpoint (#4937 )	6 years ago
R.B. Boyer	2afc2a3c3b	acl: fixes ACL replication for legacy tokens without AccessorIDs (#4885 )	6 years ago
Kyle Havlovitz	e8dd89359a	agent: fix formatting	6 years ago
Kyle Havlovitz	62691ebc82	config: remote connect replication_token	6 years ago
R.B. Boyer	9211d2701d	fix comment typos (#4890 )	6 years ago
Kyle Havlovitz	8337e3d8c0	Merge pull request #4872 from hashicorp/node-snapshot-fix Node ID/datacenter snapshot fix	6 years ago
Matt Keeler	db2cf01406	Adds documentation for the new ACL APIs (#4851 ) * Update the ACL API docs * Add a CreateTime to the anon token Also require acl:read permissions at least to perform rule translation. Don’t want someone DoSing the system with an open endpoint that actually does a bit of work. * Fix one place where I was referring to id instead of AccessorID * Add godocs for the API package additions. * Minor updates: removed some extra commas and updated the acl intro paragraph * minor tweaks * Updated the language to be clearer * Updated the language to be clearer for policy page * I was also confused by that! Your updates are much clearer. Co-Authored-By: kaitlincarter-hc <43049322+kaitlincarter-hc@users.noreply.github.com> * Sounds much better. Co-Authored-By: kaitlincarter-hc <43049322+kaitlincarter-hc@users.noreply.github.com> * Updated sidebar layout and deprecated warning	6 years ago
Matt Keeler	f9cf0eb36e	Remaining ACL Unit Tests (#4852 ) * Add leader token upgrade test and fix various ACL enablement bugs * Update the leader ACL initialization tests. * Add a StateStore ACL tests for ACLTokenSet and ACLTokenGetBy* functions * Advertise the agents acl support status with the agent/self endpoint. * Make batch token upsert CAS’able to prevent consistency issues with token auto-upgrade * Finish up the ACL state store token tests * Finish the ACL state store unit tests Also rename some things to make them more consistent. * Do as much ACL replication testing as I can.	6 years ago
Kyle Havlovitz	bd6d0e598f	fsm: update snapshot/restore test to include ID and datacenter	6 years ago
Kyle Havlovitz	6483356329	fsm: add missing ID/datacenter to persistNodes	6 years ago
Matt Keeler	d238cb181c	New ACL API Tests (#4848 ) * A few API mods and unit tests. * Update the unit tests to verify query/write metadata and to fix the rules endpoint tests. * Make sure the full information for the replication status is in the api packge	6 years ago
Matt Keeler	790cf90ee5	Fix the NonVoter Bootstrap test (#4786 )	6 years ago
banks	1757fbc0aa	Release v1.4.0-rc1	6 years ago
Kyle Havlovitz	819566f6b7	fsm: add Intention operations to transactions for internal use	6 years ago
Matt Keeler	34b53e7099	A few misc fixes found by go vet	6 years ago
Matt Keeler	18b29c45c4	New ACLs (#4791 ) This PR is almost a complete rewrite of the ACL system within Consul. It brings the features more in line with other HashiCorp products. Obviously there is quite a bit left to do here but most of it is related docs, testing and finishing the last few commands in the CLI. I will update the PR description and check off the todos as I finish them over the next few days/week. Description At a high level this PR is mainly to split ACL tokens from Policies and to split the concepts of Authorization from Identities. A lot of this PR is mostly just to support CRUD operations on ACLTokens and ACLPolicies. These in and of themselves are not particularly interesting. The bigger conceptual changes are in how tokens get resolved, how backwards compatibility is handled and the separation of policy from identity which could lead the way to allowing for alternative identity providers. On the surface and with a new cluster the ACL system will look very similar to that of Nomads. Both have tokens and policies. Both have local tokens. The ACL management APIs for both are very similar. I even ripped off Nomad's ACL bootstrap resetting procedure. There are a few key differences though. Nomad requires token and policy replication where Consul only requires policy replication with token replication being opt-in. In Consul local tokens only work with token replication being enabled though. All policies in Nomad are globally applicable. In Consul all policies are stored and replicated globally but can be scoped to a subset of the datacenters. This allows for more granular access management. Unlike Nomad, Consul has legacy baggage in the form of the original ACL system. The ramifications of this are: A server running the new system must still support other clients using the legacy system. A client running the new system must be able to use the legacy RPCs when the servers in its datacenter are running the legacy system. The primary ACL DC's servers running in legacy mode needs to be a gate that keeps everything else in the entire multi-DC cluster running in legacy mode. So not only does this PR implement the new ACL system but has a legacy mode built in for when the cluster isn't ready for new ACLs. Also detecting that new ACLs can be used is automatic and requires no configuration on the part of administrators. This process is detailed more in the "Transitioning from Legacy to New ACL Mode" section below.	6 years ago
Pierre Souchay	fab55bee2b	dns: implements prefix lookups for DNS TTL (#4605 ) This will fix https://github.com/hashicorp/consul/issues/4509 and allow forinstance lb-* to match services lb-001 or lb-service-007.	6 years ago
Jack Pearkes	8c684db488	New command: consul debug (#4754 ) * agent/debug: add package for debugging, host info * api: add v1/agent/host endpoint * agent: add v1/agent/host endpoint * command/debug: implementation of static capture * command/debug: tests and only configured targets * agent/debug: add basic test for host metrics * command/debug: add methods for dynamic data capture * api: add debug/pprof endpoints * command/debug: add pprof * command/debug: timing, wg, logs to disk * vendor: add gopsutil/disk * command/debug: add a usage section * website: add docs for consul debug * agent/host: require operator:read * api/host: improve docs and no retry timing * command/debug: fail on extra arguments * command/debug: fixup file permissions to 0644 * command/debug: remove server flags * command/debug: improve clarity of usage section * api/debug: add Trace for profiling, fix profile * command/debug: capture profile and trace at the same time * command/debug: add index document * command/debug: use "clusters" in place of members * command/debug: remove address in output * command/debug: improve comment on metrics sleep * command/debug: clarify usage * agent: always register pprof handlers and protect This will allow us to avoid a restart of a target agent for profiling by always registering the pprof handlers. Given this is a potentially sensitive path, it is protected with an operator:read ACL and enable debug being set to true on the target agent. enable_debug still requires a restart. If ACLs are disabled, enable_debug is sufficient. * command/debug: use trace.out instead of .prof More in line with golang docs. * agent: fix comment wording * agent: wrap table driven tests in t.run()	6 years ago
Kyle Havlovitz	c617326470	re-add Connect multi-dc config changes This reverts commit `8bcfbaffb6`.	6 years ago
R.B. Boyer	307d91934c	fix some test hangs (#4785 ) The default http.Client uses infinite timeouts, so if TestHTTPAPI_MethodNotAllowed_OSS experienced anything going wrong about setup it could hang forever. Switching to hard coding various http.Client timeouts to non-infinite values at least bounds the failure time.	6 years ago
banks	469768ae39	Release v1.3.0	6 years ago
Jack Pearkes	8bcfbaffb6	Revert "Connect multi-dc config" (#4784 )	6 years ago
Aestek	25f04fbd21	[Security] Add finer control over script checks (#4715 ) * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording	6 years ago
Paul Banks	298af6dca7	Quick fix for cache age flakiness in CI	6 years ago
Rebecca Zanzig	34e5516834	Support multiple tags for health and catalog http api endpoints (#4717 ) * Support multiple tags for health and catalog api endpoints Fixes #1781. Adds a `ServiceTags` field to the ServiceSpecificRequest to support multiple tags, updates the filter logic in the catalog store, and propagates these change through to the health and catalog endpoints. Note: Leaves `ServiceTag` in the struct, since it is being used as part of the DNS lookup, which in turn uses the health check. * Update the api package to support multiple tags Includes additional tests. * Update new tests to use the `require` library * Update HealthConnect check after a bad merge	6 years ago
Pierre Souchay	51b33ef015	[Performance On Large clusters] Reduce updates on large services (#4720 ) * [Performance On Large clusters] Checks do update services/nodes only when really modified to avoid too many updates on very large clusters In a large cluster, when having a few thousands of nodes, the anti-entropy mechanism performs lots of changes (several per seconds) while there is no real change. This patch wants to improve this in order to increase Consul scalability when using many blocking requests on health for instance. * [Performance for large clusters] Only updates index of service if service is really modified * [Performance for large clusters] Only updates index of nodes if node is really modified * Added comments / ensure IsSame() has clear semantics * Avoid having modified boolean, return nil directly if stutures are Same * Fixed unstable unit tests TestLeader_ChangeServerID * Rewrite TestNode_IsSame() for better readability as suggested by @banks * Rename ServiceNode.IsSame() into IsSameService() + added unit tests * Do not duplicate TestStructs_ServiceNode_Conversions() and increase test coverage of IsSameService * Clearer documentation in IsSameService * Take into account ServiceProxy into ServiceNode.IsSameService() * Fixed IsSameService() with all new structures	6 years ago
Paul Banks	51c0001aad	[WIP] Initial draft of Sidecar Service and Managed Proxy deprecation docs (#4752 ) * Initial draft of Sidecar Service and Managed Proxy deprecation docs * Service definition deprecation notices and sidecar service * gRPC and sidecar service config options; Deprecate managed proxy options * Envoy Docs: Basic envoy command; envoy getting started/intro * Remove change that snuck in * Envoy custom config example * Add agent/service API docs; deprecate proxy config endpoint * Misc grep cleanup for managed proxies; capitalize Envoy * Updates to getting started guide * Add missing link * Refactor Envoy guide into a separate guide and add bootstrap reference notes. * Add limitations to Envoy docs; Highlight no fixes for known managed proxy issues on deprecation page; clarify snake cae stuff; Sidecar Service lifecycle	6 years ago
Pierre Souchay	251156eb68	Added SOA configuration for DNS settings. (#4714 ) This will allow to fine TUNE SOA settings sent by Consul in DNS responses, for instance to be able to control negative ttl. Will fix: https://github.com/hashicorp/consul/issues/4713 # Example Override all settings: * min_ttl: 0 => 60s * retry: 600 (10m) => 300s (5 minutes), * expire: 86400 (24h) => 43200 (12h) * refresh: 3600 (1h) => 1800 (30 minutes) ``` consul agent -dev -hcl 'dns_config={soa={min_ttl=60,retry=300,expire=43200,refresh=1800}}' ``` Result: ``` dig +multiline @localhost -p 8600 service.consul ; <<>> DiG 9.12.1 <<>> +multiline @localhost -p 8600 service.consul ; (2 servers found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 36557 ;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;service.consul. IN A ;; AUTHORITY SECTION: consul. 0 IN SOA ns.consul. hostmaster.consul. ( 1537959133 ; serial 1800 ; refresh (30 minutes) 300 ; retry (5 minutes) 43200 ; expire (12 hours) 60 ; minimum (1 minute) ) ;; Query time: 4 msec ;; SERVER: 127.0.0.1#8600(127.0.0.1) ;; WHEN: Wed Sep 26 12:52:13 CEST 2018 ;; MSG SIZE rcvd: 93 ```	6 years ago
Kyle Havlovitz	e4349c5710	connect/ca: more OSS split for multi-dc	6 years ago
Kyle Havlovitz	0da4f2b2e8	connect/ca: split CA initialization logic between oss/enterprise	6 years ago
Kyle Havlovitz	56dc426227	agent: add primary_datacenter and connect replication config options	6 years ago
Kyle Havlovitz	98d95cfa80	connect: add ExternalTrustDomain to CARoot fields	6 years ago
Kyle Havlovitz	46c829b879	docs: deprecate acl_datacenter and replace it with primary_datacenter	6 years ago
Paul Banks	c9217c958e	merge feedback: fix typos; actually use deliverLatest added previously but not plumbed in	6 years ago
Paul Banks	161482d2cd	Fix up tests broken by master merge; add proxy tests to services command (and fix it!); actually run the proxycfg.Manager	6 years ago
Paul Banks	a28e4a33b2	Fix bug in leaf-cert cache type where multiple client tokens collide (#4736 ) * Fix bug in leaf-cert cache type where multiple clients with different tokens would share certs and block incorrectly * Use hash for issued certs key to avoid ambiguity concatenating	6 years ago
Paul Banks	dca1303d05	Connect Envoy Command (#4735 ) * Plumb xDS server and proxyxfg into the agent startup * Add `consul connect envoy` command to allow running Envoy as a connect sidecar. * Add test for help tabs; typos and style fixups from review	6 years ago
Paul Banks	1909a95118	xDS Server Implementation (#4731 ) * Vendor updates for gRPC and xDS server * xDS server implementation for serving Envoy as a Connect proxy * Address initial review comments * consistent envoy package aliases; typos fixed; override TLS and authz for custom listeners * Moar Typos * Moar typos	6 years ago
Paul Banks	8336b5e6b9	XDS Server Config (#4730 ) * Config for the coming XDS server * Default gRPC to 8502 for -dev mode; Re-merge the command Info output that shows gRPC.	6 years ago
Paul Banks	0f27ffd163	Proxy Config Manager (#4729 ) * Proxy Config Manager This component watches for local state changes on the agent and ensures that each service registered locally with Kind == connect-proxy has it's state being actively populated in the cache. This serves two purposes: 1. For the built-in proxy, it ensures that the state needed to accept connections is available in RAM shortly after registration and likely before the proxy actually starts accepting traffic. 2. For (future - next PR) xDS server and other possible future proxies that require _push_ based config discovery, this provides a mechanism to subscribe and be notified about updates to a proxy instance's config including upstream service discovery results. * Address review comments * Better comments; Better delivery of latest snapshot for slow watchers; Embed Config * Comment typos * Add upstream Stringer for funsies	6 years ago
Paul Banks	96b9b95a19	Add cache.Notify to abstract watching for cache updates for types that support blocking semantics. (#4695 )	6 years ago
Paul Banks	e812f5516a	Add -sidecar-for and new /agent/service/:service_id endpoint (#4691 ) - A new endpoint `/v1/agent/service/:service_id` which is a generic way to look up the service for a single instance. The primary value here is that it: - supports hash-based blocking and so; - replaces `/agent/connect/proxy/:proxy_id` as the mechanism the built-in proxy uses to read its config. - It's not proxy specific and so works for any service. - It has a temporary shim to call through to the existing endpoint to preserve current managed proxy config defaulting behaviour until that is removed entirely (tested). - The built-in proxy now uses the new endpoint exclusively for it's config - The built-in proxy now has a `-sidecar-for` flag that allows the service ID of the _target_ service to be specified, on the condition that there is exactly one "sidecar" proxy (that is one that has `Proxy.DestinationServiceID` set) for the service registered. - Several fixes for edge cases for SidecarService - A fix for `Alias` checks - when running locally they didn't update their state until some external thing updated the target. If the target service has no checks registered as below, then the alias never made it past critical.	6 years ago
Paul Banks	1e7eace066	Add SidecarService Syntax sugar to Service Definition (#4686 ) * Added new Config for SidecarService in ServiceDefinitions. * WIP: all the code needed for SidecarService is written... none of it is tested other than config :). Need API updates too. * Test coverage for the new sidecarServiceFromNodeService method. * Test API registratrion with SidecarService * Recursive Key Translation 🤦 * Add tests for nested sidecar defintion arrays to ensure they are translated correctly * Use dedicated internal state rather than Service Meta for tracking sidecars for deregistration. Add tests for deregistration. * API struct for agent register. No other endpoint should be affected yet. * Additional test cases to cover updates to API registrations	6 years ago
Paul Banks	b83bbf248c	Add Proxy Upstreams to Service Definition (#4639 ) * Refactor Service Definition ProxyDestination. This includes: - Refactoring all internal structs used - Updated tests for both deprecated and new input for: - Agent Services endpoint response - Agent Service endpoint response - Agent Register endpoint - Unmanaged deprecated field - Unmanaged new fields - Managed deprecated upstreams - Managed new - Catalog Register - Unmanaged deprecated field - Unmanaged new fields - Managed deprecated upstreams - Managed new - Catalog Services endpoint response - Catalog Node endpoint response - Catalog Service endpoint response - Updated API tests for all of the above too (both deprecated and new forms of register) TODO: - config package changes for on-disk service definitions - proxy config endpoint - built-in proxy support for new fields * Agent proxy config endpoint updated with upstreams * Config file changes for upstreams. * Add upstream opaque config and update all tests to ensure it works everywhere. * Built in proxy working with new Upstreams config * Command fixes and deprecations * Fix key translation, upstream type defaults and a spate of other subtele bugs found with ned to end test scripts... TODO: tests still failing on one case that needs a fix. I think it's key translation for upstreams nested in Managed proxy struct. * Fix translated keys in API registration. ≈ * Fixes from docs - omit some empty undocumented fields in API - Bring back ServiceProxyDestination in Catalog responses to not break backwards compat - this was removed assuming it was only used internally. * Documentation updates for Upstreams in service definition * Fixes for tests broken by many refactors. * Enable travis on f-connect branch in this branch too. * Add consistent Deprecation comments to ProxyDestination uses * Update version number on deprecation notices, and correct upstream datacenter field with explanation in docs	6 years ago
Paul Banks	b06ddc9187	Rename proxy package (re-run of #4550 ) (#4638 ) * Rename agent/proxy package to reflect that it is limited to managed proxy processes Rationale: we have several other components of the agent that relate to Connect proxies for example the ProxyConfigManager component needed for Envoy work. Those things are pretty separate from the focus of this package so far which is only concerned with managing external proxy processes so it's nota good fit to put code for that in here, yet there is a naming clash if we have other packages related to proxy functionality that are not in the `agent/proxy` package. Happy to bikeshed the name. I started by calling it `managedproxy` but `managedproxy.Manager` is especially unpleasant. `proxyprocess` seems good in that it's more specific about purpose but less clearly connected with the concept of "managed proxies". The names in use are cleaner though e.g. `proxyprocess.Manager`. This rename was completed automatically using golang.org/x/tools/cmd/gomvpkg. Depends on #4541 * Fix missed windows tagged files	6 years ago
Paul Banks	88388d760d	Support Agent Caching for Service Discovery Results (#4541 ) * Add cache types for catalog/services and health/services and basic test that caching works * Support non-blocking cache types with Cache-Control semantics. * Update API docs to include caching info for every endpoint. * Comment updates per PR feedback. * Add note on caching to the 10,000 foot view on the architecture page to make the new data path more clear. * Document prepared query staleness quirk and force all background requests to AllowStale so we can spread service discovery load across servers.	6 years ago
Igal Shprincis	e1fe3af37f	watch: don't set TLSConfig.Address explicitly (#4727 ) Don't set the value of TLSConfig.Address explicitly. This will make sure env vars like CONSUL_TLS_SERVER_NAME are taken into account for the connection. Fixes #4718.	6 years ago
Paul Banks	e8ba527f23	Add a Close method to cache that stops background goroutines. (#4746 ) In a real agent the `cache` instance is alive until the agent shuts down so this is not a real leak in production, however in out test suite, every testAgent that is started and stops leaks goroutines that never get cleaned up which accumulate consuming CPU and memory through subsequent test in the `agent` package which doesn't help our test flakiness. This adds a Close method that doesn't invalidate or clean up the cache, and still allows concurrent blocking queries to run (for up to 10 mins which might still affect tests). But at least it doesn't maintain them forever with background refresh and an expiry watcher routine. It would be nice to cancel any outstanding blocking requests as well when we close but that requires much more invasive surgery right into our RPC protocol since we don't have a way to cancel requests currently. Unscientifically this seems to make tests pass a bit quicker and more reliably locally but I can't really be sure of that!	6 years ago
Paul O'Connor	6b7f03911e	Fix prometheus error message (#4745 )	6 years ago
R.B. Boyer	491826ddbc	cli: forward SIGTERM to child process of 'lock' and 'watch' subcommands (#4737 ) cli: forward SIGTERM to child process of 'lock' and 'watch' subcommands on unix This also removes the signal handler for SIGKILL as it's impossible to receive these signals.	6 years ago
Alex Dadgar	43d0f96c42	do not bootstrap with non voters	6 years ago
Kyle Havlovitz	57deb28ade	connect/ca: tighten up the intermediate signing verification	6 years ago
Kyle Havlovitz	2919519665	connect/ca: add intermediate functions to Vault ca provider	6 years ago
Kyle Havlovitz	52e8652ac5	connect/ca: add intermediate functions to Consul CA provider	6 years ago
Kyle Havlovitz	d515d25856	Merge pull request #4644 from hashicorp/ca-refactor connect/ca: rework initialization/root generation in providers	6 years ago
mkeeler	48d287ef69	Release v1.2.3	6 years ago
Paul Banks	74f2a80a42	Fix CA pruning when CA config uses string durations. (#4669 ) * Fix CA pruning when CA config uses string durations. The tl;dr here is: - Configuring LeafCertTTL with a string like "72h" is how we do it by default and should be supported - Most of our tests managed to escape this by defining them as time.Duration directly - Out actual default value is a string - Since this is stored in a map[string]interface{} config, when it is written to Raft it goes through a msgpack encode/decode cycle (even though it's written from server not over RPC). - msgpack decode leaves the string as a `[]uint8` - Some of our parsers required string and failed - So after 1 hour, a default configured server would throw an error about pruning old CAs - If a new CA was configured that set LeafCertTTL as a time.Duration, things might be OK after that, but if a new CA was just configured from config file, intialization would cause same issue but always fail still so would never prune the old CA. - Mostly this is just a janky error that got passed tests due to many levels of complicated encoding/decoding. tl;dr of the tl;dr: Yay for type safety. Map[string]interface{} combined with msgpack always goes wrong but we somehow get bitten every time in a new way :D We already fixed this once! The main CA config had the same problem so @kyhavlov already wrote the mapstructure DecodeHook that fixes it. It wasn't used in several places it needed to be and one of those is notw in `structs` which caused a dependency cycle so I've moved them. This adds a whole new test thta explicitly tests the case that broke here. It also adds tests that would have failed in other places before (Consul and Vaul provider parsing functions). I'm not sure if they would ever be affected as it is now as we've not seen things broken with them but it seems better to explicitly test that and support it to not be bitten a third time! * Typo fix * Fix bad Uint8 usage	6 years ago
Hans Hasselberg	8e235a72b4	Allow disabling the HTTP API again. (#4655 ) If you provide an invalid HTTP configuration consul will still start again instead of failing. But if you do so the build-in proxy won't be able to start which you might need for connect.	6 years ago
Kyle Havlovitz	5c7fbc284d	connect/ca: hash the consul provider ID and include isRoot	6 years ago
Pierre Souchay	1a906ef34e	Fix more unstable tests in agent and command	6 years ago
Kyle Havlovitz	c112a72880	connect/ca: some cleanup and reorganizing of the new methods	6 years ago
Pierre Souchay	2fe728c7bd	Ensure that Proxies ARE always cleaned up, event with DeregisterCriticalServiceAfter (#4649 ) This fixes https://github.com/hashicorp/consul/issues/4648	6 years ago
Matt Keeler	d3ee66eed4	Add ECS option to EDNS responses where appropriate (#4647 ) This implements parts of RFC 7871 where Consul is acting as an authoritative name server (or forwarding resolver when recursors are configured) If ECS opt is present in the request we will mirror it back and return a response with a scope of 0 (global) or with the same prefix length as the request (indicating its valid specifically for that subnet). We only mirror the prefix-length (non-global) for prepared queries as those could potentially use nearness checks that could be affected by the subnet. In the future we could get more sophisticated with determining the scope bits and allow for better caching of prepared queries that don’t rely on nearness checks. The other thing this does not do is implement the part of the ECS RFC related to originating ECS headers when acting as a intermediate DNS server (forwarding resolver). That would take a quite a bit more effort and in general provide very little value. Consul will currently forward the ECS headers between recursors and the clients transparently, we just don't originate them for non-ECS clients to get potentially more accurate "location aware" results.	6 years ago
Pierre Souchay	22500f242e	Fix unstable tests in agent, api, and command/watch	6 years ago
Mitchell Hashimoto	49b165965d	Merge pull request #4642 from hashicorp/f-ui-meta agent: aggregate service instance meta for UI purposes	6 years ago
Mitchell Hashimoto	b95348c4b1	agent: ExternalSources instead of Meta	6 years ago
Matt Keeler	cc8327ed9a	Ensure that errors setting up the DNS servers get propagated back to the shell (#4598 ) Fixes: #4578 Prior to this fix if there was an error binding to ports for the DNS servers the error would be swallowed by the gated log writer and never output. This fix propagates the DNS server errors back to the shell with a multierror.	6 years ago
Pierre Souchay	eddcf228ea	Implementation of Weights Data structures (#4468 ) * Implementation of Weights Data structures Adding this datastructure will allow us to resolve the issues #1088 and #4198 This new structure defaults to values: ``` { Passing: 1, Warning: 0 } ``` Which means, use weight of 0 for a Service in Warning State while use Weight 1 for a Healthy Service. Thus it remains compatible with previous Consul versions. * Implemented weights for DNS SRV Records * DNS properly support agents with weight support while server does not (backwards compatibility) * Use Warning value of Weights of 1 by default When using DNS interface with only_passing = false, all nodes with non-Critical healthcheck used to have a weight value of 1. While having weight.Warning = 0 as default value, this is probably a bad idea as it breaks ascending compatibility. Thus, we put a default value of 1 to be consistent with existing behaviour. * Added documentation for new weight field in service description * Better documentation about weights as suggested by @banks * Return weight = 1 for unknown Check states as suggested by @banks * Fixed typo (of -> or) in error message as requested by @mkeeler * Fixed unstable unit test TestRetryJoin * Fixed unstable tests * Fixed wrong Fatalf format in `testrpc/wait.go` * Added notes regarding DNS SRV lookup limitations regarding number of instances * Documentation fixes and clarification regarding SRV records with weights as requested by @banks * Rephrase docs	6 years ago
Kyle Havlovitz	546bdf8663	connect/ca: add Configure/GenerateRoot to provider interface	6 years ago
Mitchell Hashimoto	e9ea190df0	agent: aggregate service instance meta for UI purposes	6 years ago
Mitchell Hashimoto	99eb154f6f	agent: configure k8s go-discover	6 years ago
Martin	feb3ce4ee0	Use target service name instead of ID as connect proxy service name (#4620 )	6 years ago
Pierre Souchay	9a2ae6e8eb	Fixed more flaky tests in ./agent/consul (#4617 )	6 years ago
Pierre Souchay	92acdaa94c	Fixed flaky tests (#4626 )	6 years ago
Siva Prasad	ca35d04472	Adds a new command line flag -log-file for file based logging. (#4581 ) * Added log-file flag to capture Consul logs in a user specified file * Refactored code. * Refactored code. Added flags to rotate logs based on bytes and duration * Added the flags for log file and log rotation on the webpage * Fixed TestSantize from failing due to the addition of 3 flags * Introduced changes : mutex, data-dir log writes, rotation logic * Added test for logfile and updated the default log destination for docs * Log name now uses UnixNano * TestLogFile is now uses t.Parallel() * Removed unnecessary int64Val function * Updated docs to reflect default log name for log-file * No longer writes to data-dir and adds .log if the filename has no extension	6 years ago
Freddy	d7a404f2ee	Bugfix: Use "%#v" when formatting structs (#4600 )	6 years ago
Siva Prasad	b1a34f899f	TestAgentAntiEntropy: Wait until Consul service is up on the agent. (#4591 ) * Anti-Entropy test wait for Consul service added * Reverted some tests back to using WaitForLeader	6 years ago
Pierre Souchay	5e0218ccf4	Fix unit test TestOperatorAutopilotGetConfigCommand (#4594 )	6 years ago
Pierre Souchay	aea31d3c5d	Fixed unstable test TestUiNodeInfo (#4586 )	6 years ago
Pierre Souchay	b898131723	[BUGFIX] Avoid returning empty data on startup of a non-leader server (#4554 ) Ensure that DB is properly initialized when performing stale queries Addresses: - https://github.com/hashicorp/consul-replicate/issues/82 - https://github.com/hashicorp/consul/issues/3975 - https://github.com/hashicorp/consul-template/issues/1131	6 years ago
Miroslav Bagljas	3c23979afd	Fixes #4483 : Add support for Authorization: Bearer token Header (#4502 ) Added Authorization Bearer token support as per RFC6750 * appended Authorization header token parsing after X-Consul-Token * added test cases * updated website documentation to mention Authorization header * improve tests, improve Bearer parsing	6 years ago
Matt Keeler	e81c85c051	Fix #4515 : Segfault when serf_wan port was -1 but reconnect_time_wan was set (#4531 ) Fixes #4515 This just slightly refactors the logic to only attempt to set the serf wan reconnect timeout when the rest of the serf wan settings are configured - thus avoiding a segfault.	6 years ago
Kyle Havlovitz	e5e1f867e5	Merge branch 'master' into ca-snapshot-fix	6 years ago
Kyle Havlovitz	f186edc42c	fsm: add connect service config to snapshot/restore test	6 years ago
nickmy9729	beddf03b26	Added code to allow snapshot inclusion of NodeMeta (#4527 )	6 years ago
Kyle Havlovitz	b51d76f469	fsm: add missing CA config to snapshot/restore logic	6 years ago
Kyle Havlovitz	4b35d877ca	autopilot: don't follow the normal server removal rules for nonvoters	6 years ago
Kyle Havlovitz	ea14482376	Fix stats fetcher healthcheck RPCs not being independent	6 years ago
Pierre Souchay	0d6de257a2	Display more information about check being not properly added when it fails (#4405 ) * Display more information about check being not properly added when it fails It follows an incident where we add lots of error messages: [WARN] consul.fsm: EnsureRegistration failed: failed inserting check: Missing service registration That seems related to Consul failing to restart on respective agents. Having Node information as well as service information would help diagnose the issue. * Renamed ensureCheckIfNodeMatches() as requested by @banks	6 years ago
Freddy	6d43d24edb	Improve reliability of tests with TestAgent (#4525 ) - Add WaitForTestAgent to tests flaky due to missing serfHealth registration - Fix bug in retries calling Fatalf with *testing.T - Convert TestLockCommand_ChildExitCode to table driven test	6 years ago
Pierre Souchay	ef3b81ab13	Allow to rename nodes with IDs, will fix #3974 and #4413 (#4415 ) * Allow to rename nodes with IDs, will fix #3974 and #4413 This change allow to rename any well behaving recent agent with an ID to be renamed safely, ie: without taking the name of another one with case insensitive comparison. Deprecated behaviour warning ---------------------------- Due to asceding compatibility, it is still possible however to "take" the name of another name by not providing any ID. Note that when not providing any ID, it is possible to have 2 nodes having similar names with case differences, ie: myNode and mynode which might lead to DB corruption on Consul server side and lead to server not properly restarting. See #3983 and #4399 for Context about this change. Disabling registration of nodes without IDs as specified in #4414 should probably be the way to go eventually. * Removed the case-insensitive search when adding a node within the else block since it breaks the test TestAgentAntiEntropy_Services While the else case is probably legit, it will be fixed with #4414 in a later release. * Added again the test in the else to avoid duplicated names, but enforce this test only for nodes having IDs. Thus most tests without any ID will work, and allows us fixing * Added more tests regarding request with/without IDs. `TestStateStore_EnsureNode` now test registration and renaming with IDs `TestStateStore_EnsureNodeDeprecated` tests registration without IDs and tests removing an ID from a node as well as updated a node without its ID (deprecated behaviour kept for backwards compatibility) * Do not allow renaming in case of conflict, including when other node has no ID * Fixed function GetNodeID that was not working due to wrong type when searching node from its ID Thus, all tests about renaming were not working properly. Added the full test cas that allowed me to detect it. * Better error messages, more tests when nodeID is not a valid UUID in GetNodeID() * Added separate TestStateStore_GetNodeID to test GetNodeID. More complete test coverage for GetNodeID * Added new unit test `TestStateStore_ensureNoNodeWithSimilarNameTxn` Also fixed comments to be clearer after remarks from @banks * Fixed error message in unit test to match test case * Use uuid.ParseUUID to parse Node.ID as requested by @mkeeler	6 years ago
Siva Prasad	c88900aaa9	PR to fix TestAgent_IndexChurn and TestPreparedQuery_Wrapper. (#4512 ) * Fixes TestAgent_IndexChurn * Fixes TestPreparedQuery_Wrapper * Increased sleep in agent_test for IndexChurn to 500ms * Made the comment about joinWAN operation much less of a cliffhanger	6 years ago
Armon Dadgar	4f1fd34e9e	consul: Update buffer sizes	6 years ago
Siva Prasad	288d350a73	Revert "CA initialization while boostrapping and TestLeader_ChangeServerID fix." (#4497 ) * Revert "BUGFIX: Unit test relying on WaitForLeader() did not work due to wrong test (#4472)" This reverts commit `cec5d72396`. * Revert "CA initialization while boostrapping and TestLeader_ChangeServerID fix. (#4493)" This reverts commit `589b589b53`.	6 years ago
Pierre Souchay	cec5d72396	BUGFIX: Unit test relying on WaitForLeader() did not work due to wrong test (#4472 ) - Improve resilience of testrpc.WaitForLeader() - Add additionall retry to CI - Increase "go test" timeout to 8m - Add wait for cluster leader to several tests in the agent package - Add retry to some tests in the api and command packages	6 years ago
Siva Prasad	589b589b53	CA initialization while boostrapping and TestLeader_ChangeServerID fix. (#4493 ) * connect: fix an issue with Consul CA bootstrapping being interrupted * streamline change server id test	6 years ago
Siva Prasad	865068a358	DNS : Fixes recursors answering the DNS query to properly return the correct response. (#4461 ) * Fixes the DNS recursor properly resolving the requests * Added a test case for the recursor bug * Refactored code && added a test case for all failing recursors * Inner indentation moved into else if check	6 years ago
Paul Banks	71dd3b408a	Fixes memory leak when blocking on /event/list (#4482 )	6 years ago
mkeeler	e716d1b5f8	Release v1.2.2	6 years ago
Matt Keeler	870a6ad6a8	Handle resolving proxy tokens when parsing HTTP requests (#4453 ) Fixes: #4441 This fixes the issue with Connect Managed Proxies + ACLs being broken. The underlying problem was that the token parsed for most http endpoints was sent untouched to the servers via the RPC request. These changes make it so that at the HTTP endpoint when parsing the token we additionally attempt to convert potential proxy tokens into regular tokens before sending to the RPC endpoint. Proxy tokens are only valid on the agent with the managed proxy so the resolution has to happen before it gets forwarded anywhere.	6 years ago
Matt Keeler	0e0227792b	Gossip tuneables (#4444 ) Expose a few gossip tuneables for both lan and wan interfaces gossip_nodes gossip_interval probe_timeout probe_interval retransmit_mult suspicion_mult	6 years ago
Kyle Havlovitz	fa0d8aff33	fix inconsistency in TestConnectCAConfig_GetSet	6 years ago
Paul Banks	8dd50d5b2d	Add config option to disable HTTP printable char path check (#4442 )	6 years ago
Kyle Havlovitz	ed87949385	Merge pull request #4400 from hashicorp/leaf-cert-ttl Add configurable leaf cert TTL to Connect CA	6 years ago
Kyle Havlovitz	f67a4d59c0	connect/ca: simplify passing of leaf cert TTL	6 years ago
Siva Prasad	f4a1c381a5	Vendoring update for go-discover. (#4412 ) * New Providers added and updated vendoring for go-discover * Vendor.json formatted using make vendorfmt * Docs/Agent/auto-join: Added documentation for the new providers introduced in this PR * Updated the golang.org/x/sys/unix in the vendor directory * Agent: TestGoDiscoverRegistration updated to reflect the addition of new providers * Deleted terraform.tfstate from vendor. * Deleted terraform.tfstate.backup Deleted terraform state file artifacts from unknown runs. * Updated x/sys/windows vendor for Windows binary compilation	6 years ago
Paul Banks	8cbeb29e73	Fixes #4421 : General solution to stop blocking queries with index 0 (#4437 ) * Fix theoretical cache collision bug if/when we use more cache types with same result type * Generalized fix for blocking query handling when state store methods return zero index * Refactor test retry to only affect CI * Undo make file merge * Add hint to error message returned to end-user requests if Connect is not enabled when they try to request cert * Explicit error for Roots endpoint if connect is disabled * Fix tests that were asserting old behaviour	6 years ago
Paul Banks	5635227fa6	Allow config-file based Service Definitions for unmanaged proxies and Connect-natice apps. (#4443 )	6 years ago
Paul Banks	d5e934f9ff	Ooops that was meant to be to a branch no master... EMORECOFFEE Revert "Add config option to disable HTTP printable char path check" This reverts commit `eebe45a47b`.	6 years ago
Paul Banks	eebe45a47b	Add config option to disable HTTP printable char path check	6 years ago
Paul Banks	e954450dec	Merge pull request #4353 from azam/add-serf-lan-wan-port-args Make RPC, Serf LAN, Serf WAN port configurable from CLI	6 years ago
Kyle Havlovitz	ce10de036e	connect/ca: check LeafCertTTL when rotating expired roots	6 years ago
Mitchell Hashimoto	7fa6bb022f	Merge pull request #4320 from hashicorp/f-alias-check Add "Alias" Check Type	6 years ago
azam	342bcb1c24	Make Serf LAN & WAN port configurable from CLI Make RPC port accessible to CLI Add tests and documentation for server-port, serf-lan-port, serf-wan-port CLI arguments	6 years ago
Mitchell Hashimoto	b3854fdd28	agent/local: silly spacing on select statements	6 years ago
Mitchell Hashimoto	8c72bb0cdf	agent/local: address remaining test feedback	6 years ago
Matt Keeler	560c9c26f7	Use the agent logger instead of log module	6 years ago
Matt Keeler	ca5851318d	Update a couple erroneous tests.	6 years ago
Mitchell Hashimoto	9f128e40d6	agent/local: don't use time.After in test since notify is instant	6 years ago
Matt Keeler	3fe5f566f2	Persist proxies from config files Also change how loadProxies works. Now it will load all persisted proxies into a map, then when loading config file proxies will look up the previous proxy token in that map.	6 years ago
Kyle Havlovitz	d6ca015a42	connect/ca: add configurable leaf cert TTL	6 years ago
Matt Keeler	c891e264ca	Fix issue with choosing a client addr that is 0.0.0.0 or ::	6 years ago
Mitchell Hashimoto	9a90400821	agent/checks: prevent overflow of backoff	6 years ago
Mitchell Hashimoto	d6ecd97d1d	agent: use the correct ACL token for alias checks	6 years ago
Mitchell Hashimoto	f97bfd5be8	agent: address some basic feedback	6 years ago
Mitchell Hashimoto	19ced12668	agent: alias checks have no interval	6 years ago
Mitchell Hashimoto	5bc27feb0b	agent/structs: check is alias if node is empty	6 years ago
Mitchell Hashimoto	36e330941a	agent/checks: support node-only checks	6 years ago

... 3 4 5 6 7 ...

1593 Commits (9f58504f1c4e0dce5e43c72da6ae6dafa09f2849)