591 Commits (CC-5545/upgrade-hds-packages)

Author	SHA1	Message	Date
Daniel Nephin	5b6f806f4f	agent: addServiceIternalRequest ... Move fields that are only relevant for addServiceInternal onto a new struct and embed AddServiceRequest.	4 years ago
Daniel Nephin	3d39359bcb	agent: move deprecated AddServiceFromSource to a test file ... The method is only used in tests, and only exists for legacy calls. There was one other package which used this method in tests. Export the AddServiceRequest and a couple of its fields so the new function can be used in those tests.	4 years ago
Daniel Nephin	e44fd1cb92	agent: use a single method for Agent.AddService	4 years ago
Daniel Nephin	6757231b82	agent: rename AddService->AddServiceFromSource ... In preparation for extracting a single AddService func that accepts a request struct.	4 years ago
Daniel Nephin	e66af1a559	agent/consuk: Rename RPCRate -> RPCRateLimit ... so that the field name is consistent across config structs.	4 years ago
Daniel Nephin	5684223e36	agent/consul: make Client/Server config reloading more obvious ... I believe this commit also fixes a bug. Previously RPCMaxConnsPerClient was not being re-read from the RuntimeConfig, so passing it to Server.ReloadConfig was never changing the value. Also improve the test runtime by not doing a lot of unnecessary work.	4 years ago
Daniel Nephin	a04cefaa28	Remove an unnecessary else	4 years ago
Daniel Nephin	4b8b2a4291	xds: remove Server.Initialize ... Requiring a call to initialize to set a single field is not really substantially different from having to set that field to a value.	4 years ago
Daniel Nephin	375aed5ed6	xds: Pass in logger ... small cleanup in tests	4 years ago
Daniel Nephin	d64425d2e4	Merge pull request #9213 from hashicorp/dnephin/resolve-tokens-take-2 ... acl: Remove some unused things and document delegate method	4 years ago
Michael Montgomery	e4f603dfae	Merge branch 'master' into 6074-allow-config-MaxHeaderBytes	4 years ago
Kenia	27f6899ec8	Create consul version metric with version label (#9350) ... * create consul version metric with version label * agent/agent.go: add pre-release Version as well as label Co-Authored-By: Radha13 <kumari.radha3@gmail.com> * verion and pre-release version labels. * hyphen/- breaks prometheus * Add Prometheus gauge defintion for version metric * Add new metric to telemetry docs Co-authored-by: Radha Kumari <kumari.radha3@gmail.com> Co-authored-by: Aestek <thib.gilles@gmail.com> Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>	4 years ago
Michael Montgomery	585c84e9ff	Merge branch 'master' into 6074-allow-config-MaxHeaderBytes	4 years ago
Daniel Nephin	738bf9efdc	agent: fix bug with multiple listeners ... Previously the listener was being passed to a closure in a loop without capturing the loop variable. The result is only the last listener is used, so the http/https servers only listen on one address. This problem is fixed by capturing the variable by passing it into a function.	4 years ago
Daniel Nephin	0ee86935f0	Remove two unused delegate methods	4 years ago
Matt Keeler	66fd23d67f	Refactor to call non-voting servers read replicas (#9191) ... Co-authored-by: Kit Patella <kit@jepsen.io>	4 years ago
Michael Montgomery	5b6ac035ff	Resolves #6074. Adds new option to configure HTTP Server's MaxHeaderBytes with option `-http-max-header-bytes` ... Adds tests for behavior	4 years ago
Daniel Nephin	bd44952c2e	streaming: disable streaming when requesting connect events ... Until the correct events are created for terminating gateways.	4 years ago
Daniel Nephin	853667e7d8	health: change the name of UseStreamingBackend config ... Remove it from the cache section, and update the docs.	4 years ago
Daniel Nephin	3a55c30a05	Merge pull request #8924 from ShimmerGlass/fix-sidecar-deregister-after-restart ... Fix: service LocallyRegisteredAsSidecar property is not persisted	4 years ago
Mathilde Gilles	1c8369b3c3	Fix: service LocallyRegisteredAsSidecar property is not persisted ... When a service is deregistered, we check whever matching services were registered as sidecar along with it and deregister them as well. To determine if a service is indeed a sidecar we check the structs.ServiceNode.LocallyRegisteredAsSidecar property. However, to avoid interal API leakage, it is excluded from JSON serialization, meaning it is not persisted to disk either. When the agent is restarted, this property lost and sidecars are no longer deregistered along with their parent service. To fix this, we now specifically save this property in the persisted service file.	4 years ago
Daniel Nephin	ea77eccb14	Merge pull request #8825 from hashicorp/streaming/add-config ... streaming: add config and docs	4 years ago
Daniel Nephin	e7d505dc33	config: add field for enabling streaming in the client ... agent: register the new streaming cache-type	4 years ago
Kit Patella	adeabf2399	Merge pull request #8877 from hashicorp/mkcp/telemetry/consul.api.http ... Add flag for disabling 1.9 metrics backwards compatibility and warnings when set to default	4 years ago
Matt Keeler	38f5ddce2a	Add per-agent reconnect timeouts (#8781) ... This allows for client agent to be run in a more stateless manner where they may be abruptly terminated and not expected to come back. If advertising a per-agent reconnect timeout using the advertise_reconnect_timeout configuration when that agent leaves, other agents will wait only that amount of time for the agent to come back before reaping it. This has the advantageous side effect of causing servers to deregister the node/services/checks for that agent sooner than if the global reconnect_timeout was used.	4 years ago
Daniel Nephin	b93577c94f	config: add field for enabling streaming RPC endpoint	4 years ago
Kit Patella	7fe2f80b4b	add config flag to disable 1.9 metrics backwards compatibility. Add warnings on start and reload on default value	4 years ago
Daniel Nephin	529f252d5c	rpcclient: Add health.Client and use it in http and dns ... This new package provides a client agent implementation of an interface for fetching the health of services. This approach has a number of benefits: 1. It provides a much more explicit interface. Instead of everything dependency on `RPC()` and `Cache.Get()` for many unrelated things they can depend on a type that are named according to the behaviour it provides. 2. It gives us a single place to vary the behaviour and migrate to a new form of RPC (gRPC). The current implementation has two options (cache, or direct RPC), and in the future we will have more. It is also a great opporunity to start adding `context.Context` args to these operations, which in the future will allow us to cancel the operations. 3. As a concequence of the first, in the Server agent where we make these calls we can replace the current in-memory RPC calls with a thin adapter for the real method. This removes the `net/rpc` machinery from the call in places where it is not needed. This new package is quite small right now, but I think we can expect it to grow to a more reasonable size as other RPC calls are replaced. This change also happens to replace two very similar implementations with a single implementation.	4 years ago
Paul Banks	e4db845246	Refactor uiserver to separate package, cleaner Reloading	4 years ago
Paul Banks	f6d55e1d25	Fix reload test; address other PR feedback	4 years ago
Paul Banks	526bab6164	Add config changes for UI metrics	4 years ago
R.B. Boyer	7eef25daf5	agent: when enable_central_service_config is enabled ensure agent reload doesn't revert check state to critical (#8747) ... Likely introduced when #7345 landed.	4 years ago
Daniel Nephin	c18516ad7d	Merge pull request #8680 from hashicorp/dnephin/replace-consul-opts-with-base-deps ... agent: Repalce ConsulOptions with a new struct from agent.BaseDeps	4 years ago
Daniel Nephin	282fbdfa75	api: rename HTTPServer to HTTPHandlers ... Resolves a TODO about naming. This type is a set of handlers for an http.Server, it is not itself a Server. It provides http.Handler functions.	4 years ago
Daniel Nephin	cdd392d77f	agent/consul: pass dependencies directly from agent ... In an upcoming change we will need to pass a grpc.ClientConnPool from BaseDeps into Server. While looking at that change I noticed all of the existing consulOption fields are already on BaseDeps. Instead of duplicating the fields, we can create a struct used by agent/consul, and use that struct in BaseDeps. This allows us to pass along dependencies without translating them into different representations. I also looked at moving all of BaseDeps in agent/consul, however that created some circular imports. Resolving those cycles wouldn't be too bad (it was only an error in agent/consul being imported from cache-types), however this change seems a little better by starting to introduce some structure to BaseDeps. This change is also a small step in reducing the scope of Agent. Also remove some constants that were only used by tests, and move the relevant comment to where the live configuration is set. Removed some validation from NewServer and NewClient, as these are not really runtime errors. They would be code errors, which will cause a panic anyway, so no reason to handle them specially here.	4 years ago
Daniel Nephin	4c9ed41eab	Merge pull request #8554 from hashicorp/dnephin/agent-setup-persisted-tokens ... agent: move token persistence from agent into token.Store	4 years ago
Daniel Nephin	6ca45e1a61	agent: add apiServers type for managing HTTP servers ... Remove Server field from HTTPServer. The field is no longer used.	4 years ago
Daniel Nephin	330be5b740	agent/token: Move token persistence out of agent ... And into token.Store. This change isolates any awareness of token persistence in a single place. It is a small step in allowing Agent.New to accept its dependencies.	4 years ago
Matt Keeler	91d680b830	Merge of auto-config and auto-encrypt code (#8523) ... auto-encrypt is now handled as a special case of auto-config. This also is moving all the cert-monitor code into the auto-config package.	4 years ago
Daniel Nephin	72bf350069	Merge pull request #8552 from pierresouchay/reload_cache_throttling_config ... Ensure that Cache options are reloaded when `consul reload` is performed	4 years ago
R.B. Boyer	74d5df7c7a	xds: use envoy's rbac filter to handle intentions entirely within envoy (#8569)	4 years ago
Matt Keeler	f97cc0445a	Move RPC router from Client/Server and into BaseDeps (#8559) ... This will allow it to be a shared component which is needed for AutoConfig	4 years ago
Pierre Souchay	d2be9d38da	Ensure that Cache options are reloaded when `consul reload` is performed. ... This will apply cache throttling parameters are properly applied: * cache.EntryFetchMaxBurst * cache.EntryFetchRate When values are updated, a log is displayed in info.	4 years ago
Daniel Nephin	e16375216d	config: use logging.Config in RuntimeConfig ... To add structure to RuntimeConfig, and remove the need to translate into a third type.	4 years ago
Daniel Nephin	f2373a5575	logging: move init of grpclog ... This line initializes global state. Moving it out of the constructor and closer to where logging is setup helps keep related things together.	4 years ago
Daniel Nephin	63bad36de7	testing: disable global metrics sink in tests ... This might be better handled by allowing configuration for the InMemSink interval and retail, and disabling the global. For now this is a smaller change to remove the goroutine leak caused by tests because go-metrics does not provide any way of shutting down the global goroutine.	4 years ago
Daniel Nephin	5d4df54296	agent: extract dependency creation from New ... With this change, Agent.New() accepts many of the dependencies instead of creating them in New. Accepting fully constructed dependencies from a constructor makes the type easier to test, and easier to change. There are still a number of dependencies created in Start() which can be addressed in a follow up.	4 years ago
Daniel Nephin	35f1ecee0b	config: Move remote-script-checks warning to config ... Previously it was done in Agent.Start, but it can be done much earlier	4 years ago
Daniel Nephin	27b36bfc4e	config: move NodeName validation to config validation ... Previsouly it was done in Agent.Start, which is much later then it needs to be. The new 'dns' package was required, because otherwise there would be an import cycle. In the future we should move more of the dns server into the dns package.	4 years ago
Daniel Nephin	399c77dfb6	agent: rename vars in newConsulConfig ... 'base' is a bit misleading, since it is the return value. Renamed to cfg.	4 years ago
Daniel Nephin	7b5b170a0d	agent: Move setupKeyring functions to keyring.go ... There are a couple reasons for this change: 1. agent.go is way too big. Smaller files makes code eaasier to read because tools that show usage also include filename which can give a lot more context to someone trying to understand which functions call other functions. 2. these two functions call into a large number of functions already in keyring.go.	4 years ago
Daniel Nephin	9919e5dfa5	agent: unmethod consulConfig ... To allow us to move newConsulConfig out of Agent.	4 years ago
Daniel Nephin	8f596f5551	Fix conflict in merged PRs ... One PR renamed the var from config->cfg, and another used the old name config, which caused the build to fail on master.	4 years ago
Daniel Nephin	190fcc14a3	Merge pull request #8463 from hashicorp/dnephin/unmethod-make-node-id ... agent: convert NodeID methods to functions	4 years ago
Daniel Nephin	37eacf8192	auto-config: reduce awareness of config ... This is a small step to allowing Agent to accept its dependencies instead of creating them in New. There were two fields in autoconfig.Config that were used exclusively to load config. These were replaced with a single function, allowing us to move LoadConfig back to the config package. Also removed the WithX functions for building a Config. Since these were simple assignment, it appeared we were not getting much value from them.	4 years ago
Daniel Nephin	875d8bde42	agent: convert NodeID methods to functions ... Making these functions allows us to cleanup how an agent is initialized. They only make use of a config and a logger, so they do not need to be agent methods. Also cleanup the testing to use t.Run and require.	4 years ago
Daniel Nephin	0738eb8596	Extract nodeID functions to a different file ... In preparation for turning them into functions. To reduce the scope of Agent, and refactor how Agent is created and started.	4 years ago
Daniel Nephin	38980ebb4c	config: Make Source an interface ... This will allow us to accept config from auto-config without needing to go through a serialziation cycle.	4 years ago
Daniel Nephin	3b82ad0955	Rename NewClient/NewServer ... Now that duplicate constructors have been removed we can use the shorter names for the single constructor.	4 years ago
Daniel Nephin	0420d91cdd	Remove LogOutput from Agent ... Now that it is no longer used, we can remove this unnecessary field. This is a pre-step in cleanup up RuntimeConfig->Consul.Config, which is a pre-step to adding a gRPCHandler component to Server for streaming. Removing this field also allows us to remove one of the return values from logging.Setup.	4 years ago
Daniel Nephin	5acf01ceeb	Remove LogOutput from Server	4 years ago
Daniel Nephin	e8ee2cf2f7	Pass a logger to ConnPool and yamux, instead of an io.Writer ... Allowing us to remove the LogOutput field from config.	4 years ago
Daniel Nephin	ed8210fe4d	api: Use a Logger instead of an io.Writer in api.Watch ... So that we can pass around only a Logger, not a LogOutput	4 years ago
Daniel Nephin	1e17a0c3e1	config: Remove unused field	4 years ago
Matt Keeler	1a78cf9b4c	Ensure certificates retrieved through the cache get persisted with auto-config (#8409)	4 years ago
Matt Keeler	34034b76f5	Agent Auto Config: Implement Certificate Generation (#8360) ... Most of the groundwork was laid in previous PRs between adding the cert-monitor package to extracting the logic of signing certificates out of the connect_ca_endpoint.go code and into a method on the server. This also refactors the auto-config package a bit to split things out into multiple files.	4 years ago
Pierre Souchay	505de6dc29	Added ratelimit to handle throtling cache (#8226) ... This implements a solution for #7863 It does: Add a new config cache.entry_fetch_rate to limit the number of calls/s for a given cache entry, default value = rate.Inf Add cache.entry_fetch_max_burst size of rate limit (default value = 2) The new configuration now supports the following syntax for instance to allow 1 query every 3s: command line HCL: -hcl 'cache = { entry_fetch_rate = 0.333}' in JSON { "cache": { "entry_fetch_rate": 0.333 } }	4 years ago
Matt Keeler	2ee9fe0a4d	Move generation of the CA Configuration from the agent code into a method on the RuntimeConfig (#8363) ... This allows this to be reused elsewhere.	4 years ago
Matt Keeler	9da8c51ac5	Fix issue with changing the agent token causing failure to renew the auto-encrypt certificate ... The fallback method would still work but it would get into a state where it would let the certificate expire for 10s before getting a new one. And the new one used the less secure RPC endpoint. This is also a pretty large refactoring of the auto encrypt code. I was going to write some tests around the certificate monitoring but it was going to be impossible to get a TestAgent configured in such a way that I could write a test that ran in less than an hour or two to exercise the functionality. Moving the certificate monitoring into its own package will allow for dependency injection and in particular mocking the cache types to control how it hands back certificates and how long those certificates should live. This will allow for exercising the main loop more than would be possible with it coupled so tightly with the Agent.	4 years ago
Daniel Nephin	653c938edc	watch: extract makeWatchPlan to facilitate testing ... There is a bug in here now that slices in opaque config are unsliced. But to test that bug fix we need a function that can be easily tested.	4 years ago
Daniel Nephin	f22f3d300d	Merge pull request #8231 from hashicorp/dnephin/unembed-HTTPServer-Server ... agent/http: un-embed the http.Server	4 years ago
Daniel Nephin	df4088291c	agent/http: Update TestSetupHTTPServer_HTTP2 ... To remove the need to store the http.Server. This will allow us to remove the http.Server field from the HTTPServer struct.	4 years ago
Daniel Nephin	5247ef4c70	Remove ACLsEnabled from delegate interface ... In all cases (oss/ent, client/server) this method was returning a value from config. Since the value is consistent, it doesn't need to be part of the delegate interface.	4 years ago
Pierre Souchay	20d1ea7d2d	Upgrade go-connlimit to v0.3.0 / return http 429 on too many connections (#8221) ... Fixes #7527 I want to highlight this and explain what I think the implications are and make sure we are aware: * `HTTPConnStateFunc` closes the connection when it is beyond the limit. `Close` does not block. * `HTTPConnStateFuncWithDefault429Handler(10 * time.Millisecond)` blocks until the following is done (worst case): 1) `conn.SetDeadline(10*time.Millisecond)` so that 2) `conn.Write(429error)` is guaranteed to timeout after 10ms, so that the http 429 can be written and 3) `conn.Close` can happen The implication of this change is that accepting any new connection is worst case delayed by 10ms. But only after a client reached the limit already.	4 years ago
Daniel Nephin	a5e45defb1	agent/http: un-embed the HTTPServer ... The embedded HTTPServer struct is not used by the large HTTPServer struct. It is used by tests and the agent. This change is a small first step in the process of removing that field. The eventual goal is to reduce the scope of HTTPServer making it easier to test, and split into separate packages.	4 years ago
Matt Keeler	a5a9560bbd	Initialize the agent leaf cert cache result with a state to prevent unnecessary second certificate signing	4 years ago
Matt Keeler	39b567a55a	Fix auto_encrypt IP/DNS SANs ... The initial auto encrypt CSR wasn’t containing the user supplied IP and DNS SANs. This fixes that. Also We were configuring a default :: IP SAN. This should be ::1 instead and was fixed.	4 years ago
Daniel Nephin	a891ee8428	Merge pull request #8176 from hashicorp/dnephin/add-linter-unparam-1 ... lint: add unparam linter and fix some of the issues	4 years ago
Matt Keeler	25a4f3c83b	Allow cancelling blocking queries in response to shutting down.	4 years ago
Daniel Nephin	010a609912	Fix a bunch of unparam lint issues	4 years ago
Matt Keeler	e2cfa93f02	Don’t leak metrics go routines in tests (#8182)	4 years ago
Matt Keeler	d6e05482ab	Allow cancelling startup when performing auto-config (#8157) ... Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>	5 years ago
Matt Keeler	3dbbd2d37d	Implement Client Agent Auto Config ... There are a couple of things in here. First, just like auto encrypt, any Cluster.AutoConfig RPC will implicitly use the less secure RPC mechanism. This drastically modifies how the Consul Agent starts up and moves most of the responsibilities (other than signal handling) from the cli command and into the Agent.	5 years ago
Matt Keeler	8b7d669a27	Allow the Agent its its child Client/Server to share a connection pool ... This is needed so that we can make an AutoConfig RPC at the Agent level prior to creating the Client/Server.	5 years ago
Matt Keeler	51c3a605ad	Merge pull request #8035 from hashicorp/feature/auto-config/server-rpc	5 years ago
Matt Keeler	9b01f9423c	Implement the insecure version of the Cluster.AutoConfig RPC endpoint ... Right now this is only hooked into the insecure RPC server and requires JWT authorization. If no JWT authorizer is setup in the configuration then we inject a disabled “authorizer” to always report that JWT authorization is disabled.	5 years ago
Daniel Nephin	d345cd8d30	ci: Add ineffsign linter ... And fix an additional ineffective assignment that was not caught by staticcheck	5 years ago
R.B. Boyer	ffb9c7d6f7	acl: remove the deprecated `acl_enforce_version_8` option (#7991) ... Fixes #7292	5 years ago
Jono Sosulska	c554ba9e10	Replace whitelist/blacklist terminology with allowlist/denylist (#7971) ... * Replace whitelist/blacklist terminology with allowlist/denylist	5 years ago
Daniel Nephin	c88fae0aac	ci: Add staticcheck and fix most errors ... Three of the checks are temporarily disabled to limit the size of the diff, and allow us to enable all the other checks in CI. In a follow up we can fix the issues reported by the other checks one at a time, and enable them.	5 years ago
Pierre Souchay	d6649e42af	Stop all watches before shuting down anything dring shutdown. (#7526) ... This will prevent watches from being triggered. ```changelog * fix(agent): stop all watches before shuting down ```	5 years ago
Pierre Souchay	e9d176db2a	Allow to restrict servers that can join a given Serf Consul cluster. (#7628) ... Based on work done in https://github.com/hashicorp/memberlist/pull/196 this allows to restrict the IP ranges that can join a given Serf cluster and be a member of the cluster. Restrictions on IPs can be done separatly using 2 new differents flags and config options to restrict IPs for LAN and WAN Serf.	5 years ago
Matt Keeler	acccdbe45c	Fix identity resolution on clients and in secondary dcs (#7862) ... Previously this happened to be using the method on the Server/Client that was meant to allow the ACLResolver to locally resolve tokens. On Servers that had tokens (primary or secondary dc + token replication) this function would lookup the token from raft and return the ACLIdentity. On clients this was always a noop. We inadvertently used this function instead of creating a new one when we added logging accessor ids for permission denied RPC requests. With this commit, a new method is used for resolving the identity properly via the ACLResolver which may still resolve locally in the case of being on a server with tokens but also supports remote token resolution.	5 years ago
Kyle Havlovitz	f14c54e25e	Add TLS option and DNS SAN support to ingress config ... xds: Only set TLS context for ingress listener when requested	5 years ago
Matt Keeler	7a4c73acaf	Updates to allow for using an enterprise specific token as the agents token ... This is needed to allow for managed Consul instances to register themselves in the catalog with one of the managed service provider tokens.	5 years ago
Matt Keeler	bec3fb7c18	Some boilerplate to allow for ACL Bootstrap disabling configurability	5 years ago
Kit Patella	e2467f4b2c	Merge pull request #7656 from hashicorp/feature/audit/oss-merge ... agent: stub out auditing functionality in OSS	5 years ago
Kit Patella	3b105435b8	agent,config: port enterprise only fields to embedded enterprise structs	5 years ago
Daniel Nephin	5fe7043439	agent/cache: Make all cache options RegisterOptions ... Previously the SupportsBlocking option was specified by a method on the type, and all the other options were specified from RegisterOptions. This change moves RegisterOptions to a method on the type, and moves SupportsBlocking into the options struct. Currently there are only 2 cache-types. So all cache-types can implement this method by embedding a struct with those predefined values. In the future if a cache type needs to be registered more than once with different options it can remove the embedded type and implement the method in a way that allows for paramaterization.	5 years ago
Kit Patella	927f584761	agent: stub out auditing functionality in OSS	5 years ago
Kyle Havlovitz	e9e8c0e730	Ingress Gateways for TCP services (#7509) ... * Implements a simple, tcp ingress gateway workflow This adds a new type of gateway for allowing Ingress traffic into Connect from external services. Co-authored-by: Chris Piraino <cpiraino@hashicorp.com>	5 years ago
Daniel Nephin	f46d1b5c94	agent/structs: Remove ServiceID.Init and CheckID.Init ... The Init method provided the same functionality as the New constructor. The constructor is both more widely used, and more idiomatic, so remove the Init method. This change is in preparation for fixing printing of these IDs.	5 years ago
Daniel Nephin	329d76fd0e	Remove SnapshotRPC passthrough ... The caller has access to the delegate, so we do not gain anything by wrapping the call in Agent.	5 years ago
Pierre Souchay	2a8bf45e38	agent: show warning when enable_script_checks is enabled without safty net (#7437) ... In order to enforce a bit security on Consul agents, add a new method in agent to highlight possible security issues. This does not return an error for now, but might in the future. For now, it detects issues such as: https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/ This would display this kind of messages: ``` 2020-03-11T18:27:49.873+0100 [ERROR] agent: [SECURITY] issue: error="using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/" ```	5 years ago
Andy Lindeman	fb0a990e4d	agent: rewrite checks with proxy address, not local service address (#7518) ... Exposing checks is supposed to allow a Consul agent bound to a different IP address (e.g., in a different Kubernetes pod) to access healthchecks through the proxy while the underlying service binds to localhost. This is an important security feature that makes sure no external traffic reaches the service except through the proxy. However, as far as I can tell, this is subtly broken in the case where the Consul agent cannot reach the proxy over localhost. If a proxy is configured with: `{ LocalServiceAddress: "127.0.0.1", Checks: true }`, as is typical with a sidecar proxy, the Consul checks are currently rewritten to `127.0.0.1:<random port>`. A Consul agent that does not share the loopback address cannot reach this address. Just to make sure I was not misunderstanding, I tried configuring the proxy with `{ LocalServiceAddress: "<pod ip>", Checks: true }`. In this case, while the checks are rewritten as expected and the agent can reach the dynamic port, the proxy can no longer reach its backend because the traffic is no longer on the loopback interface. I think rewriting the checks to use `proxy.Address`, the proxy's own address, is more correct in this case. That is the IP where the proxy can be reached, both by other proxies and by a Consul agent running on a different IP. The local service address should continue to use `127.0.0.1` in most cases.	5 years ago
Shaker Islam	ac309d55f4	docs: document exported functions in agent.go (closes #7101) (#7366) ... and fix one linter error	5 years ago
Daniel Nephin	231c99f7b4	Document Agent.LogOutput	5 years ago
Daniel Nephin	bb8833a2d5	agent: Remove unused Encrypted from interface ... It appears to be unused. It looks like it has been around a while, I geuss at some point we stopped using this method.	5 years ago
Daniel Nephin	266bdf7465	agent: Remove xdsServer field ... The field is only referenced from a single method, it can be a local var	5 years ago
R.B. Boyer	6adad71125	wan federation via mesh gateways (#6884) ... This is like a Möbius strip of code due to the fact that low-level components (serf/memberlist) are connected to high-level components (the catalog and mesh-gateways) in a twisty maze of references which make it hard to dive into. With that in mind here's a high level summary of what you'll find in the patch: There are several distinct chunks of code that are affected: * new flags and config options for the server * retry join WAN is slightly different * retry join code is shared to discover primary mesh gateways from secondary datacenters * because retry join logic runs in the *agent* and the results of that operation for primary mesh gateways are needed in the *server* there are some methods like `RefreshPrimaryGatewayFallbackAddresses` that must occur at multiple layers of abstraction just to pass the data down to the right layer. * new cache type `FederationStateListMeshGatewaysName` for use in `proxycfg/xds` layers * the function signature for RPC dialing picked up a new required field (the node name of the destination) * several new RPCs for manipulating a FederationState object: `FederationState:{Apply,Get,List,ListMeshGateways}` * 3 read-only internal APIs for debugging use to invoke those RPCs from curl * raft and fsm changes to persist these FederationStates * replication for FederationStates as they are canonically stored in the Primary and replicated to the Secondaries. * a special derivative of anti-entropy that runs in secondaries to snapshot their local mesh gateway `CheckServiceNodes` and sync them into their upstream FederationState in the primary (this works in conjunction with the replication to distribute addresses for all mesh gateways in all DCs to all other DCs) * a "gateway locator" convenience object to make use of this data to choose the addresses of gateways to use for any given RPC or gossip operation to a remote DC. This gets data from the "retry join" logic in the agent and also directly calls into the FSM. * RPC (`:8300`) on the server sniffs the first byte of a new connection to determine if it's actually doing native TLS. If so it checks the ALPN header for protocol determination (just like how the existing system uses the type-byte marker). * 2 new kinds of protocols are exclusively decoded via this native TLS mechanism: one for ferrying "packet" operations (udp-like) from the gossip layer and one for "stream" operations (tcp-like). The packet operations re-use sockets (using length-prefixing) to cut down on TLS re-negotiation overhead. * the server instances specially wrap the `memberlist.NetTransport` when running with gateway federation enabled (in a `wanfed.Transport`). The general gist is that if it tries to dial a node in the SAME datacenter (deduced by looking at the suffix of the node name) there is no change. If dialing a DIFFERENT datacenter it is wrapped up in a TLS+ALPN blob and sent through some mesh gateways to eventually end up in a server's :8300 port. * a new flag when launching a mesh gateway via `consul connect envoy` to indicate that the servers are to be exposed. This sets a special service meta when registering the gateway into the catalog. * `proxycfg/xds` notice this metadata blob to activate additional watches for the FederationState objects as well as the location of all of the consul servers in that datacenter. * `xds:` if the extra metadata is in place additional clusters are defined in a DC to bulk sink all traffic to another DC's gateways. For the current datacenter we listen on a wildcard name (`server.<dc>.consul`) that load balances all servers as well as one mini-cluster per node (`<node>.server.<dc>.consul`) * the `consul tls cert create` command got a new flag (`-node`) to help create an additional SAN in certs that can be used with this flavor of federation.	5 years ago
Pierre Souchay	864f7efffa	agent: configuration reload preserves check's statuses for services (#7345) ... This fixes issue #7318 Between versions 1.5.2 and 1.5.3, a regression has been introduced regarding health of services. A patch #6144 had been issued for HealthChecks of nodes, but not for healthchecks of services. What happened when a reload was: 1. save all healthcheck statuses 2. cleanup everything 3. add new services with healthchecks In step 3, the state of healthchecks was taken into account locally, so at step 3, but since we cleaned up at step 2, state was lost. This PR introduces the snap parameter, so step 3 can use information from step 1	5 years ago
Hans Hasselberg	315d57bfb1	agent: sensible keyring error (#7272) ... Fixes #7231. Before an agent would always emit a warning when there is an encrypt key in the configuration and an existing keyring stored, which is happening on restart. Now it only emits that warning when the encrypt key from the configuration is not part of the keyring.	5 years ago
Akshay Ganeshen	8beb716414	feat: support sending body in HTTP checks (#6602)	5 years ago
Freddy	cb77fc6d01	Add managed service provider token (#7218) ... Stubs for enterprise-only ACL token to be used by managed service providers.	5 years ago
Hans Hasselberg	5531678e9e	Security fixes (#7182) ... * Mitigate HTTP/RPC Services Allow Unbounded Resource Usage Fixes #7159. Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com> Co-authored-by: Paul Banks <banks@banksco.de>	5 years ago
R.B. Boyer	d78b5008ce	various tweaks on top of the hclog work (#7165)	5 years ago
Chris Piraino	401221de58	Allow users to configure either unstructured or JSON logging (#7130) ... * hclog Allow users to choose between unstructured and JSON logging	5 years ago
Kit Patella	0d336edb65	Add accessorID of token when ops are denied by ACL system (#7117) ... * agent: add and edit doc comments * agent: add ACL token accessorID to debugging traces * agent: polish acl debugging * agent: minor fix + string fmt over value interp * agent: undo export & fix logging field names * agent: remove note and migrate up to code review * Update agent/consul/acl.go Co-Authored-By: Matt Keeler <mkeeler@users.noreply.github.com> * agent: incorporate review feedback * Update agent/acl.go Co-Authored-By: R.B. Boyer <public@richardboyer.net> Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com> Co-authored-by: R.B. Boyer <public@richardboyer.net>	5 years ago
Matt Keeler	c09693e545	Updates to Config Entries and Connect for Namespaces (#7116)	5 years ago
Hans Hasselberg	11a571de95	agent: setup grpc server with auto_encrypt certs and add -https-port (#7086) ... * setup grpc server with TLS config used across consul. * add -https-port flag	5 years ago
Aestek	8fc736038a	agent: remove service sidecars in Agent.cleanupRegistration (#7022) ... Sidecar proxies were left behind when cleaning up after an unsuccessful registration. There are now also removed when the service is cleanup up.	5 years ago
Hans Hasselberg	87f32c8ba6	auto_encrypt: set dns and ip san for k8s and provide configuration (#6944) ... * Add CreateCSRWithSAN * Use CreateCSRWithSAN in auto_encrypt and cache * Copy DNSNames and IPAddresses to cert * Verify auto_encrypt.sign returns cert with SAN * provide configuration options for auto_encrypt dnssan and ipsan * rename CreateCSRWithSAN to CreateCSR	5 years ago
Aestek	ba8fd8296f	Add support for dual stack IPv4/IPv6 network (#6640) ... * Use consts for well known tagged adress keys * Add ipv4 and ipv6 tagged addresses for node lan and wan * Add ipv4 and ipv6 tagged addresses for service lan and wan * Use IPv4 and IPv6 address in DNS	5 years ago
Matej Urbas	ce023359fe	agent: configurable MaxQueryTime and DefaultQueryTime. (#3777)	5 years ago
Matt Keeler	a78f7d7a34	OSS changes for implementing token based namespace inferencing ... remove debug log	5 years ago
Matt Keeler	5934f803bf	Sync of OSS changes to support namespaces (#6909)	5 years ago
Hans Hasselberg	2ad0831b34	agent: fewer file local differences between enterprise and oss (#6820) (#6898) ... * Increase number to test ignore. Consul Enterprise has more flags and since we are trying to reduce the differences between both code bases, we are increasing the number in oss. The semantics don't change, it is just a cosmetic thing. * Introduce agent.initEnterprise for enterprise related hooks. * Sync test with ent version. * Fix import order. * revert error wording.	5 years ago
Sarah Adams	aed5cb7669	give feedback to CLI user on forceleave command if node does not exist (#6841)	5 years ago
Matt Keeler	8f1f15a827	Track the correct check id for idempotent service/check updates	5 years ago
Sarah Christoff	5e1c6e907b	Set MinQuorum variable in Autopilot (#6654) ... * Add MinQuorum to Autopilot	5 years ago
Freddy	60f6ec0c2f	Store check type in catalog (#6561)	5 years ago
PHBourquin	039615641e	Checks to passing/critical only after reaching a consecutive success/failure threshold (#5739) ... A check may be set to become passing/critical only if a specified number of successive checks return passing/critical in a row. Status will stay identical as before until the threshold is reached. This feature is available for HTTP, TCP, gRPC, Docker & Monitor checks.	5 years ago
Sarah Christoff	5e26971864	Prune Unhealthy Agents (#6571) ... * Add -prune flag to ForceLeave	5 years ago
Freddy	fdd10dd8b8	Expose HTTP-based paths through Connect proxy (#6446) ... Fixes: #5396 This PR adds a proxy configuration stanza called expose. These flags register listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only listening on the loopback interface, while still accepting traffic from non Connect-enabled services. Under expose there is a boolean checks flag that would automatically expose all registered HTTP and gRPC check paths. This stanza also accepts a paths list to expose individual paths. The primary use case for this functionality would be to expose paths for third parties like Prometheus or the kubelet. Listeners for requests to exposed paths are be configured dynamically at run time. Any time a proxy, or check can be registered, a listener can also be created. In this initial implementation requests to these paths are not authenticated/encrypted.	5 years ago
R.B. Boyer	5882e21b2b	agent: tolerate more failure scenarios during service registration with central config enabled (#6472) ... Also: * Finished threading replaceExistingChecks setting (from GH-4905) through service manager. * Respected the original configSource value that was used to register a service or a check when restoring persisted data. * Run several existing tests with and without central config enabled (not exhaustive yet). * Switch to ioutil.ReadFile for all types of agent persistence.	5 years ago
Aestek	7c7b7f24fd	Add option to register services and their checks idempotently (#4905)	5 years ago
Alvin Huang	c516fabfac	revert commits on master (#6413)	5 years ago
tradel	a4312d2e6e	add domain and nodeName to agent cert request	5 years ago
R.B. Boyer	91da908d2f	test: fix TestAgent.Start() to not segfault if the DNSServer cannot ListenAndServe (#6409) ... The embedded `Server` field on a `DNSServer` is only set inside of the `ListenAndServe` method. If that method fails for reasons like the address being in use and is not bindable, then the `Server` field will not be set and the overall `Agent.Start()` will fail. This will trigger the inner loop of `TestAgent.Start()` to invoke `ShutdownEndpoints` which will attempt to pretty print the DNS servers using fields on that inner `Server` field. Because it was never set, this causes a nil pointer dereference and crashes the test.	5 years ago
Mike Morris	65be58703c	connect: remove managed proxies (#6220) ... * connect: remove managed proxies implementation and all supporting config options and structs * connect: remove deprecated ProxyDestination * command: remove CONNECT_PROXY_TOKEN env var * agent: remove entire proxyprocess proxy manager * test: remove all managed proxy tests * test: remove irrelevant managed proxy note from TestService_ServerTLSConfig * test: update ContentHash to reflect managed proxy removal * test: remove deprecated ProxyDestination test * telemetry: remove managed proxy note * http: remove /v1/agent/connect/proxy endpoint * ci: remove deprecated test exclusion * website: update managed proxies deprecation page to note removal * website: remove managed proxy configuration API docs * website: remove managed proxy note from built-in proxy config * website: add note on removing proxy subdirectory of data_dir	5 years ago
Alvin Huang	ef6b80bab2	resolve circleci config conflicts	5 years ago
Paul Banks	f38da47c55	Allow raft TrailingLogs to be configured. (#6186) ... This fixes pathological cases where the write throughput and snapshot size are both so large that more than 10k log entries are written in the time it takes to restore the snapshot from disk. In this case followers that restart can never catch up with leader replication again and enter a loop of constantly downloading a full snapshot and restoring it only to find that snapshot is already out of date and the leader has truncated its logs so a new snapshot is sent etc. In general if you need to adjust this, you are probably abusing Consul for purposes outside its design envelope and should reconsider your usage to reduce data size and/or write volume.	5 years ago
Alvin Huang	6f1953d96d	Merge branch 'master' into release/1-6	5 years ago
R.B. Boyer	b16d7f00bc	agent: avoid reverting any check updates that occur while a service is being added or the config is reloaded (#6144)	5 years ago
Matt Keeler	3b6d5e382a	Implement caching for config entry lists ... Update agent/cache-types/config_entry.go Co-Authored-By: R.B. Boyer <public@richardboyer.net>	5 years ago
R.B. Boyer	4bdb690a25	activate most discovery chain features in xDS for envoy (#6024)	5 years ago
Matt Keeler	3943e38133	Implement Kind based ServiceDump and caching of the ServiceDump RPC	5 years ago
hashicorp-ci	43bda6fb76	Merge Consul OSS branch 'master' at commit e91f73f592	5 years ago
Hans Hasselberg	33a7df3330	tls: auto_encrypt enables automatic RPC cert provisioning for consul clients (#5597)	6 years ago
hashicorp-ci	f4304e2e5b	Merge Consul OSS branch 'master' at commit 4eb73973b6	6 years ago
Pierre Souchay	0e907f5aa8	Support for maximum size for Output of checks (#5233) ... * Support for maximum size for Output of checks This PR allows users to limit the size of output produced by checks at the agent and check level. When set at the agent level, it will limit the output for all checks monitored by the agent. When set at the check level, it can override the agent max for a specific check but only if it is lower than the agent max. Default value is 4k, and input must be at least 1.	6 years ago
hashicorp-ci	4d185baf55	Merge Consul OSS branch 'master' at commit 88b15d84f9 ... skip-checks: true	6 years ago
Matt Keeler	43c5ba0304	New Cache Types (#5995) ... * Add a cache type for the Catalog.ListServices endpoint * Add a cache type for the Catalog.ListDatacenters endpoint	6 years ago
Hans Hasselberg	f13fe4b304	agent: transfer leadership when establishLeadership fails (#5247)	6 years ago
Pierre Souchay	4a4c63bda0	Ensure Consul is IPv6 compliant (#5468)	6 years ago
Pierre Souchay	e892981418	agent: Improve startup message to avoid confusing users when no error occurs (#5896) ... * Improve startup message to avoid confusing users when no error occurs Several times, some users not very familiar with Consul get confused by error message at startup: `[INFO] agent: (LAN) joined: 1 Err: <nil>` Having `Err: <nil>` seems weird to many users, I propose to have the following instead: * Success: `[INFO] agent: (LAN) joined: 1` * Error: `[WARN] agent: (LAN) couldn't join: %d Err: ERROR`	6 years ago
R.B. Boyer	cc1aa3f973	acl: adding Roles to Tokens (#5514) ... Roles are named and can express the same bundle of permissions that can currently be assigned to a Token (lists of Policies and Service Identities). The difference with a Role is that it not itself a bearer token, but just another entity that can be tied to a Token. This lets an operator potentially curate a set of smaller reusable Policies and compose them together into reusable Roles, rather than always exploding that same list of Policies on any Token that needs similar permissions. This also refactors the acl replication code to be semi-generic to avoid 3x copypasta.	6 years ago
Matt Keeler	15e80e4e76	Implement bootstrapping proxy defaults from the config file (#5714)	6 years ago
Matt Keeler	222afeae4c	Move the watch package into the api module (#5664) ... * Move the watch package into the api module It was already just a thin wrapper around the API anyways. The biggest change was to the testing. Instead of using a test agent directly from the agent package it now uses the binary on the PATH just like the other API tests. The other big changes were to fix up the connect based watch tests so that we didn’t need to pull in the connect package (and therefore all of Consul)	6 years ago
Kyle Havlovitz	d8f8400fe1	Merge pull request #5700 from hashicorp/service-reg-manager ... Use centralized service config on agent service registrations	6 years ago
Kyle Havlovitz	43bfc20dc8	Test an index=0 value in cache.Notify	6 years ago
Aestek	f669bb7b0f	Add support for DNS config hot-reload (#4875) ... The DNS config parameters `recursors` and `dns_config.*` are now hot reloaded on SIGHUP or `consul reload` and do not need an agent restart to be modified. Config is stored in an atomic.Value and loaded at the beginning of each request. Reloading only affects requests that start _after_ the reload. Ongoing requests are not affected. To match the current behavior the recursor handler is loaded and unloaded as needed on config reload.	6 years ago
Kyle Havlovitz	cba47aa0ca	Fix a race in the ready logic	6 years ago
Kyle Havlovitz	c269369760	Make central service config opt-in and rework the initial registration	6 years ago
Kyle Havlovitz	88e1d8ce03	Fill out the service manager functionality and fix tests	6 years ago
Kyle Havlovitz	7c25869e67	Add the service registration manager to the agent	6 years ago
Pierre Souchay	88d4383410	Ensure we remove Connect proxy before deregistering the service itself (#5482) ... This will fix https://github.com/hashicorp/consul/issues/5351	6 years ago
Valentin Fritz	21f149de8b	Fix checks removal when removing service (#5457) ... Fix my recently discovered issue described here: #5456	6 years ago
Hans Hasselberg	7e11dd82aa	agent: enable reloading of tls config (#5419) ... This PR introduces reloading tls configuration. Consul will now be able to reload the TLS configuration which previously required a restart. It is not yet possible to turn TLS ON or OFF with these changes. Only when TLS is already turned on, the configuration can be reloaded. Most importantly the certificates and CAs.	6 years ago
R.B. Boyer	f4a3b9d518	fix typos reported by golangci-lint:misspell (#5434)	6 years ago
Aestek	2aac4d5168	Register and deregisters services and their checks atomically in the local state (#5012) ... Prevent race between register and deregister requests by saving them together in the local state on registration. Also adds more cleaning in case of failure when registering services / checks.	6 years ago
Matt Keeler	118adbb123	ACL Token Persistence and Reloading (#5328) ... This PR adds two features which will be useful for operators when ACLs are in use. 1. Tokens set in configuration files are now reloadable. 2. If `acl.enable_token_persistence` is set to `true` in the configuration, tokens set via the `v1/agent/token` endpoint are now persisted to disk and loaded when the agent starts (or during configuration reload) Note that token persistence is opt-in so our users who do not want tokens on the local disk will see no change. Some other secondary changes: * Refactored a bunch of places where the replication token is retrieved from the token store. This token isn't just for replicating ACLs and now it is named accordingly. * Allowed better paths in the `v1/agent/token/` API. Instead of paths like: `v1/agent/token/acl_replication_token` the path can now be just `v1/agent/token/replication`. The old paths remain to be valid. * Added a couple new API functions to set tokens via the new paths. Deprecated the old ones and pointed to the new names. The names are also generally better and don't imply that what you are setting is for ACLs but rather are setting ACL tokens. There is a minor semantic difference there especially for the replication token as again, its no longer used only for ACL token/policy replication. The new functions will detect 404s and fallback to using the older token paths when talking to pre-1.4.3 agents. * Docs updated to reflect the API additions and to show using the new endpoints. * Updated the ACL CLI set-agent-tokens command to use the non-deprecated APIs.	6 years ago
Hans Hasselberg	80e7d63fc2	Centralise tls configuration part 2 (#5374) ... This PR is based on #5366 and continues to centralise the tls configuration in order to be reloadable eventually! This PR is another refactoring. No tests are changed, beyond calling other functions or cosmetic stuff. I added a bunch of tests, even though they might be redundant.	6 years ago
Hans Hasselberg	786b3b1095	Centralise tls configuration part 1 (#5366) ... In order to be able to reload the TLS configuration, we need one way to generate the different configurations. This PR introduces a `tlsutil.Configurator` which holds a `tlsutil.Config`. Afterwards it is responsible for rendering every `tls.Config`. In this particular PR I moved `IncomingHTTPSConfig`, `IncomingTLSConfig`, and `OutgoingTLSWrapper` into `tlsutil.Configurator`. This PR is a pure refactoring - not a single feature added. And not a single test added. I only slightly modified existing tests as necessary.	6 years ago
Aestek	f1cdfbe40e	Allow DNS interface to use agent cache (#5300) ... Adds two new configuration parameters "dns_config.use_cache" and "dns_config.cache_max_age" controlling how DNS requests use the agent cache when querying servers.	6 years ago
R.B. Boyer	de0f585583	agent: only enable TLS on gRPC if the HTTPS API port is enabled (#5287) ... Currently the gRPC server assumes that if you have configured TLS certs on the agent (for RPC) that you want gRPC to be encrypted. If gRPC is bound to localhost this can be overkill. For the API we let the user choose to offer HTTP or HTTPS API endpoints independently of the TLS cert configuration for a similar reason. This setting will let someone encrypt RPC traffic with TLS but avoid encrypting local gRPC traffic if that is what they want to do by only enabling TLS on gRPC if the HTTPS API port is enabled.	6 years ago
R.B. Boyer	db8a871309	Merge pull request #5237 from hashicorp/term-grpc-stream-on-token-failure ... Check ACLs more often for xDS endpoints.	6 years ago
Paul Banks	ef9f27cbc8	connect: tame thundering herd of CSRs on CA rotation (#5228) ... * Support rate limiting and concurrency limiting CSR requests on servers; handle CA rotations gracefully with jitter and backoff-on-rate-limit in client * Add CSR rate limiting docs * Fix config naming and add tests for new CA configs	6 years ago
R.B. Boyer	d3eb781384	Check ACLs more often for xDS endpoints. ... For established xDS gRPC streams recheck ACLs for each DiscoveryRequest or DiscoveryResponse. If more than 5 minutes has elapsed since the last ACL check, recheck even without an incoming DiscoveryRequest or DiscoveryResponse. ACL failures will terminate the stream.	6 years ago
Paul Banks	0638e09b6e	connect: agent leaf cert caching improvements (#5091) ... * Add State storage and LastResult argument into Cache so that cache.Types can safely store additional data that is eventually expired. * New Leaf cache type working and basic tests passing. TODO: more extensive testing for the Root change jitter across blocking requests, test concurrent fetches for different leaves interact nicely with rootsWatcher. * Add multi-client and delayed rotation tests. * Typos and cleanup error handling in roots watch * Add comment about how the FetchResult can be used and change ca leaf state to use a non-pointer state. * Plumb test override of root CA jitter through TestAgent so that tests are deterministic again! * Fix failing config test	6 years ago
Paul Banks	bb7145f27d	agent: add default weights to service in local state to prevent AE churn (#5126) ... * Add default weights when adding a service with no weights to local state to prevent constant AE re-sync. This fix was contributed by @42wim in https://github.com/hashicorp/consul/pull/5096 but was merged against the wrong base. This adds it to master and adds a test to cover the behaviour. * Fix tests that broke due to comparing internal state which now has default weights	6 years ago
Aestek	8709213d6e	Prevent status flap when re-registering a check (#4904) ... Fixes point `#2` of: https://github.com/hashicorp/consul/issues/4903 When registering a service each healthcheck status is saved and restored (https://github.com/hashicorp/consul/blob/master/agent/agent.go#L1914) to avoid unnecessary flaps in health state. This change extends this feature to single check registration by moving this protection in `AddCheck()` so that both `PUT /v1/agent/service/register` and `PUT /v1/agent/check/register` behave in the same idempotent way. #### Steps to reproduce 1. Register a check : ``` curl -X PUT \ http://127.0.0.1:8500/v1/agent/check/register \ -H 'Content-Type: application/json' \ -d '{ "Name": "my_check", "ServiceID": "srv", "Interval": "10s", "Args": ["true"] }' ``` 2. The check will initialize and change to `passing` 3. Run the same request again 4. The check status will quickly go from `critical` to `passing` (the delay for this transission is determined by https://github.com/hashicorp/consul/blob/master/agent/checks/check.go#L95)	6 years ago
Matt Keeler	18b29c45c4	New ACLs (#4791) ... This PR is almost a complete rewrite of the ACL system within Consul. It brings the features more in line with other HashiCorp products. Obviously there is quite a bit left to do here but most of it is related docs, testing and finishing the last few commands in the CLI. I will update the PR description and check off the todos as I finish them over the next few days/week. Description At a high level this PR is mainly to split ACL tokens from Policies and to split the concepts of Authorization from Identities. A lot of this PR is mostly just to support CRUD operations on ACLTokens and ACLPolicies. These in and of themselves are not particularly interesting. The bigger conceptual changes are in how tokens get resolved, how backwards compatibility is handled and the separation of policy from identity which could lead the way to allowing for alternative identity providers. On the surface and with a new cluster the ACL system will look very similar to that of Nomads. Both have tokens and policies. Both have local tokens. The ACL management APIs for both are very similar. I even ripped off Nomad's ACL bootstrap resetting procedure. There are a few key differences though. Nomad requires token and policy replication where Consul only requires policy replication with token replication being opt-in. In Consul local tokens only work with token replication being enabled though. All policies in Nomad are globally applicable. In Consul all policies are stored and replicated globally but can be scoped to a subset of the datacenters. This allows for more granular access management. Unlike Nomad, Consul has legacy baggage in the form of the original ACL system. The ramifications of this are: A server running the new system must still support other clients using the legacy system. A client running the new system must be able to use the legacy RPCs when the servers in its datacenter are running the legacy system. The primary ACL DC's servers running in legacy mode needs to be a gate that keeps everything else in the entire multi-DC cluster running in legacy mode. So not only does this PR implement the new ACL system but has a legacy mode built in for when the cluster isn't ready for new ACLs. Also detecting that new ACLs can be used is automatic and requires no configuration on the part of administrators. This process is detailed more in the "Transitioning from Legacy to New ACL Mode" section below.	6 years ago
Kyle Havlovitz	c617326470	re-add Connect multi-dc config changes ... This reverts commit 8bcfbaffb6.	6 years ago
Jack Pearkes	8bcfbaffb6	Revert "Connect multi-dc config" (#4784)	6 years ago
Aestek	25f04fbd21	[Security] Add finer control over script checks (#4715) ... * Add -enable-local-script-checks options These options allow for a finer control over when script checks are enabled by giving the option to only allow them when they are declared from the local file system. * Add documentation for the new option * Nitpick doc wording	6 years ago
Paul Banks	51c0001aad	[WIP] Initial draft of Sidecar Service and Managed Proxy deprecation docs (#4752) ... * Initial draft of Sidecar Service and Managed Proxy deprecation docs * Service definition deprecation notices and sidecar service * gRPC and sidecar service config options; Deprecate managed proxy options * Envoy Docs: Basic envoy command; envoy getting started/intro * Remove change that snuck in * Envoy custom config example * Add agent/service API docs; deprecate proxy config endpoint * Misc grep cleanup for managed proxies; capitalize Envoy * Updates to getting started guide * Add missing link * Refactor Envoy guide into a separate guide and add bootstrap reference notes. * Add limitations to Envoy docs; Highlight no fixes for known managed proxy issues on deprecation page; clarify snake cae stuff; Sidecar Service lifecycle	6 years ago
Kyle Havlovitz	56dc426227	agent: add primary_datacenter and connect replication config options	6 years ago
Kyle Havlovitz	46c829b879	docs: deprecate acl_datacenter and replace it with primary_datacenter	6 years ago
Paul Banks	161482d2cd	Fix up tests broken by master merge; add proxy tests to services command (and fix it!); actually run the proxycfg.Manager	6 years ago
Paul Banks	dca1303d05	Connect Envoy Command (#4735) ... * Plumb xDS server and proxyxfg into the agent startup * Add `consul connect envoy` command to allow running Envoy as a connect sidecar. * Add test for help tabs; typos and style fixups from review	6 years ago
Paul Banks	e812f5516a	Add -sidecar-for and new /agent/service/:service_id endpoint (#4691) ... - A new endpoint `/v1/agent/service/:service_id` which is a generic way to look up the service for a single instance. The primary value here is that it: - **supports hash-based blocking** and so; - **replaces `/agent/connect/proxy/:proxy_id`** as the mechanism the built-in proxy uses to read its config. - It's not proxy specific and so works for any service. - It has a temporary shim to call through to the existing endpoint to preserve current managed proxy config defaulting behaviour until that is removed entirely (tested). - The built-in proxy now uses the new endpoint exclusively for it's config - The built-in proxy now has a `-sidecar-for` flag that allows the service ID of the _target_ service to be specified, on the condition that there is exactly one "sidecar" proxy (that is one that has `Proxy.DestinationServiceID` set) for the service registered. - Several fixes for edge cases for SidecarService - A fix for `Alias` checks - when running locally they didn't update their state until some external thing updated the target. If the target service has no checks registered as below, then the alias never made it past critical.	6 years ago
Paul Banks	1e7eace066	Add SidecarService Syntax sugar to Service Definition (#4686) ... * Added new Config for SidecarService in ServiceDefinitions. * WIP: all the code needed for SidecarService is written... none of it is tested other than config :). Need API updates too. * Test coverage for the new sidecarServiceFromNodeService method. * Test API registratrion with SidecarService * Recursive Key Translation 🤦 * Add tests for nested sidecar defintion arrays to ensure they are translated correctly * Use dedicated internal state rather than Service Meta for tracking sidecars for deregistration. Add tests for deregistration. * API struct for agent register. No other endpoint should be affected yet. * Additional test cases to cover updates to API registrations	6 years ago
Paul Banks	b06ddc9187	Rename proxy package (re-run of #4550) (#4638) ... * Rename agent/proxy package to reflect that it is limited to managed proxy processes Rationale: we have several other components of the agent that relate to Connect proxies for example the ProxyConfigManager component needed for Envoy work. Those things are pretty separate from the focus of this package so far which is only concerned with managing external proxy processes so it's nota good fit to put code for that in here, yet there is a naming clash if we have other packages related to proxy functionality that are not in the `agent/proxy` package. Happy to bikeshed the name. I started by calling it `managedproxy` but `managedproxy.Manager` is especially unpleasant. `proxyprocess` seems good in that it's more specific about purpose but less clearly connected with the concept of "managed proxies". The names in use are cleaner though e.g. `proxyprocess.Manager`. This rename was completed automatically using golang.org/x/tools/cmd/gomvpkg. Depends on #4541 * Fix missed windows tagged files	6 years ago
Paul Banks	88388d760d	Support Agent Caching for Service Discovery Results (#4541) ... * Add cache types for catalog/services and health/services and basic test that caching works * Support non-blocking cache types with Cache-Control semantics. * Update API docs to include caching info for every endpoint. * Comment updates per PR feedback. * Add note on caching to the 10,000 foot view on the architecture page to make the new data path more clear. * Document prepared query staleness quirk and force all background requests to AllowStale so we can spread service discovery load across servers.	6 years ago
Paul Banks	e8ba527f23	Add a Close method to cache that stops background goroutines. (#4746) ... In a real agent the `cache` instance is alive until the agent shuts down so this is not a real leak in production, however in out test suite, every testAgent that is started and stops leaks goroutines that never get cleaned up which accumulate consuming CPU and memory through subsequent test in the `agent` package which doesn't help our test flakiness. This adds a Close method that doesn't invalidate or clean up the cache, and still allows concurrent blocking queries to run (for up to 10 mins which might still affect tests). But at least it doesn't maintain them forever with background refresh and an expiry watcher routine. It would be nice to cancel any outstanding blocking requests as well when we close but that requires much more invasive surgery right into our RPC protocol since we don't have a way to cancel requests currently. Unscientifically this seems to make tests pass a bit quicker and more reliably locally but I can't really be sure of that!	6 years ago
Hans Hasselberg	8e235a72b4	Allow disabling the HTTP API again. (#4655) ... If you provide an invalid HTTP configuration consul will still start again instead of failing. But if you do so the build-in proxy won't be able to start which you might need for connect.	6 years ago
Pierre Souchay	2fe728c7bd	Ensure that Proxies ARE always cleaned up, event with DeregisterCriticalServiceAfter (#4649) ... This fixes https://github.com/hashicorp/consul/issues/4648	6 years ago
Matt Keeler	cc8327ed9a	Ensure that errors setting up the DNS servers get propagated back to the shell (#4598) ... Fixes: #4578 Prior to this fix if there was an error binding to ports for the DNS servers the error would be swallowed by the gated log writer and never output. This fix propagates the DNS server errors back to the shell with a multierror.	6 years ago
Matt Keeler	e81c85c051	Fix #4515: Segfault when serf_wan port was -1 but reconnect_time_wan was set (#4531) ... Fixes #4515 This just slightly refactors the logic to only attempt to set the serf wan reconnect timeout when the rest of the serf wan settings are configured - thus avoiding a segfault.	6 years ago
Matt Keeler	0e0227792b	Gossip tuneables (#4444) ... Expose a few gossip tuneables for both lan and wan interfaces gossip_nodes gossip_interval probe_timeout probe_interval retransmit_mult suspicion_mult	6 years ago
Mitchell Hashimoto	7fa6bb022f	Merge pull request #4320 from hashicorp/f-alias-check ... Add "Alias" Check Type	6 years ago
Matt Keeler	560c9c26f7	Use the agent logger instead of log module	6 years ago
Matt Keeler	3fe5f566f2	Persist proxies from config files ... Also change how loadProxies works. Now it will load all persisted proxies into a map, then when loading config file proxies will look up the previous proxy token in that map.	6 years ago
Matt Keeler	c891e264ca	Fix issue with choosing a client addr that is 0.0.0.0 or ::	6 years ago
Mitchell Hashimoto	d6ecd97d1d	agent: use the correct ACL token for alias checks	6 years ago
Mitchell Hashimoto	19ced12668	agent: alias checks have no interval	6 years ago
Mitchell Hashimoto	4a67beb734	agent: run alias checks	6 years ago
Paul Banks	9015cd62ab	Merge pull request #4381 from hashicorp/proxy-check-default ... Proxy check default	6 years ago
Matt Keeler	7572ca0f37	Merge pull request #4374 from hashicorp/feature/proxy-env-vars ... Setup managed proxy environment with API client env vars	6 years ago
Paul Banks	bb9a5c703b	Default managed proxy TCP check address sanely when proxy is bound to 0.0.0.0. ... This also provides a mechanism to configure custom address or disable the check entirely from managed proxy config.	6 years ago
Matt Keeler	c54b43bef3	PR Updates ... Proxy now doesn’t need to know anything about the api as we pass env vars to it instead of the api config.	6 years ago
Matt Keeler	3b6eef8ec6	Pass around an API Config object and convert to env vars for the managed proxy	6 years ago
Matt Keeler	962f6a1816	Remove https://prefix from TLSConfig.Address	6 years ago
mkeeler	6813a99081	Merge remote-tracking branch 'connect/f-connect'	7 years ago
Mitchell Hashimoto	a76f652fd2	agent: convert the proxy bind_port to int if it is a float	7 years ago
Paul Banks	17789d4fe3	register TCP check for managed proxies	7 years ago
Paul Banks	280f14d64c	Make proxy only listen after initial certs are fetched	7 years ago
Paul Banks	420ae3df69	Limit proxy telemetry config to only be visible with authenticated with a proxy token	7 years ago
Paul Banks	c6ef6a61c9	Refactor to use embedded struct.	7 years ago
Paul Banks	8aeb7bd206	Disable TestAgent proxy execution properly	7 years ago
Mitchell Hashimoto	a82726f0b8	agent: RemoveProxy also removes the proxy service	7 years ago
Mitchell Hashimoto	ed98d65c2b	agent/proxy: AllowRoot to disable executing managed proxies when root	7 years ago
Paul Banks	e57aa52ca6	Warn about killing proxies in dev mode	7 years ago
Paul Banks	d1c67d90bc	Fixs a few issues that stopped this working in real life but not caught by tests: ... - Dev mode assumed no persistence of services although proxy state is persisted which caused proxies to be killed on startup as their services were no longer registered. Fixed. - Didn't snapshot the ProxyID which meant that proxies were adopted OK from snapshot but failed to restart if they died since there was no proxyID in the ENV on restart - Dev mode with no persistence just kills all proxies on shutdown since it can't recover them later - Naming things	7 years ago
Paul Banks	85d6502ab3	Don't kill proxies on agent shutdown; backport manager close fix	7 years ago
Paul Banks	cdc7cfaa36	Abandon daemonize for simpler solution (preserving history): ... Reverts: - bdb274852ae469c89092d6050697c0ff97178465 - 2c689179c4f61c11f0016214c0fc127a0b813bfe - d62e25c4a7ab753914b6baccd66f88ffd10949a3 - c727ffbcc98e3e0bf41e1a7bdd40169bd2d22191 - 31b4d18933fd0acbe157e28d03ad59c2abf9a1fb - 85c3f8df3eabc00f490cd392213c3b928a85aa44	7 years ago
Paul Banks	8cf4b3a6eb	Sanity check that we are never trying to self-exec a test binary. Add daemonize bypass for TestAgent so that we don't have to jump through ridiculous self-execution hooks for every package that might possibly invoke a managed proxy	7 years ago
Paul Banks	2b377dc624	Run daemon processes as a detached child. ... This turns out to have a lot more subtelty than we accounted for. The test suite is especially prone to races now we can only poll the child and many extra levels of indirectoin are needed to correctly run daemon process without it becoming a Zombie. I ran this test suite in a loop with parallel enabled to verify for races (-race doesn't find any as they are logical inter-process ones not actual data races). I made it through ~50 runs before hitting an error due to timing which is much better than before. I want to go back and see if we can do better though. Just getting this up.	7 years ago
Paul Banks	e21723a891	Persist proxy state through agent restart	7 years ago
Mitchell Hashimoto	cd39f09693	agent: leaf endpoint accepts name, not service ID ... This change is important so that requests can made representing a service that may not be registered with the same local agent.	7 years ago
Paul Banks	a80559e439	Make invalid clusterID be fatal	7 years ago
Paul Banks	4aeab3897c	Fixed many tests after rebase. Some still failing and seem unrelated to any connect changes.	7 years ago
Mitchell Hashimoto	cfcd733609	agent/cache: implement refresh backoff	7 years ago
Mitchell Hashimoto	e54e69d11f	agent: verify local proxy tokens for CA leaf + tests	7 years ago
Mitchell Hashimoto	a099c27b07	agent: verify proxy token for ProxyConfig endpoint + tests	7 years ago
Mitchell Hashimoto	aaca1fbcf5	agent: increase timer for blocking cache endpoints	7 years ago
Mitchell Hashimoto	f5e7993249	agent: clarify why we Kill still	7 years ago
Mitchell Hashimoto	2809203408	agent: restore proxy snapshot but still Kill proxies	7 years ago
Mitchell Hashimoto	f5ccc65295	agent: only set the proxy manager data dir if its set	7 years ago
Mitchell Hashimoto	1e7f253b53	agent/proxy: write pid file whenever the daemon process changes	7 years ago
Mitchell Hashimoto	6cdacd1fd9	agent/proxy: send logs to the correct location for daemon proxies	7 years ago
Mitchell Hashimoto	171bf8d599	agent: clean up defaulting of proxy configuration ... This cleans up and unifies how proxy settings defaults are applied.	7 years ago
Mitchell Hashimoto	6ae95d754c	agent: use os.Executable	7 years ago
Mitchell Hashimoto	6539280f2a	agent: fix crash that could happen if proxy was nil on load	7 years ago
Mitchell Hashimoto	1a2b28602c	agent: start proxy manager	7 years ago
Mitchell Hashimoto	fcd2ab2338	agent/proxy: manager and basic tests, not great coverage yet coming soon	7 years ago
Mitchell Hashimoto	476ea7b04a	agent: start/stop proxies	7 years ago
Paul Banks	90c574ebaa	Wire up agent leaf endpoint to cache framework to support blocking.	7 years ago
Paul Banks	1b197d934a	Don't allow connect watches in agent/cli yet	7 years ago