112 Commits (9edb625331f1c48e5fa43f8dadec131ed8b9df8e)

Author	SHA1	Message	Date
Paul Glass	a884d0d7c7	Use agent token for service/check deregistration during anti-entropy (#16097) ... Use only the agent token for deregistration during anti-entropy The previous behavior had the agent attempt to use the "service" token (i.e. from the `token` field in a service definition file), and if that was not set then it would use the agent token. The previous behavior was problematic because, if the service token had been deleted, the deregistration request would fail. The agent would retry the deregistration during each anti-entropy sync, and the situation would never resolve. The new behavior is to only/always use the agent token for service and check deregistration during anti-entropy. This approach is: * Simpler: No fallback logic to try different tokens * Faster (slightly): No time spent attempting the service token * Correct: The agent token is able to deregister services on that agent's node, because: * node:write permissions allow deregistration of services/checks on that node. * The agent token must have node:write permission, or else the agent is not be able to (de)register itself into the catalog Co-authored-by: Vesa Hagström <weeezes@gmail.com>	2 years ago
Paul Glass	f5231b9157	Add new config_file_service_registration token (#15828)	2 years ago
Chris S. Kim	a7b34d50fc	Output user-friendly name for anonymous token (#15884)	2 years ago
Semir Patel	bafa5c7156	Pass remote addr of incoming HTTP requests through to RPC(..) calls (#15700)	2 years ago
Freddy	72b6d69652	Merge pull request #13499 from maxb/delete-unused-metric ... Delete definition of metric `consul.acl.blocked.node.deregistration`	2 years ago
cskh	59e81a728e	chore: removed unused method AddService (#13905) ... - This AddService is not used anywhere. AddServiceWithChecks is place of AddService - Test code is updated	2 years ago
Max Bowsher	7b97b8abd2	Delete definition of metric `consul.acl.blocked.node.registration` ... Although the metric is defined, there is no code which ever sets its value - the code in question is genuinely asymmetric - there are 3 types of object for which registration can be tracked, but only 2 for which deregistration can be tracked.	2 years ago
Dan Upton	e00e3a0bc3	Move ACLResolveResult into acl/resolver package (#13467) ... Having this type live in the agent/consul package makes it difficult to put anything that relies on token resolution (e.g. the new gRPC services) in separate packages without introducing import cycles. For example, if package foo imports agent/consul for the ACLResolveResult type it means that agent/consul cannot import foo to register its service. We've previously worked around this by wrapping the ACLResolver to "downgrade" its return type to an acl.Authorizer - aside from the added complexity, this also loses the resolved identity information. In the future, we may want to move the whole ACLResolver into the acl/resolver package. For now, putting the result type there at least, fixes the immediate import cycle issues.	2 years ago
Mark Anderson	98a2e282be	Fixup acl.EnterpriseMeta ... Signed-off-by: Mark Anderson <manderson@hashicorp.com>	3 years ago
Dhia Ayachi	16b19dd82d	auto-reload configuration when config files change (#12329) ... * add config watcher to the config package * add logging to watcher * add test and refactor to add WatcherEvent. * add all API calls and fix a bug with recreated files * add tests for watcher * remove the unnecessary use of context * Add debug log and a test for file rename * use inode to detect if the file is recreated/replaced and only listen to create events. * tidy ups (#1535) * tidy ups * Add tests for inode reconcile * fix linux vs windows syscall * fix linux vs windows syscall * fix windows compile error * increase timeout * use ctime ID * remove remove/creation test as it's a use case that fail in linux * fix linux/windows to use Ino/CreationTime * fix the watcher to only overwrite current file id * fix linter error * fix remove/create test * set reconcile loop to 200 Milliseconds * fix watcher to not trigger event on remove, add more tests * on a remove event try to add the file back to the watcher and trigger the handler if success * fix race condition * fix flaky test * fix race conditions * set level to info * fix when file is removed and get an event for it after * fix to trigger handler when we get a remove but re-add fail * fix error message * add tests for directory watch and fixes * detect if a file is a symlink and return an error on Add * rename Watcher to FileWatcher and remove symlink deref * add fsnotify@v1.5.1 * fix go mod * do not reset timer on errors, rename OS specific files * rename New func * events trigger on write and rename * add missing test * fix flaking tests * fix flaky test * check reconcile when removed * delete invalid file * fix test to create files with different mod time. * back date file instead of sleeping * add watching file in agent command. * fix watcher call to use new API * add configuration and stop watcher when server stop * add certs as watched files * move FileWatcher to the agent start instead of the command code * stop watcher before replacing it * save watched files in agent * add add and remove interfaces to the file watcher * fix remove to not return an error * use `Add` and `Remove` to update certs files * fix tests * close events channel on the file watcher even when the context is done * extract `NotAutoReloadableRuntimeConfig` is a separate struct * fix linter errors * add Ca configs and outgoing verify to the not auto reloadable config * add some logs and fix to use background context * add tests to auto-config reload * remove stale test * add tests to changes to config files * add check to see if old cert files still trigger updates * rename `NotAutoReloadableRuntimeConfig` to `StaticRuntimeConfig` * fix to re add both key and cert file. Add test to cover this case. * review suggestion Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com> * add check to static runtime config changes * fix test * add changelog file * fix review comments * Apply suggestions from code review Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com> * update flag description Co-authored-by: FFMMM <FFMMM@users.noreply.github.com> * fix compilation error * add static runtime config support * fix test * fix review comments * fix log test * Update .changelog/12329.txt Co-authored-by: Dan Upton <daniel@floppy.co> * transfer tests to runtime_test.go * fix filewatcher Replace to not deadlock. * avoid having lingering locks Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com> * split ReloadConfig func * fix warning message Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com> * convert `FileWatcher` into an interface * fix compilation errors * fix tests * extract func for adding and removing files Co-authored-by: Ashwin Venkatesh <ashwin@hashicorp.com> Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com> Co-authored-by: FFMMM <FFMMM@users.noreply.github.com> Co-authored-by: Daniel Upton <daniel@floppy.co>	3 years ago
R.B. Boyer	fa4577d1a9	local: fixes a data race in anti-entropy sync (#12324) ... The race detector noticed this initially in `TestAgentConfigWatcherSidecarProxy` but it is not restricted to just tests. The two main changes here were: - ensure that before we mutate the internal `agent/local` representation of a Service (for tags or VIPs) we clone those fields - ensure that there's no function argument joint ownership between the caller of a function and the local state when calling `AddService`, `AddCheck`, and related using `copystructure` for now.	3 years ago
Daniel Nephin	edca8d61a3	acl: remove ResolveTokenToIdentity ... By exposing the AccessorID from the primary ResolveToken method we can remove this duplication.	3 years ago
Chris S. Kim	98ea6d1cf1	Fix race with tags (#12041)	3 years ago
Chris S. Kim	a0acf9978f	Fix races in anti-entropy tests (#12028)	3 years ago
Dhia Ayachi	1eac39ae9c	clone the service under lock to avoid a data race (#11940) ... * clone the service under lock to avoid a data race * add change log * create a struct and copy the pointer to mutate it to avoid a data race * fix failing test * revert added space * add comments, to clarify the data race.	3 years ago
Jared Kirschner	b393c90ce7	Clarify service and check error messages (use ID) ... Error messages related to service and check operations previously included the following substrings: - service %q - check %q From this error message, it isn't clear that the expected field is the ID for the entity, not the name. For example, if the user has a service named test, the error message would read 'Unknown service "test"'. This is misleading - a service with that *name* does exist, but not with that *ID*. The substrings above have been modified to make it clear that ID is needed, not name: - service with ID %q - check with ID %q	3 years ago
Kyle Havlovitz	4f2cfee4b0	consul: add virtual IP generation for connect services	3 years ago
Will Jordan	68efecafed	Update node info sync comment (#11465)	3 years ago
Daniel Nephin	eb632c53a2	structs: rename the last helper method. ... This one gets used a bunch, but we can rename it to make the behaviour more obvious.	3 years ago
R.B. Boyer	097e1645e3	agent: ensure that most agent behavior correctly respects partition configuration (#10880)	3 years ago
R.B. Boyer	188e8dc51f	agent/structs: add a bunch more EnterpriseMeta helper functions to help with partitioning (#10669)	3 years ago
Daniel Nephin	df5e18c9c5	agent/local: only fallback to agent token for deletes ... Fallback to the default user token for synching registrations.	4 years ago
Daniel Nephin	0b017e2497	agent/local: do not persist the agent tokens ... Only default to the user token and agent token for the sync. Change the exported methods to only return the stored tokens associated with a specific check or service.	4 years ago
Daniel Nephin	8a5163b184	local: default to the agent token instead of the user token ... When de-registering in anti-entropy sync, when there is no service or check token. The agent token will fall back to the default (aka user) token if no agent token is set, so the existing behaviour still works, but it will prefer the agent token over the user token if both are set. ref: https://www.consul.io/docs/agent/options#acl_tokens The agent token seems more approrpiate in this case, since this is an "internal operation", not something initiated by the user.	4 years ago
Daniel Nephin	d113f0e690	structs: Fix printing of IDs ... These types are used as values (not pointers) in other structs. Using a pointer receiver causes problems when the value is printed. fmt will not call the String method if it is passed a value and the String method has a pointer receiver. By using a value receiver the correct string is printed. Also remove some unused methods.	4 years ago
Daniel Nephin	33b81067f8	local: mark service and checks as InSync when added ... If the existing service and checks are the same as the new registration.	4 years ago
Kit Patella	15af5ead0b	trim help strings to save a few bytes	4 years ago
Kit Patella	5da2f1efa8	finish adding static server metrics	4 years ago
Kit Patella	06d59c03b9	add the service name in the agent rather than in the definitions themselves	4 years ago
Kit Patella	24a2471029	first pass on agent-configured prometheusDefs and adding defs for every consul metric	4 years ago
Matt Keeler	a3a653342b	Fix a bunch of linter warnings	4 years ago
Matt Keeler	c048e86bb2	Switch to using the external autopilot module	4 years ago
Freddy	d72f72dcd5	Notify alias checks when aliased service is [de]registered (#8456)	4 years ago
R.B. Boyer	9cfa4a3fc9	tests: ensure that the ServiceExists helper function normalizes entmeta (#8025) ... This fixes a unit test failure over in enterprise due to https://github.com/hashicorp/consul/pull/7384	5 years ago
Pierre Souchay	9813ae512b	checks: when a service does not exists in an alias, consider it failing (#7384) ... In current implementation of Consul, check alias cannot determine if a service exists or not. Because a service without any check is semantically considered as passing, so when no healthchecks are found for an agent, the check was considered as passing. But this make little sense as the current implementation does not make any difference between: * a non-existing service (passing) * a service without any check (passing as well) In order to make it work, we have to ensure that when a check did not find any healthcheck, the service does indeed exists. If it does not, lets consider the check as failing.	5 years ago
Matt Keeler	acccdbe45c	Fix identity resolution on clients and in secondary dcs (#7862) ... Previously this happened to be using the method on the Server/Client that was meant to allow the ACLResolver to locally resolve tokens. On Servers that had tokens (primary or secondary dc + token replication) this function would lookup the token from raft and return the ACLIdentity. On clients this was always a noop. We inadvertently used this function instead of creating a new one when we added logging accessor ids for permission denied RPC requests. With this commit, a new method is used for resolving the identity properly via the ACLResolver which may still resolve locally in the case of being on a server with tokens but also supports remote token resolution.	5 years ago
Freddy	3956cff60f	Fix check deletion in anti-entropy sync (#7690) ... * Incorporate entMeta into service equality check	5 years ago
Daniel Nephin	f46d1b5c94	agent/structs: Remove ServiceID.Init and CheckID.Init ... The Init method provided the same functionality as the New constructor. The constructor is both more widely used, and more idiomatic, so remove the Init method. This change is in preparation for fixing printing of these IDs.	5 years ago
Hans Hasselberg	6a18f01b42	agent: ensure node info sync and full sync. (#7189) ... This fixes #7020. There are two problems this PR solves: * if the node info changes it is highly likely to get service and check registration permission errors unless those service tokens have node:write. Hopefully services you register don’t have this permission. * the timer for a full sync gets reset for every partial sync which means that many partial syncs are preventing a full sync from happening Instead of syncing node info last, after services and checks, and possibly saving one RPC because it is included in every service sync, I am syncing node info first. It is only ever going to be a single RPC that we are only doing when node info has changed. This way we are guaranteed to sync node info even when something goes wrong with services or checks which is more likely because there are more syncs happening for them.	5 years ago
R.B. Boyer	d78b5008ce	various tweaks on top of the hclog work (#7165)	5 years ago
Chris Piraino	401221de58	Allow users to configure either unstructured or JSON logging (#7130) ... * hclog Allow users to choose between unstructured and JSON logging	5 years ago
Kit Patella	0d336edb65	Add accessorID of token when ops are denied by ACL system (#7117) ... * agent: add and edit doc comments * agent: add ACL token accessorID to debugging traces * agent: polish acl debugging * agent: minor fix + string fmt over value interp * agent: undo export & fix logging field names * agent: remove note and migrate up to code review * Update agent/consul/acl.go Co-Authored-By: Matt Keeler <mkeeler@users.noreply.github.com> * agent: incorporate review feedback * Update agent/acl.go Co-Authored-By: R.B. Boyer <public@richardboyer.net> Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com> Co-authored-by: R.B. Boyer <public@richardboyer.net>	5 years ago
Chris Piraino	59f1462801	Fix segfault when removing both a service and associated check (#7108) ... * Fix segfault when removing both a service and associated check updateSyncState creates entries in the services and checks maps for remote services/checks that are not found locally, so that we can then make sure to delete them in our reconciliation process. However, the values added to the map are missing key fields that the rest of the code expects to not be nil. * Add comment stating Check field can be nil	5 years ago
Aestek	5dc8875bd3	agent: do not deregister service checks twice (#6168) ... Deregistering a service from the catalog automatically deregisters its checks, however the agent still performs a deregister call for each service checks even after the service has been deregistered. With ACLs enabled this results in logs like: "message:consul: "Catalog.Deregister" RPC failed to server server_ip:8300: rpc error making call: rpc error making call: Unknown check 'check_id'" This change removes associated checks from the agent state when deregistering a service, which results in less calls to the servers and supresses the error logs.	5 years ago
Matt Keeler	5934f803bf	Sync of OSS changes to support namespaces (#6909)	5 years ago
Freddy	fdd10dd8b8	Expose HTTP-based paths through Connect proxy (#6446) ... Fixes: #5396 This PR adds a proxy configuration stanza called expose. These flags register listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only listening on the loopback interface, while still accepting traffic from non Connect-enabled services. Under expose there is a boolean checks flag that would automatically expose all registered HTTP and gRPC check paths. This stanza also accepts a paths list to expose individual paths. The primary use case for this functionality would be to expose paths for third parties like Prometheus or the kubelet. Listeners for requests to exposed paths are be configured dynamically at run time. Any time a proxy, or check can be registered, a listener can also be created. In this initial implementation requests to these paths are not authenticated/encrypted.	5 years ago
Mike Morris	65be58703c	connect: remove managed proxies (#6220) ... * connect: remove managed proxies implementation and all supporting config options and structs * connect: remove deprecated ProxyDestination * command: remove CONNECT_PROXY_TOKEN env var * agent: remove entire proxyprocess proxy manager * test: remove all managed proxy tests * test: remove irrelevant managed proxy note from TestService_ServerTLSConfig * test: update ContentHash to reflect managed proxy removal * test: remove deprecated ProxyDestination test * telemetry: remove managed proxy note * http: remove /v1/agent/connect/proxy endpoint * ci: remove deprecated test exclusion * website: update managed proxies deprecation page to note removal * website: remove managed proxy configuration API docs * website: remove managed proxy note from built-in proxy config * website: add note on removing proxy subdirectory of data_dir	5 years ago
Aestek	24a0f2bba2	ae: use stale requests when performing full sync (#5873) ... Read requests performed during anti antropy full sync currently target the leader only. This generates a non-negligible load on the leader when the DC is large enough and can be offloaded to the followers following the "eventually consistent" policy for the agent state. We switch the AE read calls to use stale requests with a small (2s) MaxStaleDuration value and make sure we do not read too fast after a write.	6 years ago
R.B. Boyer	40336fd353	agent: fix several data races and bugs related to node-local alias checks (#5876) ... The observed bug was that a full restart of a consul datacenter (servers and clients) in conjunction with a restart of a connect-flavored application with bring-your-own-service-registration logic would very frequently cause the envoy sidecar service check to never reflect the aliased service. Over the course of investigation several bugs and unfortunate interactions were corrected: (1) local.CheckState objects were only shallow copied, but the key piece of data that gets read and updated is one of the things not copied (the underlying Check with a Status field). When the stock code was run with the race detector enabled this highly-relevant-to-the-test-scenario field was found to be racy. Changes: a) update the existing Clone method to include the Check field b) copy-on-write when those fields need to change rather than incrementally updating them in place. This made the observed behavior occur slightly less often. (2) If anything about how the runLocal method for node-local alias check logic was ever flawed, there was no fallback option. Those checks are purely edge-triggered and failure to properly notice a single edge transition would leave the alias check incorrect until the next flap of the aliased check. The change was to introduce a fallback timer to act as a control loop to double check the alias check matches the aliased check every minute (borrowing the duration from the non-local alias check logic body). This made the observed behavior eventually go away when it did occur. (3) Originally I thought there were two main actions involved in the data race: A. The act of adding the original check (from disk recovery) and its first health evaluation. B. The act of the HTTP API requests coming in and resetting the local state when re-registering the same services and checks. It took awhile for me to realize that there's a third action at work: C. The goroutines associated with the original check and the later checks. The actual sequence of actions that was causing the bad behavior was that the API actions result in the original check to be removed and re-added _without waiting for the original goroutine to terminate_. This means for brief windows of time during check definition edits there are two goroutines that can be sending updates for the alias check status. In extremely unlikely scenarios the original goroutine sees the aliased check start up in `critical` before being removed but does not get the notification about the nearly immediate update of that check to `passing`. This is interlaced wit the new goroutine coming up, initializing its base case to `passing` from the current state and then listening for new notifications of edge triggers. If the original goroutine "finishes" its update, it then commits one more write into the local state of `critical` and exits leaving the alias check no longer reflecting the underlying check. The correction here is to enforce that the old goroutines must terminate before spawning the new one for alias checks.	6 years ago
Freddy	f2213f60e0	Update alias checks on local add and remove	6 years ago