github
/
consul
mirror of https://github.com/hashicorp/consul
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
consul/agent
History
Derek Menteer 0ac8ae6c3b
Fix xDS deadlock due to syncLoop termination. (#20867) 
* Fix xDS deadlock due to syncLoop termination.

This fixes an issue where agentless xDS streams can deadlock permanently until
a server is restarted. When this issue occurs, no new proxies are able to
successfully connect to the server.

Effectively, the trigger for this deadlock stems from the following return
statement:
https://github.com/hashicorp/consul/blob/v1.18.0/agent/proxycfg-sources/catalog/config_source.go#L199-L202

When this happens, the entire `syncLoop()` terminates and stops consuming from
the following channel:
https://github.com/hashicorp/consul/blob/v1.18.0/agent/proxycfg-sources/catalog/config_source.go#L182-L192

Which results in the `ConfigSource.cleanup()` function never receiving a
response and holding a mutex indefinitely:
https://github.com/hashicorp/consul/blob/v1.18.0/agent/proxycfg-sources/catalog/config_source.go#L241-L247

Because this mutex is shared, it effectively deadlocks the server's ability to
process new xDS streams.

----

The fix to this issue involves removing the `chan chan struct{}` used like an
RPC-over-channels pattern and replacing it with two distinct channels:

+ `stopSyncLoopCh` - indicates that the `syncLoop()` should terminate soon.  +
`syncLoopDoneCh` - indicates that the `syncLoop()` has terminated.

Splitting these two concepts out and deferring a `close(syncLoopDoneCh)` in the
`syncLoop()` function ensures that the deadlock above should no longer occur.

We also now evict xDS connections of all proxies for the corresponding
`syncLoop()` whenever it encounters an irrecoverable error. This is done by
hoisting the new `syncLoopDoneCh` upwards so that it's visible to the xDS delta
processing. Prior to this fix, the behavior was to simply orphan them so they
would never receive catalog-registration or service-defaults updates.

* Add changelog.
 		2024-03-15 13:57:11 -05:00
..
ae server: when the v2 catalog experiment is enabled reject api and rpc requests that are for the v1 catalog (#19129) 2023-10-11 10:44:03 -05:00
auto-config [NET-6593] agent: check for minimum RSA key size (#20112) 2024-01-10 12:15:36 +00:00
blockingquery [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
cache xds controller: setup watches for and compute leaf cert references in ProxyStateTemplate, and wire up leaf cert manager dependency (#18756) 2023-09-12 12:56:43 -07:00
cache-types xds controller: setup watches for and compute leaf cert references in ProxyStateTemplate, and wire up leaf cert manager dependency (#18756) 2023-09-12 12:56:43 -07:00
cacheshim xds controller: setup watches for and compute leaf cert references in ProxyStateTemplate, and wire up leaf cert manager dependency (#18756) 2023-09-12 12:56:43 -07:00
checks Remove old build tags (#19128) 2023-10-10 10:58:06 -04:00
config [CC-7411] Fix environment variable precedence when linking to HCP (#20527) 2024-02-13 14:06:18 -06:00
configentry Hash based config entry replication (#19795) 2023-12-12 08:29:13 -05:00
connect [NET-6593] agent: check for minimum RSA key size (#20112) 2024-01-10 12:15:36 +00:00
consul NET-6821 Disable Terminating Gateway Auto Host Header Rewrite (#20802) 2024-03-12 15:37:20 -05:00
debug [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
discovery DNS v2 - split up router into multiple responsibilities & break up router tests into multiple files. (#20688) 2024-03-01 15:36:37 +00:00
dns DNS v2 - split up router into multiple responsibilities & break up router tests into multiple files. (#20688) 2024-03-01 15:36:37 +00:00
envoyextensions NET-6946 / NET-6941 - Replace usage of deprecated Envoy fields envoy.config.route.v3.HeaderMatcher.safe_regex_match and envoy.type.matcher.v3.RegexMatcher.google_re2 (#20013) 2024-01-03 09:53:39 -07:00
exec Remove old build tags (#19128) 2023-10-10 10:58:06 -04:00
grpc-external Various bug-fixes and improvements (#20866) 2024-03-15 13:10:48 -05:00
grpc-internal In-Memory gRPC (#19942) 2024-01-12 11:54:07 -05:00
grpc-middleware security: upgrade google.golang.org/protobuf to 1.33.0 (#20801) 2024-03-06 23:04:42 +00:00
hcp [CC-7411] Fix environment variable precedence when linking to HCP (#20527) 2024-02-13 14:06:18 -06:00
leafcert xds controller: setup watches for and compute leaf cert references in ProxyStateTemplate, and wire up leaf cert manager dependency (#18756) 2023-09-12 12:56:43 -07:00
local bug: prevent go routine leakage due to existing DeferCheck (#18558) 2023-08-23 10:33:07 -04:00
log-drop [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
metadata [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
metrics [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
mock [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
pool [NET-6459] Fix issue with wanfed lan ip conflicts. (#19503) 2023-11-06 08:47:12 -06:00
proxycfg Fix xDS deadlock due to syncLoop termination. (#20867) 2024-03-15 13:57:11 -05:00
proxycfg-glue Add default intention policy (#20544) 2024-02-08 20:25:42 +00:00
proxycfg-sources Fix xDS deadlock due to syncLoop termination. (#20867) 2024-03-15 13:57:11 -05:00
router [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
routine-leak-checker [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
rpc Panic for unregistered types (#20476) 2024-02-06 11:23:06 -05:00
rpcclient OSS -> CE (community edition) changes (#18517) 2023-08-22 09:46:03 -05:00
structs NET-6821 Disable Terminating Gateway Auto Host Header Rewrite (#20802) 2024-03-12 15:37:20 -05:00
submatview [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
systemd [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
token Remove old build tags (#19128) 2023-10-10 10:58:06 -04:00
uiserver NET-5398: Update UI server to include if v2 is enabled (#20353) 2024-01-26 14:38:51 -07:00
xds Fix xDS deadlock due to syncLoop termination. (#20867) 2024-03-15 13:57:11 -05:00
xdsv2 add traffic permissions excludes and tests (#20453) 2024-02-07 20:21:44 +00:00
acl.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
acl_ce.go Remove old build tags (#19128) 2023-10-10 10:58:06 -04:00
acl_endpoint.go [NET-6249] Add templated policies description (#19735) 2023-11-27 10:34:22 -05:00
acl_endpoint_test.go Various race condition and test fixes. (#20212) 2024-01-16 08:57:43 -06:00
acl_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
agent.go NET-7644/NET-7634 - Implement query lookup for tagged addresses on nodes and services including WAN translation. (#20583) 2024-02-12 14:27:25 -05:00
agent_ce.go Remove old build tags (#19128) 2023-10-10 10:58:06 -04:00
agent_ce_test.go Remove old build tags (#19128) 2023-10-10 10:58:06 -04:00
agent_endpoint.go Add default intention policy (#20544) 2024-02-08 20:25:42 +00:00
agent_endpoint_ce.go Remove old build tags (#19128) 2023-10-10 10:58:06 -04:00
agent_endpoint_ce_test.go Remove old build tags (#19128) 2023-10-10 10:58:06 -04:00
agent_endpoint_test.go Add default intention policy (#20544) 2024-02-08 20:25:42 +00:00
agent_test.go Move HCP Manager lifecycle management out of Link controller (#20401) 2024-02-12 10:48:23 -05:00
apiserver.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
apiserver_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
catalog_endpoint.go NET-7644/NET-7634 - Implement query lookup for tagged addresses on nodes and services including WAN translation. (#20583) 2024-02-12 14:27:25 -05:00
catalog_endpoint_ce.go Remove old build tags (#19128) 2023-10-10 10:58:06 -04:00
catalog_endpoint_test.go NET-6821 Disable Terminating Gateway Auto Host Header Rewrite (#20802) 2024-03-12 15:37:20 -05:00
check.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
config_endpoint.go NET-5824 Exported services api (#20015) 2024-01-23 10:06:59 +05:30
config_endpoint_test.go Fix issue with persisting proxy-defaults (#20481) 2024-02-05 16:00:19 -06:00
connect_ca_endpoint.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
connect_ca_endpoint_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
coordinate_endpoint.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
coordinate_endpoint_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
delegate_mock_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
denylist.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
denylist_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
discovery_chain_endpoint.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
discovery_chain_endpoint_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
dns.go NET-7813 - DNS : SERVFAIL when resolving PTR records (#20679) 2024-02-21 17:44:04 +00:00
dns_catalogv2_test.go [CE] Misc cleanup for V2 DNS (#20640) 2024-02-14 12:40:38 -05:00
dns_ce.go DNS v2 - add virtual ip questions (#20245) 2024-01-17 23:46:18 +00:00
dns_ce_test.go feat(v2dns): enable peering queries (#20581) 2024-02-12 14:25:45 -05:00
dns_node_lookup_test.go fix(v2dns): add node ttl to workloads, comment cleanup, and changelog (#20643) 2024-02-14 17:38:11 -05:00
dns_reverse_lookup_test.go NET-7813 - DNS : SERVFAIL when resolving PTR records (#20679) 2024-02-21 17:44:04 +00:00
dns_service_lookup_test.go NET-7813 - DNS : SERVFAIL when resolving PTR records (#20679) 2024-02-21 17:44:04 +00:00
dns_test.go NET-8056 - v2 DNS Testing Improvements (#20710) 2024-03-01 05:42:42 -07:00
enterprise_delegate_ce.go Remove old build tags (#19128) 2023-10-10 10:58:06 -04:00
event_endpoint.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
event_endpoint_test.go Retry lint fixes (#19151) 2023-12-06 12:11:32 -05:00
federation_state_endpoint.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
health_endpoint.go NET-7644/NET-7634 - Implement query lookup for tagged addresses on nodes and services including WAN translation. (#20583) 2024-02-12 14:27:25 -05:00
health_endpoint_test.go Retry lint fixes (#19151) 2023-12-06 12:11:32 -05:00
http.go Fix audit-log encoding issue (CC-7337) (#20345) 2024-02-06 16:40:07 +05:30
http_ce.go Fix audit-log encoding issue (CC-7337) (#20345) 2024-02-06 16:40:07 +05:30
http_ce_test.go OSS -> CE (community edition) changes (#18517) 2023-08-22 09:46:03 -05:00
http_decode_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
http_register.go NET-5824 Exported services api (#20015) 2024-01-23 10:06:59 +05:30
http_test.go Retry lint fixes (#19151) 2023-12-06 12:11:32 -05:00
intentions_endpoint.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
intentions_endpoint_ce_test.go Remove old build tags (#19128) 2023-10-10 10:58:06 -04:00
intentions_endpoint_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
keyring.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
keyring_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
kvs_endpoint.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
kvs_endpoint_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
metrics.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
metrics_test.go Various race condition and test fixes. (#20212) 2024-01-16 08:57:43 -06:00
nodeid.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
nodeid_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
notify.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
notify_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
operator_endpoint.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
operator_endpoint_ce.go Remove old build tags (#19128) 2023-10-10 10:58:06 -04:00
operator_endpoint_ce_test.go Remove old build tags (#19128) 2023-10-10 10:58:06 -04:00
operator_endpoint_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
peering_endpoint.go OSS -> CE (community edition) changes (#18517) 2023-08-22 09:46:03 -05:00
peering_endpoint_ce_test.go Remove old build tags (#19128) 2023-10-10 10:58:06 -04:00
peering_endpoint_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
prepared_query_endpoint.go NET-7644/NET-7634 - Implement query lookup for tagged addresses on nodes and services including WAN translation. (#20583) 2024-02-12 14:27:25 -05:00
prepared_query_endpoint_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
proxycfg_test.go Fix xDS deadlock due to syncLoop termination. (#20867) 2024-03-15 13:57:11 -05:00
reload.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
remote_exec.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
remote_exec_test.go Retry lint fixes (#19151) 2023-12-06 12:11:32 -05:00
retry_join.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
retry_join_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
service_checks_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
service_manager.go Ensure that upstream configuration is properly normalized. (#19076) 2023-10-06 13:59:47 -05:00
service_manager_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
session_endpoint.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
session_endpoint_test.go Retry lint fixes (#19151) 2023-12-06 12:11:32 -05:00
setup.go [CE] feat(v2dns): add v2 style query metrics (#20608) 2024-02-13 12:08:01 -05:00
setup_ce.go Remove old build tags (#19128) 2023-10-10 10:58:06 -04:00
sidecar_service.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
sidecar_service_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
signal_unix.go Remove old build tags (#19128) 2023-10-10 10:58:06 -04:00
signal_windows.go Remove old build tags (#19128) 2023-10-10 10:58:06 -04:00
snapshot_endpoint.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
snapshot_endpoint_test.go Fix more test flakes (#19533) 2023-11-07 10:15:50 -06:00
status_endpoint.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
status_endpoint_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
streaming_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
testagent.go Various race condition and test fixes. (#20212) 2024-01-16 08:57:43 -06:00
testagent_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
translate_addr.go NET-7644/NET-7634 - Implement query lookup for tagged addresses on nodes and services including WAN translation. (#20583) 2024-02-12 14:27:25 -05:00
txn_endpoint.go Add TCP+TLS Healthchecks (#18381) 2023-09-05 13:34:44 -07:00
txn_endpoint_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
ui_endpoint.go [NET-5688] APIGateway UI Topology Fixes (#19657) 2023-11-28 21:27:14 +00:00
ui_endpoint_ce_test.go Remove old build tags (#19128) 2023-10-10 10:58:06 -04:00
ui_endpoint_test.go server: when the v2 catalog experiment is enabled reject api and rpc requests that are for the v1 catalog (#19129) 2023-10-11 10:44:03 -05:00
user_event.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
user_event_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
util.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
util_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
watch_handler.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
watch_handler_test.go [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00