Commit Graph

2010 Commits (5d5bb6cca6db69f5c94c8671235c4f843cab86f4)

Author SHA1 Message Date
Frank Schroeder 727b6444ad dns: fix data races in DNS compression tests
Make the DisableCompression value configurable at runtime
to allow tests to change it without restarting/recreating
the server.
2017-07-07 09:22:34 +02:00
Frank Schroeder 0763788b82 agent: fix data race between consul server and local state 2017-07-07 09:22:34 +02:00
Frank Schroeder 119f6a1ed7 rpc: monkey patch fix for data races for localState
The tests that use the localState of the agent access the internal
variables and call methods which are not guarded by locks creating
data races in tests. While the use of internal variables is somewhat
easy to spot the fact that not all methods are thread-safe is a
surprise.

A proper fix requires the localState struct to be moved into its own
package so that tests in the agent can only access the external
interface.

However, the localState is currently dependent on the agent.Config
which would create a circular dependency. Therefore, the Config
struct needs to be moved first for this to happen.

This patch literally monkey patches the use of the lock around the
cases which have data races and marks them with a
// todo(fs): data race comment.
2017-07-07 09:22:34 +02:00
Frank Schroeder 2159d499e3 rpc: try shutting down leader first to avoid hang in TestLeader_LeftServer 2017-07-07 09:22:34 +02:00
Frank Schroeder f12fac278e rpc: fix logging and try quicker timing of TestServer_JoinSeparateLanAndWanAddresses 2017-07-07 09:22:34 +02:00
Frank Schroeder bae4b1d045 rpc: less agressive raft timeouts
Allowing more time for raft to consolidate should
drop the number of leader elections.
2017-07-07 09:22:34 +02:00
Frank Schroeder 457b98a099 rpc: run agent/consul tests in parallel 2017-07-07 09:22:34 +02:00
Frank Schroeder 13eeeb720d rpc: refactor sessionTimers and fix racy tests
The sessionTimers map was secured by a lock which wasn't used
properly in the tests. This lead to data races and failing tests
when accessing the length or the members of the map.

This patch adds a separate SessionTimers struct which is safe
for concurrent use and which ecapsulates the behavior of the
sessionTimers map.
2017-07-07 09:22:34 +02:00
Frank Schroeder 05f756853e rpc: fix TestServer_Leave
wait for the leader election.
2017-07-07 09:22:34 +02:00
Frank Schroeder 583959392b rpc: fix TestSession_Renew
make the timing less tight
2017-07-07 09:22:34 +02:00
Frank Schroeder ff2c29c0be rpc: fix TestReadyForConsistentRead
timing was too tight. Standardized name.
2017-07-07 09:22:34 +02:00
Frank Schroeder fcab525053 rpc: fix for 'no leader' in TLS tests
Ensure both servers know about each other before looking
for a leader.
2017-07-07 09:22:34 +02:00
Frank Schroeder b2a71fd8b0 rpc: fix TestServer_JoinWAN_Flood
The second server in the first data center should not be
in bootstrap mode.
2017-07-07 09:22:34 +02:00
Frank Schroeder 8369b6cb9d rpc: provide unique node names for server and client 2017-07-07 09:22:34 +02:00
Frank Schroeder 534977239b rpc: prefix log output with test name 2017-07-07 09:22:34 +02:00
Frank Schroeder c8ef588d8d rpc: discover serf wan port before starting serf lan
When using dynamic ports for the serf clusters then
the actual bind port of the serf WAN cluster needs to
be discovered before the serf LAN cluster is started
since the serf LAN cluster announces the port of the WAN
cluster.
2017-07-07 09:22:34 +02:00
Frank Schroeder 53eab7e970 rpc: bind rpc test server to port 0 2017-07-07 09:22:34 +02:00
Frank Schroeder e9e2c599db rpc: refactor: unify test server setup 2017-07-07 09:22:34 +02:00
Frank Schroeder c803146550 rpc: fix typos 2017-07-07 09:22:34 +02:00
Frank Schroeder a0368e3827 agent: refactor: log to stderr during tests 2017-07-07 09:22:34 +02:00
Frank Schroeder 2afafe6911 agent: refactor: use handler for test http tls server 2017-07-07 09:22:34 +02:00
Frank Schroeder 63447a0cf3 agent: refactor: make address translation part of the agent 2017-07-07 09:22:34 +02:00
Preetha Appan f549c06764 Rename to raftNotifyCh, fix typo 2017-07-06 09:10:36 -05:00
Preetha Appan f2171a6720 Fixes deadlock between barrier write and leader notify channel read . Fixes #3230 2017-07-05 17:09:18 -05:00
Grégoire Seux 940b1513a6 Correctly forward Host header in healthcheck (#3203)
Host header must be set explicitely on http requests

Change-Id: I91a32f0fb1ec3fbc713adf0e10869797e91172c7
Signed-off-by: Grégoire Seux <g.seux@criteo.com>
2017-06-29 16:26:08 -07:00
Preetha Appan 07db760d53 Fix missing formatting directive causing go vet to fail 2017-06-27 16:32:38 -05:00
Frank Schroeder 6dd68f7ad7
Revert "agent: add allowStale option for HTTP API (#3142)"
This reverts commit 1e0fd27a74.
2017-06-27 07:04:55 +02:00
Frank Schröder 4b51d00458 agent: fix DNS recursor tests (#3190)
The makeRecursor function was using an unreliable mechanism
to start a server with a random port. This patch changes this
so that the server starts on port 0 to let the kernel pick
a free port.

In addition, to similar functions for starting a test DNS
server were folded into one.
2017-06-25 10:42:37 -07:00
James Phillips 4a3604a3ee
Removes some useless comments. 2017-06-25 10:32:35 -07:00
James Phillips 6977e40077 Fixes watch tracking during reloads and fixes address issue. (#3189)
This patch fixes watch registration through the config file and a broken log line when the watch registration fails. It also plumbs all the watch loading through a common function and tweaks the
unit test to create the watch before the reload.
2017-06-24 12:52:41 -07:00
James Phillips 380c8b957d Changes host-based node IDs from opt-out to opt-in. (#3187) 2017-06-24 09:36:53 -07:00
James Phillips 42f60b04bb Revert "discover: move instance discover code into separate package (#3144)" (#3180)
This reverts commit 26bfb2d00a.
2017-06-23 01:38:55 -07:00
James Phillips e4b11682bc Fixes broken HTTP header and method for health checks. (#3178)
* Fixes broken HTTP header and method for health checks.
* Adds a fuzz utility and test to make sure copy is complete.
2017-06-23 01:15:48 -07:00
wojtkiewicz 1e0fd27a74
agent: add allowStale option for HTTP API (#3142)
This patch adds an "allowStale" option to the HTTP API
configuration which allows stale reads to provide linear
read scalability.

Fixes #3142
2017-06-22 10:31:13 +02:00
wojtkiewicz 37785028be
agent: add "http_config"
This patch adds an "http_config" object to the config file
and moves the "http_api_response_headers" option there.

"http_api_response_headers" is now deprecated in favor of
"http_config.response_headers"
2017-06-22 10:31:11 +02:00
James Phillips 0e695aef94 Fixes checked in web assets and associated build scripts. (#3173) 2017-06-21 14:43:07 -07:00
Frank Schröder 26bfb2d00a discover: move instance discover code into separate package (#3144)
This patch moves the code that discovers instances from metadata
information to github.com/hashicorp/go-discover with
sub-packages for each provider.
2017-06-21 10:40:38 +02:00
Frank Schröder 31a310f551 agent: notify systemd after JoinLAN (#2121)
This patch adds support for notifying systemd via the
NOTIFY_SOCKET by sending 'READY=1' to the socket after
a successful JoinLAN.

Fixes #2121
2017-06-21 06:43:55 +02:00
Frank Schroeder ea5b0f2c7c agent: fix 'consul leave' shutdown race (#2880)
When the agent is triggered to shutdown via an external 'consul leave'
command delivered via the HTTP API then the client expects to receive a
response when the agent is down. This creates a race on when to shutdown
the agent itself like the RPC server, the checks and the state and the
external endpoints like DNS and HTTP.

This patch splits the shutdown process into two parts:

 * shutdown the agent
 * shutdown the endpoints (http and dns)

They can be executed multiple times, concurrently and in any order but
should be executed first agent, then endpoints to provide consistent
behavior across all use cases. Both calls have to be executed for a
proper shutdown.

This could be partially hidden in a single function but would introduce
some magic that happens behind the scenes which one has to know of but
isn't obvious.

Fixes #2880
2017-06-21 05:52:51 +02:00
Frank Schroeder 7abe308c66 agent: drop unused constant 2017-06-21 05:42:39 +02:00
Frank Schroeder c4fc581e07 agent: make registerEndpoint private
This is only used for testing.
2017-06-21 05:42:39 +02:00
Frank Schroeder 2b41f2e3a3 agent: make the RPC endpoint overwrite mechanism more transparent
This patch hides the RPC handler overwrite mechanism from the
rest of the code so that it works in all cases and that there
is no cooperation required from the tested code, i.e. we can
drop a.getEndpoint().
2017-06-21 05:42:39 +02:00
Frank Schroeder e15f9f9d90 agent: rename agent var 2017-06-21 05:42:39 +02:00
Frank Schroeder c49a15d0f3 agent: move structs into consul/structs pkg
* CheckDefinition
 * ServiceDefinition
 * CheckType
2017-06-21 05:42:39 +02:00
Frank Schroeder 4273fb8444 agent: move NotifyGroup into the agent pkg 2017-06-21 05:42:39 +02:00
Frank Schroeder 82a132da60 agent: move conn pool for muxed connections into separate pkg 2017-06-21 05:42:39 +02:00
Frank Schroeder 80971c8a85 agent: move the SnapshotReplyFn out of the way
When splitting up the consul package into server and client
the SnapshotReplyFn needs to be in a separate package to avoid
a circular dependency.
2017-06-21 05:42:39 +02:00
Frank Schroeder 04b9392b00 agent: use the delegate interface for local state 2017-06-21 05:42:39 +02:00
Frank Schroeder d77d2be13e agent: rename clientServer interface to delegate 2017-06-21 05:42:39 +02:00
preetapan 0c2a83e1ec Merge pull request #3154 from hashicorp/issue_2644_redux
Fix stale reads on server startup. Consistent reads will now wait for up to config.RPCHoldTimeout for the server to get past its raft log, before returning an error. Servers that are starting up will eventually catch up. 
This fixes issue #2644
2017-06-20 19:47:12 -05:00
Preetha Appan f658231ab9 Minor fixes per code review 2017-06-20 19:43:07 -05:00
Frank Schroeder b083ce17c7
Revert "agent: fix 'consul leave' shutdown race (#2880)"
This reverts commit 90c83a32b5.
2017-06-19 21:34:08 +02:00
Frank Schroeder 90c83a32b5 agent: fix 'consul leave' shutdown race (#2880)
When the agent is triggered to shutdown via an external 'consul leave'
command delivered via the HTTP API then the client expects to receive a
response when the agent is down. This creates a race on when to shutdown
the agent itself like the RPC server, the checks and the state and the
external endpoints like DNS and HTTP. Ideally, the external endpoints
should be shutdown before the internal state but if the goal is to
respond reliably that the agent is down then this is not possible.

This patch splits the agent shutdown into two parts implemented in a
single method to keep it simple and unambiguos for the caller. The first
stage shuts down the internal state, checks, RPC server, ...
synchronously and then triggers the shutdown of the external endpoints
asychronously. This way the caller is guaranteed that the internal state
services are down when Shutdown returns and there remains enough time to
send a response.

Fixes #2880
2017-06-19 21:24:26 +02:00
Preetha Appan b3b2e9dcb4 Added unit test to verify consistentRead method behavior 2017-06-16 11:58:12 -05:00
Preetha Appan 44f5086873 Code review feedback, fixed major logic bug 2017-06-16 10:49:54 -05:00
Preetha Appan 72af7b9bc4 Redo bug fix for stale reads on server startup, leveraging RPCHOldtimeout instead of maxQueryTime, plus tests 2017-06-15 22:41:30 -05:00
Kyle Havlovitz 5d99ee80ca Add an option to disable keyring file (#3145)
Also disables keyring file in dev mode.
2017-06-15 15:24:04 -07:00
Seth Vargo a8f1e7ebf9 Add EDNS0 support (#3131)
This is a refactor of GH-1980. Originally I tried to do a straight
rebase, but the code has changed too much.
2017-06-14 16:22:54 -07:00
Preetha Appan cf5006c4f4 Fixed static asset build target and checked in new executable for assetfs 2017-06-12 12:57:02 -05:00
Frank Schroeder 1c75cf1af5 pkg refactor
command/agent/*                  -> agent/*
    command/consul/*                 -> agent/consul/*
    command/agent/command{,_test}.go -> command/agent{,_test}.go
    command/base/command.go          -> command/base.go
    command/base/*                   -> command/*
    commands.go                      -> command/commands.go

The script which did the refactor is:

(
	cd $GOPATH/src/github.com/hashicorp/consul
	git mv command/agent/command.go command/agent.go
	git mv command/agent/command_test.go command/agent_test.go
	git mv command/agent/flag_slice_value{,_test}.go command/
	git mv command/agent .
	git mv command/base/command.go command/base.go
	git mv command/base/config_util{,_test}.go command/
	git mv commands.go command/
	git mv consul agent
	rmdir command/base/

	gsed -i -e 's|package agent|package command|' command/agent{,_test}.go
	gsed -i -e 's|package agent|package command|' command/flag_slice_value{,_test}.go
	gsed -i -e 's|package base|package command|' command/base.go command/config_util{,_test}.go
	gsed -i -e 's|package main|package command|' command/commands.go

	gsed -i -e 's|base.Command|BaseCommand|' command/commands.go
	gsed -i -e 's|agent.Command|AgentCommand|' command/commands.go
	gsed -i -e 's|\tCommand:|\tBaseCommand:|' command/commands.go
	gsed -i -e 's|base\.||' command/commands.go
	gsed -i -e 's|command\.||' command/commands.go

	gsed -i -e 's|command|c|' main.go
	gsed -i -e 's|range Commands|range command.Commands|' main.go
	gsed -i -e 's|Commands: Commands|Commands: command.Commands|' main.go

	gsed -i -e 's|base\.BoolValue|BoolValue|' command/operator_autopilot_set.go
	gsed -i -e 's|base\.DurationValue|DurationValue|' command/operator_autopilot_set.go
	gsed -i -e 's|base\.StringValue|StringValue|' command/operator_autopilot_set.go
	gsed -i -e 's|base\.UintValue|UintValue|' command/operator_autopilot_set.go

	gsed -i -e 's|\bCommand\b|BaseCommand|' command/base.go
	gsed -i -e 's|BaseCommand Options|Command Options|' command/base.go
	gsed -i -e 's|base.Command|BaseCommand|' command/*.go
	gsed -i -e 's|c\.Command|c.BaseCommand|g' command/*.go
	gsed -i -e 's|\tCommand:|\tBaseCommand:|' command/*_test.go
	gsed -i -e 's|base\.||' command/*_test.go

	gsed -i -e 's|\bCommand\b|AgentCommand|' command/agent{,_test}.go
	gsed -i -e 's|cmd.AgentCommand|cmd.BaseCommand|' command/agent.go

	gsed -i -e 's|cli.AgentCommand = new(Command)|cli.Command = new(AgentCommand)|' command/agent_test.go
	gsed -i -e 's|exec.AgentCommand|exec.Command|' command/agent_test.go
	gsed -i -e 's|exec.BaseCommand|exec.Command|' command/agent_test.go
	gsed -i -e 's|NewTestAgent|agent.NewTestAgent|' command/agent_test.go
	gsed -i -e 's|= TestConfig|= agent.TestConfig|' command/agent_test.go
	gsed -i -e 's|: RetryJoin|: agent.RetryJoin|' command/agent_test.go

	gsed -i -e 's|\.\./\.\./|../|' command/config_util_test.go

	gsed -i -e 's|\bverifyUniqueListeners|VerifyUniqueListeners|' agent/config{,_test}.go command/agent.go
	gsed -i -e 's|\bserfLANKeyring\b|SerfLANKeyring|g' agent/{agent,keyring,testagent}.go command/agent.go
	gsed -i -e 's|\bserfWANKeyring\b|SerfWANKeyring|g' agent/{agent,keyring,testagent}.go command/agent.go
	gsed -i -e 's|\bNewAgent\b|agent.New|g' command/agent{,_test}.go
	gsed -i -e 's|\bNewAgent|New|' agent/{acl_test,agent,testagent}.go

	gsed -i -e 's|\bAgent\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bBool\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bConfig\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bDefaultConfig\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bDevConfig\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bMergeConfig\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bReadConfigPaths\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bParseMetaPair\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bSerfLANKeyring\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bSerfWANKeyring\b|agent.&|g' command/agent{,_test}.go

	gsed -i -e 's|circonus\.agent|circonus|g' command/agent{,_test}.go
	gsed -i -e 's|logger\.agent|logger|g' command/agent{,_test}.go
	gsed -i -e 's|metrics\.agent|metrics|g' command/agent{,_test}.go
	gsed -i -e 's|// agent.Agent|// agent|' command/agent{,_test}.go
	gsed -i -e 's|a\.agent\.Config|a.Config|' command/agent{,_test}.go

	gsed -i -e 's|agent\.AppendSliceValue|AppendSliceValue|' command/{configtest,validate}.go

	gsed -i -e 's|consul/consul|agent/consul|' GNUmakefile

	gsed -i -e 's|\.\./test|../../test|' agent/consul/server_test.go

	# fix imports
	f=$(grep -rl 'github.com/hashicorp/consul/command/agent' * | grep '\.go')
	gsed -i -e 's|github.com/hashicorp/consul/command/agent|github.com/hashicorp/consul/agent|' $f
	goimports -w $f

	f=$(grep -rl 'github.com/hashicorp/consul/consul' * | grep '\.go')
	gsed -i -e 's|github.com/hashicorp/consul/consul|github.com/hashicorp/consul/agent/consul|' $f
	goimports -w $f

	goimports -w command/*.go main.go
)
2017-06-10 18:52:45 +02:00