mirror of https://github.com/hashicorp/consul
122 lines
4.8 KiB
Plaintext
122 lines
4.8 KiB
Plaintext
|
---
|
||
|
|
layout: docs
|
||
|
|
page_title: Jepsen Testing
|
||
|
|
description: >-
|
||
|
|
Jepsen is a tool, written by Kyle Kingsbury, designed to test the partition
|
||
|
|
tolerance of distributed systems. It creates network partitions while fuzzing
|
||
|
|
the system with random operations. The results are analyzed to see if the
|
||
|
|
system violates any of the consistency properties it claims to have.
|
||
|
|
---
|
||
|
|
|
||
|
|
# Jepsen Testing
|
||
|
|
|
||
|
|
[Jepsen](http://aphyr.com/posts/281-call-me-maybe-carly-rae-jepsen-and-the-perils-of-network-partitions)
|
||
|
|
is a tool, written by Kyle Kingsbury, designed to test the partition
|
||
|
|
tolerance of distributed systems. It creates network partitions while fuzzing
|
||
|
|
the system with random operations. The results are analyzed to see if the system
|
||
|
|
violates any of the consistency properties it claims to have.
|
||
|
|
|
||
|
|
As part of our Consul testing, we ran a Jepsen test to determine if
|
||
|
|
any consistency issues could be uncovered. In our testing, Consul
|
||
|
|
gracefully recovered from partitions without introducing any consistency
|
||
|
|
issues.
|
||
|
|
|
||
|
|
## Running the tests
|
||
|
|
|
||
|
|
At the moment, testing with Jepsen is rather complex as it requires
|
||
|
|
setting up multiple virtual machines, SSH keys, DNS configuration,
|
||
|
|
and a working Clojure environment. We hope to contribute our Consul
|
||
|
|
testing code upstream and to provide a Vagrant environment for Jepsen
|
||
|
|
testing soon.
|
||
|
|
|
||
|
|
## Output
|
||
|
|
|
||
|
|
Below is the output captured from Jepsen. We ran Jepsen multiple times,
|
||
|
|
and it passed each time. This output is only representative of a single
|
||
|
|
run and has been edited for length. Please reach out on [Consul's Discuss](https://discuss.hashicorp.com/c/consul)
|
||
|
|
if you would like to reproduce the Jepsen results.
|
||
|
|
|
||
|
|
<!--googleoff: all-->
|
||
|
|
|
||
|
|
```shell-session
|
||
|
|
$ lein test :only jepsen.system.consul-test
|
||
|
|
|
||
|
|
lein test jepsen.system.consul-test
|
||
|
|
INFO jepsen.os.debian - :n5 setting up debian
|
||
|
|
INFO jepsen.os.debian - :n3 setting up debian
|
||
|
|
INFO jepsen.os.debian - :n4 setting up debian
|
||
|
|
INFO jepsen.os.debian - :n1 setting up debian
|
||
|
|
INFO jepsen.os.debian - :n2 setting up debian
|
||
|
|
INFO jepsen.os.debian - :n4 debian set up
|
||
|
|
INFO jepsen.os.debian - :n5 debian set up
|
||
|
|
INFO jepsen.os.debian - :n3 debian set up
|
||
|
|
INFO jepsen.os.debian - :n1 debian set up
|
||
|
|
INFO jepsen.os.debian - :n2 debian set up
|
||
|
|
INFO jepsen.system.consul - :n1 consul nuked
|
||
|
|
INFO jepsen.system.consul - :n4 consul nuked
|
||
|
|
INFO jepsen.system.consul - :n5 consul nuked
|
||
|
|
INFO jepsen.system.consul - :n3 consul nuked
|
||
|
|
INFO jepsen.system.consul - :n2 consul nuked
|
||
|
|
INFO jepsen.system.consul - Running nodes: {:n1 false, :n2 false, :n3 false, :n4 false, :n5 false}
|
||
|
|
INFO jepsen.system.consul - :n2 consul nuked
|
||
|
|
INFO jepsen.system.consul - :n3 consul nuked
|
||
|
|
INFO jepsen.system.consul - :n4 consul nuked
|
||
|
|
INFO jepsen.system.consul - :n5 consul nuked
|
||
|
|
INFO jepsen.system.consul - :n1 consul nuked
|
||
|
|
INFO jepsen.system.consul - :n1 starting consul
|
||
|
|
INFO jepsen.system.consul - :n2 starting consul
|
||
|
|
INFO jepsen.system.consul - :n4 starting consul
|
||
|
|
INFO jepsen.system.consul - :n5 starting consul
|
||
|
|
INFO jepsen.system.consul - :n3 starting consul
|
||
|
|
INFO jepsen.system.consul - :n3 consul ready
|
||
|
|
INFO jepsen.system.consul - :n2 consul ready
|
||
|
|
INFO jepsen.system.consul - Running nodes: {:n1 true, :n2 true, :n3 true, :n4 true, :n5 true}
|
||
|
|
INFO jepsen.system.consul - :n5 consul ready
|
||
|
|
INFO jepsen.system.consul - :n1 consul ready
|
||
|
|
INFO jepsen.system.consul - :n4 consul ready
|
||
|
|
INFO jepsen.core - Worker 0 starting
|
||
|
|
INFO jepsen.core - Worker 2 starting
|
||
|
|
INFO jepsen.core - Worker 1 starting
|
||
|
|
INFO jepsen.core - Worker 3 starting
|
||
|
|
INFO jepsen.core - Worker 4 starting
|
||
|
|
INFO jepsen.util - 2 :invoke :read nil
|
||
|
|
INFO jepsen.util - 3 :invoke :cas [4 4]
|
||
|
|
INFO jepsen.util - 0 :invoke :write 4
|
||
|
|
INFO jepsen.util - 1 :invoke :write 1
|
||
|
|
INFO jepsen.util - 4 :invoke :cas [4 0]
|
||
|
|
INFO jepsen.util - 2 :ok :read nil
|
||
|
|
INFO jepsen.util - 4 :fail :cas [4 0]
|
||
|
|
(Log Truncated...)
|
||
|
|
INFO jepsen.util - 4 :invoke :cas [3 3]
|
||
|
|
INFO jepsen.util - 4 :fail :cas [3 3]
|
||
|
|
INFO jepsen.util - :nemesis :info :stop nil
|
||
|
|
INFO jepsen.util - :nemesis :info :stop "fully connected"
|
||
|
|
INFO jepsen.util - 0 :fail :read nil
|
||
|
|
INFO jepsen.util - 1 :fail :write 0
|
||
|
|
INFO jepsen.util - :nemesis :info :stop nil
|
||
|
|
INFO jepsen.util - :nemesis :info :stop "fully connected"
|
||
|
|
INFO jepsen.core - nemesis done
|
||
|
|
INFO jepsen.core - Worker 3 done
|
||
|
|
INFO jepsen.util - 1 :invoke :read nil
|
||
|
|
INFO jepsen.core - Worker 2 done
|
||
|
|
INFO jepsen.core - Worker 4 done
|
||
|
|
INFO jepsen.core - Worker 0 done
|
||
|
|
INFO jepsen.util - 1 :ok :read 3
|
||
|
|
INFO jepsen.core - Worker 1 done
|
||
|
|
INFO jepsen.core - Run complete, writing
|
||
|
|
INFO jepsen.core - Analyzing
|
||
|
|
(Log Truncated...)
|
||
|
|
INFO jepsen.core - Analysis complete
|
||
|
|
INFO jepsen.system.consul - :n3 consul nuked
|
||
|
|
INFO jepsen.system.consul - :n2 consul nuked
|
||
|
|
INFO jepsen.system.consul - :n4 consul nuked
|
||
|
|
INFO jepsen.system.consul - :n1 consul nuked
|
||
|
|
INFO jepsen.system.consul - :n5 consul nuked
|
||
|
|
1964 element history linearizable. :D
|
||
|
|
|
||
|
|
Ran 1 tests containing 1 assertions.
|
||
|
|
0 failures, 0 errors.
|
||
|
|
```
|
||
|
|
|
||
|
|
<!--googleon: all-->
|