mirror of https://github.com/hashicorp/consul
Armon Dadgar
11 years ago
parent
779e6df7b8
commit
262c039425
@ -0,0 +1,117 @@
|
|||||||
|
---
|
||||||
|
layout: "docs"
|
||||||
|
page_title: "Consul Architecture"
|
||||||
|
sidebar_current: "docs-internals-architecture"
|
||||||
|
---
|
||||||
|
|
||||||
|
# Consul Architecture
|
||||||
|
|
||||||
|
Consul is a complex system that has many different moving parts. To help
|
||||||
|
users and developers of Consul form a mental model of how it works, this
|
||||||
|
page documents the system architecture.
|
||||||
|
|
||||||
|
<div class="alert alert-block alert-warning">
|
||||||
|
<strong>Advanced Topic!</strong> This page covers technical details of
|
||||||
|
the internals of Consul. You don't need to know these details to effectively
|
||||||
|
operate and use Consul. These details are documented here for those who wish
|
||||||
|
to learn about them without having to go spelunking through the source code.
|
||||||
|
</div>
|
||||||
|
|
||||||
|
## Glossary
|
||||||
|
|
||||||
|
Before describing the architecture, we provide a glossary of terms to help
|
||||||
|
clarify what is being discussed:
|
||||||
|
|
||||||
|
* Agent - An agent is the long running daemon on every member of the Consul cluster.
|
||||||
|
It is started by running `consul agent`. The agent is able to run in either *client*,
|
||||||
|
or *server* mode. Since all nodes must be running an agent, it is simpler to refer to
|
||||||
|
the node as either being a client or server, but other are instances of the agent. All
|
||||||
|
agents can run the DNS or HTTP interfaces, and are responsible for running checks and
|
||||||
|
keeping services in sync.
|
||||||
|
|
||||||
|
* Client - A client is an agent that forwards all RPC's to a server. The client is relatively
|
||||||
|
stateless. The only background activity a client performs is taking part of LAN gossip pool.
|
||||||
|
This has a minimal resource overhead and consumes only a small amount of network bandwidth.
|
||||||
|
|
||||||
|
* Server - An agent that is server mode. When in server mode, there is an expanded set
|
||||||
|
of responsibilities including participated in the Raft quorum, maintaining cluster state,
|
||||||
|
responding to RPC queries, WAN gossip to other datacenters, forwarding of queries to leaders
|
||||||
|
or remote datacenters.
|
||||||
|
|
||||||
|
* Datacenter - A data center seems obvious, but there are subtle details such as multiple
|
||||||
|
availability zones in EC2. We define a data center to be a networking environment that is
|
||||||
|
private, low latency, and high badwidth. This excludes communication that would traverse
|
||||||
|
the public internet.
|
||||||
|
|
||||||
|
* Consensus - When used in our documentation we use consensus to mean agreement upon
|
||||||
|
the elected leader as well as agreement on the ordering of transactions. Since these
|
||||||
|
transactions are applied to a FSM, we implicitly include the consistency of a replicated
|
||||||
|
state machine. Consensus is described in more detail on [Wikipedia](http://en.wikipedia.org/wiki/Consensus_(computer_science)),
|
||||||
|
as well as our [implementation here](/docs/internals/consensus.html).
|
||||||
|
|
||||||
|
* Gossip - Consul is built on top of [Serf](http://www.serfdom.io/), which provides a full
|
||||||
|
[gossip protocol](http://en.wikipedia.org/wiki/Gossip_protocol) that is used for multiple purposes.
|
||||||
|
Serf provides membership, failure detection, and event broadcast mechanisms. Our use of these
|
||||||
|
is described more in the [gossip documentation](/docs/internals/gossip.html). It is enough to know
|
||||||
|
gossip involves random node-to-node communication, primary over UDP.
|
||||||
|
|
||||||
|
* LAN Gossip - This is used to mean that there is a gossip pool, containing nodes that
|
||||||
|
are all located on the same local area network or datacenter.
|
||||||
|
|
||||||
|
* WAN Gossip - This is used to mean that there is a gossip pool, containing servers that
|
||||||
|
are primary located in different datacenters and must communicate over the internet or
|
||||||
|
wide area network.
|
||||||
|
|
||||||
|
* RPC - RPC is short for a Remote Procedure Call. This is a request / response mechanism
|
||||||
|
allowing a client to make a request from a server.
|
||||||
|
|
||||||
|
## 10,000 foot view
|
||||||
|
|
||||||
|
From a 10,000 foot altitude the architecture of Consul looks like this:
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Lets break down this image and describe each piece. First of all we can see
|
||||||
|
that there are two datacenters, one and two respectively. Consul has first
|
||||||
|
class support for multiple data centers and expects this to be the common case.
|
||||||
|
|
||||||
|
Within each datacenter we have a mixture of clients, and servers. It is expected
|
||||||
|
that there be between three and five servers. This strikes a balance between
|
||||||
|
availability in the case of failure and performance, as consensus gets progressively
|
||||||
|
slower as more machines are added. However, there is no limit to the number of clients,
|
||||||
|
and they can easily scale into the thousands or tens of thousands.
|
||||||
|
|
||||||
|
All the nodes that are in a datacenter participate in a [gossip protocol](/docs/internals/gossip.html).
|
||||||
|
This means is there is a Serf cluster that contains all the nodes for a given datacenter. This serves
|
||||||
|
a few purposes: first, there is no need to configure clients with the addresses of servers,
|
||||||
|
that discovery is done automatically using Serf. Second, the work of detecting node failures
|
||||||
|
is not placed on the servers but is distributed. This makes the failure detection much more
|
||||||
|
scalable than naive heartbeating schemes. Thirdly, it is used as a messaging layer to notify
|
||||||
|
when important events such as leader election take place.
|
||||||
|
|
||||||
|
The servers in each datacenter are all part of a single Raft peer set. This means that
|
||||||
|
they work together to elect a leader, which has extra duties. The leader is responsible for
|
||||||
|
processing all queries and transactions. Transactions must also be replicated to all peers
|
||||||
|
as part of the [consensus protocol](/docs/internals/consensus.html). Because of this requirement,
|
||||||
|
when a non-leader server receives an RPC request it forwards it to the cluster leader.
|
||||||
|
|
||||||
|
The server nodes also operate as part of a WAN gossip. This pool is different from the LAN pool,
|
||||||
|
as it is optimized for the higher latency of the internet, and is expected to only contain
|
||||||
|
other Consul server nodes. The purpose of this pool is to allow datacenters to discover each
|
||||||
|
other in a low touch manner. Bringing a new datacenter online is as easy as joining the existing
|
||||||
|
WAN gossip. Because the servers are all operating in this pool, it also enables cross-dc requests.
|
||||||
|
When a server receives a request for a different datacenter, it forwards it to a random server
|
||||||
|
in the correct datacenter. That server may then forward to the local leader.
|
||||||
|
|
||||||
|
This results in a very low coupling between datacenters, but because of a Serf failure detection,
|
||||||
|
connection caching and multiplexing, cross-dc requests are relatively fast and reliable.
|
||||||
|
|
||||||
|
## Getting in depth
|
||||||
|
|
||||||
|
At this point we've covered the high level architecture of Consul, but there are much
|
||||||
|
more details to each of the sub-systems. The [consensus protocol](/docs/internals/consensus.html) is
|
||||||
|
documented in detail, as is the [gossip protocol](/docs/internals/gossip.html). The [documentation](/docs/internals/security.html)
|
||||||
|
for the security model and protocols used for is also available.
|
||||||
|
|
||||||
|
For other details, either consult the code, ask in IRC or reach out to the mailing list.
|
||||||
|
|
||||||
@ -0,0 +1,103 @@
|
|||||||
|
---
|
||||||
|
layout: "docs"
|
||||||
|
page_title: "Consensus Protocol"
|
||||||
|
sidebar_current: "docs-internals-consensus"
|
||||||
|
---
|
||||||
|
|
||||||
|
# Consensus Protocol
|
||||||
|
|
||||||
|
Serf uses a [gossip protocol](http://en.wikipedia.org/wiki/Gossip_protocol)
|
||||||
|
to broadcast messages to the cluster. This page documents the details of
|
||||||
|
this internal protocol. The gossip protocol is based on
|
||||||
|
["SWIM: Scalable Weakly-consistent Infection-style Process Group Membership Protocol"](http://www.cs.cornell.edu/~asdas/research/dsn02-swim.pdf),
|
||||||
|
with a few minor adaptations, mostly to increase propagation speed
|
||||||
|
and convergence rate.
|
||||||
|
|
||||||
|
<div class="alert alert-block alert-warning">
|
||||||
|
<strong>Advanced Topic!</strong> This page covers the technical details of
|
||||||
|
the internals of Serf. You don't need to know these details to effectively
|
||||||
|
operate and use Serf. These details are documented here for those who wish
|
||||||
|
to learn about them without having to go spelunking through the source code.
|
||||||
|
</div>
|
||||||
|
|
||||||
|
## SWIM Protocol Overview
|
||||||
|
|
||||||
|
Serf begins by joining an existing cluster or starting a new
|
||||||
|
cluster. If starting a new cluster, additional nodes are expected to join
|
||||||
|
it. New nodes in an existing cluster must be given the address of at
|
||||||
|
least one existing member in order to join the cluster. The new member
|
||||||
|
does a full state sync with the existing member over TCP and begins gossiping its
|
||||||
|
existence to the cluster.
|
||||||
|
|
||||||
|
Gossip is done over UDP with a configurable but fixed fanout and interval.
|
||||||
|
This ensures that network usage is constant with regards to number of nodes.
|
||||||
|
Complete state exchanges with a random node are done periodically over
|
||||||
|
TCP, but much less often than gossip messages. This increases the likelihood
|
||||||
|
that the membership list converges properly since the full state is exchanged
|
||||||
|
and merged. The interval between full state exchanges is configurable or can
|
||||||
|
be disabled entirely.
|
||||||
|
|
||||||
|
Failure detection is done by periodic random probing using a configurable interval.
|
||||||
|
If the node fails to ack within a reasonable time (typically some multiple
|
||||||
|
of RTT), then an indirect probe is attempted. An indirect probe asks a
|
||||||
|
configurable number of random nodes to probe the same node, in case there
|
||||||
|
are network issues causing our own node to fail the probe. If both our
|
||||||
|
probe and the indirect probes fail within a reasonable time, then the
|
||||||
|
node is marked "suspicious" and this knowledge is gossiped to the cluster.
|
||||||
|
A suspicious node is still considered a member of cluster. If the suspect member
|
||||||
|
of the cluster does not dispute the suspicion within a configurable period of
|
||||||
|
time, the node is finally considered dead, and this state is then gossiped
|
||||||
|
to the cluster.
|
||||||
|
|
||||||
|
This is a brief and incomplete description of the protocol. For a better idea,
|
||||||
|
please read the
|
||||||
|
[SWIM paper](http://www.cs.cornell.edu/~asdas/research/dsn02-swim.pdf)
|
||||||
|
in its entirety, along with the Serf source code.
|
||||||
|
|
||||||
|
## SWIM Modifications
|
||||||
|
|
||||||
|
As mentioned earlier, the gossip protocol is based on SWIM but includes
|
||||||
|
minor changes, mostly to increase propogation speed and convergence rates.
|
||||||
|
|
||||||
|
The changes from SWIM are noted here:
|
||||||
|
|
||||||
|
* Serf does a full state sync over TCP periodically. SWIM only propagates
|
||||||
|
changes over gossip. While both are eventually consistent, Serf is able to
|
||||||
|
more quickly reach convergence, as well as gracefully recover from network
|
||||||
|
partitions.
|
||||||
|
|
||||||
|
* Serf has a dedicated gossip layer separate from the failure detection
|
||||||
|
protocol. SWIM only piggybacks gossip messages on top of probe/ack messages.
|
||||||
|
Serf uses piggybacking along with dedicated gossip messages. This
|
||||||
|
feature lets you have a higher gossip rate (for example once per 200ms)
|
||||||
|
and a slower failure detection rate (such as once per second), resulting
|
||||||
|
in overall faster convergence rates and data propagation speeds.
|
||||||
|
|
||||||
|
* Serf keeps the state of dead nodes around for a set amount of time,
|
||||||
|
so that when full syncs are requested, the requester also receives information
|
||||||
|
about dead nodes. Because SWIM doesn't do full syncs, SWIM deletes dead node
|
||||||
|
state immediately upon learning that the node is dead. This change again helps
|
||||||
|
the cluster converge more quickly.
|
||||||
|
|
||||||
|
## Serf-Specific Messages
|
||||||
|
|
||||||
|
On top of the SWIM-based gossip layer, Serf sends some custom message types.
|
||||||
|
|
||||||
|
Serf makes heavy use of [lamport clocks](http://en.wikipedia.org/wiki/Lamport_timestamps)
|
||||||
|
to maintain some notion of message ordering despite being eventually
|
||||||
|
consistent. Every message sent by Serf contains a lamport clock time.
|
||||||
|
|
||||||
|
When a node gracefully leaves the cluster, Serf sends a _leave intent_ through
|
||||||
|
the gossip layer. Because the underlying gossip layer makes no differentiation
|
||||||
|
between a node leaving the cluster and a node being detected as failed, this
|
||||||
|
allows the higher level Serf layer to detect a failure versus a graceful
|
||||||
|
leave.
|
||||||
|
|
||||||
|
When a node joins the cluster, Serf sends a _join intent_. The purpose
|
||||||
|
of this intent is solely to attach a lamport clock time to a join so that
|
||||||
|
it can be ordered properly in case a leave comes out of order.
|
||||||
|
|
||||||
|
For custom events, Serf sends a _user event_ message. This message contains
|
||||||
|
a lamport time, event name, and event payload. Because user events are sent
|
||||||
|
along the gossip layer, which uses UDP, the payload and entire message framing
|
||||||
|
must fit within a single UDP packet.
|
||||||
|
After Width: | Height: | Size: 77 KiB |
Loading…
Reference in new issue