mirror of https://github.com/hashicorp/consul
178 lines
13 KiB
Markdown
178 lines
13 KiB
Markdown
---
|
||
layout: docs
|
||
page_title: Fault Tolerance in Consul
|
||
description: >-
|
||
Fault tolerance is a system's ability to operate without interruption despite component failure. Learn how a set of Consul servers provide fault tolerance through use of a quorum, and how to further improve control plane resilience through use of infrastructure zones and Enterprise redundancy zones.
|
||
---
|
||
|
||
# Fault tolerance
|
||
|
||
|
||
You must give careful consideration to reliability in the architecture frameworks that you build. When you build a resilient platform, it minimizes the remediation actions you need to take when a failure occurs. This document provides useful information on how to design and operate a resilient Consul cluster, including the methods and functionalities for this goal.
|
||
|
||
Consul has many features that operate both locally and remotely that can help you offer a resilient service across multiple datacenters.
|
||
|
||
|
||
## Introduction
|
||
|
||
Fault tolerance is the ability of a system to continue operating without interruption
|
||
despite the failure of one or more components. In Consul, the number of server agents determines the fault tolerance.
|
||
|
||
|
||
Each Consul datacenter depends on a set of Consul voting server agents.
|
||
The voting servers ensure Consul has a consistent, fault-tolerant state
|
||
by requiring a majority of voting servers, known as a quorum, to agree upon any state changes.
|
||
Examples of state changes include: adding or removing services,
|
||
adding or removing nodes, and changes in service or node health status.
|
||
|
||
Without a quorum, Consul experiences an outage:
|
||
it cannot provide most of its capabilities because they rely on
|
||
the availability of this state information.
|
||
If Consul has an outage, normal operation can be restored by following the
|
||
[Disaster recovery for Consul clusters guide](/consul/tutorials/datacenter-operations/recovery-outage).
|
||
|
||
If Consul is deployed with 3 servers, the quorum size is 2. The deployment can lose 1
|
||
server and still maintain quorum, so it has a fault tolerance of 1.
|
||
If Consul is instead deployed with 5 servers, the quorum size increases to 3, so
|
||
the fault tolerance increases to 2.
|
||
To learn more about the relationship between the
|
||
number of servers, quorum, and fault tolerance, refer to the
|
||
[consensus protocol documentation](/consul/docs/architecture/consensus#deployment_table).
|
||
|
||
Effectively mitigating your risk is more nuanced than just increasing the fault tolerance
|
||
because the infrastructure costs can outweigh the improved resiliency. You must also consider correlated risks at the infrastructure-level. There are occasions when multiple servers fail at the same time. That means that a single failure could cause a Consul outage, even if your server-level fault tolerance is 2.
|
||
|
||
Different options for your resilient datacenter present trade-offs between operational complexity, computing cost, and Consul request performance. Consider these factors when designing your resilient architecture.
|
||
|
||
## Fault tolerance
|
||
|
||
The following sections explore several options for increasing Consul's fault tolerance. For enhanced reliability, we recommend taking a holistic approach by layering these multiple functionalities together.
|
||
|
||
- Spread servers across infrastructure [availability zones](#availability-zones).
|
||
- Use a [minimum quorum size](#quorum-size) to avoid performance impacts.
|
||
- Use [redundancy zones](#redundancy-zones) to improve fault tolerance. <EnterpriseAlert inline />
|
||
- Use [Autopilot](#autopilot) to automatically prune failed servers and maintain quorum size.
|
||
- Use [cluster peering](#cluster-peering) to provide service redundancy.
|
||
|
||
### Availability zones
|
||
|
||
|
||
The cloud or on-premise infrastructure underlying your [Consul datacenter](/consul/docs/install/glossary#datacenter) can run across multiple availability zones.
|
||
|
||
An availability zone is meant to share no points of failure with other zones by:
|
||
- Having power, cooling, and networking systems independent from other zones
|
||
- Being physically distant enough from other zones so that large-scale disruptions
|
||
such as natural disasters (flooding, earthquakes) are very unlikely to affect multiple zones
|
||
|
||
Availability zones are available in the regions of most cloud providers and in some on-premise installations.
|
||
If possible, spread your Consul voting servers across 3 availability zones
|
||
to protect your Consul datacenter from a single zone-level failure.
|
||
For example, if deploying 5 Consul servers across 3 availability zones, place no more than 2 servers in each zone.
|
||
If one zone fails, at most 2 servers are lost and quorum will be maintained by the 3 remaining servers.
|
||
|
||
To distribute your Consul servers across availability zones, modify your infrastructure configuration with your infrastructure provider. No change is needed to your Consul server's agent configuration.
|
||
|
||
Additionally, you should leverage resources that can automatically restore your compute instance,
|
||
such as autoscaling groups, virtual machine scale sets, or compute engine autoscaler.
|
||
Customize autoscaling resources to re-deploy servers into specific availability zones and ensure the desired numbers of servers are available at all times.
|
||
|
||
### Quorum size
|
||
|
||
For most production use cases, we recommend using a minimum quorum of either 3 or 5 voting servers,
|
||
yielding a server-level fault tolerance of 1 or 2 respectively.
|
||
|
||
Even though it would improve fault tolerance,
|
||
adding voting servers beyond 5 is **not recommended** because it decreases Consul's performance—
|
||
it requires Consul to involve more servers in every state change or consistent read.
|
||
|
||
Consul Enterprise users can use redundancy zones to improve fault tolerance without this performance penalty.
|
||
|
||
### Redundancy zones <EnterpriseAlert inline />
|
||
|
||
Use Consul Enterprise [redundancy zones](/consul/docs/enterprise/redundancy) to improve fault tolerance without the performance penalty of increasing the number of voting servers.
|
||
|
||
|
||
|
||
|
||
Each redundancy zone should be assigned 2 or more Consul servers.
|
||
If all servers are healthy, only one server per redundancy zone will be an active voter;
|
||
all other servers will be backup voters.
|
||
If a zone's voter is lost, it will be replaced by:
|
||
- A backup voter within the same zone, if any. Otherwise,
|
||
- A backup voter within another zone, if any.
|
||
|
||
Consul can replace lost voters with backup voters within 30 seconds in most cases.
|
||
Because this replacement process is not instantaneous,
|
||
redundancy zones do not improve immediate fault tolerance—
|
||
the number of healthy voting servers that can fail at once without causing an outage.
|
||
Instead, redundancy zones improve optimistic fault tolerance:
|
||
the number of healthy active and back-up voting servers that can fail gradually without causing an outage.
|
||
|
||
The relationship between these two types of fault tolerance is:
|
||
|
||
_Optimistic fault tolerance = immediate fault tolerance + the number of healthy backup voters_
|
||
|
||
For example, consider a Consul datacenter with 3 redundancy zones and 2 servers per zone.
|
||
There will be 3 voting servers (1 per zone), meaning a quorum size of 2 and an immediate fault tolerance of 1.
|
||
There will also be 3 backup voters (1 per zone), each of which increase the optimistic fault tolerance.
|
||
Therefore, the optimistic fault tolerance is 4.
|
||
This provides performance similar to a 3 server setup with fault tolerance similar to a 7 server setup.
|
||
|
||
We recommend associating each Consul redundancy zone with an infrastructure availability zone
|
||
to also gain the infrastructure-level fault tolerance benefits provided by availability zones.
|
||
However, Consul redundancy zones can be used even without the backing of infrastructure availability zones.
|
||
|
||
For more information on redundancy zones, refer to:
|
||
- [Redundancy zone documentation](/consul/docs/enterprise/redundancy)
|
||
for a more detailed explanation
|
||
- [Redundancy zone tutorial](/consul/tutorials/enterprise/redundancy-zones)
|
||
to learn how to use them
|
||
|
||
### Autopilot
|
||
|
||
Autopilot is a set of functions that introduce servers to a cluster, cleans up dead servers, and monitors the state of the Raft protocol in the Consul cluster.
|
||
|
||
When you enable Autopilot's dead server cleanup, Autopilot marks failed servers as `Left` and removes them from the Raft peer set to prevent them from interfering with the quorum size. Autopilot does that as soon as a replacement Consul server comes online. This behavior is beneficial when server nodes failed and have been redeployed but Consul considers them as new nodes because their IP address and hostnames have changed. Autopilot keeps the cluster peer set size correct and the quorum requirement simple.
|
||
|
||
To illustrate the Autopilot advantage, consider a scenario where Consul has a cluster of five server nodes. The quorum is three, which means the cluster can lose two server nodes before the cluster fails. The following events happen:
|
||
|
||
1. Two server nodes fail.
|
||
1. Two replacement nodes are deployed with new hostnames and IPs.
|
||
1. The two replacement nodes rejoin the Consul cluster.
|
||
1. Consul treats the replacement nodes as extra nodes, unrelated to the previously failed nodes.
|
||
|
||
_With Autopilot not enabled_, the following happens:
|
||
|
||
1. Consul does not immediately clean up the failed nodes when the replacement nodes join the cluster.
|
||
1. The cluster now has the three surviving nodes, the two failed nodes, and the two replacement nodes, for a total of seven nodes.
|
||
- The quorum is increased to four, which means the cluster can only afford to lose one node until after the two failed nodes are deleted in seventy-two hours.
|
||
- The redundancy level has decreased from its initial state.
|
||
|
||
_With Autopilot enabled_, the following happens:
|
||
|
||
1. Consul immediately cleans up the failed nodes when the replacement nodes join the cluster.
|
||
1. The cluster now has the three surviving nodes and the two replacement nodes, for a total of five nodes.
|
||
- The quorum stays at three, which means the cluster can afford to lose two nodes before it fails.
|
||
- The redundancy level remains the same.
|
||
|
||
### Cluster peering
|
||
|
||
Linking multiple Consul clusters together to provide service redundancy is the most effective method to prevent disruption from failure. This method is enhanced when you design individual Consul clusters with resilience in mind. Consul clusters interconnect in two ways: WAN federation and cluster peering. We recommend using cluster peering whenever possible.
|
||
|
||
Cluster peering lets you connect two or more independent Consul clusters using mesh gateways, so that services can communicate between non-identical partitions in different datacenters.
|
||
|
||
|
||
|
||
|
||
Cluster peering is the preferred way to interconnect clusters because it is operationally easier to configure and manage than WAN federation. Cluster peering communication between two datacenters runs only on one port on the related Consul mesh gateway, which makes it operationally easy to expose for routing purposes.
|
||
|
||
When you use cluster peering to connect admin partitions between datacenters, use Consul’s dynamic traffic management functionalities `service-splitter`, `service-router` and `service-failover` to configure your service mesh to automatically forward or failover service traffic between peer clusters. Consul can then manage the traffic intended for the service and do [failover](/consul/docs/connect/config-entries/service-resolver#spec-failover), [load-balancing](/consul/docs/connect/config-entries/service-resolver#spec-loadbalancer), or [redirection](/consul/docs/connect/config-entries/service-resolver#spec-redirect).
|
||
|
||
Cluster peering also extends service discovery across different datacenters independent of service mesh functions. After you peer datacenters, you can refer to services between datacenters with `<service>.virtual.peer.consul` in Consul DNS. For Consul Enterprise, your query string may need to include the namespace, partition, or both. Refer to the [Consul DNS documentation](/consul/docs/services/discovery/dns-static-lookups#service-virtual-ip-lookups) for details on building virtual service lookups.
|
||
|
||
For more information on cluster peering, refer to:
|
||
- [Cluster peering documentation](/consul/docs/connect/cluster-peering)
|
||
for a more detailed explanation
|
||
- [Cluster peering tutorial](/consul/tutorials/implement-multi-tenancy/cluster-peering)
|
||
to learn how to implement cluster peering
|