- `disable_log_cache` ((#raft_logstore_disable_log_cache)) Disables the in-memory cache for recent logs. We recommend using it for performance testing purposes, as no significant improvement has been measured when the cache is disabled. While the in-memory log cache theoretically prevents disk reads for recent logs, recent logs are also stored in the OS page cache, which does not slow either the `boltdb` or `wal` backend's ability to read them.
This topic describes how to safely configure and test the WAL backend in your Consul deployment.
The overall process for enabling the WAL LogStore backend for one server consists of the following steps. In production environments, we recommend starting by enabling the backend on a single server . If you eventually choose to expand the test to further servers, you must repeat these steps for each one.
1. Enable log verification.
1. Select target server to enable WAL.
1. Stop target server gracefully.
1. Remove data directory from target server.
1. Update target server's configuration.
1. Start the target server.
1. Monitor target server raft metrics and logs.
!> **Experimental feature:** The WAL LogStore backend is experimental and may contain bugs that could cause data loss. Follow this guide to manage risk during testing.
## Requirements
- Consul v1.15 or later is required for all servers in the datacenter. Refer to the [standard upgrade procedure](/consul/docs/upgrading/instructions/general-process) and the [1.15 upgrade notes](/consul/docs/upgrading/upgrade-specific#consul-1-15-x) for additional information.
- A Consul cluster with at least three nodes are required to safely test the WAL backend without downtime.
We recommend taking the following additional measures:
- Take a snapshot prior to testing.
- Monitor Consul server metrics and logs, and set an alert on specific log events that occur when WAL is enabled. Refer to [Monitor Raft metrics and logs for WAL](/consul/docs/agent/wal-logstore/monitoring) for more information.
- Enable WAL in a pre-production environment and run it for a several days before enabling it in production.
WAL LogStore is now enabled by default
## Known issues
@ -39,113 +16,3 @@ The following issues exist in Consul 1.15.0 and 1.15.1.
* A follower that is disconnected may be unable to catch up if it is using the WAL backend.
* Restoring user snapshots can break replication to WAL-enabled followers.
* Restoring user snapshots can cause a WAL-enabled leader to panic.
## Risks
While their likelihood remains low to very low, be aware of the following risks before implementing the WAL backend:
- If WAL corrupts data on a Consul server agent, the server data cannot be recovered. Restart the server with an empty data directory and reload its state from the leader to resolve the issue.
- WAL may corrupt data or contain a defect that causes the server to panic and crash. WAL may not restart if the defect recurs when WAL reads from the logs on startup. Restart the server with an empty data directory and reload its state from the leader to resolve the issue.
- If WAL corrupts data, clients may read corrupted data from the Consul server, such as invalid IP addresses or unmatched tokens. This outcome is unlikely even if a recurring defect causes WAL to corrupt data because replication uses objects cached in memory instead of reads from disk. Restart the server with an empty data directory and reload its state from the leader to resolve the issue.
- If you enable a Consul CE server to use WAL or enable WAL on a voting server with Consul Enterprise, WAL may corrupt the server's state, become the leader, and replicate the corrupted state to all other servers. In this case, restoring from backup is required to recover a completely uncorrupted state. Test WAL on a non-voting server in Enterprise to prevent this outcome. You can add a new non-voting server to the cluster to test with if there are no existing ones.
## Enable log verification
You must enable log verification on all voting servers in Enterprise and all servers in CE because the leader writes verification checkpoints.
1. On each voting server, add the following to the server's configuration file:
```hcl
raft_logstore {
verification {
enabled = true
interval = "60s"
}
}
```
1. Restart the server to apply the changes. The `consul reload` command is not sufficient to apply `raft_logstore` configuration changes.
1. Run the `consul operator raft list-peers` command to wait for each server to become a healthy voter before moving on to the next. This may take a few minutes for large snapshots.
When complete, the server's logs should contain verifier reports that appear like the following example:
If you are using Consul CE, or Consul Enterprise without non-voting servers, select a follower server to enable WAL. As noted in [Risks](#risks), Consul Enterprise users with non-voting servers should first select a non-voting server, or consider adding another server as a non-voter to test on.
Retrieve the current state of the servers by running the following command:
```shell-session
$ consul operator raft list-peers
```
## Stop target server
Stop the target server gracefully. For example, if you are using `systemd`,
run the following command:
```shell-session
$ systemctl stop consul
```
If your environment uses configuration management automation that might interfere with this process, such as Chef or Puppet, you must disable them until you have completely enabled WAL as a storage backend.
## Remove data directory from target server
Temporarily moving the data directory to a different location is less destructive than deleting it. We recommend moving it in cases where you unsuccessfully enable WAL. Do not use the old data directory (`/data-dir/raft.bak`) for recovery after restarting the server. We recommend eventually deleting the old directory.
The following example assumes the `data_dir` in the server's configuration is `/data-dir` and renames it to `/data-dir.bak`.
```shell-session
$ mv /data-dir/raft /data-dir/raft.bak
```
When switching backends, you must always remove _the entire raft directory_, not just the `raft.db` file or `wal` directory. The log must always be consistent with the snapshots to avoid undefined behavior or data loss.
## Update target server configuration
Add the following to the target server's configuration file:
```hcl
raft_logstore {
backend = "wal"
verification {
enabled = true
interval = "60s"
}
}
```
## Start target server
Start the target server. For example, if you are using `systemd`, run the following command:
```shell-session
$ systemctl start consul
```
Watch for the server to become a healthy voter again.
```shell-session
$ consul operator raft list-peers
```
## Monitor target server Raft metrics and logs
Refer to [Monitor Raft metrics and logs for WAL](/consul/docs/agent/wal-logstore/monitoring) for details.
We recommend leaving the cluster in the test configuration for several days or weeks, as long as you observe no errors. An extended test provides more confidence that WAL operates correctly under varied workloads and during routine server restarts. If you observe any errors, end the test immediately and report them.
If you disabled configuration management automation, consider reenabling it during the testing phase to pick up other updates for the host. You must ensure that it does not revert the Consul configuration file and remove the altered backend configuration. One way to do this is add the `raft_logstore` block to a separate file that is not managed by your automation. This file can either be added to the directory if [`-config-dir`](/consul/docs/agent/config/cli-flags#_config_dir) is used or as an additional [`-config-file`](/consul/docs/agent/config/cli-flags#_config_file) argument.
## Next steps
- If you observe any verification errors, performance anomalies, or other suspicious behavior from the target server during the test, you should immediately follow [the procedure to revert back to BoltDB](/consul/docs/agent/wal-logstore/revert-to-boltdb). Report failures through GitHub.
- If you do not see errors and would like to expand the test further, you can repeat the above procedure on another target server. We suggest waiting after each test expansion and slowly rolling WAL out to other parts of your environment. Once the majority of your servers use WAL, any bugs not yet discovered may result in cluster unavailability.
- If you wish to permanently enable WAL on all servers, repeat the steps described in this topic for each server. Even if `backend = "wal"` is set in logs, servers continue to use BoltDB if they find an existing raft.db file in the data directory.
The experimental WAL (write-ahead log) LogStore backend shipped in Consul 1.15 is intended to replace the BoltDB backend, improving performance and log storage issues.
---
# Experimental WAL LogStore backend overview
# WAL LogStore backend overview
This topic provides an overview of the WAL (write-ahead log) LogStore backend.
The WAL backend is an experimental feature. Refer to
The WAL backend is now the default Consul LogStore when a boltdb database is not already in place. Refer to
[Requirements](/consul/docs/agent/wal-logstore/enable#requirements) for
supported environments and known issues.
We do not recommend enabling the WAL backend in production without following
[our guide for safe
testing](/consul/docs/agent/wal-logstore/enable).
## WAL versus BoltDB
WAL implements a traditional log with rotating, append-only log files. WAL resolves many issues with the existing `LogStore` provided by the BoltDB backend. The BoltDB `LogStore` is a copy-on-write BTree, which is not optimized for append-only, write-heavy workloads.
This topic describes how to monitor Raft metrics and logs if you are testing the WAL backend. We strongly recommend monitoring the Consul cluster, especially the target server, for evidence that the WAL backend is not functioning correctly. Refer to [Enable the experimental WAL LogStore backend](/consul/docs/agent/wal-logstore/enable) for additional information about the WAL backend.
!> **Upgrade warning:** The WAL LogStore backend is experimental.
This topic describes how to monitor Raft metrics and logs if you are testing the WAL backend. We strongly recommend monitoring the Consul cluster, especially the target server, for evidence that the WAL backend is not functioning correctly.
## Monitor for checksum failures
@ -39,7 +37,7 @@ The `consul.raft.logstore.verifier.write_checksum_failures` metric increments wh
## Resolve checksum failures
If either type of corruption is detected, complete the instructions for [reverting to BoltDB](/consul/docs/agent/wal-logstore/revert-to-boltdb). If the server already uses BoltDB, the errors likely indicate a latent bug in BoltDB or a bug in the verification code. In both cases, you should follow the revert instructions.
If either type of corruption is detected, complete the instructions for [reverting to BoltDB](/consul/docs/agent/wal-logstore/revert-to-boltdb). If the server already uses BoltDB, the errors likely indicate a latent bug in BoltDB or a bug in the verification code. In both cases, you should follow the revert instructions.
Dhia Ayachi
3 months ago