Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

135 lines
3.3 KiB

Adds support for snapshots and restores. (#2396) * Updates Raft library to get new snapshot/restore API. * Basic backup and restore working, but need some cleanup. * Breaks out a snapshot module and adds a SHA256 integrity check. * Adds snapshot ACL and fills in some missing comments. * Require a consistent read for snapshots. * Make sure snapshot works if ACLs aren't enabled. * Adds a bit of package documentation. * Returns an empty response from restore to avoid EOF errors. * Adds API client support for snapshots. * Makes internal file names match on-disk file snapshots. * Adds DC and token coverage for snapshot API test. * Adds missing documentation. * Adds a unit test for the snapshot client endpoint. * Moves the connection pool out of the client for easier testing. * Fixes an incidental issue in the prepared query unit test. I realized I had two servers in bootstrap mode so this wasn't a good setup. * Adds a half close to the TCP stream and fixes panic on error. * Adds client and endpoint tests for snapshots. * Moves the pool back into the snapshot RPC client. * Adds a TLS test and fixes half-closes for TLS connections. * Tweaks some comments. * Adds a low-level snapshot test. This is independent of Consul so we can pull this out into a library later if we want to. * Cleans up snapshot and archive and completes archive tests. * Sends a clear error for snapshot operations in dev mode. Snapshots require the Raft snapshots to be readable, which isn't supported in dev mode. Send a clear error instead of a deep-down Raft one. * Adds docs for the snapshot endpoint. * Adds a stale mode and index feedback for snapshot saves. This gives folks a way to extract data even if the cluster has no leader. * Changes the internal format of a snapshot from zip to tgz. * Pulls in Raft fix to cancel inflight before a restore. * Pulls in new Raft restore interface. * Adds metadata to snapshot saves and a verify function. * Adds basic save and restore snapshot CLI commands. * Gets rid of tarball extensions and adds restore message. * Fixes an incidental bad link in the KV docs. * Adds documentation for the snapshot CLI commands. * Scuttle any request body when a snapshot is saved. * Fixes archive unit test error message check. * Allows for nil output writers in snapshot RPC handlers. * Renames hash list Decode to DecodeAndVerify. * Closes the client connection for snapshot ops. * Lowers timeout for restore ops. * Updates Raft vendor to get new Restore signature and integrates with Consul. * Bounces the leader's internal state when we do a restore.
8 years ago
package api
import (
"bytes"
"strings"
"testing"
)
func TestAPI_Snapshot(t *testing.T) {
Adds support for snapshots and restores. (#2396) * Updates Raft library to get new snapshot/restore API. * Basic backup and restore working, but need some cleanup. * Breaks out a snapshot module and adds a SHA256 integrity check. * Adds snapshot ACL and fills in some missing comments. * Require a consistent read for snapshots. * Make sure snapshot works if ACLs aren't enabled. * Adds a bit of package documentation. * Returns an empty response from restore to avoid EOF errors. * Adds API client support for snapshots. * Makes internal file names match on-disk file snapshots. * Adds DC and token coverage for snapshot API test. * Adds missing documentation. * Adds a unit test for the snapshot client endpoint. * Moves the connection pool out of the client for easier testing. * Fixes an incidental issue in the prepared query unit test. I realized I had two servers in bootstrap mode so this wasn't a good setup. * Adds a half close to the TCP stream and fixes panic on error. * Adds client and endpoint tests for snapshots. * Moves the pool back into the snapshot RPC client. * Adds a TLS test and fixes half-closes for TLS connections. * Tweaks some comments. * Adds a low-level snapshot test. This is independent of Consul so we can pull this out into a library later if we want to. * Cleans up snapshot and archive and completes archive tests. * Sends a clear error for snapshot operations in dev mode. Snapshots require the Raft snapshots to be readable, which isn't supported in dev mode. Send a clear error instead of a deep-down Raft one. * Adds docs for the snapshot endpoint. * Adds a stale mode and index feedback for snapshot saves. This gives folks a way to extract data even if the cluster has no leader. * Changes the internal format of a snapshot from zip to tgz. * Pulls in Raft fix to cancel inflight before a restore. * Pulls in new Raft restore interface. * Adds metadata to snapshot saves and a verify function. * Adds basic save and restore snapshot CLI commands. * Gets rid of tarball extensions and adds restore message. * Fixes an incidental bad link in the KV docs. * Adds documentation for the snapshot CLI commands. * Scuttle any request body when a snapshot is saved. * Fixes archive unit test error message check. * Allows for nil output writers in snapshot RPC handlers. * Renames hash list Decode to DecodeAndVerify. * Closes the client connection for snapshot ops. * Lowers timeout for restore ops. * Updates Raft vendor to get new Restore signature and integrates with Consul. * Bounces the leader's internal state when we do a restore.
8 years ago
t.Parallel()
c, s := makeClient(t)
defer s.Stop()
// Place an initial key into the store.
kv := c.KV()
key := &KVPair{Key: testKey(), Value: []byte("hello")}
if _, err := kv.Put(key, nil); err != nil {
t.Fatalf("err: %v", err)
}
// Make sure it reads back.
pair, _, err := kv.Get(key.Key, nil)
if err != nil {
t.Fatalf("err: %v", err)
}
if pair == nil {
t.Fatalf("expected value: %#v", pair)
}
if !bytes.Equal(pair.Value, []byte("hello")) {
t.Fatalf("unexpected value: %#v", pair)
}
// Take a snapshot.
snapshot := c.Snapshot()
snap, qm, err := snapshot.Save(nil)
if err != nil {
t.Fatalf("err: %v", err)
}
defer snap.Close()
// Sanity check th query metadata.
if qm.LastIndex == 0 || !qm.KnownLeader ||
qm.RequestTime == 0 {
t.Fatalf("bad: %v", qm)
}
// Overwrite the key's value.
key.Value = []byte("goodbye")
if _, err := kv.Put(key, nil); err != nil {
t.Fatalf("err: %v", err)
}
// Read the key back and look for the new value.
pair, _, err = kv.Get(key.Key, nil)
if err != nil {
t.Fatalf("err: %v", err)
}
if pair == nil {
t.Fatalf("expected value: %#v", pair)
}
if !bytes.Equal(pair.Value, []byte("goodbye")) {
t.Fatalf("unexpected value: %#v", pair)
}
// Restore the snapshot.
if err := snapshot.Restore(nil, snap); err != nil {
t.Fatalf("err: %v", err)
}
// Read the key back and look for the original value.
pair, _, err = kv.Get(key.Key, nil)
if err != nil {
t.Fatalf("err: %v", err)
}
if pair == nil {
t.Fatalf("expected value: %#v", pair)
}
if !bytes.Equal(pair.Value, []byte("hello")) {
t.Fatalf("unexpected value: %#v", pair)
}
}
func TestAPI_Snapshot_Options(t *testing.T) {
Adds support for snapshots and restores. (#2396) * Updates Raft library to get new snapshot/restore API. * Basic backup and restore working, but need some cleanup. * Breaks out a snapshot module and adds a SHA256 integrity check. * Adds snapshot ACL and fills in some missing comments. * Require a consistent read for snapshots. * Make sure snapshot works if ACLs aren't enabled. * Adds a bit of package documentation. * Returns an empty response from restore to avoid EOF errors. * Adds API client support for snapshots. * Makes internal file names match on-disk file snapshots. * Adds DC and token coverage for snapshot API test. * Adds missing documentation. * Adds a unit test for the snapshot client endpoint. * Moves the connection pool out of the client for easier testing. * Fixes an incidental issue in the prepared query unit test. I realized I had two servers in bootstrap mode so this wasn't a good setup. * Adds a half close to the TCP stream and fixes panic on error. * Adds client and endpoint tests for snapshots. * Moves the pool back into the snapshot RPC client. * Adds a TLS test and fixes half-closes for TLS connections. * Tweaks some comments. * Adds a low-level snapshot test. This is independent of Consul so we can pull this out into a library later if we want to. * Cleans up snapshot and archive and completes archive tests. * Sends a clear error for snapshot operations in dev mode. Snapshots require the Raft snapshots to be readable, which isn't supported in dev mode. Send a clear error instead of a deep-down Raft one. * Adds docs for the snapshot endpoint. * Adds a stale mode and index feedback for snapshot saves. This gives folks a way to extract data even if the cluster has no leader. * Changes the internal format of a snapshot from zip to tgz. * Pulls in Raft fix to cancel inflight before a restore. * Pulls in new Raft restore interface. * Adds metadata to snapshot saves and a verify function. * Adds basic save and restore snapshot CLI commands. * Gets rid of tarball extensions and adds restore message. * Fixes an incidental bad link in the KV docs. * Adds documentation for the snapshot CLI commands. * Scuttle any request body when a snapshot is saved. * Fixes archive unit test error message check. * Allows for nil output writers in snapshot RPC handlers. * Renames hash list Decode to DecodeAndVerify. * Closes the client connection for snapshot ops. * Lowers timeout for restore ops. * Updates Raft vendor to get new Restore signature and integrates with Consul. * Bounces the leader's internal state when we do a restore.
8 years ago
t.Parallel()
c, s := makeACLClient(t)
defer s.Stop()
// Try to take a snapshot with a bad token.
snapshot := c.Snapshot()
_, _, err := snapshot.Save(&QueryOptions{Token: "anonymous"})
if err == nil || !strings.Contains(err.Error(), "Permission denied") {
t.Fatalf("err: %v", err)
}
// Now try an unknown DC.
_, _, err = snapshot.Save(&QueryOptions{Datacenter: "nope"})
if err == nil || !strings.Contains(err.Error(), "No path to datacenter") {
t.Fatalf("err: %v", err)
}
// This should work with a valid token.
snap, _, err := snapshot.Save(&QueryOptions{Token: "root"})
if err != nil {
t.Fatalf("err: %v", err)
}
defer snap.Close()
// This should work with a stale snapshot. This doesn't have good feedback
// that the stale option was sent, but it makes sure nothing bad happens.
snap, _, err = snapshot.Save(&QueryOptions{Token: "root", AllowStale: true})
if err != nil {
t.Fatalf("err: %v", err)
}
defer snap.Close()
// Try to restore a snapshot with a bad token.
null := bytes.NewReader([]byte(""))
err = snapshot.Restore(&WriteOptions{Token: "anonymous"}, null)
if err == nil || !strings.Contains(err.Error(), "Permission denied") {
t.Fatalf("err: %v", err)
}
// Now try an unknown DC.
null = bytes.NewReader([]byte(""))
err = snapshot.Restore(&WriteOptions{Datacenter: "nope"}, null)
if err == nil || !strings.Contains(err.Error(), "No path to datacenter") {
t.Fatalf("err: %v", err)
}
// This should work.
if err := snapshot.Restore(&WriteOptions{Token: "root"}, snap); err != nil {
t.Fatalf("err: %v", err)
}
}