k3s/examples/cockroachdb

PLEASE NOTE: This document applies to the HEAD of the source tree

If you are using a released version of Kubernetes, you should refer to the docs that go with that version.

The latest release of this document can be found [here](http://releases.k8s.io/release-1.4/examples/cockroachdb/README.md).

Documentation for other releases can be found at releases.k8s.io.

CockroachDB on Kubernetes as a PetSet

This example deploys CockroachDB on Kubernetes as a PetSet. CockroachDB is a distributed, scalable NewSQL database. Please see the homepage and the documentation for details.

Limitations

PetSet limitations

Standard PetSet limitations apply: There is currently no possibility to use node-local storage (outside of single-node tests), and so there is likely a performance hit associated with running CockroachDB on some external storage. Note that CockroachDB already does replication and thus should not be deployed on a persistent volume which already replicates internally. High-performance use cases on a private Kubernetes cluster should consider a DaemonSet deployment.

Recovery after persistent storage failure

A persistent storage failure (e.g. losing the hard drive) is gracefully handled by CockroachDB as long as enough replicas survive (two out of three by default). Due to the bootstrapping in this deployment, a storage failure of the first node is special in that the administrator must manually prepopulate the "new" storage medium by running an instance of CockroachDB with the --join parameter. If this is not done, the first node will bootstrap a new cluster, which will lead to a lot of trouble.

Dynamic provisioning

The deployment is written for a use case in which dynamic provisioning is available. When that is not the case, the persistent volume claims need to be created manually. See minikube.sh for the necessary steps.

Testing locally on minikube

Follow the steps in minikube.sh (or simply run that file).

Accessing the database

Along with our PetSet configuration, we expose a standard Kubernetes service that offers a load-balanced virtual IP for clients to access the database with. In our example, we've called this service cockroachdb-public.

Start up a client pod and open up an interactive, (mostly) Postgres-flavor SQL shell using:

$ kubectl run -it cockroach-client --image=cockroachdb/cockroach --restart=Never --command -- bash
root@cockroach-client # ./cockroach sql --host cockroachdb-public

You can see example SQL statements for inserting and querying data in the included demo script, but can use almost any Postgres-style SQL commands. Some more basic examples can be found within CockroachDB's documentation.

Simulating failures

When all (or enough) nodes are up, simulate a failure like this:

kubectl exec cockroachdb-0 -- /bin/bash -c "while true; do kill 1; done"

You can then reconnect to the database as demonstrated above and verify that no data was lost. The example runs with three-fold replication, so it can tolerate one failure of any given node at a time. Note also that there is a brief period of time immediately after the creation of the cluster during which the three-fold replication is established, and during which killing a node may lead to unavailability.

The demo script gives an example of killing one instance of the database and ensuring the other replicas have all data that was written.

Scaling up or down

Simply edit the PetSet (but note that you may need to create a new persistent volume claim first). If you ran minikube.sh, there's a spare volume so you can immediately scale up by one. Convince yourself that the new node immediately serves reads and writes.

Cleaning up when you're done

Because all of the resources in this example have been tagged with the label app=cockroachdb, we can clean up everything that we created in one quick command using a selector on that label:

kubectl delete petsets,pods,persistentvolumes,persistentvolumeclaims,services -l app=cockroachdb