mirror of https://github.com/hashicorp/consul
Browse Source
* First commit * Fix navigation * Add some commands * Structure draft * Complete usage doc structure * Fix link * Apply suggestions from code review Co-authored-by: Aimee Ukasick <aimee.ukasick@hashicorp.com> * Apply suggestions from code review Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> * Apply suggestions from code review * Replace tutorial path * Apply suggestions from code review Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> --------- Co-authored-by: Aimee Ukasick <aimee.ukasick@hashicorp.com> Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> Co-authored-by: boruszak <jeffrey.boruszak@hashicorp.com>pull/21614/head
danielehc
3 months ago
committed by
GitHub
7 changed files with 413 additions and 8 deletions
@ -0,0 +1,396 @@ |
|||||||
|
--- |
||||||
|
layout: docs |
||||||
|
page_title: Application leader election |
||||||
|
description: >- |
||||||
|
Learn how to perform client-side leader elections using sessions and Consul key/value (KV) store. |
||||||
|
--- |
||||||
|
|
||||||
|
# Application leader election |
||||||
|
|
||||||
|
This topic describes the process for building client-side leader elections for service instances using Consul's [session mechanism for building distributed locks](/consul/docs/dynamic-app-config/sessions) and the [Consul key/value store](/consul/docs/dynamic-app-config/kv), which is Consul's key/value datastore. |
||||||
|
|
||||||
|
This topic is not related to Consul's leader election. For more information about the Raft leader election used internally by Consul, refer to |
||||||
|
[consensus protocol](/consul/docs/architecture/consensus) documentation. |
||||||
|
|
||||||
|
## Background |
||||||
|
|
||||||
|
Some distributed applications, like HDFS or ActiveMQ, require setting up one instance as a leader to ensure application data is current and stable. |
||||||
|
|
||||||
|
Consul's support for [sessions](/consul/docs/dynamic-app-config/sessions) and [watches](/consul/docs/dynamic-app-config/watches) allows you to build a client-side leader election process where clients use a lock on a key in the KV datastore to ensure mutual exclusion and to gracefully handle failures. |
||||||
|
|
||||||
|
All service instances that are participating should coordinate on a key format. We recommend the following pattern: |
||||||
|
|
||||||
|
```plaintext |
||||||
|
service/<service name>/leader |
||||||
|
``` |
||||||
|
|
||||||
|
## Requirements |
||||||
|
|
||||||
|
- A running Consul server |
||||||
|
- A path in the Consul KV datastore to acquire locks and to store information about the leader. The instructions on this page use the following key: `service/leader`. |
||||||
|
- If ACLs are enabled, a token with the following permissions: |
||||||
|
- `session:write` permissions over the service session name |
||||||
|
- `key:write` permissions over the key |
||||||
|
- The `curl` command |
||||||
|
|
||||||
|
Expose the token using the `CONSUL_HTTP_TOKEN` environment variable. |
||||||
|
|
||||||
|
## Client-side leader election procedure |
||||||
|
|
||||||
|
The workflow for building a client-side leader election process has the following steps: |
||||||
|
|
||||||
|
- For each client trying to acquire the lock: |
||||||
|
1. [Create a session](#create-a-new-session) associated with the client node. |
||||||
|
1. [Acquire the lock](#acquire-the-lock) on the designated key in the KV store using the `acquire` parameter. |
||||||
|
1. [Watch the KV key](#watch-the-kv-key-for-locks) to verify if the lock was released. If no lock is present, try to acquire a lock. |
||||||
|
|
||||||
|
- For the client that acquires the lock: |
||||||
|
1. Periodically, [renew the session](#renew-a-session) to avoid expiration. |
||||||
|
1. Optionally, [release the lock](#release-a-lock). |
||||||
|
|
||||||
|
- For other services: |
||||||
|
1. [Watch the KV key](#watch-the-kv-key-for-locks) to verify there is at least one process holding the lock. |
||||||
|
1. Use the values written under the KV path to identify the leader and update configurations accordingly. |
||||||
|
|
||||||
|
## Create a new session |
||||||
|
|
||||||
|
Create a configuration for the session. |
||||||
|
The minimum viable configuration requires that you specify the session name. The following example demonstrates this configuration. |
||||||
|
|
||||||
|
<CodeBlockConfig hideClipboard> |
||||||
|
|
||||||
|
```json |
||||||
|
{ |
||||||
|
"Name": "session_name" |
||||||
|
} |
||||||
|
``` |
||||||
|
|
||||||
|
</CodeBlockConfig> |
||||||
|
|
||||||
|
Create a session using the [`/session` Consul HTTP API](/consul/api-docs/session) endpoint. In the following example, the node's `hostname` is the session name. |
||||||
|
|
||||||
|
```shell-session |
||||||
|
$ curl --silent \ |
||||||
|
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \ |
||||||
|
--data '{"Name": "'`hostname`'"}' \ |
||||||
|
--request PUT \ |
||||||
|
http://127.0.0.1:8500/v1/session/create | jq |
||||||
|
``` |
||||||
|
|
||||||
|
|
||||||
|
The command returns a JSON object containing the ID of the newly created session. |
||||||
|
|
||||||
|
```json |
||||||
|
{ |
||||||
|
"ID": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6" |
||||||
|
} |
||||||
|
``` |
||||||
|
|
||||||
|
### Verify session |
||||||
|
|
||||||
|
Use the `/v1/session/list` endpoint to retrieve existing sessions. |
||||||
|
|
||||||
|
```shell-session |
||||||
|
$ curl --silent \ |
||||||
|
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \ |
||||||
|
--request GET \ |
||||||
|
http://127.0.0.1:8500/v1/session/list | jq |
||||||
|
``` |
||||||
|
|
||||||
|
The command returns a JSON array containing all available sessions in the system. |
||||||
|
|
||||||
|
<CodeBlockConfig hideClipboard> |
||||||
|
|
||||||
|
```json |
||||||
|
[ |
||||||
|
{ |
||||||
|
"ID": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6", |
||||||
|
"Name": "hashicups-db-0", |
||||||
|
"Node": "hashicups-db-0", |
||||||
|
"LockDelay": 15000000000, |
||||||
|
"Behavior": "release", |
||||||
|
"TTL": "", |
||||||
|
"NodeChecks": [ |
||||||
|
"serfHealth" |
||||||
|
], |
||||||
|
"ServiceChecks": null, |
||||||
|
"CreateIndex": 11956, |
||||||
|
"ModifyIndex": 11956 |
||||||
|
} |
||||||
|
] |
||||||
|
``` |
||||||
|
|
||||||
|
</CodeBlockConfig> |
||||||
|
|
||||||
|
You can verify from the output that the session is associated with the `hashicups-db-0` node, which is the client agent where the API request was made. |
||||||
|
|
||||||
|
With the exception of the `Name`, all parameters are set to their default values. The session is created without a `TTL` value, which means that it never expires and requires you to delete it explicitly. |
||||||
|
|
||||||
|
Depending on your needs you can create sessions specifying more parameters such as: |
||||||
|
|
||||||
|
- `TTL` - If provided, the session is invalidated and deleted if it is not renewed before the TTL expires. |
||||||
|
- `ServiceChecks` - Specifies a list of service checks to monitor. The session is invalidated if the checks return a critical state. |
||||||
|
|
||||||
|
By setting these extra parameters, you can create a client-side leader election workflow that automatically releases the lock after a specified amount of time since the last renew, or that automatically releases locks when the service holding them fails. |
||||||
|
|
||||||
|
For a full list of parameters available refer to the [`/session/create` endpoint documentation](/consul/api-docs/session#create-session). |
||||||
|
|
||||||
|
## Acquire the lock |
||||||
|
|
||||||
|
Create the data object to associate to the lock request. |
||||||
|
|
||||||
|
The data of the request should be a JSON object representing the local instance. This value is opaque to Consul, but it should contain whatever information clients require to communicate with your application. For example, it could be a JSON object that contains the node's name and the application's port. |
||||||
|
|
||||||
|
<CodeBlockConfig hideClipboard> |
||||||
|
|
||||||
|
```json |
||||||
|
{ |
||||||
|
"Node": "node-name", |
||||||
|
"Port": "8080" |
||||||
|
} |
||||||
|
``` |
||||||
|
|
||||||
|
</CodeBlockConfig> |
||||||
|
|
||||||
|
<Tabs> |
||||||
|
<Tab heading="API" group="api"> |
||||||
|
|
||||||
|
Acquire a lock for a given key using the PUT method on a [KV entry](/consul/api-docs/kv) with the |
||||||
|
`?acquire=<session>` query parameter. |
||||||
|
|
||||||
|
```shell-session |
||||||
|
$ curl --silent \ |
||||||
|
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \ |
||||||
|
--data '{"Node": "'`hostname`'"}' \ |
||||||
|
--request PUT \ |
||||||
|
http://localhost:8500/v1/kv/service/leader?acquire=d21d60ad-c2d2-b32a-7432-ca4048a4a7d6 | jq |
||||||
|
``` |
||||||
|
|
||||||
|
This request returns either `true` or `false`. If `true`, the lock was acquired and |
||||||
|
the local service instance is now the leader. If `false`, a different node acquired |
||||||
|
the lock. |
||||||
|
|
||||||
|
|
||||||
|
</Tab> |
||||||
|
<Tab heading="CLI" group="cli"> |
||||||
|
|
||||||
|
```shell-session |
||||||
|
$ consul kv put -acquire -session=d21d60ad-c2d2-b32a-7432-ca4048a4a7d6 /service/leader '{"Node": "'`hostname`'"}' |
||||||
|
``` |
||||||
|
|
||||||
|
In case of success, the command exits with exit code `0` and outputs the following message. |
||||||
|
|
||||||
|
<CodeBlockConfig hideClipboard> |
||||||
|
|
||||||
|
```plaintext |
||||||
|
Success! Lock acquired on: service/leader |
||||||
|
``` |
||||||
|
|
||||||
|
</CodeBlockConfig> |
||||||
|
|
||||||
|
If the lock was already acquired by another node, the command exits with exit code `1` and outputs the following message. |
||||||
|
|
||||||
|
<CodeBlockConfig hideClipboard> |
||||||
|
|
||||||
|
```plaintext |
||||||
|
Error! Did not acquire lock |
||||||
|
``` |
||||||
|
|
||||||
|
</CodeBlockConfig> |
||||||
|
|
||||||
|
</Tab> |
||||||
|
</Tabs> |
||||||
|
|
||||||
|
This example used the node's `hostname` as the key data. This data can be used by the other services to create configuration files. |
||||||
|
|
||||||
|
Be aware that this locking system has no enforcement mechanism that requires clients to acquire a lock before they perform an operation. Any client can read, write, and delete a key without owning the corresponding lock. |
||||||
|
|
||||||
|
## Watch the KV key for locks |
||||||
|
|
||||||
|
Existing locks need to be monitored by all nodes involved in the client-side leader elections, as well as by the other nodes that need to know the identity of the leader. |
||||||
|
|
||||||
|
- Lock holders need to monitor the lock because the session might get invalidated by an operator. |
||||||
|
- Other services that want to acquire the lock need to monitor it to check if the lock is released so they can try acquire the lock. |
||||||
|
- Other nodes need to monitor the lock to see if the value of the key changed and update their configuration accordingly. |
||||||
|
|
||||||
|
<Tabs> |
||||||
|
<Tab heading="API" group="api"> |
||||||
|
|
||||||
|
Monitor the lock using the GET method on a [KV entry](/consul/api-docs/kv) with the blocking query enabled. |
||||||
|
|
||||||
|
First, verify the latest index for the current value. |
||||||
|
|
||||||
|
```shell-session |
||||||
|
$ curl --silent \ |
||||||
|
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \ |
||||||
|
--request GET \ |
||||||
|
http://127.0.0.1:8500/v1/kv/service/leader?index=1 | jq |
||||||
|
``` |
||||||
|
|
||||||
|
The command outputs the key data, including the `ModifyIndex` for the object. |
||||||
|
|
||||||
|
<CodeBlockConfig hideClipboard highlight="9"> |
||||||
|
|
||||||
|
```json |
||||||
|
[ |
||||||
|
{ |
||||||
|
"LockIndex": 0, |
||||||
|
"Key": "service/leader", |
||||||
|
"Flags": 0, |
||||||
|
"Value": "eyJOb2RlIjogImhhc2hpY3Vwcy1kYi0wIn0=", |
||||||
|
"Session": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6", |
||||||
|
"CreateIndex": 12399, |
||||||
|
"ModifyIndex": 13061 |
||||||
|
} |
||||||
|
] |
||||||
|
``` |
||||||
|
|
||||||
|
</CodeBlockConfig> |
||||||
|
|
||||||
|
Using the value of the `ModifyIndex`, run a [blocking query](/consul/api-docs/features/blocking) against the lock. |
||||||
|
|
||||||
|
```shell-session |
||||||
|
$ curl --silent \ |
||||||
|
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \ |
||||||
|
--request GET \ |
||||||
|
http://127.0.0.1:8500/v1/kv/service/leader?index=13061 | jq |
||||||
|
``` |
||||||
|
The command hangs until a change is made on the KV path and after that the path data prints on the console. |
||||||
|
|
||||||
|
<CodeBlockConfig hideClipboard highlight="9"> |
||||||
|
|
||||||
|
```json |
||||||
|
[ |
||||||
|
{ |
||||||
|
"LockIndex": 0, |
||||||
|
"Key": "service/leader", |
||||||
|
"Flags": 0, |
||||||
|
"Value": "eyJOb2RlIjogImhhc2hpY3Vwcy1kYi0xIn0=", |
||||||
|
"Session": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6", |
||||||
|
"CreateIndex": 12399, |
||||||
|
"ModifyIndex": 13329 |
||||||
|
} |
||||||
|
] |
||||||
|
``` |
||||||
|
|
||||||
|
</CodeBlockConfig> |
||||||
|
|
||||||
|
For automation purposes, add logic to the blocking query mechanism to trigger a command every time a change is returned. |
||||||
|
A better approach is to use the CLI command `consul watch`. |
||||||
|
|
||||||
|
</Tab> |
||||||
|
<Tab heading="CLI" group="cli"> |
||||||
|
|
||||||
|
Monitor the lock using the [`consul watch`](/consul/commands/watch) command. |
||||||
|
|
||||||
|
```shell-session |
||||||
|
$ consul watch -type=key -key=service/leader cat | jq |
||||||
|
``` |
||||||
|
|
||||||
|
In this example, the command output prints to the shell. However, it is possible to pass more complex option to the command as well as a script that contains more complex logic to react to the lock data change. |
||||||
|
|
||||||
|
An example output for the command is: |
||||||
|
|
||||||
|
<CodeBlockConfig hideClipboard> |
||||||
|
|
||||||
|
```json |
||||||
|
{ |
||||||
|
"Key": "service/leader", |
||||||
|
"CreateIndex": 12399, |
||||||
|
"ModifyIndex": 13061, |
||||||
|
"LockIndex": 0, |
||||||
|
"Flags": 0, |
||||||
|
"Value": "eyJOb2RlIjogImhhc2hpY3Vwcy1kYi0wIn0=", |
||||||
|
"Session": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6" |
||||||
|
} |
||||||
|
``` |
||||||
|
|
||||||
|
</CodeBlockConfig> |
||||||
|
|
||||||
|
The `consul watch` command polls the KV path for changes and runs the specified command on the output when a change is made. |
||||||
|
|
||||||
|
</Tab> |
||||||
|
</Tabs> |
||||||
|
|
||||||
|
From the output, notice that once the lock is acquired, the `Session` parameter contains the ID of the session that holds the lock. |
||||||
|
|
||||||
|
## Renew a session |
||||||
|
|
||||||
|
If a session is created with a `TTL` value set, you need to renew the session before the TTL expires. |
||||||
|
|
||||||
|
Use the [`/v1/session/renew`](/consul/api-docs/session#renew-session) endpoint to renew existing sessions. |
||||||
|
|
||||||
|
```shell-session |
||||||
|
$ curl --silent \ |
||||||
|
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \ |
||||||
|
--request PUT \ |
||||||
|
http://127.0.0.1:8500/v1/session/renew/f027470f-2759-6b53-542d-066ae4185e67 | jq |
||||||
|
``` |
||||||
|
|
||||||
|
If the command succeeds, the session information in JSON format is printed. |
||||||
|
|
||||||
|
<CodeBlockConfig hideClipboard> |
||||||
|
|
||||||
|
```json |
||||||
|
[ |
||||||
|
{ |
||||||
|
"ID": "f027470f-2759-6b53-542d-066ae4185e67", |
||||||
|
"Name": "test", |
||||||
|
"Node": "consul-server-0", |
||||||
|
"LockDelay": 15000000000, |
||||||
|
"Behavior": "release", |
||||||
|
"TTL": "30s", |
||||||
|
"NodeChecks": [ |
||||||
|
"serfHealth" |
||||||
|
], |
||||||
|
"ServiceChecks": null, |
||||||
|
"CreateIndex": 11842, |
||||||
|
"ModifyIndex": 11842 |
||||||
|
} |
||||||
|
] |
||||||
|
``` |
||||||
|
|
||||||
|
</CodeBlockConfig> |
||||||
|
|
||||||
|
## Release a lock |
||||||
|
|
||||||
|
A lock associated with a session with no `TTL` value set might never be released, even when the service holding it fails. |
||||||
|
|
||||||
|
In such cases, you need to manually release the lock. |
||||||
|
|
||||||
|
<Tabs> |
||||||
|
<Tab heading="API" group="api"> |
||||||
|
|
||||||
|
```shell-session |
||||||
|
$ curl --silent \ |
||||||
|
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \ |
||||||
|
--data '{"Node": "'`hostname`'"}' \ |
||||||
|
--request PUT \ |
||||||
|
http://localhost:8500/v1/kv/service/leader?release=d21d60ad-c2d2-b32a-7432-ca4048a4a7d6 | jq |
||||||
|
``` |
||||||
|
|
||||||
|
The command prints `true` on success. |
||||||
|
|
||||||
|
</Tab> |
||||||
|
<Tab heading="CLI" group="cli"> |
||||||
|
|
||||||
|
```shell-session |
||||||
|
$ consul kv put -release -session=d21d60ad-c2d2-b32a-7432-ca4048a4a7d6 service/leader '{"Node": "'`hostname`'"}' |
||||||
|
``` |
||||||
|
|
||||||
|
On success, the command outputs a success message. |
||||||
|
|
||||||
|
<CodeBlockConfig hideClipboard> |
||||||
|
|
||||||
|
```plaintext |
||||||
|
Success! Lock released on: service/leader |
||||||
|
``` |
||||||
|
|
||||||
|
</CodeBlockConfig> |
||||||
|
|
||||||
|
</Tab> |
||||||
|
</Tabs> |
||||||
|
|
||||||
|
After a lock is released, the key data do not show a value for `Session` in the results. |
||||||
|
Other clients can use this as a way to coordinate their lock requests. |
||||||
|
|
||||||
Loading…
Reference in new issue