mirror of https://github.com/hashicorp/consul
Browse Source
* First commit * Fix navigation * Add some commands * Structure draft * Complete usage doc structure * Fix link * Apply suggestions from code review Co-authored-by: Aimee Ukasick <aimee.ukasick@hashicorp.com> * Apply suggestions from code review Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> * Apply suggestions from code review * Replace tutorial path * Apply suggestions from code review Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> --------- Co-authored-by: Aimee Ukasick <aimee.ukasick@hashicorp.com> Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> Co-authored-by: boruszak <jeffrey.boruszak@hashicorp.com>pull/21614/head
danielehc
3 months ago
committed by
GitHub
7 changed files with 413 additions and 8 deletions
@ -0,0 +1,396 @@
|
||||
--- |
||||
layout: docs |
||||
page_title: Application leader election |
||||
description: >- |
||||
Learn how to perform client-side leader elections using sessions and Consul key/value (KV) store. |
||||
--- |
||||
|
||||
# Application leader election |
||||
|
||||
This topic describes the process for building client-side leader elections for service instances using Consul's [session mechanism for building distributed locks](/consul/docs/dynamic-app-config/sessions) and the [Consul key/value store](/consul/docs/dynamic-app-config/kv), which is Consul's key/value datastore. |
||||
|
||||
This topic is not related to Consul's leader election. For more information about the Raft leader election used internally by Consul, refer to |
||||
[consensus protocol](/consul/docs/architecture/consensus) documentation. |
||||
|
||||
## Background |
||||
|
||||
Some distributed applications, like HDFS or ActiveMQ, require setting up one instance as a leader to ensure application data is current and stable. |
||||
|
||||
Consul's support for [sessions](/consul/docs/dynamic-app-config/sessions) and [watches](/consul/docs/dynamic-app-config/watches) allows you to build a client-side leader election process where clients use a lock on a key in the KV datastore to ensure mutual exclusion and to gracefully handle failures. |
||||
|
||||
All service instances that are participating should coordinate on a key format. We recommend the following pattern: |
||||
|
||||
```plaintext |
||||
service/<service name>/leader |
||||
``` |
||||
|
||||
## Requirements |
||||
|
||||
- A running Consul server |
||||
- A path in the Consul KV datastore to acquire locks and to store information about the leader. The instructions on this page use the following key: `service/leader`. |
||||
- If ACLs are enabled, a token with the following permissions: |
||||
- `session:write` permissions over the service session name |
||||
- `key:write` permissions over the key |
||||
- The `curl` command |
||||
|
||||
Expose the token using the `CONSUL_HTTP_TOKEN` environment variable. |
||||
|
||||
## Client-side leader election procedure |
||||
|
||||
The workflow for building a client-side leader election process has the following steps: |
||||
|
||||
- For each client trying to acquire the lock: |
||||
1. [Create a session](#create-a-new-session) associated with the client node. |
||||
1. [Acquire the lock](#acquire-the-lock) on the designated key in the KV store using the `acquire` parameter. |
||||
1. [Watch the KV key](#watch-the-kv-key-for-locks) to verify if the lock was released. If no lock is present, try to acquire a lock. |
||||
|
||||
- For the client that acquires the lock: |
||||
1. Periodically, [renew the session](#renew-a-session) to avoid expiration. |
||||
1. Optionally, [release the lock](#release-a-lock). |
||||
|
||||
- For other services: |
||||
1. [Watch the KV key](#watch-the-kv-key-for-locks) to verify there is at least one process holding the lock. |
||||
1. Use the values written under the KV path to identify the leader and update configurations accordingly. |
||||
|
||||
## Create a new session |
||||
|
||||
Create a configuration for the session. |
||||
The minimum viable configuration requires that you specify the session name. The following example demonstrates this configuration. |
||||
|
||||
<CodeBlockConfig hideClipboard> |
||||
|
||||
```json |
||||
{ |
||||
"Name": "session_name" |
||||
} |
||||
``` |
||||
|
||||
</CodeBlockConfig> |
||||
|
||||
Create a session using the [`/session` Consul HTTP API](/consul/api-docs/session) endpoint. In the following example, the node's `hostname` is the session name. |
||||
|
||||
```shell-session |
||||
$ curl --silent \ |
||||
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \ |
||||
--data '{"Name": "'`hostname`'"}' \ |
||||
--request PUT \ |
||||
http://127.0.0.1:8500/v1/session/create | jq |
||||
``` |
||||
|
||||
|
||||
The command returns a JSON object containing the ID of the newly created session. |
||||
|
||||
```json |
||||
{ |
||||
"ID": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6" |
||||
} |
||||
``` |
||||
|
||||
### Verify session |
||||
|
||||
Use the `/v1/session/list` endpoint to retrieve existing sessions. |
||||
|
||||
```shell-session |
||||
$ curl --silent \ |
||||
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \ |
||||
--request GET \ |
||||
http://127.0.0.1:8500/v1/session/list | jq |
||||
``` |
||||
|
||||
The command returns a JSON array containing all available sessions in the system. |
||||
|
||||
<CodeBlockConfig hideClipboard> |
||||
|
||||
```json |
||||
[ |
||||
{ |
||||
"ID": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6", |
||||
"Name": "hashicups-db-0", |
||||
"Node": "hashicups-db-0", |
||||
"LockDelay": 15000000000, |
||||
"Behavior": "release", |
||||
"TTL": "", |
||||
"NodeChecks": [ |
||||
"serfHealth" |
||||
], |
||||
"ServiceChecks": null, |
||||
"CreateIndex": 11956, |
||||
"ModifyIndex": 11956 |
||||
} |
||||
] |
||||
``` |
||||
|
||||
</CodeBlockConfig> |
||||
|
||||
You can verify from the output that the session is associated with the `hashicups-db-0` node, which is the client agent where the API request was made. |
||||
|
||||
With the exception of the `Name`, all parameters are set to their default values. The session is created without a `TTL` value, which means that it never expires and requires you to delete it explicitly. |
||||
|
||||
Depending on your needs you can create sessions specifying more parameters such as: |
||||
|
||||
- `TTL` - If provided, the session is invalidated and deleted if it is not renewed before the TTL expires. |
||||
- `ServiceChecks` - Specifies a list of service checks to monitor. The session is invalidated if the checks return a critical state. |
||||
|
||||
By setting these extra parameters, you can create a client-side leader election workflow that automatically releases the lock after a specified amount of time since the last renew, or that automatically releases locks when the service holding them fails. |
||||
|
||||
For a full list of parameters available refer to the [`/session/create` endpoint documentation](/consul/api-docs/session#create-session). |
||||
|
||||
## Acquire the lock |
||||
|
||||
Create the data object to associate to the lock request. |
||||
|
||||
The data of the request should be a JSON object representing the local instance. This value is opaque to Consul, but it should contain whatever information clients require to communicate with your application. For example, it could be a JSON object that contains the node's name and the application's port. |
||||
|
||||
<CodeBlockConfig hideClipboard> |
||||
|
||||
```json |
||||
{ |
||||
"Node": "node-name", |
||||
"Port": "8080" |
||||
} |
||||
``` |
||||
|
||||
</CodeBlockConfig> |
||||
|
||||
<Tabs> |
||||
<Tab heading="API" group="api"> |
||||
|
||||
Acquire a lock for a given key using the PUT method on a [KV entry](/consul/api-docs/kv) with the |
||||
`?acquire=<session>` query parameter. |
||||
|
||||
```shell-session |
||||
$ curl --silent \ |
||||
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \ |
||||
--data '{"Node": "'`hostname`'"}' \ |
||||
--request PUT \ |
||||
http://localhost:8500/v1/kv/service/leader?acquire=d21d60ad-c2d2-b32a-7432-ca4048a4a7d6 | jq |
||||
``` |
||||
|
||||
This request returns either `true` or `false`. If `true`, the lock was acquired and |
||||
the local service instance is now the leader. If `false`, a different node acquired |
||||
the lock. |
||||
|
||||
|
||||
</Tab> |
||||
<Tab heading="CLI" group="cli"> |
||||
|
||||
```shell-session |
||||
$ consul kv put -acquire -session=d21d60ad-c2d2-b32a-7432-ca4048a4a7d6 /service/leader '{"Node": "'`hostname`'"}' |
||||
``` |
||||
|
||||
In case of success, the command exits with exit code `0` and outputs the following message. |
||||
|
||||
<CodeBlockConfig hideClipboard> |
||||
|
||||
```plaintext |
||||
Success! Lock acquired on: service/leader |
||||
``` |
||||
|
||||
</CodeBlockConfig> |
||||
|
||||
If the lock was already acquired by another node, the command exits with exit code `1` and outputs the following message. |
||||
|
||||
<CodeBlockConfig hideClipboard> |
||||
|
||||
```plaintext |
||||
Error! Did not acquire lock |
||||
``` |
||||
|
||||
</CodeBlockConfig> |
||||
|
||||
</Tab> |
||||
</Tabs> |
||||
|
||||
This example used the node's `hostname` as the key data. This data can be used by the other services to create configuration files. |
||||
|
||||
Be aware that this locking system has no enforcement mechanism that requires clients to acquire a lock before they perform an operation. Any client can read, write, and delete a key without owning the corresponding lock. |
||||
|
||||
## Watch the KV key for locks |
||||
|
||||
Existing locks need to be monitored by all nodes involved in the client-side leader elections, as well as by the other nodes that need to know the identity of the leader. |
||||
|
||||
- Lock holders need to monitor the lock because the session might get invalidated by an operator. |
||||
- Other services that want to acquire the lock need to monitor it to check if the lock is released so they can try acquire the lock. |
||||
- Other nodes need to monitor the lock to see if the value of the key changed and update their configuration accordingly. |
||||
|
||||
<Tabs> |
||||
<Tab heading="API" group="api"> |
||||
|
||||
Monitor the lock using the GET method on a [KV entry](/consul/api-docs/kv) with the blocking query enabled. |
||||
|
||||
First, verify the latest index for the current value. |
||||
|
||||
```shell-session |
||||
$ curl --silent \ |
||||
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \ |
||||
--request GET \ |
||||
http://127.0.0.1:8500/v1/kv/service/leader?index=1 | jq |
||||
``` |
||||
|
||||
The command outputs the key data, including the `ModifyIndex` for the object. |
||||
|
||||
<CodeBlockConfig hideClipboard highlight="9"> |
||||
|
||||
```json |
||||
[ |
||||
{ |
||||
"LockIndex": 0, |
||||
"Key": "service/leader", |
||||
"Flags": 0, |
||||
"Value": "eyJOb2RlIjogImhhc2hpY3Vwcy1kYi0wIn0=", |
||||
"Session": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6", |
||||
"CreateIndex": 12399, |
||||
"ModifyIndex": 13061 |
||||
} |
||||
] |
||||
``` |
||||
|
||||
</CodeBlockConfig> |
||||
|
||||
Using the value of the `ModifyIndex`, run a [blocking query](/consul/api-docs/features/blocking) against the lock. |
||||
|
||||
```shell-session |
||||
$ curl --silent \ |
||||
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \ |
||||
--request GET \ |
||||
http://127.0.0.1:8500/v1/kv/service/leader?index=13061 | jq |
||||
``` |
||||
The command hangs until a change is made on the KV path and after that the path data prints on the console. |
||||
|
||||
<CodeBlockConfig hideClipboard highlight="9"> |
||||
|
||||
```json |
||||
[ |
||||
{ |
||||
"LockIndex": 0, |
||||
"Key": "service/leader", |
||||
"Flags": 0, |
||||
"Value": "eyJOb2RlIjogImhhc2hpY3Vwcy1kYi0xIn0=", |
||||
"Session": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6", |
||||
"CreateIndex": 12399, |
||||
"ModifyIndex": 13329 |
||||
} |
||||
] |
||||
``` |
||||
|
||||
</CodeBlockConfig> |
||||
|
||||
For automation purposes, add logic to the blocking query mechanism to trigger a command every time a change is returned. |
||||
A better approach is to use the CLI command `consul watch`. |
||||
|
||||
</Tab> |
||||
<Tab heading="CLI" group="cli"> |
||||
|
||||
Monitor the lock using the [`consul watch`](/consul/commands/watch) command. |
||||
|
||||
```shell-session |
||||
$ consul watch -type=key -key=service/leader cat | jq |
||||
``` |
||||
|
||||
In this example, the command output prints to the shell. However, it is possible to pass more complex option to the command as well as a script that contains more complex logic to react to the lock data change. |
||||
|
||||
An example output for the command is: |
||||
|
||||
<CodeBlockConfig hideClipboard> |
||||
|
||||
```json |
||||
{ |
||||
"Key": "service/leader", |
||||
"CreateIndex": 12399, |
||||
"ModifyIndex": 13061, |
||||
"LockIndex": 0, |
||||
"Flags": 0, |
||||
"Value": "eyJOb2RlIjogImhhc2hpY3Vwcy1kYi0wIn0=", |
||||
"Session": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6" |
||||
} |
||||
``` |
||||
|
||||
</CodeBlockConfig> |
||||
|
||||
The `consul watch` command polls the KV path for changes and runs the specified command on the output when a change is made. |
||||
|
||||
</Tab> |
||||
</Tabs> |
||||
|
||||
From the output, notice that once the lock is acquired, the `Session` parameter contains the ID of the session that holds the lock. |
||||
|
||||
## Renew a session |
||||
|
||||
If a session is created with a `TTL` value set, you need to renew the session before the TTL expires. |
||||
|
||||
Use the [`/v1/session/renew`](/consul/api-docs/session#renew-session) endpoint to renew existing sessions. |
||||
|
||||
```shell-session |
||||
$ curl --silent \ |
||||
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \ |
||||
--request PUT \ |
||||
http://127.0.0.1:8500/v1/session/renew/f027470f-2759-6b53-542d-066ae4185e67 | jq |
||||
``` |
||||
|
||||
If the command succeeds, the session information in JSON format is printed. |
||||
|
||||
<CodeBlockConfig hideClipboard> |
||||
|
||||
```json |
||||
[ |
||||
{ |
||||
"ID": "f027470f-2759-6b53-542d-066ae4185e67", |
||||
"Name": "test", |
||||
"Node": "consul-server-0", |
||||
"LockDelay": 15000000000, |
||||
"Behavior": "release", |
||||
"TTL": "30s", |
||||
"NodeChecks": [ |
||||
"serfHealth" |
||||
], |
||||
"ServiceChecks": null, |
||||
"CreateIndex": 11842, |
||||
"ModifyIndex": 11842 |
||||
} |
||||
] |
||||
``` |
||||
|
||||
</CodeBlockConfig> |
||||
|
||||
## Release a lock |
||||
|
||||
A lock associated with a session with no `TTL` value set might never be released, even when the service holding it fails. |
||||
|
||||
In such cases, you need to manually release the lock. |
||||
|
||||
<Tabs> |
||||
<Tab heading="API" group="api"> |
||||
|
||||
```shell-session |
||||
$ curl --silent \ |
||||
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \ |
||||
--data '{"Node": "'`hostname`'"}' \ |
||||
--request PUT \ |
||||
http://localhost:8500/v1/kv/service/leader?release=d21d60ad-c2d2-b32a-7432-ca4048a4a7d6 | jq |
||||
``` |
||||
|
||||
The command prints `true` on success. |
||||
|
||||
</Tab> |
||||
<Tab heading="CLI" group="cli"> |
||||
|
||||
```shell-session |
||||
$ consul kv put -release -session=d21d60ad-c2d2-b32a-7432-ca4048a4a7d6 service/leader '{"Node": "'`hostname`'"}' |
||||
``` |
||||
|
||||
On success, the command outputs a success message. |
||||
|
||||
<CodeBlockConfig hideClipboard> |
||||
|
||||
```plaintext |
||||
Success! Lock released on: service/leader |
||||
``` |
||||
|
||||
</CodeBlockConfig> |
||||
|
||||
</Tab> |
||||
</Tabs> |
||||
|
||||
After a lock is released, the key data do not show a value for `Session` in the results. |
||||
Other clients can use this as a way to coordinate their lock requests. |
||||
|
||||
Loading…
Reference in new issue