From e2bb1b76cc77bd051d6ba69d4cbe398e4f87950d Mon Sep 17 00:00:00 2001 From: danielehc <40759828+danielehc@users.noreply.github.com> Date: Thu, 15 Aug 2024 09:07:30 +0200 Subject: [PATCH] CE-657 - Move Application leader election tutorial to docs (#21366) * First commit * Fix navigation * Add some commands * Structure draft * Complete usage doc structure * Fix link * Apply suggestions from code review Co-authored-by: Aimee Ukasick * Apply suggestions from code review Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> * Apply suggestions from code review * Replace tutorial path * Apply suggestions from code review Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> --------- Co-authored-by: Aimee Ukasick Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> Co-authored-by: boruszak --- website/content/api-docs/kv.mdx | 2 +- website/content/api-docs/session.mdx | 2 +- website/content/commands/lock.mdx | 2 +- .../docs/dynamic-app-config/kv/index.mdx | 2 +- .../sessions/application-leader-election.mdx | 396 ++++++++++++++++++ .../{sessions.mdx => sessions/index.mdx} | 6 +- website/data/docs-nav-data.json | 11 +- 7 files changed, 413 insertions(+), 8 deletions(-) create mode 100644 website/content/docs/dynamic-app-config/sessions/application-leader-election.mdx rename website/content/docs/dynamic-app-config/{sessions.mdx => sessions/index.mdx} (97%) diff --git a/website/content/api-docs/kv.mdx b/website/content/api-docs/kv.mdx index 912a8b6944..3eef6eb27f 100644 --- a/website/content/api-docs/kv.mdx +++ b/website/content/api-docs/kv.mdx @@ -212,7 +212,7 @@ The corresponding CLI command is [`consul kv put`](/consul/commands/kv/put). session has locked the key.** For an example of how to use the lock feature, check the - [Leader Election tutorial](/consul/tutorials/developer-configuration/application-leader-elections). + [Leader Election tutorial](/consul/docs/dynamic-app-config/sessions/application-leader-election). - `release` `(string: "")` - Supply a session ID to use in a release operation. This is useful when paired with `?acquire=` as it allows clients to yield a lock. This diff --git a/website/content/api-docs/session.mdx b/website/content/api-docs/session.mdx index 30dd005a1c..3f13f178d7 100644 --- a/website/content/api-docs/session.mdx +++ b/website/content/api-docs/session.mdx @@ -77,7 +77,7 @@ The table below shows this endpoint's support for 86400s). If provided, the session is invalidated if it is not renewed before the TTL expires. The lowest practical TTL should be used to keep the number of managed sessions low. When locks are forcibly expired, such as when following - the [leader election pattern](/consul/tutorials/developer-configuration/application-leader-elections) in an application, + the [leader election pattern](/consul/docs/dynamic-app-config/sessions/application-leader-election) in an application, sessions may not be reaped for up to double this TTL, so long TTL values (> 1 hour) should be avoided. Valid time units include "s", "m" and "h". diff --git a/website/content/commands/lock.mdx b/website/content/commands/lock.mdx index d4f405324c..85a1568051 100644 --- a/website/content/commands/lock.mdx +++ b/website/content/commands/lock.mdx @@ -18,7 +18,7 @@ communication is disrupted, the child process is terminated. The number of lock holders is configurable with the `-n` flag. By default, a single holder is allowed, and a lock is used for mutual exclusion. This -uses the [leader election algorithm](/consul/tutorials/developer-configuration/application-leader-elections). +uses the [leader election algorithm](/consul/docs/dynamic-app-config/sessions/application-leader-election). If the lock holder count is more than one, then a semaphore is used instead. A semaphore allows more than a single holder, but this is less efficient than diff --git a/website/content/docs/dynamic-app-config/kv/index.mdx b/website/content/docs/dynamic-app-config/kv/index.mdx index 982c772938..677037321d 100644 --- a/website/content/docs/dynamic-app-config/kv/index.mdx +++ b/website/content/docs/dynamic-app-config/kv/index.mdx @@ -111,7 +111,7 @@ increment to the `LockIndex` and the session value is updated to reflect the session holding the lock. Review the session documentation for more information on the [integration](/consul/docs/dynamic-app-config/sessions#k-v-integration). -Review the following tutorials to learn how to use Consul sessions for [application leader election](/consul/tutorials/developer-configuration/application-leader-elections) and +Review the following tutorials to learn how to use Consul sessions for [application leader election](/consul/docs/dynamic-app-config/sessions/application-leader-election) and to [build distributed semaphores](/consul/tutorials/developer-configuration/distributed-semaphore). ### Vault diff --git a/website/content/docs/dynamic-app-config/sessions/application-leader-election.mdx b/website/content/docs/dynamic-app-config/sessions/application-leader-election.mdx new file mode 100644 index 0000000000..5b14bcdc9e --- /dev/null +++ b/website/content/docs/dynamic-app-config/sessions/application-leader-election.mdx @@ -0,0 +1,396 @@ +--- +layout: docs +page_title: Application leader election +description: >- + Learn how to perform client-side leader elections using sessions and Consul key/value (KV) store. +--- + +# Application leader election + +This topic describes the process for building client-side leader elections for service instances using Consul's [session mechanism for building distributed locks](/consul/docs/dynamic-app-config/sessions) and the [Consul key/value store](/consul/docs/dynamic-app-config/kv), which is Consul's key/value datastore. + +This topic is not related to Consul's leader election. For more information about the Raft leader election used internally by Consul, refer to +[consensus protocol](/consul/docs/architecture/consensus) documentation. + +## Background + +Some distributed applications, like HDFS or ActiveMQ, require setting up one instance as a leader to ensure application data is current and stable. + +Consul's support for [sessions](/consul/docs/dynamic-app-config/sessions) and [watches](/consul/docs/dynamic-app-config/watches) allows you to build a client-side leader election process where clients use a lock on a key in the KV datastore to ensure mutual exclusion and to gracefully handle failures. + +All service instances that are participating should coordinate on a key format. We recommend the following pattern: + +```plaintext +service//leader +``` + +## Requirements + +- A running Consul server +- A path in the Consul KV datastore to acquire locks and to store information about the leader. The instructions on this page use the following key: `service/leader`. +- If ACLs are enabled, a token with the following permissions: + - `session:write` permissions over the service session name + - `key:write` permissions over the key + - The `curl` command + +Expose the token using the `CONSUL_HTTP_TOKEN` environment variable. + +## Client-side leader election procedure + +The workflow for building a client-side leader election process has the following steps: + +- For each client trying to acquire the lock: + 1. [Create a session](#create-a-new-session) associated with the client node. + 1. [Acquire the lock](#acquire-the-lock) on the designated key in the KV store using the `acquire` parameter. + 1. [Watch the KV key](#watch-the-kv-key-for-locks) to verify if the lock was released. If no lock is present, try to acquire a lock. + +- For the client that acquires the lock: + 1. Periodically, [renew the session](#renew-a-session) to avoid expiration. + 1. Optionally, [release the lock](#release-a-lock). + +- For other services: + 1. [Watch the KV key](#watch-the-kv-key-for-locks) to verify there is at least one process holding the lock. + 1. Use the values written under the KV path to identify the leader and update configurations accordingly. + +## Create a new session + +Create a configuration for the session. +The minimum viable configuration requires that you specify the session name. The following example demonstrates this configuration. + + + +```json +{ + "Name": "session_name" +} +``` + + + +Create a session using the [`/session` Consul HTTP API](/consul/api-docs/session) endpoint. In the following example, the node's `hostname` is the session name. + +```shell-session +$ curl --silent \ + --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \ + --data '{"Name": "'`hostname`'"}' \ + --request PUT \ + http://127.0.0.1:8500/v1/session/create | jq +``` + + +The command returns a JSON object containing the ID of the newly created session. + +```json +{ + "ID": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6" +} +``` + +### Verify session + +Use the `/v1/session/list` endpoint to retrieve existing sessions. + +```shell-session +$ curl --silent \ + --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \ + --request GET \ + http://127.0.0.1:8500/v1/session/list | jq +``` + +The command returns a JSON array containing all available sessions in the system. + + + +```json +[ + { + "ID": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6", + "Name": "hashicups-db-0", + "Node": "hashicups-db-0", + "LockDelay": 15000000000, + "Behavior": "release", + "TTL": "", + "NodeChecks": [ + "serfHealth" + ], + "ServiceChecks": null, + "CreateIndex": 11956, + "ModifyIndex": 11956 + } +] +``` + + + +You can verify from the output that the session is associated with the `hashicups-db-0` node, which is the client agent where the API request was made. + +With the exception of the `Name`, all parameters are set to their default values. The session is created without a `TTL` value, which means that it never expires and requires you to delete it explicitly. + +Depending on your needs you can create sessions specifying more parameters such as: + +- `TTL` - If provided, the session is invalidated and deleted if it is not renewed before the TTL expires. +- `ServiceChecks` - Specifies a list of service checks to monitor. The session is invalidated if the checks return a critical state. + +By setting these extra parameters, you can create a client-side leader election workflow that automatically releases the lock after a specified amount of time since the last renew, or that automatically releases locks when the service holding them fails. + +For a full list of parameters available refer to the [`/session/create` endpoint documentation](/consul/api-docs/session#create-session). + +## Acquire the lock + +Create the data object to associate to the lock request. + +The data of the request should be a JSON object representing the local instance. This value is opaque to Consul, but it should contain whatever information clients require to communicate with your application. For example, it could be a JSON object that contains the node's name and the application's port. + + + +```json +{ + "Node": "node-name", + "Port": "8080" +} +``` + + + + + + +Acquire a lock for a given key using the PUT method on a [KV entry](/consul/api-docs/kv) with the +`?acquire=` query parameter. + +```shell-session +$ curl --silent \ + --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \ + --data '{"Node": "'`hostname`'"}' \ + --request PUT \ + http://localhost:8500/v1/kv/service/leader?acquire=d21d60ad-c2d2-b32a-7432-ca4048a4a7d6 | jq +``` + +This request returns either `true` or `false`. If `true`, the lock was acquired and +the local service instance is now the leader. If `false`, a different node acquired +the lock. + + + + + +```shell-session +$ consul kv put -acquire -session=d21d60ad-c2d2-b32a-7432-ca4048a4a7d6 /service/leader '{"Node": "'`hostname`'"}' +``` + +In case of success, the command exits with exit code `0` and outputs the following message. + + + +```plaintext +Success! Lock acquired on: service/leader +``` + + + +If the lock was already acquired by another node, the command exits with exit code `1` and outputs the following message. + + + +```plaintext +Error! Did not acquire lock +``` + + + + + + +This example used the node's `hostname` as the key data. This data can be used by the other services to create configuration files. + +Be aware that this locking system has no enforcement mechanism that requires clients to acquire a lock before they perform an operation. Any client can read, write, and delete a key without owning the corresponding lock. + +## Watch the KV key for locks + +Existing locks need to be monitored by all nodes involved in the client-side leader elections, as well as by the other nodes that need to know the identity of the leader. + + - Lock holders need to monitor the lock because the session might get invalidated by an operator. + - Other services that want to acquire the lock need to monitor it to check if the lock is released so they can try acquire the lock. + - Other nodes need to monitor the lock to see if the value of the key changed and update their configuration accordingly. + + + + +Monitor the lock using the GET method on a [KV entry](/consul/api-docs/kv) with the blocking query enabled. + +First, verify the latest index for the current value. + +```shell-session +$ curl --silent \ + --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \ + --request GET \ + http://127.0.0.1:8500/v1/kv/service/leader?index=1 | jq +``` + +The command outputs the key data, including the `ModifyIndex` for the object. + + + +```json +[ + { + "LockIndex": 0, + "Key": "service/leader", + "Flags": 0, + "Value": "eyJOb2RlIjogImhhc2hpY3Vwcy1kYi0wIn0=", + "Session": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6", + "CreateIndex": 12399, + "ModifyIndex": 13061 + } +] +``` + + + +Using the value of the `ModifyIndex`, run a [blocking query](/consul/api-docs/features/blocking) against the lock. + +```shell-session +$ curl --silent \ + --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \ + --request GET \ + http://127.0.0.1:8500/v1/kv/service/leader?index=13061 | jq +``` +The command hangs until a change is made on the KV path and after that the path data prints on the console. + + + +```json +[ + { + "LockIndex": 0, + "Key": "service/leader", + "Flags": 0, + "Value": "eyJOb2RlIjogImhhc2hpY3Vwcy1kYi0xIn0=", + "Session": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6", + "CreateIndex": 12399, + "ModifyIndex": 13329 + } +] +``` + + + +For automation purposes, add logic to the blocking query mechanism to trigger a command every time a change is returned. +A better approach is to use the CLI command `consul watch`. + + + + +Monitor the lock using the [`consul watch`](/consul/commands/watch) command. + +```shell-session +$ consul watch -type=key -key=service/leader cat | jq +``` + +In this example, the command output prints to the shell. However, it is possible to pass more complex option to the command as well as a script that contains more complex logic to react to the lock data change. + +An example output for the command is: + + + +```json +{ + "Key": "service/leader", + "CreateIndex": 12399, + "ModifyIndex": 13061, + "LockIndex": 0, + "Flags": 0, + "Value": "eyJOb2RlIjogImhhc2hpY3Vwcy1kYi0wIn0=", + "Session": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6" +} +``` + + + +The `consul watch` command polls the KV path for changes and runs the specified command on the output when a change is made. + + + + +From the output, notice that once the lock is acquired, the `Session` parameter contains the ID of the session that holds the lock. + +## Renew a session + +If a session is created with a `TTL` value set, you need to renew the session before the TTL expires. + +Use the [`/v1/session/renew`](/consul/api-docs/session#renew-session) endpoint to renew existing sessions. + +```shell-session +$ curl --silent \ + --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \ + --request PUT \ + http://127.0.0.1:8500/v1/session/renew/f027470f-2759-6b53-542d-066ae4185e67 | jq +``` + +If the command succeeds, the session information in JSON format is printed. + + + +```json +[ + { + "ID": "f027470f-2759-6b53-542d-066ae4185e67", + "Name": "test", + "Node": "consul-server-0", + "LockDelay": 15000000000, + "Behavior": "release", + "TTL": "30s", + "NodeChecks": [ + "serfHealth" + ], + "ServiceChecks": null, + "CreateIndex": 11842, + "ModifyIndex": 11842 + } +] +``` + + + +## Release a lock + +A lock associated with a session with no `TTL` value set might never be released, even when the service holding it fails. + +In such cases, you need to manually release the lock. + + + + +```shell-session +$ curl --silent \ + --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \ + --data '{"Node": "'`hostname`'"}' \ + --request PUT \ + http://localhost:8500/v1/kv/service/leader?release=d21d60ad-c2d2-b32a-7432-ca4048a4a7d6 | jq +``` + +The command prints `true` on success. + + + + +```shell-session +$ consul kv put -release -session=d21d60ad-c2d2-b32a-7432-ca4048a4a7d6 service/leader '{"Node": "'`hostname`'"}' +``` + +On success, the command outputs a success message. + + + +```plaintext +Success! Lock released on: service/leader +``` + + + + + + +After a lock is released, the key data do not show a value for `Session` in the results. +Other clients can use this as a way to coordinate their lock requests. + diff --git a/website/content/docs/dynamic-app-config/sessions.mdx b/website/content/docs/dynamic-app-config/sessions/index.mdx similarity index 97% rename from website/content/docs/dynamic-app-config/sessions.mdx rename to website/content/docs/dynamic-app-config/sessions/index.mdx index eb0c4cf495..3eb26f558f 100644 --- a/website/content/docs/dynamic-app-config/sessions.mdx +++ b/website/content/docs/dynamic-app-config/sessions/index.mdx @@ -134,9 +134,9 @@ the goal of Consul to protect against misbehaving clients. ## Leader Election -The primitives provided by sessions and the locking mechanisms of the KV -store can be used to build client-side leader election algorithms. -These are covered in more detail in the [Leader Election guide](/consul/tutorials/developer-configuration/application-leader-elections). +You can use the primitives provided by sessions and the locking mechanisms of the KV +store to build client-side leader election algorithms. +These are covered in more detail in the [Leader Election guide](/consul/docs/dynamic-app-config/sessions/application-leader-election). ## Prepared Query Integration diff --git a/website/data/docs-nav-data.json b/website/data/docs-nav-data.json index c883f29a5b..8af621f8d3 100644 --- a/website/data/docs-nav-data.json +++ b/website/data/docs-nav-data.json @@ -1054,7 +1054,16 @@ }, { "title": "Sessions", - "path": "dynamic-app-config/sessions" + "routes": [ + { + "title": "Overview", + "path": "dynamic-app-config/sessions" + }, + { + "title": "Application leader election", + "path": "dynamic-app-config/sessions/application-leader-election" + } + ] }, { "title": "Watches",