Commit Graph

703 Commits (717621698d7b883aa05e36031c92b96c103c0f40)

Author SHA1 Message Date
James Phillips c01a3871c9 Adds support for snapshots and restores. (#2396)
* Updates Raft library to get new snapshot/restore API.

* Basic backup and restore working, but need some cleanup.

* Breaks out a snapshot module and adds a SHA256 integrity check.

* Adds snapshot ACL and fills in some missing comments.

* Require a consistent read for snapshots.

* Make sure snapshot works if ACLs aren't enabled.

* Adds a bit of package documentation.

* Returns an empty response from restore to avoid EOF errors.

* Adds API client support for snapshots.

* Makes internal file names match on-disk file snapshots.

* Adds DC and token coverage for snapshot API test.

* Adds missing documentation.

* Adds a unit test for the snapshot client endpoint.

* Moves the connection pool out of the client for easier testing.

* Fixes an incidental issue in the prepared query unit test.

I realized I had two servers in bootstrap mode so this wasn't a good setup.

* Adds a half close to the TCP stream and fixes panic on error.

* Adds client and endpoint tests for snapshots.

* Moves the pool back into the snapshot RPC client.

* Adds a TLS test and fixes half-closes for TLS connections.

* Tweaks some comments.

* Adds a low-level snapshot test.

This is independent of Consul so we can pull this out into a library
later if we want to.

* Cleans up snapshot and archive and completes archive tests.

* Sends a clear error for snapshot operations in dev mode.

Snapshots require the Raft snapshots to be readable, which isn't supported
in dev mode. Send a clear error instead of a deep-down Raft one.

* Adds docs for the snapshot endpoint.

* Adds a stale mode and index feedback for snapshot saves.

This gives folks a way to extract data even if the cluster has no
leader.

* Changes the internal format of a snapshot from zip to tgz.

* Pulls in Raft fix to cancel inflight before a restore.

* Pulls in new Raft restore interface.

* Adds metadata to snapshot saves and a verify function.

* Adds basic save and restore snapshot CLI commands.

* Gets rid of tarball extensions and adds restore message.

* Fixes an incidental bad link in the KV docs.

* Adds documentation for the snapshot CLI commands.

* Scuttle any request body when a snapshot is saved.

* Fixes archive unit test error message check.

* Allows for nil output writers in snapshot RPC handlers.

* Renames hash list Decode to DecodeAndVerify.

* Closes the client connection for snapshot ops.

* Lowers timeout for restore ops.

* Updates Raft vendor to get new Restore signature and integrates with Consul.

* Bounces the leader's internal state when we do a restore.
2016-10-25 19:20:24 -07:00
Kyle Havlovitz f6c971f46c Fix race condition in TestClient_WatchList and TestClient_WatchGet 2016-10-25 17:48:11 -07:00
James Phillips 554c898338 Merge pull request #2382 from zaunerc/master
Add info about return values for function Get().
2016-10-20 09:40:58 -07:00
James Phillips 556ffc588b Adds comment about SessionOpts. 2016-10-05 17:53:27 -07:00
James Phillips 25455c97c3 Merge pull request #2372 from HotelsDotCom/flexible-lock-sessions
More flexible Session configuration when using api.Lock
2016-10-05 17:52:12 -07:00
Christoph Zauner faf85f602b Add info about return values for function Get(). 2016-10-03 08:24:04 +00:00
Maxim Ivanov 2ed58a2747 More flexible Session configuration when using api.Lock 2016-09-29 14:07:54 +01:00
aksentyev 4f96fa0aea CreateIndex, ModifyIndex added for CatalogService 2016-09-28 00:40:25 +03:00
Seth Vargo 40de5858d3
Fix typo 2016-09-26 16:06:56 -07:00
Seth Vargo 55f1b4ac44
Rename session name to session ID 2016-09-26 16:06:55 -07:00
Seth Vargo 12e46503f7
Update ModifyIndex comment 2016-09-26 16:06:55 -07:00
Seth Vargo 227019ec59
Add documentation to KVPair
Based on discussions with @slackpad, some of the values in the KVPair are
ReadOnly. This commit updates the docs to reflect that.
2016-09-26 16:06:52 -07:00
James Phillips 84e8fc5848
Removes Raft types from public API interface.
This will cause a lot of breakage because we've currently vendored a
branch of the Raft library.
2016-08-30 14:59:16 -07:00
James Phillips 3c9188c38b
Makes the Raft configuration API easier to consume. 2016-08-30 11:30:56 -07:00
James Phillips 0558b9c44b
Fixes a stale comment. 2016-08-30 00:10:46 -07:00
James Phillips e5850d8a26
Adds new consul operator endpoint, CLI, and ACL and some basic Raft commands. 2016-08-30 00:02:50 -07:00
James Phillips defa2a6180 Merge pull request #2226 from abhinavdahiya/rm-health-unknown
Fixes #1775; Removes 'unknown' state
2016-08-17 17:51:04 -07:00
James Phillips 4c7a0ed3b0
Merge branch 'master' into f-deregister-critical 2016-08-16 12:53:21 -07:00
James Phillips 9f7a973ace
Adds an `X-Consul-Translate-Addresses` to signal translation is enabled. 2016-08-16 11:31:41 -07:00
James Phillips 1d5eb36689
Adds missing TaggedAddress structures to API client. 2016-08-16 10:30:30 -07:00
James Phillips 231f5a957f
Fixes a typo and adds an admonition about only being in Consul 0.7+. 2016-08-16 09:27:20 -07:00
James Phillips 4a3d7db24f
Adds ability to deregister a service based on critical check state longer than a timeout. 2016-08-16 01:00:26 -07:00
Hasyimi Bahrudin 105cea9bae Add api environment variables as constants for consistency 2016-08-03 14:40:31 +08:00
Hasyimi Bahrudin bb6bb19aad Remove redundant hardcoded environment variables
The following hardcoded environment variables are removed:

* CONSUL_RPC_ADDR
* CONSUL_HTTP_ADDR
2016-08-03 14:40:26 +08:00
Jeff Mitchell f1a67836d8 Use header to send Consul token rather than query param. 2016-08-02 16:54:59 -04:00
Abhinav Dahiya 7c6c3b38bc Fixes #1775; Removes 'unknown' state
Signed-off-by: Abhinav Dahiya <abhinavdtu2012@gmail.com>
2016-07-30 19:33:14 +05:30
Ryan Uber 499537addf api: add query templates 2016-07-02 16:05:41 -07:00
Ryan Uber 72b43eba26 api: add Near parameter to PQ's 2016-07-01 16:41:46 -07:00
James Phillips 778b975e7a Adds a get-tree verb to KV transaction operations. 2016-05-13 16:57:39 -07:00
James Phillips a37bf9de56 Adds a read-only optimized path for transactions. 2016-05-13 00:34:05 -07:00
James Phillips 04d99cd702 Makes get fail a transaction if the key doesn't exist. 2016-05-11 14:18:31 -07:00
James Phillips 4882a9fe43 De-nests the KV output structure (removes DirEnt member). 2016-05-11 13:48:03 -07:00
James Phillips 960b9d6fb6 Switches to "KV" instead of "KV" for the KV operations. 2016-05-11 10:58:27 -07:00
James Phillips 38d0f6676f Refactors TxnRequest/TxnResponse into a form that will allow non-KV ops.
This isn't needed/used yet, but it's a good hook to get in there so we
can add more atomic operations in the future. The Go API hides this detail
so that feels like a KV-specific API. The implications on the REST API are
pretty minimal.
2016-05-11 01:39:10 -07:00
James Phillips 69f58ad04a Moves txn code into a new endpoint, not specific to KV. 2016-05-10 21:58:02 -07:00
James Phillips 44ab1aa418 Adds type for API ops and an example transaction. 2016-05-10 13:36:48 -07:00
James Phillips e491245062 Performs basic plumbing of KVS transactions through all the layers. 2016-05-09 22:15:49 -07:00
Sean Chittenden 4255a0826d Correct a small typo 2016-04-23 20:18:19 -07:00
Sean Chittenden f1873c21d7 consul/ uses structs.Health*, the api uses api.Health* 2016-04-23 16:06:58 -07:00
Sean Chittenden e63d3a1275 Update Check API to use constants
Use constants where appropriate to advocate their use.  Also add a deprecation notice re: `updateTTL`.
2016-04-23 16:01:59 -07:00
James Phillips 3340d7ccd7 Merge pull request #1876 from hashicorp/f-tls-helper
Adds TLS config helper to API client.
2016-03-24 11:34:24 -07:00
James Phillips 0eb6279c5e Improves the comment for the Address field. 2016-03-24 11:33:44 -07:00
Diptanu Choudhury 4811a72d80 Added some constants in the api for check health statuses 2016-03-24 11:26:07 -07:00
James Phillips 1cfed981b3 Adds TLS config helper to API client. 2016-03-24 11:24:18 -07:00
James Phillips abacccd6e5 Switches default for API client to pooled connections. 2016-03-10 12:29:50 -08:00
James Phillips 23247b3699 Adds support for new PUT API for checks. 2016-03-04 15:18:25 -08:00
James Phillips afdeb2f1fc Adds support for EnableTagOverride to the API client. 2016-02-16 11:45:29 -08:00
Seth Vargo cae0ff354a Use HTTPS + www. where appropriate 2016-01-13 17:44:01 -05:00
James Phillips 58fb27e70f Factors server error checking into a new function. 2016-01-06 11:35:16 -08:00
James Phillips c048c5aca6 Makes the API behave better with small wait values. 2016-01-06 11:05:11 -08:00
James Phillips 49342dc973 Makes the timeout behavior more intuitive.
Previously, it would try once "up to" the timeout, but in practice it would
just fall through. This modifies the behavior to block until the timeout has
been reached.
2016-01-06 09:40:20 -08:00
James Phillips ca08ba3aee Adds the ability for semaphore monitors to ride out brief periods of 500 errors. 2016-01-05 18:22:36 -08:00
James Phillips 8caa9e4c7e Adds "try" support to locks and semaphores. 2016-01-05 18:22:36 -08:00
James Phillips ebff3fe0d0 Builds the insecure SSL transport off the clean default transport. 2015-12-17 12:15:18 -08:00
Jeff Mitchell 4604af6aa5 Revert "Adds client and transport pooling in the API so we don't leak connections." 2015-12-17 10:56:50 -05:00
James Phillips f6546a0e46 Makes the insecure transport work like the default one. 2015-12-17 06:48:18 -08:00
James Phillips d57e60a123 Adds client and transport pooling in the API so we don't leak connections. 2015-12-16 22:27:07 -08:00
James Phillips 380658f6f1 Adds a unit test for lock monitor retries. 2015-11-30 20:16:40 -08:00
James Phillips c2a8fee76c Adds custom retry time for lock monitors. 2015-11-30 20:16:22 -08:00
James Phillips 86e6230de6 Adds a retry capability to lock monitors in the API client. 2015-11-30 17:09:41 -08:00
James Phillips 87ccd1eb34 Updates API docs for version 0.6.0. 2015-11-28 20:04:29 -08:00
Craig Wickesser 7890a3dda8 Fix version support
Updated the version that the library supports.
2015-11-23 13:03:38 -05:00
James Phillips 95c708f65e Adds Docker checks support to client API.
Also changed `DockerContainerId` to `DockerContainerID`, and updated the agent
API docs to reflect their support for Docker checks.
2015-11-18 07:40:02 -08:00
James Phillips 1059a8b3a3 Removes a useless empty import and fixes some stale comments. 2015-11-17 08:29:20 -08:00
James Phillips 8fc6a6a986 Switches to helpers for prepared query API wrappers. 2015-11-16 23:12:44 -08:00
James Phillips 712a3dba2f Adds API client support for prepared queries. 2015-11-15 17:06:00 -08:00
James Phillips 787f946f8d Adds support for coordinates to client API. 2015-10-23 15:23:01 -07:00
Mikhail Dyakov 01aba571a1 Handle error in api README example 2015-10-23 21:20:01 +02:00
Jeff Mitchell 1e3840b044 Update cleanhttp repo location 2015-10-22 14:14:22 -04:00
Jeff Mitchell 9a5fd5424a Use cleanhttp to get rid of DefaultTransport 2015-10-22 10:47:50 -04:00
James Phillips 2c36c4f969 Fixes a typo. 2015-10-15 14:59:09 -07:00
James Phillips 181c216f53 Fixes index management for KVS. 2015-10-15 14:59:09 -07:00
Jeff Mitchell f49fc095ef Don't use http.DefaultClient
Two of the changes are in tests; the one of consequence is in the API.
As explained in #1308 this can cause conflicts with downstream programs.

Fixes #1308.
2015-10-15 17:49:35 -04:00
James Phillips c74355fec0 Merge pull request #1041 from rboyer/api-fix-session-renew
Fix Session.Renew to care about 404s properly
2015-09-25 13:15:54 -07:00
Dale Wijnand 5a28ebcaa3 Fix a bunch of typos. 2015-09-15 13:22:08 +01:00
R.B. Boyer 8b072467eb Correct the Session.Renew{,Periodic} to handle session expiration better 2015-09-14 08:52:32 -05:00
Anthony Scalisi 10e028d599 remove various typos 2015-09-11 12:29:54 -07:00
Peter Fern b023904298 Add TCP check type
Adds the ability to simply check whether a TCP socket accepts
connections to determine if it is healthy.  This is a light-weight -
though less comprehensive than scripting - method of checking network
service health.

The check parameter `tcp` should be set to the `address:port`
combination for the service to be tested.  Supports both IPv6 and IPv4,
in the case of a hostname that resolves to both, connections will be
attempted via both protocol versions, with the first successful
connection returning a successful check result.

Example check:

```json
{
  "check": {
    "id": "ssh",
    "name": "SSH (TCP)",
    "tcp": "example.com:22",
    "interval": "10s"
  }
}
```
2015-07-24 14:06:05 +10:00
Robert Gogolok 2b8d0c4785 api: run ACL tests by default 2015-06-13 23:51:30 +02:00
James Phillips de29cb4968 Fixes a broken unit test introduced by #834. 2015-06-09 12:01:20 -07:00
Armon Dadgar c2f8f954e2 Merge pull request #1008 from highlyunavailable/master
Check to see why a lock acquisition failed
2015-06-09 07:59:28 -04:00
Ryan Uber 20a2d19198 Merge pull request #834 from apoydence/master
Returns an error for a key with a '/'
2015-06-05 15:23:57 -07:00
Tiru Srikantha 3d97ea0bae Check to see why a lock acquisition failed 2015-06-05 13:23:23 -07:00
highlyunavailable 16de0a909b Remove unused DefaultSemaphoreRetryTime
Fixes #932

DefaultSemaphoreRetryTime is actually unused, unlike DefaultLockRetryTime.
2015-05-15 08:25:02 -07:00
Armon Dadgar dc6795a55a api: improve test reliability 2015-05-12 12:10:25 -07:00
Ryan Uber 72ee584df3 Fix tests after merge 2015-05-11 18:53:09 -07:00
Armon Dadgar 8d86290ebf Fixing merge conflict 2015-05-11 16:48:10 -07:00
Armon Dadgar db134f6190 Merge pull request #912 from rojojo23/renew-periodic
modify RenewPeriodic to retry failed Renew until TTL elapses
2015-05-08 11:38:51 -07:00
Ryan Uber 565cac0340 api: run tests in parallel 2015-05-08 10:27:24 -07:00
Robert Johnson 379a69d7a4 modify RenewPeriodic to retry failed Renews until TTL elapses 2015-05-06 16:40:45 -07:00
Armon Dadgar 5921fcb29b api: Fixing test compilation issue 2015-05-04 15:27:22 -07:00
Armon Dadgar 0450854f12 Merge pull request #861 from fraenkel/reclaim_lock
Reclaim locks
2015-05-04 15:23:29 -07:00
Michael Fraenkel 0e25381494 Reclaim locks
- When attempting to lock with the same session that already owns the lock,
  Lock() will re-acquire the lock.
2015-04-29 10:10:24 -07:00
Ryan Mills 275af975e8 Allow specifying a status field in the agent/service/register and agent/check/register endpoints.
This status must be one of the valid check statuses: 'passing', 'warning', 'critical', 'unknown'.
If the status field is not present or the empty string, the default of 'critical' is used.
2015-04-12 02:00:31 +00:00
Andrew Poydence b5f6451e74 Returns an error for a key with a '/' 2015-03-31 21:08:06 -06:00
Ryan Uber fa7414e276 testutil: Use HTTPAddr 2015-03-19 17:03:07 -07:00
Ryan Uber 78f9f53bf1 testutil: exposing the API address from the test server 2015-03-19 17:03:06 -07:00
Ryan Uber 37f6301856 testutil: initial pass at moving test server harness into testutil 2015-03-19 17:03:06 -07:00
Seth Vargo b4f8c98fbb Read select environment variables when generating the default configuration 2015-03-18 11:12:10 -04:00
Chris Grieger f70ccdb4d2 Add 'ServiceAddress' field to 'CatalogService's truct 2015-03-04 10:29:57 +01:00
Ryan Uber 33390779eb api: remove unneeded static config 2015-02-24 09:50:35 -08:00
Armon Dadgar 5f627fca1b api: Refactoring into shared write logic 2015-02-18 15:15:02 -08:00
Armon Dadgar c0c9fb58ce api: Refactoring into shared query logic 2015-02-18 15:15:02 -08:00
Ryan Uber 5c2a764cbe api: allow timeout field to be specified for checks 2015-02-05 23:34:30 -08:00
Nicholas Capo 8cbca78b29 api/agent: Support HTTP service checks in client api 2015-02-03 20:13:55 +00:00
Armon Dadgar 17d1c392e4 CHANGELOG update, gofmt 2015-01-22 13:42:22 -08:00
Armon Dadgar e1a5d537d4 Merge pull request #625 from hashicorp/f-maintcmd
New "maint" command
2015-01-22 11:56:49 -08:00
Dr. Stefan Schimanski acd9efbc40 Add test case for service Address field 2015-01-22 11:50:20 +01:00
Dr. Stefan Schimanski a85575dcaa Add agent service Address field to the api
The Address field was introduced in #570. This patch extends this to the api.
2015-01-22 10:26:27 +01:00
Ryan Uber 089c4396c4 api: support reason flag for maintenance mode 2015-01-21 13:02:47 -08:00
Ryan Uber e088240c95 api: finish up maintenance tests 2015-01-21 11:08:57 -08:00
Ryan Uber e47b64197e api: add node/service maintenance mode functions 2015-01-21 10:51:43 -08:00
Ryan Uber 197a5a9a9a api: support multiple checks during service/check registration 2015-01-20 21:48:45 -08:00
Armon Dadgar d998bb3d67 Fixing merge conflict 2015-01-20 13:01:13 -08:00
Armon Dadgar 2e281dc51f api: Fixing some comments 2015-01-20 12:58:09 -08:00
Armon Dadgar 7195de51a2 api: Refactor to share session renew code 2015-01-19 16:53:47 -10:00
Armon Dadgar 14691f7e29 api: Detect conflicting existing values for lock/semaphore 2015-01-19 15:32:19 -10:00
Armon Dadgar 4fef14163d api: Make channels receive only 2015-01-19 14:37:36 -10:00
Ryan Uber b30af9536d Merge pull request #608 from imkira/fixBodyLeak
fixed: body not closed for non HTTP 200 responses
2015-01-19 09:33:06 -08:00
Mário Freitas f45105383c fixed: requireOK should close Body on error 2015-01-19 11:53:21 +09:00
Mário Freitas f30237a92e Revert "fixed: body not closed for non HTTP 200 responses"
This reverts commit dafa61ad10.
2015-01-19 11:51:51 +09:00
Ryan Uber 4b635738bf api: simple test for unix socket connectivity 2015-01-16 10:56:47 -08:00
Ryan Uber 4c3ec248a5 agent: fixing up tests 2015-01-16 09:58:37 -08:00
Mário Freitas dafa61ad10 fixed: body not closed for non HTTP 200 responses 2015-01-16 18:54:55 +09:00
Ryan Uber 4675cdf01c agent: beginning refactor 2015-01-16 00:45:03 -08:00
Jeff Mitchell 9fcea08dbb Ensure a socket is created for permissions adjustment tests and fix some
items pointed out in the code review

This code is copyright 2014 Akamai Technologies, Inc. <opensource@akamai.com>
2015-01-15 14:33:44 +00:00
Jeff Mitchell 70dd5a1e81 Unix socket-based HTTP API test functionality. As a consequence this
also required making some hardcoded values into more generic
functionality, which is generally a good thing. I verified that each
test function that I modified still passed.:

This code is copyright 2014 Akamai Technologies, Inc. <opensource@akamai.com>
2015-01-14 19:49:37 +00:00
Jeff Mitchell 11a3ce0bdd RPC and HTTP interfaces fully generically-sockified so Unix is supported.
Client works for RPC; will honor CONSUL_RPC_ADDR. HTTP works via consul/api;
honors CONSUL_HTTP_ADDR.

The format of a Unix socket in configuration data is:
"unix://[/path/to/socket];[username or uid];[gid];[mode]"

Obviously, the user must have appropriate permissions to create the socket
file in the given path and assign the requested uid/gid. Also note that Go does
not support gid lookups from group name, so gid must be numeric. See
https://codereview.appspot.com/101310044

When connecting from the client, the format is just the first part of the
above line:
"unix://[/path/to/socket]"

This code is copyright 2014 Akamai Technologies, Inc. <opensource@akamai.com>
2015-01-14 19:31:21 +00:00
Armon Dadgar 7ed1449b6c api: Adding Destroy to cleanup a semaphore 2015-01-13 14:18:28 -08:00
Armon Dadgar 9608108e64 api: Adding Destroy to cleanup a lock 2015-01-13 14:01:50 -08:00
Armon Dadgar 94d7022a88 api: Add support for DeleteCAS 2015-01-13 13:57:48 -08:00
Armon Dadgar f2e1594814 Merge pull request #600 from hashicorp/f-api-semaphore
Adding Semaphore support to API
2015-01-13 11:59:22 -08:00
Armon Dadgar ed6abe05fb api: Changing default semaphore key 2015-01-13 11:51:12 -08:00
Armon Dadgar 4a038927ee api: More reliable session check 2015-01-13 11:50:57 -08:00
Armon Dadgar c19b0e2ab2 api: Enable debug output from Consul for tests 2015-01-13 11:50:09 -08:00
Ryan Uber 78328ec149 api: make node health test more reliable 2015-01-13 11:25:32 -08:00
Armon Dadgar 5107f5d1f9 api: More semaphore tests 2015-01-12 18:18:32 -08:00
Armon Dadgar 8ad16ca390 api: Adding semaphore tests and fixes 2015-01-12 18:13:52 -08:00
Armon Dadgar b39374acae api: First pass at semaphore 2015-01-12 17:43:54 -08:00
Armon Dadgar c291acd96e api: Minor cleanups in lock 2015-01-12 17:43:13 -08:00
Armon Dadgar 3cba4df127 Merge pull request #594 from hashicorp/f-api-lock
api: Adding Lock helpers for leader election
2015-01-12 15:47:35 -08:00
Armon Dadgar 7225088c7d api: Fixing typo 2015-01-12 15:46:40 -08:00
Armon Dadgar e05ab983ab api: Fixing typos 2015-01-09 17:42:24 -08:00
Armon Dadgar 854aef82b0 api: Adding Lock helpers for leader election 2015-01-09 17:35:17 -08:00
Ryan Uber fab30406be api: test for a known leader as well as index > 0 2015-01-09 16:39:35 -08:00
Ryan Uber cb1b722813 api: improve reliability of catalog endpoint tests 2015-01-07 18:29:42 -08:00
Armon Dadgar bb3829c2d9 api: Adding constants for session behaviors 2015-01-07 14:01:27 -08:00
Ryan Uber 60ebd63760 Import HTTP basic auth patch from armon/consul-api#16 2015-01-07 13:01:44 -08:00
Ryan Uber 6c4ff58d2a api: improve test reliability 2015-01-06 16:51:28 -08:00
Ryan Uber 0c6c06cb0c api: use WaitForResult() to improve health tests 2015-01-06 16:18:50 -08:00
Ryan Uber f49189252e api: run consul instance on testing ports 2015-01-06 16:18:29 -08:00
Ryan Uber 8455077e5d api: add harnessing for tests
This is necessary as consul-api's tests require a real consul instance
to be running. We can't directly import an agent to fire up an instance,
due to the way this would create an import cycle. These tests instead
will start a consul instance using the binary in $PATH (if it exists).
2015-01-06 15:52:06 -08:00
Ryan Uber 5555e0eb9b api: initial import from armon/consul-api 2015-01-06 10:40:00 -08:00