Commit Graph

143 Commits (9c74f140616808b1cc79d68402fc76f20ce2084d)

Author SHA1 Message Date
R.B. Boyer 7b0548dd8d
server: suppress spurious blocking query returns where multiple config entries are involved (#12362)
Starting from and extending the mechanism introduced in #12110 we can specially handle the 3 main special Consul RPC endpoints that react to many config entries in a single blocking query in Connect:

- `DiscoveryChain.Get`
- `ConfigEntry.ResolveServiceConfig`
- `Intentions.Match`

All of these will internally watch for many config entries, and at least one of those will likely be not found in any given query. Because these are blends of multiple reads the exact solution from #12110 isn't perfectly aligned, but we can tweak the approach slightly and regain the utility of that mechanism.

### No Config Entries Found

In this case, despite looking for many config entries none may be found at all. Unlike #12110 in this scenario we do not return an empty reply to the caller, but instead synthesize a struct from default values to return. This can be handled nearly identically to #12110 with the first 1-2 replies being non-empty payloads followed by the standard spurious wakeup suppression mechanism from #12110.

### No Change Since Last Wakeup

Once a blocking query loop on the server has completed and slept at least once, there is a further optimization we can make here to detect if any of the config entries that were present at specific versions for the prior execution of the loop are identical for the loop we just woke up for. In that scenario we can return a slightly different internal sentinel error and basically externally handle it similar to #12110.

This would mean that even if 20 discovery chain read RPC handling goroutines wakeup due to the creation of an unrelated config entry, the only ones that will terminate and reply with a blob of data are those that genuinely have new data to report.

### Extra Endpoints

Since this pattern is pretty reusable, other key config-entry-adjacent endpoints used by `agent/proxycfg` also were updated:

- `ConfigEntry.List`
- `Internal.IntentionUpstreams` (tproxy)
2022-02-25 15:46:34 -06:00
Dhia Ayachi cd9d8d44a5
file watcher to be used for configuration auto-reload feature (#12301)
* add config watcher to the config package

* add logging to watcher

* add test and refactor to add WatcherEvent.

* add all API calls and fix a bug with recreated files

* add tests for watcher

* remove the unnecessary use of context

* Add debug log and a test for file rename

* use inode to detect if the file is recreated/replaced and only listen to create events.

* tidy ups (#1535)

* tidy ups

* Add tests for inode reconcile

* fix linux vs windows syscall

* fix linux vs windows syscall

* fix windows compile error

* increase timeout

* use ctime ID

* remove remove/creation test as it's a use case that fail in linux

* fix linux/windows to use Ino/CreationTime

* fix the watcher to only overwrite current file id

* fix linter error

* fix remove/create test

* set reconcile loop to 200 Milliseconds

* fix watcher to not trigger event on remove, add more tests

* on a remove event try to add the file back to the watcher and trigger the handler if success

* fix race condition

* fix flaky test

* fix race conditions

* set level to info

* fix when file is removed and get an event for it after

* fix to trigger handler when we get a remove but re-add fail

* fix error message

* add tests for directory watch and fixes

* detect if a file is a symlink and return an error on Add

* rename Watcher to FileWatcher and remove symlink deref

* add fsnotify@v1.5.1

* fix go mod

* fix flaky test

* Apply suggestions from code review

Co-authored-by: Ashwin Venkatesh <ashwin@hashicorp.com>

* fix a possible stack overflow

* do not reset timer on errors, rename OS specific files

* start the watcher when creating it

* fix data race in tests

* rename New func

* do not call handler when a remove event happen

* events trigger on write and rename

* fix watcher tests

* make handler async

* remove recursive call

* do not produce events for sub directories

* trim "/" at the end of a directory when adding

* add missing test

* fix logging

* add todo

* fix failing test

* fix flaking tests

* fix flaky test

* add logs

* fix log text

* increase timeout

* reconcile when remove

* check reconcile when removed

* fix reconcile move test

* fix logging

* delete invalid file

* Apply suggestions from code review

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* fix review comments

* fix is watched to properly catch a remove

* change test timeout

* fix test and rename id

* fix test to create files with different mod time.

* fix deadlock when stopping watcher

* Apply suggestions from code review

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* fix a deadlock when calling stop while emitting event is blocked

* make sure to close the event channel after the event loop is done

* add go doc

* back date file instead of sleeping

* Apply suggestions from code review

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* check error

Co-authored-by: Ashwin Venkatesh <ashwin@hashicorp.com>
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2022-02-21 11:36:52 -05:00
R.B. Boyer 80dfcb1bcd
raft: update to v1.3.5 (#12325)
This includes closing some leadership transfer gaps and adding snapshot
restore progress logging.
2022-02-14 13:48:52 -06:00
FFMMM 78264a8030
Vendor in rpc mono repo for net/rpc fork, go-msgpack, msgpackrpc. (#12311)
This commit syncs ENT changes to the OSS repo.

Original commit details in ENT:

```
commit 569d25f7f4578981c3801e6e067295668210f748
Author: FFMMM <FFMMM@users.noreply.github.com>
Date:   Thu Feb 10 10:23:33 2022 -0800

    Vendor fork net rpc (#1538)

    * replace net/rpc w consul-net-rpc/net/rpc

    Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>

    * replace msgpackrpc and go-msgpack with fork from mono repo

    Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>

    * gofmt all files touched

    Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>
```

Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>
2022-02-14 09:45:45 -08:00
Dhia Ayachi 25c36af222
update serf to v0.9.7 (#12057)
* update serf to v0.9.7

* add change log

* update changelog
2022-01-18 13:03:22 -05:00
dependabot[bot] f3ac9dafa6
build(deps): bump github.com/ryanuber/columnize (#12062)
Bumps [github.com/ryanuber/columnize](https://github.com/ryanuber/columnize) from 2.1.0+incompatible to 2.1.2+incompatible.
- [Release notes](https://github.com/ryanuber/columnize/releases)
- [Commits](https://github.com/ryanuber/columnize/compare/v2.1.0...v2.1.2)

---
updated-dependencies:
- dependency-name: github.com/ryanuber/columnize
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-01-14 12:39:13 -05:00
dependabot[bot] 2c8de28005
build(deps): bump github.com/kr/text from 0.1.0 to 0.2.0 (#12063)
Bumps [github.com/kr/text](https://github.com/kr/text) from 0.1.0 to 0.2.0.
- [Release notes](https://github.com/kr/text/releases)
- [Commits](https://github.com/kr/text/compare/v0.1.0...v0.2.0)

---
updated-dependencies:
- dependency-name: github.com/kr/text
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-01-14 12:35:00 -05:00
dependabot[bot] 4947d6d29f
build(deps): bump github.com/mitchellh/pointerstructure (#12072)
Bumps [github.com/mitchellh/pointerstructure](https://github.com/mitchellh/pointerstructure) from 1.0.0 to 1.2.1.
- [Release notes](https://github.com/mitchellh/pointerstructure/releases)
- [Commits](https://github.com/mitchellh/pointerstructure/compare/v1.0.0...v1.2.1)

---
updated-dependencies:
- dependency-name: github.com/mitchellh/pointerstructure
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-01-14 11:56:26 -05:00
dependabot[bot] 97348e25a7
Bump github.com/aws/aws-sdk-go from 1.25.41 to 1.42.34 (#12083)
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.25.41 to 1.42.34.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.25.41...v1.42.34)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-01-14 11:54:41 -05:00
dependabot[bot] 151e516a8c
Bump github.com/hashicorp/go-memdb from 1.3.1 to 1.3.2 (#11066)
Bumps [github.com/hashicorp/go-memdb](https://github.com/hashicorp/go-memdb) from 1.3.1 to 1.3.2.
- [Release notes](https://github.com/hashicorp/go-memdb/releases)
- [Changelog](https://github.com/hashicorp/go-memdb/blob/master/changes.go)
- [Commits](https://github.com/hashicorp/go-memdb/compare/v1.3.1...v1.3.2)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/go-memdb
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-01-13 10:54:36 -08:00
dependabot[bot] b9ed140b4b
Bump github.com/hashicorp/go-raftchunking from 0.6.1 to 0.6.2 (#11065)
Bumps [github.com/hashicorp/go-raftchunking](https://github.com/hashicorp/go-raftchunking) from 0.6.1 to 0.6.2.
- [Release notes](https://github.com/hashicorp/go-raftchunking/releases)
- [Commits](https://github.com/hashicorp/go-raftchunking/compare/v0.6.1...v0.6.2)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/go-raftchunking
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-01-13 10:28:49 -08:00
dependabot[bot] 8878f0ea21
build(deps): bump github.com/hashicorp/go-multierror from 1.1.0 to 1.1.1
Bumps [github.com/hashicorp/go-multierror](https://github.com/hashicorp/go-multierror) from 1.1.0 to 1.1.1.
- [Release notes](https://github.com/hashicorp/go-multierror/releases)
- [Commits](https://github.com/hashicorp/go-multierror/compare/v1.1.0...v1.1.1)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/go-multierror
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-01-13 17:41:15 +00:00
Chris S. Kim ea0df29afb
Update memberlist to 0.3.1 (#12042) 2022-01-12 12:00:18 -05:00
Dhia Ayachi d0274d38a9
upgrade raft to v1.3.3 (#11958)
* upgrade raft to v1.3.3

* add change log

* reword the changelog

Co-authored-by: FFMMM <FFMMM@users.noreply.github.com>

Co-authored-by: FFMMM <FFMMM@users.noreply.github.com>
2022-01-06 14:09:13 -05:00
kisunji 59ddc44cc7 Update golang.org/x/net to address CVE-2021-44716 2021-12-15 11:54:47 -05:00
Daniel Nephin dccd3f5806 Merge remote-tracking branch 'origin/main' into serve-panic-recovery 2021-12-07 16:30:41 -05:00
Matt Keeler 42a5635bc3 Use raft-boltdb/v2 2021-12-02 16:56:15 -05:00
Mike Morris 25826e3ee4
deps: update gopsutil to fix Windows ARM and macOS non-Apple LLVM builds (#11586)
Bumps transitive dep go-ole to v1.2.6 with fixes
2021-11-16 15:40:11 -05:00
R.B. Boyer eb21649f82
partitions: various refactors to support partitioning the serf LAN pool (#11568) 2021-11-15 09:51:14 -06:00
Giulio Micheloni af7b7b5693
Merge branch 'main' into serve-panic-recovery 2021-11-06 16:12:06 +01:00
Connor efe4b21287
Support Vault Namespaces explicitly in CA config (#11477)
* Support Vault Namespaces explicitly in CA config

If there is a Namespace entry included in the Vault CA configuration,
set it as the Vault Namespace on the Vault client

Currently the only way to support Vault namespaces in the Consul CA
config is by doing one of the following:
1) Set the VAULT_NAMESPACE environment variable which will be picked up
by the Vault API client
2) Prefix all Vault paths with the namespace

Neither of these are super pleasant. The first requires direct access
and modification to the Consul runtime environment. It's possible and
expected, not super pleasant.

The second requires more indepth knowledge of Vault and how it uses
Namespaces and could be confusing for anyone without that context. It
also infers that it is not supported

* Add changelog

* Remove fmt.Fprint calls

* Make comment clearer

* Add next consul version to website docs

* Add new test for default configuration

* go mod tidy

* Add skip if vault not present

* Tweak changelog text
2021-11-05 11:42:28 -05:00
FFMMM 0954d261ae
use *telemetry.MetricsPrefix as prometheus.PrometheusOpts.Name (#11290)
Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>
2021-10-21 13:33:01 -07:00
Dhia Ayachi ab31c50915
update raft to v1.3.2 (#11375)
* update raft to v1.3.2

* add changelog

* fix changelog
2021-10-21 13:21:22 -04:00
Mike Morris 55659f3c9d
deps: update consul-api and consul-sdk (#11361)
* deps: update consul/sdk in api/go.mod

* deps: update consul/api to v1.11.0
2021-10-20 14:58:53 -04:00
Giulio Micheloni fecce25658 Separete test file and no stack trace in ret error 2021-10-16 18:02:03 +01:00
Giulio Micheloni 0c78ddacde Merge branch 'main' of https://github.com/hashicorp/consul into hashicorp-main 2021-10-16 16:59:32 +01:00
Jeff Widman 2dc62aa0c4
Bump `go-discover` to fix broken dep tree (#10898) 2021-09-16 15:31:22 -04:00
freddygv a78390a30b Update yamux 2021-08-25 19:46:12 -06:00
Giulio Micheloni 655da1fc42
Merge branch 'main' into serve-panic-recovery 2021-08-22 20:31:11 +02:00
Daniel Nephin 31bcd80528 debug: improve a couple of the test cases
Use gotest.tools/v3/fs to make better assertions about the files

Remove the TestAgent from TestDebugCommand_Prepare_ValidateTiming, since we can test that validation
without making any API calls.
2021-08-18 12:29:34 -04:00
Roopak Venkatakrishnan e43cf46267 Update x/sys to support go 1.17 2021-08-18 03:00:22 +00:00
Mike Morris 3bae53a989
deps: upgrade gogo-protobuf to v1.3.2 (#10813)
* deps: upgrade gogo-protobuf to v1.3.2

* go mod tidy using go 1.16

* proto: regen protobufs after upgrading gogo/protobuf

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
2021-08-12 14:05:46 -04:00
Giulio Micheloni 2b14a9b59a grpc Server: turn panic into error through middleware 2021-08-07 13:21:12 +01:00
Daniel Nephin 5f753dedab Update armon/go-metrics
To pickup new InMemSink.Stream method
2021-07-26 15:58:17 -04:00
Daniel Nephin 414ce3f09b Update serf
To pick up data race fixes
2021-07-14 18:58:16 -04:00
Dhia Ayachi c72ee2063e
upgrade golang crypto from 0.0.0-20200930160638-afb6bcd081ae => v0.0.0-20210513164829-c07d793c2f9a (#10390) 2021-06-14 12:38:42 -04:00
Dhia Ayachi 005ad9e46d
generate a single debug file for a long duration capture (#10279)
* debug: remove the CLI check for debug_enabled

The API allows collecting profiles even debug_enabled=false as long as
ACLs are enabled. Remove this check from the CLI so that users do not
need to set debug_enabled=true for no reason.

Also:
- fix the API client to return errors on non-200 status codes for debug
  endpoints
- improve the failure messages when pprof data can not be collected

Co-Authored-By: Dhia Ayachi <dhia@hashicorp.com>

* remove parallel test runs

parallel runs create a race condition that fail the debug tests

* snapshot the timestamp at the beginning of the capture

- timestamp used to create the capture sub folder is snapshot only at the beginning of the capture and reused for subsequent captures
- capture append to the file if it already exist

* Revert "snapshot the timestamp at the beginning of the capture"

This reverts commit c2d03346

* Refactor captureDynamic to extract capture logic for each item in a different func

* snapshot the timestamp at the beginning of the capture

- timestamp used to create the capture sub folder is snapshot only at the beginning of the capture and reused for subsequent captures
- capture append to the file if it already exist

* Revert "snapshot the timestamp at the beginning of the capture"

This reverts commit c2d03346

* Refactor captureDynamic to extract capture logic for each item in a different func

* extract wait group outside the go routine to avoid a race condition

* capture pprof in a separate go routine

* perform a single capture for pprof data for the whole duration

* add missing vendor dependency

* add a change log and fix documentation to reflect the change

* create function for timestamp dir creation and simplify error handling

* use error groups and ticker to simplify interval capture loop

* Logs, profile and traces are captured for the full duration. Metrics, Heap and Go routines are captured every interval

* refactor Logs capture routine and add log capture specific test

* improve error reporting when log test fail

* change test duration to 1s

* make time parsing in log line more robust

* refactor log time format in a const

* test on log line empty the earliest possible and return

Co-authored-by: Freddy <freddygv@users.noreply.github.com>

* rename function to captureShortLived

* more specific changelog

Co-authored-by: Paul Banks <banks@banksco.de>

* update documentation to reflect current implementation

* add test for behavior when invalid param is passed to the command

* fix argument line in test

* a more detailed description of the new behaviour

Co-authored-by: Paul Banks <banks@banksco.de>

* print success right after the capture is done

* remove an unnecessary error check

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>

* upgraded github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57 => v0.0.0-20210601050228-01bbb1931b22

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
Co-authored-by: Freddy <freddygv@users.noreply.github.com>
Co-authored-by: Paul Banks <banks@banksco.de>
2021-06-07 13:00:51 -04:00
Matt Keeler 3a0007b158
Bump raft-autopilot version to the latest. (#10306) 2021-05-27 12:59:14 -04:00
Daniel Nephin 1618912cf6 Fix some test flakes
- return errors in TestAgent.Start so that the retry works correctly
- remove duplicate logging, the error is returned already
- add a missing t.Helper() to retry.Run
- properly set a.Agent to nil so that subsequent retry attempts will actually try to start
2021-05-10 13:20:45 -04:00
Daniel Nephin 9a7fb48dcb Update a couple dependencies
To pickup bug fixes
2021-05-04 14:09:10 -04:00
Paul Banks 3ad754ca7b
Make Raft trailing logs and snapshot timing reloadable (#10129)
* WIP reloadable raft config

* Pre-define new raft gauges

* Update go-metrics to change gauge reset behaviour

* Update raft to pull in new metric and reloadable config

* Add snapshot persistance timing and installSnapshot to our 'protected' list as they can be infrequent but are important

* Update telemetry docs

* Update config and telemetry docs

* Add note to oldestLogAge on when it is visible

* Add changelog entry

* Update website/content/docs/agent/options.mdx

Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com>

Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com>
2021-05-04 15:36:53 +01:00
R.B. Boyer 71d45a3460
Support Incremental xDS mode (#9855)
This adds support for the Incremental xDS protocol when using xDS v3. This is best reviewed commit-by-commit and will not be squashed when merged.

Union of all commit messages follows to give an overarching summary:

xds: exclusively support incremental xDS when using xDS v3

Attempts to use SoTW via v3 will fail, much like attempts to use incremental via v2 will fail.
Work around a strange older envoy behavior involving empty CDS responses over incremental xDS.
xds: various cleanups and refactors that don't strictly concern the addition of incremental xDS support

Dissolve the connectionInfo struct in favor of per-connection ResourceGenerators instead.
Do a better job of ensuring the xds code uses a well configured logger that accurately describes the connected client.
xds: pull out checkStreamACLs method in advance of a later commit

xds: rewrite SoTW xDS protocol tests to use protobufs rather than hand-rolled json strings

In the test we very lightly reuse some of the more boring protobuf construction helper code that is also technically under test. The important thing of the protocol tests is testing the protocol. The actual inputs and outputs are largely already handled by the xds golden output tests now so these protocol tests don't have to do double-duty.

This also updates the SoTW protocol test to exclusively use xDS v2 which is the only variant of SoTW that will be supported in Consul 1.10.

xds: default xds.Server.AuthCheckFrequency at use-time instead of construction-time
2021-04-29 13:54:05 -05:00
R.B. Boyer 2c3d7da5dd
mod: bump to github.com/hashicorp/mdns v1.0.4 (#10018) 2021-04-14 14:17:52 -05:00
Daniel Nephin 46279547ec Update memberlist to v0.2.3
To pickup data race fixes
2021-03-24 18:20:19 -04:00
Daniel Nephin cae4b2c0eb Update go-memdb
To use a version that will not panic when an iterator is used with modifications.
2021-01-28 17:19:55 -05:00
Daniel Nephin 727a402810
Merge pull request #9302 from hashicorp/dnephin/add-service-3
agent: remove ServiceManager.Start goroutine
2021-01-28 16:59:41 -05:00
Matt Keeler f561462064
Upgrade raft-autopilot and wait for autopilot it to stop when revoking leadership (#9644)
Fixes: 9626
2021-01-27 11:14:52 -05:00
Daniel Nephin 28736e60fd lib/mutex: add mutex with TryLock and update vendor 2021-01-25 18:01:47 -05:00
Daniel Nephin aeb9c09e25 Update mapstructure 2021-01-12 12:24:56 -05:00
Pierre Souchay 408b249fc5 [bugfix] Prometheus metrics without warnings
go-metrics is updated to 0.3.6 to properly handle help in prometheus metrics

This fixes https://github.com/hashicorp/consul/issues/9303 and
https://github.com/hashicorp/consul/issues/9471
2021-01-06 13:54:05 +01:00