Commit Graph

11519 Commits (c4c06c2dab1d524de66c4006c870df456e8acae1)

Author SHA1 Message Date
Daniel Nephin c9a87be6ee agent/cache: move typeEntry lookup to the edge
This change moves all the typeEntry lookups to the first step in the exported methods,
and makes unexporter internals accept the typeEntry struct.

This change is primarily intended to make it easier to extract the container of caches
from the Cache type.

It may incidentally reduce locking in fetch, but that was not a goal.
2020-04-03 16:01:56 -04:00
David Yu b51ad875c3
[docs] Built-in Proxies not meant for production (#7579)
* [docs] Built-in Proxies not meant for production

* Adding link to Envoy for Connect

* Update website/source/docs/connect/proxies/built-in.md

Co-Authored-By: Blake Covarrubias <blake@covarrubi.as>

* Revising note

* Update website/source/docs/connect/proxies/built-in.md

period

Co-Authored-By: Hans Hasselberg <me@hans.io>

Co-authored-by: Blake Covarrubias <blake@covarrubi.as>
Co-authored-by: Hans Hasselberg <me@hans.io>
2020-04-03 11:52:05 -07:00
Daniel Nephin b9cb1e93c3
Merge pull request #7589 from pierresouchay/fix_unstable_test_TestForwardSignals
Fixed unstable test TestForwardSignals()
2020-04-03 12:43:28 -04:00
Pierre Souchay 73056fecf8 Fixed unstable test TestForwardSignals()
Sometimes, in the CI, it could receive a SIGURG, producing this line:

  FAIL: TestForwardSignals/signal-interrupt (0.06s)
        util_test.go:286: expected to read line "signal: interrupt" but got "signal: urgent I/O condition"

Only forward the signals we test to avoid this kind of false positive

Example of such unstable errors in CI:
https://circleci.com/gh/hashicorp/consul/153571
2020-04-03 14:23:03 +02:00
Pierre Souchay 09e638a9c6
tests: more tolerance to latency for unstable test `TestCacheNotifyPolling()`. (#7574) 2020-04-03 10:29:38 +02:00
Luke Kysow 08df582e20
Document bootstrapACLs deprecation 2020-04-02 16:58:55 -07:00
Freddy b61214ef24
Fix regression with gateway registration and update docs (#7582) 2020-04-02 12:52:11 -06:00
Matt Keeler 8aec09aa8f
Ensure that token clone copies the roles (#7577) 2020-04-02 12:09:35 -04:00
Chris Piraino bcb7a89fda
Update CHANGELOG.md 2020-04-02 10:14:17 -05:00
Chris Piraino 584f90bbeb
Fix flapping of mesh gateway connect-service watches (#7575) 2020-04-02 10:12:13 -05:00
Pierre Souchay 2a8bf45e38
agent: show warning when enable_script_checks is enabled without safty net (#7437)
In order to enforce a bit security on Consul agents, add a new method in agent
to highlight possible security issues.

This does not return an error for now, but might in the future.

For now, it detects issues such as:

https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/

This would display this kind of messages:

```
2020-03-11T18:27:49.873+0100 [ERROR] agent: [SECURITY] issue: error="using enable-script-checks without ACLs and without allow_write_http_from is DANGEROUS, use enable-local-script-checks instead see https://www.hashicorp.com/blog/protecting-consul-from-rce-risk-in-specific-configurations/"
```
2020-04-02 09:59:23 +02:00
Andy Lindeman fb0a990e4d
agent: rewrite checks with proxy address, not local service address (#7518)
Exposing checks is supposed to allow a Consul agent bound to a different
IP address (e.g., in a different Kubernetes pod) to access healthchecks
through the proxy while the underlying service binds to localhost. This
is an important security feature that makes sure no external traffic
reaches the service except through the proxy.

However, as far as I can tell, this is subtly broken in the case where
the Consul agent cannot reach the proxy over localhost.

If a proxy is configured with: `{ LocalServiceAddress: "127.0.0.1",
Checks: true }`, as is typical with a sidecar proxy, the Consul checks
are currently rewritten to `127.0.0.1:<random port>`. A Consul agent
that does not share the loopback address cannot reach this address. Just
to make sure I was not misunderstanding, I tried configuring the proxy
with `{ LocalServiceAddress: "<pod ip>", Checks: true }`. In this case,
while the checks are rewritten as expected and the agent can reach the
dynamic port, the proxy can no longer reach its backend because the
traffic is no longer on the loopback interface.

I think rewriting the checks to use `proxy.Address`, the proxy's own
address, is more correct in this case. That is the IP where the proxy
can be reached, both by other proxies and by a Consul agent running on
a different IP. The local service address should continue to use
`127.0.0.1` in most cases.
2020-04-02 09:35:43 +02:00
Andy Lindeman c1cb18c648
proxycfg: support path exposed with non-HTTP2 protocol (#7510)
If a proxied service is a gRPC or HTTP2 service, but a path is exposed
using the HTTP1 or TCP protocol, Envoy should not be configured with
`http2ProtocolOptions` for the cluster backing the path.

A situation where this comes up is a gRPC service whose healthcheck or
metrics route (e.g. for Prometheus) is an HTTP1 service running on
a different port. Previously, if these were exposed either using
`Expose: { Checks: true }` or `Expose: { Paths: ... }`, Envoy would
still be configured to communicate with the path over HTTP2, which would
not work properly.
2020-04-02 09:35:04 +02:00
Pierre Souchay be1c5c4b48
config: validate system limits against limits.http_max_conns_per_client (#7434)
I spent some time today on my local Mac to figure out why Consul 1.6.3+
was not accepting limits.http_max_conns_per_client.

This adds an explicit check on number of file descriptors to be sure
it might work (this is no guarantee as if many clients are reaching
the agent, it might consume even more file descriptors)

Anyway, many users are fighting with RLIMIT_NOFILE, having a clear
message would allow them to figure out what to fix.

Example of message (reload or start):

```
2020-03-11T16:38:37.062+0100 [ERROR] agent: Error starting agent: error="system allows a max of 512 file descriptors, but limits.http_max_conns_per_client: 8192 needs at least 8212"
```
2020-04-02 09:22:17 +02:00
Daniel Nephin a6cede401a
Merge pull request #7572 from hashicorp/dnephin/ci-fix-go-mod-download
ci: Fix working_directory for go mod download
2020-04-01 18:57:21 -04:00
Daniel Nephin 625ce9baf2 ci: Fix working_directory for go mod download
The previous PR which added these was accidentally performing the download
in the root directory. For the api, and sdk directories it should be in done
in the same directory that will be used to run tests. Otherwise the
wrong dependencies will be downloaded which may add unnecessary time to
the CI run.
2020-04-01 17:02:23 -04:00
Shaker Islam ac309d55f4
docs: document exported functions in agent.go (closes #7101) (#7366)
and fix one linter error
2020-04-01 22:52:23 +02:00
R.B. Boyer c2f4063103 update changelog 2020-04-01 13:16:01 -05:00
Pierre Souchay 5703aadb59
[BUGFIX] Fix race condition in freeport (#7567)
This removes a race condition in reset since pendingPorts can be set to nil in reset()

If ticker is hit at wrong time, it would crash the unit test.

We ensure in reset to avoid this race condition by cancelling the goroutine using
killTicker chan.

We also properly clean up eveything, so garbage collector can work as expected.

To reproduce existing bug:
`while go test -timeout 30s github.com/hashicorp/consul/sdk/freeport -run '^(Test.*)$'; do go clean -testcache; done`

Will crash after a few 10s runs on my machine.

Error could be seen in unit tests sometimes:

[INFO] freeport: resetting the freeport package state
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1125536]

goroutine 25 [running]:
container/list.(*List).Len(...)
	/usr/local/Cellar/go/1.14/libexec/src/container/list/list.go:66
github.com/hashicorp/consul/sdk/freeport.checkFreedPortsOnce()
	/Users/p.souchay/go/src/github.com/hashicorp/consul/sdk/freeport/freeport.go:157 +0x86
github.com/hashicorp/consul/sdk/freeport.checkFreedPorts()
	/Users/p.souchay/go/src/github.com/hashicorp/consul/sdk/freeport/freeport.go:147 +0x71
created by github.com/hashicorp/consul/sdk/freeport.initialize
	/Users/p.souchay/go/src/github.com/hashicorp/consul/sdk/freeport/freeport.go:113 +0x2cf
FAIL	github.com/hashicorp/consul/sdk/freeport	1.607s
2020-04-01 13:14:33 -05:00
Daniel Nephin fce56e4cb6
Merge pull request #7570 from pierresouchay/fix_lint_upstream_due_to_change_args_in_NewTestAgent
[FIX BUILD] fix build due to merge of #7562
2020-04-01 12:39:34 -04:00
Pierre Souchay 08acd6a03e [FIX BUILD] fix build due to merge of #7562
Due to merge #7562, upstream does not compile anymore.

Error is:

ERRO Running error: gofmt: analysis skipped: errors in package: [/Users/p.souchay/go/src/github.com/hashicorp/consul/agent/config_endpoint_test.go:188:33: too many arguments]
2020-04-01 18:29:45 +02:00
Daniel Nephin 0d8edc3e27
Merge pull request #7562 from hashicorp/dnephin/remove-tname-from-name
testing: Remove old default value from NewTestAgent() calls
2020-04-01 11:48:45 -04:00
Daniel Nephin 76f03982eb
Merge pull request #7427 from hashicorp/dnephin/website-fix-errrors-in-upgrade-docs
docs: fix some errors in upgrade-specific
2020-04-01 11:36:53 -04:00
Daniel Nephin 2c631a7a78
Merge pull request #7465 from hashicorp/dnephin/correct-docs-log-file
docs: Update config reference for log-file
2020-04-01 11:29:20 -04:00
Daniel Nephin 5f44bbcac2
Merge pull request #7519 from hashicorp/dnephin/help-to-stdout
cli: send requested help text to stdout
2020-04-01 11:26:12 -04:00
Daniel Nephin 190fc3c732
Merge pull request #7533 from hashicorp/dnephin/xds-server-1
agent/xds: small cleanup
2020-04-01 11:24:50 -04:00
Hans Hasselberg efaf0f9a18
docs: revert "docs: fix param for register checks id->checkid (#7393)" (#7565)
This reverts commit ed5202943e.
2020-04-01 11:49:20 +02:00
Rasmus Mikkelsen ed5202943e
docs: fix param for register checks id->checkid (#7393) 2020-04-01 10:59:28 +02:00
John Cowen 8d4631526c
ui: Fix token duplication bug (#7552)
We need to detect whether an object is an ember-data snapshot or just a
plain object, and we where restricted from using `instanceof` due to
ember-data's `Snapshot` class being private.

We'd chosen to go with `constructor.name` instead, which seemed to work,
but when the javascript gets minifed the name of the contructor is also
minified and therefore is not what you are expecting.

This commit reverts to our original idea of checking for the existence
and type of the `.attributes` function, which is the function we require
within the conditional, and therefore is more reliable (if the function
doesn't exist it will error out during development aswell as production)
2020-04-01 09:55:20 +01:00
Emre Savcı 2083b7b04d
agent: add len, cap while initializing arrays 2020-04-01 10:54:51 +02:00
Pierre Souchay 4d35816bee
tests: fixed unstable test TestAPI_AgentMonitor (#7561)
Similar fix as https://github.com/hashicorp/consul/pull/7541
2020-04-01 09:47:57 +02:00
Daniel Nephin e759daafdd Rename NewTestAgentWithFields to StartTestAgent
This function now only starts the agent.

Using:

git grep -l 'StartTestAgent(t, true,' | \
        xargs sed -i -e 's/StartTestAgent(t, true,/StartTestAgent(t,/g'
2020-03-31 17:14:55 -04:00
Daniel Nephin f9f6b14533 Convert the remaining calls to NewTestAgentWithFields
After removing the t.Name() parameter with sed, convert the last few tests which
use a custom name to call NewTestAgentWithFields instead.
2020-03-31 17:14:55 -04:00
Daniel Nephin 9d83a45663
Merge pull request #7557 from hashicorp/dnephin/ci-small-improvements
ci: Upload coverage from each job
2020-03-31 17:07:01 -04:00
Daniel Nephin 0c409d460f
Merge pull request #7470 from hashicorp/dnephin/dns-unused-params
dns: Remove a few unused function parameters
2020-03-31 16:56:19 -04:00
Andreas Gruhler 3cb9aa8010
docs: make example code for RedundancyZoneTag match description (#7464) 2020-03-31 22:54:04 +02:00
Pierre Souchay 54b22c638d
config: allow running `consul agent -dev -ui-dir=some_path` (#7525)
When run in with `-dev` in DevMode, it is not possible to replace
the embeded UI with another one because `-dev` implies `-ui`.

This commit allows this an slightly change the error message
about Consul 0.7.0 which is very old and does not apply to
current version anyway.
2020-03-31 22:36:20 +02:00
Geoffrey Grosenbach ab6c410064
docs: add link to Learn from downloads page (#7534)
Adds a link to Learn from the downloads page as well as a link to the Consul
Template guide.
2020-03-31 22:30:29 +02:00
Pierre Souchay eafe9a895a
tests: fixed bats warning (#7544)
This fixes this bats warning:

  duplicate test name(s) in /workdir/primary/bats/verify.bats: test_s1_upstream_made_1_connection

Test was already defined at line 42, rename it to avoid test name duplication
2020-03-31 22:29:27 +02:00
Daniel Nephin 475659a132 Remove name from NewTestAgent
Using:

git grep -l 'NewTestAgent(t, t.Name(),' | \
    xargs sed -i -e 's/NewTestAgent(t, t.Name(),/NewTestAgent(t,/g'
2020-03-31 16:13:44 -04:00
Freddy 90576060bc
Add config entry for terminating gateways (#7545)
This config entry will be used to configure terminating gateways.

It accepts the name of the gateway and a list of services the gateway will represent.

For each service users will be able to specify: its name, namespace, and additional options for TLS origination.

Co-authored-by: Kyle Havlovitz <kylehav@gmail.com>
Co-authored-by: Chris Piraino <cpiraino@hashicorp.com>
2020-03-31 13:27:32 -06:00
Daniel Nephin 66a51c9701 Upload coverage from each job
Allows us to remove the merge coverage report step
2020-03-31 14:43:13 -04:00
Kyle Havlovitz c911174327
Add config entry/state for Ingress Gateways (#7483)
* Add Ingress gateway config entry and other relevant structs

* Add api package tests for ingress gateways

* Embed EnterpriseMeta into ingress service struct

* Add namespace fields to api module and test consul config write decoding

* Don't require a port for ingress gateways

* Add snakeJSON and camelJSON cases in command test

* Run Normalize on service's ent metadata

Sadly cannot think of a way to test this in OSS.

* Every protocol requires at least 1 service

* Validate ingress protocols

* Update agent/structs/config_entry_gateways.go

Co-authored-by: Chris Piraino <cpiraino@hashicorp.com>
Co-authored-by: Freddy <freddygv@users.noreply.github.com>
2020-03-31 11:59:10 -05:00
Daniel Nephin 5035fe782a
Merge pull request #7555 from hashicorp/dnephin/ci-small-improvements
ci: small improvements
2020-03-31 12:50:21 -04:00
Daniel Nephin 2a676dc287 ci: Increase GOMAXPROCS so that the tests run faster 2020-03-31 12:32:23 -04:00
Daniel Nephin a79417f973 ci: small improvements
Run 'go mod download' first so that dependencies are pulled in a
separate step. This makes the test splitting and test output easier
to scroll through.

Remove test splitting from the api/sdk test runs. These jobs do not
use parallelism, so there is no test timing data injected into the
job. They can't be split.

Remove the dependencies from the go-test job, so that we can start
the slowest test job earlier in the workflow.
2020-03-31 11:52:58 -04:00
Daniel Nephin 9d959907a4
Merge pull request #7485 from hashicorp/dnephin/do-not-skip-tests-on-ci
ci: Make it harder to accidentally skip tests on CI, and doc why some are skipped
2020-03-31 11:15:44 -04:00
Daniel Nephin fd7383eb6e
Merge pull request #7546 from hashicorp/dnephin/logging-in-tests
testing: Improve logging from TestAgent
2020-03-31 11:08:39 -04:00
John Cowen 6ffe0e6fe4
ui: Use the `each key=""` parameter to force ember to reuse DOM (#7550)
Ember tries to reuse DOM elements when it can but as ember looks for
changes to objects rather than the DOM itself sometimes. This and the
fact that an objects identity may change even though its value hasn't,
results in ember occasionally re-mutating DOM when it doesn't need to.

The `each` helper includes a `key` attribute to hint to ember what it
should look for when deciding whether something has changed, rather than
the objects identity.

https://api.emberjs.com/ember/release/classes/Ember.Templates.helpers/methods/each#specifying-keys

We use this here to fix an issue where DOM was being redrawn after the
user had scrolled the page and was therefore resetting the scroll back
to 0 (the top of the page)
2020-03-31 14:58:59 +01:00
Daniel Nephin ad7c78f134 Remove t.Name() from TestAgent.Name
And re-add the name to the logger so that log messages from different agents
in a single can be identified.
2020-03-30 16:47:24 -04:00