Commit Graph

3676 Commits (8e9773e20b0ee92cc2a5d34f9b5b52546a4c7b58)

Author SHA1 Message Date
Matt Keeler caafc02449 hcs-1936: Prepare for adding license auto-retrieval to auto-config in enterprise 2021-05-24 13:20:30 -04:00
Matt Keeler 234d0a3c2a Preparation for changing where license management is done. 2021-05-24 10:19:31 -04:00
hc-github-team-consul-core 57e6c8a9e5 auto-updated agent/uiserver/bindata_assetfs.go from commit 600f85753 2021-05-24 11:37:54 +00:00
hc-github-team-consul-core be9b45588b auto-updated agent/uiserver/bindata_assetfs.go from commit dd4a66808 2021-05-24 10:56:33 +00:00
Daniel Nephin d5dec93992
Merge pull request #10272 from hashicorp/dnephin/backport-namespace-license-fix
Backport some ent changes for serf tags
2021-05-21 12:31:34 -04:00
Matt Keeler 50cc9fdd06 Add OSS bits for supporting specifying the enterprise license via config 2021-05-20 16:11:33 -04:00
Daniel Nephin ed37a2dbc2 Refactor of serf feature flag tags.
This refactor is to make it easier to see how serf feature flags are
encoded as serf tags, and where those feature flags are read.

- use constants for both the prefix and feature flag name. A constant
  makes it much easier for an IDE to locate the read and write location.
- isolate the feature-flag encoding logic in the metadata package, so
  that the feature flag prefix can be unexported. Only expose a function
  for encoding the flags into tags. This logic is now next to the logic
  which reads the tags.
- remove the duplicate `addEnterpriseSerfTags` functions. Both Client
  and Server structs had the same implementation. And neither
  implementation needed the method receiver.
2021-05-20 12:57:06 -04:00
Daniel Nephin 6bda3a48ba
Merge pull request #10200 from hashicorp/dnephin/backport-audit-log-config-changes
config: backport audit log config changes from enterprise
2021-05-19 10:58:28 -04:00
hc-github-team-consul-core 76f857e781 auto-updated agent/uiserver/bindata_assetfs.go from commit 39302041e 2021-05-19 10:11:29 +00:00
Joshua Shanks 00ca99bf67 GH-8728 add raft default values 2021-05-18 14:51:14 -04:00
hc-github-team-consul-core 2bb846fee5 auto-updated agent/uiserver/bindata_assetfs.go from commit 8301e79c5 2021-05-18 15:35:50 +00:00
hc-github-team-consul-core 5c119ba050 auto-updated agent/uiserver/bindata_assetfs.go from commit d1bbe0895 2021-05-17 12:32:31 +00:00
R.B. Boyer ede14b7c54
xds: emit a labeled gauge of connected xDS streams by version (#10243)
Fixes #10099
2021-05-14 13:59:13 -05:00
R.B. Boyer 597448da47
server: ensure that central service config flattening properly resets the state each time (#10239)
The prior solution to call reply.Reset() aged poorly since newer fields
were added to the reply, but not added to Reset() leading serial
blocking query loops on the server to blend replies.

This could manifest as a service-defaults protocol change from
default=>http not reverting back to default after the config entry
reponsible was deleted.
2021-05-14 10:21:44 -05:00
R.B. Boyer 7e1d7803b8
agent: ensure we hash the non-deprecated upstream fields on ServiceConfigRequest (#10240) 2021-05-14 10:15:48 -05:00
hc-github-team-consul-core 0f313f0f58 auto-updated agent/uiserver/bindata_assetfs.go from commit 04bd57617 2021-05-13 10:42:23 +00:00
Iryna Shustava d7d44f6ae7
Save exposed ports in agent's store and expose them via API (#10173)
* Save exposed HTTP or GRPC ports to the agent's store
* Add those the health checks API so we can retrieve them from the API
* Change redirect-traffic command to also exclude those ports from inbound traffic redirection when expose.checks is set to true.
2021-05-12 13:51:39 -07:00
Daniel Nephin 71fc219189
Merge pull request #10217 from hashicorp/dnephin/test-flakes
testing: attempt to fix some test flakes
2021-05-12 15:38:29 -04:00
R.B. Boyer 3b50a55533
connect: update supported envoy versions to 1.18.3, 1.17.3, 1.16.4, and 1.15.5 (#10231) 2021-05-12 14:06:06 -05:00
Kyle Havlovitz 7bc4a59177
Merge pull request #9995 from tarat44/fix-close-h2ping-client-connections
Add graceful shutdown of client connection and WaitGroup to H2PING
2021-05-11 10:45:26 -07:00
hc-github-team-consul-core 6234d01d5c auto-updated agent/uiserver/bindata_assetfs.go from commit ecbeaa87c 2021-05-11 15:39:38 +00:00
Daniel Nephin b823cd3994
Merge pull request #10188 from hashicorp/dnephin/dont-persist-agent-tokens
agent/local: do not persist the agent or user token
2021-05-10 15:58:20 -04:00
Daniel Nephin 9c8b0b451f testing: don't run t.Parallel in a goroutine
TestACLEndpoint_Login_with_TokenLocality was reguardly being reported as failed even though
it was not failing. I took another look and I suspect it is because t.Parllel was being
called in a goroutine.

This would lead to strange behaviour which apparently confused the 'go test' runner.
2021-05-10 13:30:10 -04:00
Daniel Nephin 1618912cf6 Fix some test flakes
- return errors in TestAgent.Start so that the retry works correctly
- remove duplicate logging, the error is returned already
- add a missing t.Helper() to retry.Run
- properly set a.Agent to nil so that subsequent retry attempts will actually try to start
2021-05-10 13:20:45 -04:00
hc-github-team-consul-core ece69f8e11 auto-updated agent/uiserver/bindata_assetfs.go from commit 90d1a307e 2021-05-10 10:22:54 +00:00
hc-github-team-consul-core 08b6f95fd7 auto-updated agent/uiserver/bindata_assetfs.go from commit 0e4db3f24 2021-05-07 15:59:32 +00:00
hc-github-team-consul-core a227966d85 auto-updated agent/uiserver/bindata_assetfs.go from commit e3f9a0f8e 2021-05-07 11:29:11 +00:00
hc-github-team-consul-core 43cf4fa2dc auto-updated agent/uiserver/bindata_assetfs.go from commit 5400c81e2 2021-05-07 11:11:34 +00:00
Daniel Nephin df5e18c9c5 agent/local: only fallback to agent token for deletes
Fallback to the default user token for synching registrations.
2021-05-06 18:44:05 -04:00
Daniel Nephin 49e7e37cc7 config: handled deprecated config key with a warning 2021-05-06 18:09:30 -04:00
Daniel Nephin 0ddbb98668 Backport audit log config changes from enterprise 2021-05-06 18:09:30 -04:00
Daniel Nephin d19137a429
Merge pull request #10075 from hashicorp/dnephin/handle-raft-apply-errors
rpc: some cleanup of canRetry and ForwardRPC
2021-05-06 16:59:53 -04:00
Freddy a9e49c524c
Merge pull request #10187 from hashicorp/fixup/ent-tproxy-test 2021-05-06 14:47:44 -06:00
Daniel Nephin 7d8d00b154
Merge pull request #10047 from hashicorp/dnephin/config-entry-validate
state: reduce arguments to validateProposedConfigEntryInServiceGraph
2021-05-06 14:11:21 -04:00
Daniel Nephin 45c5ba46f3 state: reduce arguments to validateProposedConfigEntryInServiceGraph 2021-05-06 13:47:40 -04:00
Daniel Nephin 6b513c1ba4 rpc: add tests for canRetry
Also accept an RPCInfo instead of interface{}. Accepting an interface
lead to a bug where the caller was expecting the arg to be the response
when in fact it was always passed the request. By accepting RPCInfo
it should indicate that this is actually the request value.

One caller of canRetry already passed an RPCInfo, the second handles
the type assertion before calling canRetry.
2021-05-06 13:30:07 -04:00
Daniel Nephin 5a6f15713c rpc: remove unnecessary arg to ForwardRPC 2021-05-06 13:30:07 -04:00
Daniel Nephin 0575ca67ca http: set consistency header properly
A recent change in 1.9.x inverted the order of these two lines, which caused the
X-Consul-Effective-Consistency header to be missing for the servie health endpoints
2021-05-06 13:19:45 -04:00
Daniel Nephin 0b017e2497 agent/local: do not persist the agent tokens
Only default to the user token and agent token for the sync. Change the
exported methods to only return the stored tokens associated with a
specific check or service.
2021-05-06 13:18:58 -04:00
hc-github-team-consul-core 702e3d23bc auto-updated agent/uiserver/bindata_assetfs.go from commit 9801551f7 2021-05-05 23:34:22 +00:00
Daniel Nephin c88a77c9a3 Fix config entry test for enterprise meta 2021-05-05 12:31:23 -06:00
freddygv c73703c08b Ensure entmeta is encoded in test correlationID 2021-05-05 12:31:23 -06:00
Mark Anderson 751fe7e314 Fixup more structs_test
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-05-05 10:13:56 -07:00
Mark Anderson f808a25a73 Fixup checksums for tests
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-05-05 10:13:41 -07:00
Mark Anderson 4d0346bc46 Fixup filtering tests
Missed annotations for SocketPath, LocalSocketPath, LocalServiceSocketPath

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-05-05 09:28:17 -07:00
Mark Anderson 4aa0a0b2a3 Fixup agent/config tests
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-05-05 09:28:17 -07:00
Daniel Nephin 347f3d2128
Merge pull request #10155 from hashicorp/dnephin/config-entry-remove-fields
config-entry: remove Kind and Name field from Mesh config entry
2021-05-04 17:27:56 -04:00
Daniel Nephin c8c85523e1 config-entries: add a test for the API client
Also fixes a bug with listing kind=mesh config entries. ValidateConfigEntryKind was only being used by
the List endpoint, and was yet another place where we have to enumerate all the kinds.

This commit removes ValidateConfigEntryKind and uses MakeConfigEntry instead. This change removes
the need to maintain two separate functions at the cost of creating an instance of the config entry which will be thrown away immediately.
2021-05-04 17:14:21 -04:00
Mark Anderson ff7fca756b Add simple test for downstream sockets
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-05-04 12:41:43 -07:00
Mark Anderson 8040f91a43 Add support for downstreams
Enhance config by adding SocketPath and LocalSocketPath config values

Supports syntax of the form:
```
services {
  name = "sock_forwarder"
  id = "sock_forwarder.1"
  socket_path = "/tmp/downstream_3.sock"
  connect {
    sidecar_service {
      proxy {
	local_service_socket_path = "/tmp/downstream.sock"
      }
    }
  }
}
```

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-05-04 12:41:43 -07:00
Mark Anderson 6be9cebad0 Add tests for xds/listeners
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-05-04 12:41:43 -07:00
Mark Anderson 583ae65d5b Convert mode to string representation
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-05-04 12:41:43 -07:00
Mark Anderson 06f0f79218 Continue working through proxy and agent
Rework/listeners, rename makeListener

Refactor, tests pass

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-05-04 12:41:43 -07:00
Mark Anderson 8b1217417a First changes for unix domain sockets upstreams
Start making structure changes to support unix domain socket address for upstreams

	upstreams = [
	  {
	    destination_name = "echo-service"
	    local_bind_socket_path = "/tmp/upstream.sock"
	    config {
	      passive_health_check {
		interval = "10s"
		max_failures = 42
	      }
	    }
	  }

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-05-04 12:41:43 -07:00
Daniel Nephin 6713afdff3 lint: fix warning by removing reference to deprecated interface 2021-05-04 14:09:14 -04:00
hc-github-team-consul-core 8f9d5c8eef auto-updated agent/uiserver/bindata_assetfs.go from commit 07e11f32e 2021-05-04 16:56:40 +00:00
hc-github-team-consul-core fade7e6583 auto-updated agent/uiserver/bindata_assetfs.go from commit fc7fe23b4 2021-05-04 16:36:21 +00:00
Freddy ed1082510d
Fixup discovery chain handling in transparent mode (#10168)
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

Previously we would associate the address of a discovery chain target
with the discovery chain's filter chain. This was broken for a few reasons:

- If the upstream is a virtual service, the client proxy has no way of
dialing it because virtual services are not targets of their discovery
chains. The targets are distinct services. This is addressed by watching
the endpoints of all upstream services, not just their discovery chain
targets.

- If multiple discovery chains resolve to the same target, that would
lead to multiple filter chains attempting to match on the target's
virtual IP. This is addressed by only matching on the upstream's virtual
IP.

NOTE: this implementation requires an intention to the redirecting
virtual service and not just to the final destination. This is how
we can know that the virtual service is an upstream to watch.

A later PR will look into traversing discovery chains when computing
upstreams so that intentions are only required to the discovery chain
targets.
2021-05-04 08:45:19 -06:00
Paul Banks 3ad754ca7b
Make Raft trailing logs and snapshot timing reloadable (#10129)
* WIP reloadable raft config

* Pre-define new raft gauges

* Update go-metrics to change gauge reset behaviour

* Update raft to pull in new metric and reloadable config

* Add snapshot persistance timing and installSnapshot to our 'protected' list as they can be infrequent but are important

* Update telemetry docs

* Update config and telemetry docs

* Add note to oldestLogAge on when it is visible

* Add changelog entry

* Update website/content/docs/agent/options.mdx

Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com>

Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com>
2021-05-04 15:36:53 +01:00
Luke Kysow 8d6cbe7281
Give descriptive error if auth method not found (#10163)
* Give descriptive error if auth method not found

Previously during a `consul login -method=blah`, if the auth method was not found, the
error returned would be "ACL not found". This is potentially confusing
because there may be many different ACLs involved in a login: the ACL of
the Consul client, perhaps the binding rule or the auth method.

Now the error will be "auth method blah not found", which is much easier
to debug.
2021-05-03 13:39:13 -07:00
Freddy 2ca3f481f8
Only consider virtual IPs for transparent proxies (#10162)
Initially we were loading every potential upstream address into Envoy
and then routing traffic to the logical upstream service. The downside
of this behavior is that traffic meant to go to a specific instance
would be load balanced across ALL instances.

Traffic to specific instance IPs should be forwarded to the original
destination and if it's a destination in the mesh then we should ensure
the appropriate certificates are used.

This PR makes transparent proxying a Kubernetes-only feature for now
since support for other environments requires generating virtual IPs,
and Consul does not do that at the moment.
2021-05-03 14:15:22 -06:00
hc-github-team-consul-core a37c1e8466 auto-updated agent/uiserver/bindata_assetfs.go from commit 211bc824a 2021-05-03 15:24:01 +00:00
Daniel Nephin 00d3283e93
Merge pull request #10149 from hashicorp/dnephin/config-use-streaming-backend-defualt-true
config: default UseStreamingBackend to true
2021-04-30 16:29:11 -04:00
tarat44 51f8db3879 create separate function with its own context to shutdown http2 client conn in h2ping check 2021-04-29 18:05:50 -04:00
Daniel Nephin a07a58a873 config-entry: use custom MarshalJSON for mesh type
So that the Kind field is added to the JSON object.
2021-04-29 17:49:43 -04:00
Daniel Nephin 62efaaab21 config-entry: remove Kind and Name field from Mesh config entry
No config entry needs a Kind field. It is only used to determine the Go type to
target. As we introduce new config entries (like this one) we can remove the kind field
and have the GetKind method return the single supported value.

In this case (similar to proxy-defaults) the Name field is also unnecessary. We always
use the same value. So we can omit the name field entirely.
2021-04-29 17:11:21 -04:00
R.B. Boyer abc1dc0fe9
connect: update supported envoy versions to 1.18.2, 1.17.2, 1.16.3, and 1.15.4 (#10101)
The only thing that needed fixing up pertained to this section of the 1.18.x release notes:

> grpc_stats: the default value for stats_for_all_methods is switched from true to false, in order to avoid possible memory exhaustion due to an untrusted downstream sending a large number of unique method names. The previous default value was deprecated in version 1.14.0. This only changes the behavior when the value is not set. The previous behavior can be used by setting the value to true. This behavior change by be overridden by setting runtime feature envoy.deprecated_features.grpc_stats_filter_enable_stats_for_all_methods_by_default.

For now to maintain status-quo I'm explicitly setting `stats_for_all_methods=true` in all versions to avoid relying upon the default.

Additionally the naming of the emitted metrics for these gRPC requests changed slightly so the integration test assertions for `case-grpc` needed adjusting.
2021-04-29 15:22:03 -05:00
R.B. Boyer 85a718da63
xds: ensure that all envoyproxy/go-control-plane protobuf symbols are linked into the final binary (#10131)
This ensures that if someone does include some extension Consul does not currently make use of, that extension is actually usable. Without linking these envoy protobufs into the main binary it can't round trip the escape hatches to send them down to envoy.

Whenenver the go-control-plane library is upgraded next we just have to re-run 'make envoy-library'.
2021-04-29 14:58:26 -05:00
R.B. Boyer 71d45a3460
Support Incremental xDS mode (#9855)
This adds support for the Incremental xDS protocol when using xDS v3. This is best reviewed commit-by-commit and will not be squashed when merged.

Union of all commit messages follows to give an overarching summary:

xds: exclusively support incremental xDS when using xDS v3

Attempts to use SoTW via v3 will fail, much like attempts to use incremental via v2 will fail.
Work around a strange older envoy behavior involving empty CDS responses over incremental xDS.
xds: various cleanups and refactors that don't strictly concern the addition of incremental xDS support

Dissolve the connectionInfo struct in favor of per-connection ResourceGenerators instead.
Do a better job of ensuring the xds code uses a well configured logger that accurately describes the connected client.
xds: pull out checkStreamACLs method in advance of a later commit

xds: rewrite SoTW xDS protocol tests to use protobufs rather than hand-rolled json strings

In the test we very lightly reuse some of the more boring protobuf construction helper code that is also technically under test. The important thing of the protocol tests is testing the protocol. The actual inputs and outputs are largely already handled by the xds golden output tests now so these protocol tests don't have to do double-duty.

This also updates the SoTW protocol test to exclusively use xDS v2 which is the only variant of SoTW that will be supported in Consul 1.10.

xds: default xds.Server.AuthCheckFrequency at use-time instead of construction-time
2021-04-29 13:54:05 -05:00
hc-github-team-consul-core cbabf82587 auto-updated agent/uiserver/bindata_assetfs.go from commit 5d89ba031 2021-04-29 09:46:32 +00:00
Freddy 701b49f772
Rename cluster config files to mesh as well (#10148) 2021-04-28 18:15:26 -06:00
Daniel Nephin bb472111fd config: default UseStreamingBackend to true 2021-04-28 18:58:02 -04:00
Freddy 078c40425f
Rename "cluster" config entry to "mesh" (#10127)
This config entry is being renamed primarily because in k8s the name
cluster could be confusing given that the config entry applies across
federated datacenters.

Additionally, this config entry will only apply to Consul as a service
mesh, so the more generic "cluster" name is not needed.
2021-04-28 16:13:29 -06:00
Daniel Nephin 9b344b3300
Merge pull request #10112 from hashicorp/dnephin/remove-streaming-from-cache
streaming: replace agent/cache with submatview.Store
2021-04-28 17:31:42 -04:00
Daniel Nephin 3a27fce2ad submatview: fix godoc and comment typos
Co-authored-by: Paul Banks <banks@banksco.de>
2021-04-28 12:27:38 -04:00
hc-github-team-consul-core 4ad37c326e auto-updated agent/uiserver/bindata_assetfs.go from commit 345765f76 2021-04-28 11:21:09 +00:00
Daniel Nephin 11398892b3 submatview: avoid sorting results unnecessarily
Previous getFromView would call view.Result when the result may not have been returned
(because the index is updated past the minIndex. This would allocate a slice and sort it
for no reason, because the values would never be returned.

Fix this by re-ordering the operations in getFromView.

The test changes in this commit were an attempt to cover the case where
an update is received but the index does not exceed the minIndex.
2021-04-27 19:03:17 -04:00
Daniel Nephin 23e1cd6f93 submatview: only return materializer from getEntry
Also rename it to readEntry now that it doesn't return the entire entry. Based on feedback
in PR review, the full entry is not used by the caller, and accessing the fields wouldn't be
safe outside the lock, so it is safer to return only the Materializer
2021-04-27 19:03:17 -04:00
Daniel Nephin 46b7d0b573 submatview: godoc 2021-04-27 19:03:17 -04:00
Daniel Nephin d9537411b8 rpcclient/health: convert tests to the new submatview.Store interface
Also fixes a minor data race in Materializer. Capture the error before
releasing the lock.
2021-04-27 19:03:17 -04:00
Daniel Nephin 79a54f208e agent: fix data race in tests caused by grpc log init
grpclog.SetLoggerV2 is meant to be called only once before any gRPC requests are received, but
each test that uses TestAgent will call NewBaseDeps again. Use a sync.Once to prevent the grpc
logging from being re-initialized by each test.

This will mean that a test can't use a fake logger to capture logs from the gRPC server.
2021-04-27 19:03:17 -04:00
Daniel Nephin dacf50038d submatview: fix two flaky tests
These tests can flake when we get a notification for an earlier event.
Retry the read from update channel a few times to make sure we get the
event we expect.
2021-04-27 19:03:17 -04:00
Daniel Nephin c932833acb rpcclient:health: fix a data race and flake in tests
Split the TestStreamingClient into the two logical components the real
client uses. This allows us to test multiple clients properly.

Previously writing of ctx from multiple Subscribe calls was showing a
data race.

Once this was fixed a test started to fail because the request had to be
made with a greater index, so that the store.Get call did not return
immediately.
2021-04-27 19:03:17 -04:00
Daniel Nephin 5fa0dea63a rpcclient/health: fix data race in a test
The idleTTL was being written and read concurrently. Instead move the idleTTL to a struct
field so that when one test patches the TTL it does not impact others.

The background goroutines for the store can outlive a test because context cancellation
is async.
2021-04-27 19:03:17 -04:00
Daniel Nephin 10ec9c2be3 rpcclient: close the grpc.ClientConn on shutdown 2021-04-27 19:03:16 -04:00
Daniel Nephin e229b877d8 health: create health.Client in Agent.New 2021-04-27 19:03:16 -04:00
Daniel Nephin a16c377ca7 rpcclient/health: move all backend routing logic to client 2021-04-27 19:03:16 -04:00
Daniel Nephin 0558586dbd health: use blocking queries for near query parameter 2021-04-27 19:03:16 -04:00
Daniel Nephin 2a26085b2c connect: do not set QuerySource.Node
Setting this field to a value is equivalent to using the 'near' query paramter.
The intent is to sort the results by proximity to the node requesting
them. However with connect we send the results to envoy, which doesn't
care about the order, so setting this field is increasing the work
performed for no gain.

It is necessary to unset this field now because we would like connect
to use streaming, but streaming does not support sorting by proximity.
2021-04-27 19:03:16 -04:00
Daniel Nephin 4fb2ba9de7 submatview: move error return to NewMaterializer
So that we don't have to create views ahead of time, when we will never use that view.
2021-04-27 19:03:16 -04:00
Daniel Nephin 55a677b7d1 rpcclient/health: integrate submatview.Store into rpcclient/health 2021-04-27 19:03:16 -04:00
Daniel Nephin 6f29fa0de8 rpcclient: move streaming cache tests 2021-04-27 19:03:16 -04:00
Daniel Nephin 6ebb0f8b74 submatview: test Store.Run 2021-04-27 19:03:16 -04:00
Daniel Nephin 9854c424eb submatview: test store with Get and Notify calls together 2021-04-27 19:03:16 -04:00
Daniel Nephin c1063999fc submatview: more test cases for Store.Get
And simplify the return value by using a Result type.
2021-04-27 19:03:16 -04:00
Daniel Nephin 832f1a2847 submatview: reduce the getFromView implementation
Remove View.Result error return value, it was always nil, and seems like it will likely always remain nill
since it is simply reading a stored value.

Also replace some cache types with local types.
2021-04-27 19:03:16 -04:00
Daniel Nephin 1d9d7d0aa5 submatview: track requests instead of notifiers
And only start expiration time when the last request ends. This makes tracking expiry simpler, and
ensures that no entry can be expired while there are active requests.
2021-04-27 19:03:16 -04:00
Daniel Nephin 7d13465c7b submatview: set up expiry of materializers 2021-04-27 19:03:16 -04:00
Daniel Nephin c574108354 submatview: setup testing structure 2021-04-27 19:03:16 -04:00
Daniel Nephin ddddbdb990 submatview: rough outline of the Get and Notify methods. 2021-04-27 19:03:16 -04:00
Daniel Nephin 3d52abb5e2 submatview: setup the interface for the store 2021-04-27 19:03:15 -04:00
Matt Keeler 65d73771a5
Add prometheus guage definitions for replication metrics. (#10109) 2021-04-23 17:05:33 -04:00
Matt Keeler ecbccdc261
Add replication metrics (#10073) 2021-04-22 11:20:53 -04:00
hashicorp-ci 8f3223a984 auto-updated agent/uiserver/bindata_assetfs.go from commit 7a095e7d2 2021-04-22 11:28:27 +00:00
hashicorp-ci 78886e8f4f auto-updated agent/uiserver/bindata_assetfs.go from commit f6f282ac7 2021-04-22 11:21:21 +00:00
Daniel Nephin 2e4dc7b705
Merge pull request #10045 from hashicorp/dnephin/state-proxy-defaults
state: remove config-entries kind index
2021-04-20 17:12:50 -04:00
Daniel Nephin 87cd3fc2e8 Handle ErrChunkingResubmit.Error properly
Previously canRetry was attempting to retrieve this error from args, however there was never
any callers that would pass an error to args.

With the change to raftApply to move this error to the error return value, it is now possible
to receive this error from the err argument.

This commit updates canRetry to check for ErrChunkingResubmit in err.
2021-04-20 13:29:31 -04:00
Daniel Nephin d2ab767fef Handle FSM.Apply errors in raftApply
Previously we were inconsistently checking the response for errors. This
PR moves the response-is-error check into raftApply, so that all callers
can look at only the error response, instead of having to know that
errors could come from two places.

This should expose a few more errors that were previously hidden because
in some calls to raftApply we were ignoring the response return value.

Also handle errors more consistently. In some cases we would log the
error before returning it. This can be very confusing because it can
result in the same error being logged multiple times. Instead return
a wrapped error.
2021-04-20 13:29:29 -04:00
Freddy f265dd635f
Disallow * as service-defaults name (#10069) 2021-04-19 14:23:01 -06:00
R.B. Boyer 70f1ba3a2b
fix test when run in enterprise (#10048) 2021-04-15 17:25:25 -05:00
freddygv a0f3591aee Don't panic on nil UpstreamConfiguration.Clone() 2021-04-15 15:46:21 -06:00
Daniel Nephin b57b3726d2 state: remove unnecessary kind index
The query can be performed using a prefix query on the ID index.

Also backport some enterprise changes to prevent conflicts.
2021-04-15 17:37:28 -04:00
Daniel Nephin 653f41ce41 state: use index constants for ConfigEntry indexes 2021-04-15 17:30:07 -04:00
hashicorp-ci 131599bfab auto-updated agent/uiserver/bindata_assetfs.go from commit daf897f1a 2021-04-15 20:19:13 +00:00
Freddy 3be304be16
Merge pull request #10016 from hashicorp/topology-update 2021-04-15 14:11:23 -06:00
Freddy 439a7fce2d
Split Upstream.Identifier() so non-empty namespace is always prepended in ent (#10031) 2021-04-15 13:54:40 -06:00
R.B. Boyer 4db8b78854
connect: update centralized upstreams representation in service-defaults (#10015) 2021-04-15 14:21:44 -05:00
Kent 'picat' Gruber 9f7190a4fe
Merge pull request #10023 from hashicorp/fix-raw-kv-xss
Add content type headers to raw KV responses
2021-04-14 18:49:14 -04:00
Daniel Nephin 88f83b41aa snapshot: fix saving of auth methods
Previously only a single auth method would be saved to the snapshot. This commit fixes the typo
and adds to the test, to show that all auth methods are now saved.
2021-04-14 16:51:21 -04:00
Kent 'picat' Gruber cf1fce3d0c Add content type headers to raw KV responses 2021-04-14 16:20:22 -04:00
freddygv f7094f91c7 Avoid returning a nil slice 2021-04-14 10:52:05 -06:00
Matt Keeler bbf5993534
Move static token resolution into the ACLResolver (#10013) 2021-04-14 12:39:35 -04:00
R.B. Boyer 06848ce67e fix broken golden tests 2021-04-14 11:36:47 -05:00
Freddy 3ec4556114
Merge pull request #10006 from hashicorp/api-ptrs 2021-04-14 10:21:08 -06:00
freddygv 8e74eaa684 Update viz endpoint to include topology from intentions 2021-04-14 10:20:15 -06:00
Freddy 55a3697b83
Merge pull request #9987 from hashicorp/remove-kube-dns-hack 2021-04-14 10:00:53 -06:00
freddygv e1808af729 Fixup tests 2021-04-13 16:08:41 -06:00
freddygv 7cb3f32672 Convert new tproxy structs in api module into ptrs
This way we avoid serializing these when empty. Otherwise users of the
latest version of the api submodule cannot interact with older versions
of Consul, because a new api client would send keys that the older Consul
doesn't recognize yet.
2021-04-13 12:44:25 -06:00
Freddy 1f119aec7c
Merge pull request #10005 from hashicorp/tproxy-fixes 2021-04-13 11:45:40 -06:00
Freddy 8fc60a6ca6
Merge pull request #10000 from hashicorp/remove-upstream-cfg-validation
Remove zero-value validation of upstream cfg structs
2021-04-13 11:00:02 -06:00
freddygv 4e509aa768 Remove todo that was todone 2021-04-13 10:19:59 -06:00
freddygv 75edc9bc7c Avoid nil panic when cluster config doesn't exist 2021-04-13 10:17:11 -06:00
freddygv 932fbddd27 Augment intention decision summary with DefaultAllow mode 2021-04-12 19:32:09 -06:00
freddygv 8857195437 Fixup wildcard ent assertion 2021-04-12 17:04:33 -06:00
Freddy 18decbba9d
Merge pull request #9999 from hashicorp/update-enabling-tproxy 2021-04-12 16:37:04 -06:00
freddygv b8ed82b808 Fixup bexpr filtering 2021-04-12 10:17:52 -06:00
freddygv d7c43049fa Remove zero-value validation of upstream cfg structs
The zero value of these flags was already being excluded in the xDS
generation of circuit breaker/outlier detection config.

See: makeThresholdsIfNeeded and ToOutlierDetection.
2021-04-12 10:08:57 -06:00
freddygv 7bd51ff536 Replace TransparentProxy bool with ProxyMode
This PR replaces the original boolean used to configure transparent
proxy mode. It was replaced with a string mode that can be set to:

- "": Empty string is the default for when the setting should be
defaulted from other configuration like config entries.
- "direct": Direct mode is how applications originally opted into the
mesh. Proxy listeners need to be dialed directly.
- "transparent": Transparent mode enables configuring Envoy as a
transparent proxy. Traffic must be captured and redirected to the
inbound and outbound listeners.

This PR also adds a struct for transparent proxy specific configuration.
Initially this is not stored as a pointer. Will revisit that decision
before GA.
2021-04-12 09:35:14 -06:00
freddygv 9e194b4b3c Avoid failing test due to undiscoverable node name 2021-04-12 09:26:55 -06:00
hashicorp-ci 2995d0e437 auto-updated agent/uiserver/bindata_assetfs.go from commit 84064f972 2021-04-12 13:08:41 +00:00
freddygv 98ba582797 Fixup mesh gateway docs 2021-04-11 15:48:04 -06:00
tarat44 1ca5fa9769 fix formatting 2021-04-11 15:12:33 -04:00
tarat44 a2e6ca1226 add WaitGroup to h2ping 2021-04-11 15:11:00 -04:00
tarat44 5307c5c3a1 close h2ping client connections 2021-04-10 00:53:53 -04:00
Tara Tufano 9deb52e868
add http2 ping health checks (#8431)
* add http2 ping checks

* fix test issue

* add h2ping check to config resources

* add new test and docs for h2ping

* fix grammatical inconsistency in H2PING documentation

* resolve rebase conflicts, add test for h2ping tls verification failure

* api documentation for h2ping

* update test config data with H2PING

* add H2PING to protocol buffers and update changelog

* fix typo in changelog entry
2021-04-09 15:12:10 -04:00
Iryna Shustava 5755c97bc7
cli: Add new `consul connect redirect-traffic` command for applying traffic redirection rules when Transparent Proxy is enabled. (#9910)
* Add new consul connect redirect-traffic command for applying traffic redirection rules when Transparent Proxy is enabled.
* Add new iptables package for applying traffic redirection rules with iptables.
2021-04-09 11:48:10 -07:00
Freddy a02245b75a
Merge pull request #9976 from hashicorp/centralized-upstream-fixups 2021-04-08 12:26:56 -06:00
Freddy e385e5992f
Merge pull request #9042 from lawliet89/tg-rewrite 2021-04-08 11:49:23 -06:00
freddygv c6d64a8078 Stable sort cidr ranges to match on 2021-04-08 11:27:57 -06:00
freddygv b21224a4c8 PR comments 2021-04-08 11:16:03 -06:00
Daniel Nephin 34f1facebb
Merge pull request #9950 from hashicorp/dnephin/state-use-txn-everywhere
state: use Txn interface everywhere
2021-04-08 12:02:03 -04:00
Daniel Nephin c40e1a2ac6
Merge pull request #9880 from hashicorp/dnephin/catalog-events-test-pattern
state: use runCase pattern for large test
2021-04-08 11:54:41 -04:00
Paul Banks 1406671290
cache: Fix bug where connection errors can cause early cache expiry (#9979)
Fixes a cache bug where TTL is not updated while a value isn't changing or cache entry is returning fetch errors.
2021-04-08 11:11:15 +01:00
Paul Banks ee04d452be
cache: fix bug where TTLs were ignored leading to leaked memory in client agents (#9978)
* Fix bug in cache where TTLs are effectively ignored

This mostly affects streaming since streaming will immediately return from Fetch calls when the state is Closed on eviction which causes the race condition every time.

However this also affects all other cache types if the fetch call happens to return between the eviction and then next time around the Get loop by any client.

There is a separate bug that allows cache items to be evicted even when there are active clients which is the trigger here.

* Add changelog entry

* Update .changelog/9978.txt
2021-04-08 11:08:56 +01:00
Paul Banks 8e00e327b0
Merge pull request #9977 from hashicorp/grpc-tuning
streaming: Grpc tuning
2021-04-08 11:05:38 +01:00
freddygv ab752c1c86 Avoid sending zero-value upstream defaults from api 2021-04-07 15:03:42 -06:00
freddygv a6388c7e2f Revert "Avoid accumulating synthetic upstreams"
This reverts commit 86672df4fa.
2021-04-07 14:30:30 -06:00
freddygv 02f6768cd2 Remove kube-dns resolution since clusterip will be a tagged addr 2021-04-07 14:15:21 -06:00
hashicorp-ci dedf2861be auto-updated agent/uiserver/bindata_assetfs.go from commit a0d12ff16 2021-04-07 16:48:59 +00:00
freddygv 86672df4fa Avoid accumulating synthetic upstreams
Synthetic upstreams from service-defaults config are stored locally in
the Upstreams list. Since these come from service-defaults they should
be cleaned up locally when no longer present in the service config
response.
2021-04-07 09:32:48 -06:00
freddygv 49a4a78fd5 Ensure mesh gateway mode override is set for upstreams for intentions 2021-04-07 09:32:48 -06:00
freddygv 5140c3e51f Finish resolving upstream defaults in proxycfg 2021-04-07 09:32:48 -06:00
freddygv 986bcccbea Pass down upstream defaults to client proxies
This is needed in case the client proxy is in TransparentProxy mode.
Typically they won't have explicit configuration for every upstream, so
this ensures the settings can be applied to all of them when generating
xDS config.
2021-04-07 09:32:47 -06:00
freddygv 77ead5cca9 Prevent wildcard destinations for proxies and upstreams 2021-04-07 09:32:47 -06:00
freddygv 24ee8a0488 Prevent requests without UpstreamIDs from being flagged as legacy.
New clients in transparent proxy mode can send requests for service
config resolution without any upstream args because they do not have
explicitly defined upstreams.

Old clients on the other hand will never send requests without the
Upstreams args unless they don't have upstreams, in which case we do not
send back upstream config.
2021-04-07 09:32:47 -06:00
freddygv 2b49cc39ed Fixup doc phrasing 2021-04-07 09:32:47 -06:00
freddygv 458eb41be1 Prevent synthetic upstreams without addresses from failing duplicate ip/port validation 2021-04-07 09:32:47 -06:00
Paul Banks 5529cb7347 Tune streaming backoff on errors to retry a bit faster when TCP connections drop 2021-04-07 14:13:30 +01:00
Paul Banks 44718456b5 Set gRPC keepalives to mirror Yamux keepalive behaviour 2021-04-07 14:09:22 +01:00
R.B. Boyer d4c401b350
missed build tag on this file (#9974) 2021-04-06 13:24:11 -05:00
R.B. Boyer 499fee73b3
connect: add toggle to globally disable wildcard outbound network access when transparent proxy is enabled (#9973)
This adds a new config entry kind "cluster" with a single special name "cluster" where this can be controlled.
2021-04-06 13:19:59 -05:00
Daniel Nephin f0ba6f858a
Merge pull request #9958 from hashicorp/dnephin/state-improve-indexer-tests
state: support additional test cases in indexer tests
2021-04-06 11:55:24 -04:00
Yong Wen Chua 409768d6e5
Merge branch 'master' of github.com:hashicorp/consul into tg-rewrite 2021-04-06 17:05:26 +08:00
R.B. Boyer e494313e7b
api: ensure v1/health/ingress/:service endpoint works properly when streaming is enabled (#9967)
The streaming cache type for service health has no way to handle v1/health/ingress/:service queries as there is no equivalent topic that would return the appropriate data.

Ensure that attempts to use this endpoint will use the old cache-type for now so that they return appropriate data when streaming is enabled.
2021-04-05 13:23:00 -05:00
Daniel Nephin 6e69829edb state: support additional test cases in indexer tests
And add a few additional cases.
2021-03-31 14:39:33 -04:00
Kyle Havlovitz a2869b280b Backport enterprise changes to prevent merge conflicts
Co-Authored-By: Kyle Havlovitz <kylehav@gmail.com>
2021-03-31 14:05:26 -04:00
Daniel Nephin 26440d9e1f
Merge pull request #9949 from hashicorp/dnephin/state-index-checks
state: convert remaining checks table indexers to functional pattern
2021-03-31 11:53:21 -04:00
Daniel Nephin 909348e546
Merge pull request #9948 from hashicorp/dnephin/state-index-service
state: convert remaining services table indexers to functions
2021-03-31 11:49:21 -04:00
Daniel Nephin 1e32dbca29
Merge pull request #9947 from hashicorp/dnephin/state-ent-index-3
state: move indexer functions out of oss files
2021-03-31 11:45:26 -04:00
hashicorp-ci 352061c72e auto-updated agent/uiserver/bindata_assetfs.go from commit ee5c3e3aa 2021-03-31 15:00:21 +00:00
Daniel Nephin 1f64b3a7de state: use tableIndex constant 2021-03-29 18:52:20 -04:00
Daniel Nephin 9514698b10 state: use ReadTxn and WriteTxn interface
Instead of *txn, so that we can replace the txn implementation with others, and so
that the function is easily documented as a read or write function.
2021-03-29 18:52:16 -04:00
Daniel Nephin 80827e8615
Merge pull request #9932 from hashicorp/dnephin/use-ent-meta-default
Set default enterprise meta in test case
2021-03-29 18:44:39 -04:00
Daniel Nephin d0e5cd66f0 state: convert checks.service index to new pattern 2021-03-29 16:38:53 -04:00
Daniel Nephin 3092c627fe state: convert checks.status indexer
As part of this change the indexer will now be case insensitive by using
the lower case value. This should be safe because previously we always
had lower case strings.

This change was made out of convenience. All the other indexers use
lowercase, so we can re-use the indexFromQuery function by using
lowercase here as well.
2021-03-29 16:38:50 -04:00
Daniel Nephin 628eed3748 state: add tests for checks indexers 2021-03-29 16:38:47 -04:00
Daniel Nephin ce631d0bba state: use constants for table checks 2021-03-29 16:38:43 -04:00
Daniel Nephin cf2646e0d1 state: pass Query in from caller
To reduce the number of arguments
2021-03-29 15:42:30 -04:00
Daniel Nephin 9eea19da59 state: convert services.kind to functional indexer pattern 2021-03-29 15:42:30 -04:00
Daniel Nephin b6553af222 state: add tests for services.kind indexer 2021-03-29 15:42:27 -04:00
Daniel Nephin 0c61abcc31 state: convert services table service and connect indexer
To the new functional indexer pattern
2021-03-29 15:42:24 -04:00
Daniel Nephin 395ebce510 state: add tests for services table service and connect indexers 2021-03-29 15:42:22 -04:00
Daniel Nephin 2d2c3e1190 state: use constant for tableServices 2021-03-29 15:42:18 -04:00
Daniel Nephin 341265ec69 state: remove duplication of Query indexer 2021-03-29 14:35:11 -04:00
Daniel Nephin 08ee12ab34 state: remove duplication in acl tables schema 2021-03-29 14:21:27 -04:00
Daniel Nephin 72960388a3 state: reduce duplication in catalog table schema 2021-03-29 14:21:23 -04:00
Daniel Nephin 7de186f291 state: share more indexer functions for config_entries 2021-03-29 14:21:20 -04:00
Daniel Nephin 024dcbef03 state: remove old schema test
This test has been replaced by TestNewDBSchema_Indexers
2021-03-29 14:21:13 -04:00
Daniel Nephin 8591feb58a state: use addNamespaceIndex again 2021-03-29 14:21:02 -04:00
hashicorp-ci 02b7eb0949 auto-updated agent/uiserver/bindata_assetfs.go from commit 4d13e31ae 2021-03-26 15:51:59 +00:00
Daniel Nephin d62ed94415 Set default enterprise meta in test case 2021-03-25 17:40:22 -04:00
Lars Lehtonen 4bad87c293 agent: use testAgent_RegisterService_TranslateKeys()
When this test was refactored it looks like there was a copy+paste error
and the test case was made to call the wrong function.
2021-03-25 16:55:13 -04:00
Lars Lehtonen 681bd8f57b agent: remove unused makeTelemetryDefaults() 2021-03-25 16:02:42 -04:00
Daniel Nephin 8d25f9ab3d
Merge pull request #9923 from hashicorp/dnephin/fix-ui-config
http: fix a bug that would cause runtimeConfig to be cached
2021-03-25 12:26:09 -04:00
Daniel Nephin ac210cdc48
Merge pull request #9911 from hashicorp/dnephin/state-index-acl-roles
state: convert ACLRoles policies index to new functional indexer pattern
2021-03-24 18:28:19 -04:00
Daniel Nephin 2e917e3f9c
Merge pull request #9916 from hashicorp/dnephin/state-index-checks-id
state: convert checks.ID index to the functional indexer pattern
2021-03-24 18:23:52 -04:00
Daniel Nephin d50037cae2 http: add a test for transform changing value 2021-03-24 15:38:11 -04:00
Daniel Nephin c98805f505 http: fix a bug that would cause runtimeConfig to be cached
This bug would result in the UI not having the correct settings in
Consul enterprise, which could produce many warnings in the logs.

This bug occured because the index page, which includes a map of configuration
was rendered when the HTTPHandler is first created. This PR changes the
UIServer to instead render the index page when the page is requested.

The rendering does not appear to be all that expensive, so rendering it
when requested should not cause much extra latency.
2021-03-24 14:48:18 -04:00
hashicorp-ci 0414a872ec auto-updated agent/uiserver/bindata_assetfs.go from commit 232921b60 2021-03-23 13:26:56 +00:00
Daniel Nephin 8743e925d5 state: add tests for checks.ID indexer 2021-03-22 18:06:43 -04:00
Daniel Nephin 30281a5332 state: use tx.First instead of tx.FirstWatch
Where appropriate. After removing the helper function a bunch of  these calls can
be changed to tx.First.
2021-03-22 18:06:33 -04:00
Daniel Nephin 1cdcfb8260 state: convert checks.ID index to new pattern 2021-03-22 18:06:08 -04:00
Hans Hasselberg 53e9c134af
introduce certopts (#9606)
* introduce cert opts

* it should be using the same signer

* lint and omit serial
2021-03-22 10:16:41 +01:00
Daniel Nephin 6324f37241 state: use uuid for acl-roles.policies index
Previously we were encoding the UUID as a string, but the index it references uses a UUID
so this index can also use an encoded UUID to save a bit of memory.
2021-03-19 19:45:37 -04:00
Daniel Nephin 43df402e51 state: convert acl-roles.policies index to new pattern 2021-03-19 19:45:37 -04:00
Daniel Nephin 00b6f0b41a state: convert acl-roles.name index to the functional indexer pattern 2021-03-19 19:45:37 -04:00
Daniel Nephin d7f5094702 state: add indexer tests for acl-roles table 2021-03-19 19:45:37 -04:00
Daniel Nephin a058c31ead state: use constants for acl-roles table and indexes 2021-03-19 19:45:37 -04:00
Daniel Nephin eb6769ccc6 state: convert acl-policies table to new pattern 2021-03-19 15:24:00 -04:00
Daniel Nephin 340462dd72 state: use constants and add tests for acl-policies table 2021-03-19 15:19:57 -04:00
Daniel Nephin 0c14f3818d state: add indexer test for services.ID index 2021-03-19 14:13:14 -04:00
Daniel Nephin dbd3cef1ed state: handle wildcard for services.ID index
When listing services, use the id_prefix directly if wildcards are allowed.

Error if a wildcard is used for a query that does not index the wildcard
2021-03-19 14:12:19 -04:00
Daniel Nephin 627c469f08 state: fix prefix index with the new pattern
Prefix queries are generally being used to match part of a partial
index. We can support these indexes by using a function that accept
different types for each subset of the index.

What I found interesting is that in the generic StringFieldIndexer the
implementation for PrefixFromArgs would remove the trailing null, but
at least in these 2 cases we actually want a null terminated string.
We simply want fewer components in the string.
2021-03-19 14:12:17 -04:00
Daniel Nephin d90845f26d state: move services.ID to new pattern 2021-03-19 14:11:59 -04:00
hashicorp-ci f3f3513d83 auto-updated agent/uiserver/bindata_assetfs.go from commit a7a56ca39 2021-03-19 15:19:17 +00:00
Daniel Nephin 4d1d19ed46 state: add tests for gateway-service table indexers 2021-03-18 12:09:42 -04:00
Daniel Nephin 11b4de719c state: use constants and remove wrapping
for GatewayServices table
2021-03-18 12:08:59 -04:00
Daniel Nephin d879fe581d state: Move UpstreamDownstream to state package 2021-03-18 12:08:59 -04:00
Daniel Nephin 65f5b99247 state: add tests for mesh-topology table indexers 2021-03-18 12:08:57 -04:00
Daniel Nephin c749c6c927 state: use constants for mesh-topology table operations 2021-03-18 12:08:03 -04:00
hashicorp-ci 43f6544c21 auto-updated agent/uiserver/bindata_assetfs.go from commit 980299d51 2021-03-18 15:23:26 +00:00
hashicorp-ci 69f6fc9f99 auto-updated agent/uiserver/bindata_assetfs.go from commit 8dc590cf1 2021-03-18 14:41:20 +00:00
Freddy 1c13aa23f1
Merge pull request #9900 from hashicorp/ent-fixes
Fixup enterprise tests from tproxy changes
2021-03-18 08:33:30 -06:00
Freddy 0bab999fe4
Merge pull request #9899 from hashicorp/wildcard-ixn-oss
Add methods to check intention has wildcard src or dst
2021-03-18 08:33:07 -06:00
freddygv 098b9af901 Fixup enterprise tests from tproxy changes 2021-03-17 23:05:00 -06:00
freddygv 9713e3ba38 Add methods to check intention has wildcard src or dst 2021-03-17 22:15:48 -06:00
freddygv eb1e0a1751 Cancel watch on all errors 2021-03-17 21:44:14 -06:00
freddygv 52bf00de8b Split up normalizing from defaulting values for upstream cfg 2021-03-17 21:37:55 -06:00
freddygv ad6c726453 Uncomment listener tests 2021-03-17 21:37:12 -06:00
freddygv f4f45af6d0 Merge master and fix upstream config protocol defaulting 2021-03-17 21:13:40 -06:00
freddygv 9bff39ba07 Temporarily silence spurious wakeup. Addressing false positive in beta. 2021-03-17 17:25:29 -06:00
freddygv 0defd17106 Merge remote-tracking branch 'origin/master' into intention-topology-endpoint 2021-03-17 17:14:38 -06:00
Freddy 8207b832df
Add TransparentProxy option to proxy definitions 2021-03-17 17:01:45 -06:00
Freddy c664938bae
Add per-upstream configuration to service-defaults 2021-03-17 16:59:51 -06:00
freddygv 7938dd82eb Add changelog and cleanup todo for beta 2021-03-17 16:45:13 -06:00
freddygv 9f0696528b Rename hasChains for clarity 2021-03-17 16:42:29 -06:00
freddygv 0da8702f34 PR comments 2021-03-17 16:18:56 -06:00
freddygv bf96d536d9 Upstreams loop is only for prepared queries and they are not CentrallyConfigured 2021-03-17 15:32:52 -06:00
freddygv 8a062e1546 Handle prepared queries in Upstreams loop and escape hatches in disco chain loop 2021-03-17 15:17:43 -06:00
freddygv ce964f8ea5 Update xds for transparent proxy 2021-03-17 13:40:49 -06:00
freddygv a54d6a9010 Update proxycfg for transparent proxy 2021-03-17 13:40:39 -06:00
freddygv 37f684664d Do not include consul as upstream or downstream 2021-03-17 13:40:04 -06:00
Daniel Nephin 69ce10602f state: add tests for config-entry indexers 2021-03-17 14:41:46 -04:00
Daniel Nephin a414649543 state: convert config-entries kind index to new pattern 2021-03-17 14:40:57 -04:00
Daniel Nephin aadf187094 state: remove config-entries namespace index
Use a prefix of the ID index instead.
2021-03-17 14:40:57 -04:00
Daniel Nephin d70bbf671a state: remove unnecessary method receiver 2021-03-17 14:40:57 -04:00
Daniel Nephin 650ac62098 state: convert config-entries table to new indexer pattern
Using functional indexes to isolate enterprise differentiation and
remove reflection.
2021-03-17 14:40:57 -04:00
Daniel Nephin 9f03e23e44
Merge pull request #9881 from hashicorp/dnephin/state-index-service-check-nodes
state: convert services.node and checks.node indexes
2021-03-17 14:12:02 -04:00
Daniel Nephin bd6332ae25
Merge pull request #9863 from hashicorp/dnephin/config-entry-kind-name
state: move ConfigEntryKindName
2021-03-17 14:09:39 -04:00
hashicorp-ci 583743424f auto-updated agent/uiserver/bindata_assetfs.go from commit 9e715842d 2021-03-17 16:03:18 +00:00
hashicorp-ci bac1afbb89 auto-updated agent/uiserver/bindata_assetfs.go from commit f9e8b26af 2021-03-17 14:45:58 +00:00
hashicorp-ci 6872c33881 auto-updated agent/uiserver/bindata_assetfs.go from commit aca797658 2021-03-17 11:27:44 +00:00
hashicorp-ci f4a96768d1 auto-updated agent/uiserver/bindata_assetfs.go from commit 41471719e 2021-03-17 10:50:59 +00:00
freddygv 3f2489c31d Refactor makePublicListener
By accepting a name the function can be used for other inbound listeners,
like the one for TransparentProxy.
2021-03-16 19:22:26 -06:00
Christopher Broglie f0307c73e5 Add support for configuring TLS ServerName for health checks
Some TLS servers require SNI, but the Golang HTTP client doesn't
include it in the ClientHello when connecting to an IP address. This
change adds a new TLSServerName field to health check definitions to
optionally set it. This fixes #9473.
2021-03-16 18:16:44 -04:00
freddygv 7892964a0c Add cache-type for Internal.IntentionUpstreams 2021-03-16 11:06:47 -06:00
Daniel Nephin 34eb6c01ff state: convert services.node and checks.node indexes
Using NodeIdentity to share the indexes with both.
2021-03-16 13:00:31 -04:00
freddygv 942334b208 Prefix match type vars to match use 2021-03-16 09:49:24 -06:00
freddygv 4cb9fdc27f Pass txn into service list queries 2021-03-16 09:33:08 -06:00
freddygv 86ff9065c1 Pass txn into intention match queries 2021-03-16 08:03:52 -06:00
freddygv 31e757de2a Replace CertURI.Authorize() calls.
AuthorizeIntentionTarget is a generalized version of the old function,
and can be evaluated against sources or destinations.
2021-03-15 18:06:04 -06:00
freddygv f5ed751c91 Fixup typo, comments, and regression 2021-03-15 17:50:47 -06:00
freddygv 4bdbcff9c0 Fixup upstream test 2021-03-15 17:20:30 -06:00
freddygv 3492f9e0d6 Finish cleanup from ServiceConfigRequest changes 2021-03-15 16:38:01 -06:00
freddygv 770c5552d6 Update service manager to pass MeshGateway with config req 2021-03-15 16:08:03 -06:00
freddygv 6090cfcf68 PR comments 2021-03-15 16:02:03 -06:00
Daniel Nephin 4d456922a9 state: use runCase pattern for large test
The TestServiceHealthEventsFromChanges function was over 1400 lines.
Attempting to debug test failures in test functions this large is
difficult. It requires scrolling to the line which defines the testcase
because the failure message only includes the line number of the
assertion, not the line number of the test case.

This is an excellent example of where test tables stop working well, and
start being a problem. To mitigate this problem, the runCase pattern can
be used. When one of these tests fails, a failure message will print the
line number of both the test case and the assertion. This allows a
developer to quickly jump to both of the relevant lines, signficanting
reducing the time it takes to debug test failures.

For example, one such failure could look like this:

    catalog_events_test.go:1610: case: service reg, new node
    catalog_events_test.go:1605: assertion failed: values are not equal
2021-03-15 17:53:16 -04:00
freddygv 7df846aa24 Pass MeshGateway config in service config request
ResolveServiceConfig is called by service manager before the proxy
registration is in the catalog. Therefore we should pass proxy
registration flags in the request rather than trying to fetch
them from the state store (where they may not exist yet).
2021-03-15 14:32:13 -06:00
freddygv 8b46d8dcbb Restore old Envoy prefix on escape hatches
This is done because after removing ID and NodeName from
ServiceConfigRequest we will no longer know whether a request coming in
is for a Consul client earlier than v1.10.
2021-03-15 14:12:57 -06:00
freddygv 93c3c1780d Only lowercase the protocol when normalizing 2021-03-15 14:12:15 -06:00
freddygv 41b2ba1e58 Add omitempty across the board for UpstreamConfig 2021-03-15 13:23:18 -06:00
freddygv 08759e46ed Add RPC endpoint for intention upstreams 2021-03-15 08:50:35 -06:00
freddygv 08737fa606 Add state store function for intention upstreams 2021-03-15 08:50:35 -06:00
freddygv 3722ce2fff Refactor IntentionDecision
This enables it to be called for many upstreams or downstreams of a
service while only querying intentions once.

Additionally, decisions are now optionally denied due to L7 permissions
being present. This enables the function to be used to filter for
potential upstreams/downstreams of a service.
2021-03-15 08:50:35 -06:00
Daniel Nephin f40b76af2d proxycfg: use rpcclient/health.Client instead of passing around cache name
This should allow us to swap out the implementation with something other
than `agent/cache` without making further code changes.
2021-03-12 11:46:04 -05:00
Daniel Nephin 566741a143 catalog_events: set the right key for connect snapshots 2021-03-12 11:35:43 -05:00
Daniel Nephin 906834ce8e proxycfg: Use streaming in connect state 2021-03-12 11:35:42 -05:00
Daniel Nephin 1a764553c0 rpcclient: use streaming for connect health 2021-03-12 11:35:42 -05:00
freddygv d80e4b27b1 Update content hash due to new field 2021-03-11 19:59:19 -07:00
freddygv 682f357185 Fixup more tests 2021-03-11 16:26:55 -07:00
freddygv 756ab4c546 Fixup protobufs and tests 2021-03-11 14:58:59 -07:00
Kyle Havlovitz 1e87c7183a
Merge pull request #9672 from hashicorp/ca-force-skip-xc
connect/ca: Allow ForceWithoutCrossSigning for all providers
2021-03-11 11:49:15 -08:00
freddygv df1f3995f8 Update service manager to store centrally configured upstreams 2021-03-11 11:37:21 -07:00
freddygv 6fd30d0384 Add TransparentProxy opt to proxy definition 2021-03-11 11:37:21 -07:00
freddygv 306ef7d252 Restore old escape hatch alias 2021-03-11 11:36:35 -07:00
freddygv e3dc2a49df Turn Limits and PassiveHealthChecks into pointers 2021-03-11 11:04:40 -07:00
hashicorp-ci f3556f6bba auto-updated agent/uiserver/bindata_assetfs.go from commit fa6687b7f 2021-03-11 09:34:21 +00:00
freddygv acec711a6a Update server-side config resolution and client-side merging 2021-03-10 21:05:11 -07:00
freddygv 1710ec87d2 finish moving UpstreamConfig and related fields to structs pkg 2021-03-10 21:04:13 -07:00
Daniel Nephin 9d924a81a9
Merge pull request #9797 from hashicorp/dnephin/state-index-node-id
state: convert nodes.ID to the new pattern of functional indexers
2021-03-10 17:34:23 -05:00
Daniel Nephin b06b3dd8f8 state: move ConfigEntryKindName
Previously this type was defined in structs, but unlike the other types in structs this type
is not used by RPC requests. By moving it to state we can better indicate that this is not
an API type, but part of the state implementation.
2021-03-10 12:27:22 -05:00
Daniel Nephin 948d1a317d
Merge pull request #9796 from hashicorp/dnephin/state-cleanup-catalog-index-oss
state: remove duplicate tableCheck indexes
2021-03-10 12:20:09 -05:00
Daniel Nephin 3a3007298f
Merge pull request #9851 from panascais-forks/fix-wan-ipv6-key
Fix advertise_addr_wan_ipv6 configuration key
2021-03-10 11:56:07 -05:00
Daniel Nephin 71b0f0a7a6 structs: remove EnterpriseMeta.GetNamespace
I added this recently without realizing that the method already existed and was named
NamespaceOrEmpty. Replace all calls to GetNamespace with NamespaceOrEmpty or NamespaceOrDefault
as appropriate.
2021-03-09 15:17:26 -05:00
Daniel Nephin 2b612a8e92
Merge pull request #9671 from hashicorp/streaming/terminating-gateway-events
state: Add terminating gateway events for streaming
2021-03-09 14:20:21 -05:00
Daniel Nephin 23421e190c state: adjust compare for catalog events
Document that this comparison should roughly match MatchesKey

Only sort by overrideKey or service name, but not both
Add namespace to the sort.

The client side also builds a map of these based on the namespace/node/service key, so the only order
that really matters is the ordering of register/dereigster events.
2021-03-09 14:00:36 -05:00
Daniel Nephin 68ec20f66a state: handle terminating gateway events properly in snapshot
Refactored out a function that can be used for both the snapshot and stream of events to translate
an event into an appropriate connect event.

Previously terminating gateway events would have used the wrong key in the snapshot, which would have
caused them to be filtered out later on.

Also removed an unused function, and some commented out code.
2021-03-09 14:00:35 -05:00
Kyle Havlovitz db572aca59 Add remaining terminating gateway tests for namespaces
Co-Authored-By: Daniel Nephin <dnephin@hashicorp.com>
2021-03-09 14:00:35 -05:00
Daniel Nephin 701285e470 Start to setup enterprise tests for terminating gateway streaming events.
Co-Authored-By: Kyle Havlovitz <kylehav@gmail.com>
2021-03-09 14:00:35 -05:00
Daniel Nephin ae368768e5 state: Add support for override of namespace
in MatchesKey
also tests for MatchesKey

Co-Authored-By: Kyle Havlovitz <kylehav@gmail.com>
2021-03-09 14:00:35 -05:00
Daniel Nephin 4756ff059d state: update calls to ensureConfigEntryTxn
The EnterpriseMeta paramter was removed after this code was written, but before it merged.

Also the table name constant has changed.
2021-03-09 14:00:35 -05:00
Daniel Nephin 30a575dd33 state: add 2 more test cases for terminate gateway streaming events
Co-Authored-By: Kyle Havlovitz <kylehav@gmail.com>
2021-03-09 14:00:34 -05:00
Kyle Havlovitz a21be5efa8 Added 6 new test cases for terminating gateway events
Co-Authored-By: Daniel Nephin <dnephin@hashicorp.com>
2021-03-09 14:00:34 -05:00
Daniel Nephin 06b1c32e25 state: Add two more tests for connect events with terminating gateways
And expand one test case to cover more.

Co-Authored-By: Kyle Havlovitz <kylehav@gmail.com>
2021-03-09 14:00:34 -05:00
Daniel Nephin eb58a39738 state: Include the override key in the sorting of events
Co-Authored-By: Kyle Havlovitz <kylehav@gmail.com>
2021-03-09 14:00:34 -05:00
Kyle Havlovitz c2481ca10f state: Add terminating gateway events on updating a config entry
Co-Authored-By: Daniel Nephin <dnephin@hashicorp.com>
2021-03-09 14:00:34 -05:00
Daniel Nephin 28de159c14 state: add first terminating catalog catalog event
Health of a terminating gateway instance changes
- Generate an event for creating/destroying this instance of the terminating gateway,
  duplicate it for each affected service

Co-Authored-By: Kyle Havlovitz <kylehav@gmail.com>
2021-03-09 14:00:33 -05:00
Silas Rech ab9c484137
Fix advertise_addr_wan_ipv6 configuration key 2021-03-09 14:56:44 +01:00
hashicorp-ci 0e9250df80 auto-updated agent/uiserver/bindata_assetfs.go from commit 33d038377 2021-03-09 09:35:32 +00:00
freddygv 87cde19b4c Create new types for service-defaults upstream cfg 2021-03-08 22:10:27 -07:00
hashicorp-ci 35daee45bc auto-updated agent/uiserver/bindata_assetfs.go from commit 308e5a480 2021-03-08 12:28:15 +00:00
Daniel Nephin a4e68e32d6 state: convert nodes.ID to new functional pattern
In preparation for adding other identifiers to the index.
2021-03-05 12:30:40 -05:00
R.B. Boyer 398b766532
xds: default to speaking xDS v3, but allow for v2 to be spoken upon request (#9658)
- Also add support for envoy 1.17.0
2021-02-26 16:23:15 -06:00
Daniel Nephin 6b95e8dfe2
Merge pull request #9188 from hashicorp/dnephin/more-streaming-tests
Add more streaming tests
2021-02-26 12:36:55 -05:00
Daniel Nephin 566efad2e7
Merge pull request #9759 from hashicorp/dnephin/streaming-default-rpc-enabled
streaming: default rpc.enable_streaming to true
2021-02-26 12:08:00 -05:00
Daniel Nephin 5c8a6311b6
Merge pull request #9703 from pierresouchay/streaming_tags_and_case_insensitive
Streaming filter tags + case insensitive lookups for Service Names
2021-02-26 12:06:26 -05:00
Daniel Nephin 55add28725 catalog_events: set the right key for connect snapshots
Add a test for catalog_event snapshot on connect topic
2021-02-25 14:30:39 -05:00
Daniel Nephin 432dd2d204 consul: Add integration tests of streaming.
Restored from streaming-rpc-final branch.

Co-authored-by: Paul Banks <banks@banksco.de>
2021-02-25 14:30:39 -05:00
Daniel Nephin b7f8e3bad2 state: Add a test for ServiceHealthSnapshot 2021-02-25 14:08:10 -05:00
Daniel Nephin 1d2d15b1e1 agent: add a test for streaming in the service health endpoint
Co-authored-by: Paul Banks <banks@banksco.de>
2021-02-25 14:08:10 -05:00
Daniel Nephin 3aec942b17 streaming: default rpc.enable_streaming to true
So that all servers will start the grpc server used by streaming
2021-02-25 14:06:04 -05:00
hashicorp-ci b76dfa1441 auto-updated agent/uiserver/bindata_assetfs.go from commit 779f7f7b6 2021-02-25 09:41:02 +00:00
John Cowen 5892e75452
ui: Remove any trailing fullstop/period DNS characters from Gateways UI API (#9752)
Previous to this commit, the API response would include Gateway
Addresses in the form `domain.name.:8080`, which due to the addition of
the port is probably not the expected response.

This commit rightTrims any `.` characters from the end of the domain
before formatting the address to include the port resulting in
`domain.name:8080`
2021-02-25 09:34:47 +00:00
R.B. Boyer be89557fb4
test: omit envoy golden test files that differ from the latest version (#9807)
Since we currently do no version switching this removes 75% of the PR
noise.

To generate all *.golden files were removed and then I ran:

    go test ./agent/xds -update
2021-02-24 14:04:31 -06:00
hashicorp-ci f0872a4ac1 auto-updated agent/uiserver/bindata_assetfs.go from commit f9b0e50ca 2021-02-24 09:08:42 +00:00
Yong Wen Chua 58b553704a
Update test fixtures 2021-02-24 16:24:32 +08:00
Yong Wen Chua 750e2921b0
Auto Rewrite Host Headers for Terminating Gateways
Tries to partially address
https://github.com/hashicorp/consul/issues/8707
2021-02-24 16:23:39 +08:00
hashicorp-ci 1413e9ddca auto-updated agent/uiserver/bindata_assetfs.go from commit 8b12d0d09 2021-02-23 09:02:41 +00:00
R.B. Boyer 3b6ffc447b
xds: remove deprecated usages of xDS (#9602)
Note that this does NOT upgrade to xDS v3. That will come in a future PR.

Additionally:

- Ignored staticcheck warnings about how github.com/golang/protobuf is deprecated.
- Shuffled some agent/xds imports in advance of a later xDS v3 upgrade.
- Remove support for envoy 1.13.x but don't add in 1.17.x yet. We have to wait until the xDS v3 support is added in a follow-up PR.

Fixes #8425
2021-02-22 15:00:15 -06:00
R.B. Boyer a0d26430cc
connect: if the token given to the vault provider returns no data avoid a panic (#9806)
Improves #9800
2021-02-22 14:08:49 -06:00
hashicorp-ci 02fe20ec25 auto-updated agent/uiserver/bindata_assetfs.go from commit 2c2e9d1ce 2021-02-22 14:58:06 +00:00
Daniel Nephin b1d3c8002b
Merge pull request #9683 from hashicorp/dnephin/fix-zombie-service-dereg
local: use agent token to deregister services
2021-02-19 18:56:40 -05:00
Daniel Nephin 8a5163b184 local: default to the agent token instead of the user token
When de-registering in anti-entropy sync, when there is no service or
check token.

The agent token will fall back to the default (aka user) token if no agent
token is set, so the existing behaviour still works, but it will prefer
the agent token over the user token if both are set.

ref: https://www.consul.io/docs/agent/options#acl_tokens

The agent token seems more approrpiate in this case, since this is an
"internal operation", not something initiated by the user.
2021-02-19 18:35:08 -05:00
Daniel Nephin 1f9247677b
Merge pull request #9763 from hashicorp/dnephin/cache-warn-on-error-in-notify
cache: log a warning when Cache.Notify handles an error
2021-02-19 18:30:36 -05:00
Daniel Nephin 340d714fd6 state: add a test case for memdb indexers 2021-02-19 17:14:46 -05:00
Daniel Nephin cc5c88213a state: support for functional indexers
These new functional indexers provide a few advantages:

1. enterprise differences can be isolated to a single function (the
   indexer function), making code easier to change
2. as a consequence of (1) we no longer need to wrap all the calls to
   Txn operations, making code easier to read.
3. by removing reflection we should increase the performance of all
   operations.

One important change is in making all the function signatures the same.

https://blog.golang.org/errors-are-values

An extra boolean return value for SingleIndexer.FromObject is superfluous.
The error value can indicate when the index value could not be created.
By removing this extra return value we can use the same signature for both
indexer functions.

This has the nice properly of a function being usable for both indexing operations.
2021-02-19 17:14:46 -05:00
Daniel Nephin b861642910 state: remove duplicate index on the checks table
By using a new pattern for more specific indexes. This allows us to use
the same index for both service checks and node checks. It removes the
abstraction around memdb.Txn operations, and isolates all of the
enterprise differences in a single place (the indexer).
2021-02-19 17:14:46 -05:00
Daniel Nephin 519bb82a00 state: remove duplicate function
catalogChecksForNodeService was a duplicate of catalogListServiceChecks
2021-02-19 17:14:46 -05:00
R.B. Boyer 39effd620c
xds: only try to create an ipv6 expose checks listener if ipv6 is supported by the kernel (#9765)
Fixes #9311

This only fails if the kernel has ipv6 hard-disabled. It is not sufficient to merely not provide an ipv6 address for a network interface.
2021-02-19 14:38:43 -06:00
hashicorp-ci 21713f1b73 auto-updated agent/uiserver/bindata_assetfs.go from commit 8263879e6 2021-02-19 16:48:03 +00:00
hashicorp-ci 087247967c auto-updated agent/uiserver/bindata_assetfs.go from commit 7be21a902 2021-02-18 18:37:38 +00:00
hashicorp-ci c309d2b6cb auto-updated agent/uiserver/bindata_assetfs.go from commit 1507dd8ab 2021-02-17 19:01:18 +00:00
hashicorp-ci 61fd873f24 auto-updated agent/uiserver/bindata_assetfs.go from commit 1b5087f6e 2021-02-17 18:56:34 +00:00
Mark Anderson b9d22f48cd
Add fields to the /acl/auth-methods endpoint. (#9741)
* A GET of the /acl/auth-method/:name endpoint returns the fields
MaxTokenTTL and TokenLocality, while a LIST (/acl/auth-methods) does
not.

The list command returns a filtered subset of the full set. This is
somewhat deliberate, so that secrets aren't shown, but the TTL and
Locality fields aren't (IMO) security critical, and it is useful for
the front end to be able to show them.

For consistency these changes mirror the 'omit empty' and string
representation choices made for the GET call.

This includes changes to the gRPC and API code in the client.

The new output looks similar to this
curl 'http://localhost:8500/v1/acl/auth-methods' | jq '.'

  {
    "MaxTokenTTL": "8m20s",
    "Name": "minikube-ttl-local2",
    "Type": "kubernetes",
    "Description": "minikube auth method",
    "TokenLocality": "local",
    "CreateIndex": 530,
    "ModifyIndex": 530,
    "Namespace": "default"
  }
]

Signed-off-by: Mark Anderson <manderson@hashicorp.com>

* Add changelog

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-02-17 08:16:57 -08:00
Daniel Nephin d5448c004d
Merge pull request #9452 from hashicorp/dnephin/config-tests-flags-and-edgecases
config: make TestBuilder_BuildAndValidate_ConfigFlagsAndEdgecases easier to work with
2021-02-16 16:43:49 -05:00
Daniel Nephin d18e00194a
Merge pull request #9720 from hashicorp/dnephin/ent-meta-ergo-1
structs: rename EnterpriseMeta constructor
2021-02-16 15:31:58 -05:00
Daniel Nephin 363d738fd2
Merge pull request #9772 from hashicorp/streamin-fix-bad-cached-snapshot
streaming: fix snapshot cache bug
2021-02-16 15:28:00 -05:00
Daniel Nephin 89383e2d98
Merge pull request #9728 from hashicorp/dnephin/state-index-table
state: document how index table is used
2021-02-16 15:27:27 -05:00
Daniel Nephin d1772ae305 structs: rename EnterpriseMeta constructor
To match the Go convention.
2021-02-16 14:45:43 -05:00
Daniel Nephin b91c5a8383 config: rename testCase fields
So that their name better reflects their purpose
2021-02-16 14:40:16 -05:00
Daniel Nephin 534631659c config: make TestLoad_IntegrationWithFlags easier to work with
Replace the large table of tests with individual calls to run(). By using
runCase, failure messages will include the line number for the test case, as
well as a line number from the test functions.

Example:
=== FAIL: agent/config TestLoad_IntegrationWithFlags/failing_case (0.01s)
    runtime_test.go:4721: case: failing case
    runtime_test.go:4864: error "data_dir cannot be empty" does not contain "I expected this error"

Previous:
    runtime_test.go:4864: error "data_dir cannot be empty" does not contain "I expected this error"

Without the line number to the testCase data, debugging these tests is
difficult. It is impossible to jump directly to the test case, and
difficult to find the location because of many similarly named cases.
2021-02-16 14:40:16 -05:00
Daniel Nephin bd728df345 move test case iteration to caller
To make the test case logic more obvious
2021-02-16 14:40:16 -05:00
Daniel Nephin f86ceea803 config: remove test cases for impossible scenarios
AEInterval is overridden by NonUserSource, so there is no way for a user
to set this value. These two cases represented impossible real world
scenarios.

Instead the test is replaced with one that shows that the AEInterval can
not be set by config.

This change allows us to remove the hcltail and jsontail fields from
testCase
2021-02-16 14:40:15 -05:00
Daniel Nephin 76787d4dd0 Fix error checking
There were two identical if lines, remove one and use a switch/case so the compiler
can catch this problem.
2021-02-16 14:40:15 -05:00
Daniel Nephin 863551beb4 config: move testing shims to LoadOpts
This will allow us to eventually port these tests to use config.Load instead of
unexported functions.
2021-02-16 14:40:15 -05:00
Daniel Nephin 4411da5ddd Remove patchActual, does not appear to be necessary
also small rename and fix comment
2021-02-16 14:40:15 -05:00
Daniel Nephin ba3a1b95e1 stream: fix a snapshot cache bug
Previously a snapshot created as part of a resumse-stream request could have incorrectly
cached the newSnapshotToFollow event. This would cause clients to error because they
received an unexpected framing event.
2021-02-16 12:52:23 -05:00
Daniel Nephin 9b3c6da9df stream: test the snapshot cache is saved correctly
when the cache entry is created from resuming a stream.
2021-02-16 12:08:43 -05:00
Daniel Nephin d9b8d30cad cache: log a warning when Cache.Notify handles an error
Without these warnings, errors are silently ignored, which can make
debugging problems more challenging.
2021-02-12 13:02:23 -05:00
Daniel Nephin c522f8defd
Merge pull request #9758 from hashicorp/dnephin/fix-streaming-bugs
http: error if near is used with streaming
2021-02-12 10:37:29 -05:00
Daniel Nephin d5cc206e39 streaming: move ServiceTag and NodeMetaFiltering to the cache-entry
So that all the client side filtering is in the same place. Previously
only the bexpr filter was in the cache-entry.

Also makes a small change to the filtering so that instead of rebuilding
slices of items, the filtering can return a bool to determine if the
event payload is saved or not.
2021-02-11 20:20:09 -05:00
Daniel Nephin 0a4304697e
Merge pull request #9727 from mbanikazemi/typos
Correcting the changed function name in comment
2021-02-11 14:46:10 -05:00
Daniel Nephin b909088500 http: error if near is used with streaming 2021-02-11 14:10:38 -05:00
hashicorp-ci bb67e0939a auto-updated agent/uiserver/bindata_assetfs.go from commit 96204a21d 2021-02-11 11:42:07 +00:00
hashicorp-ci d5024e64c3 auto-updated agent/uiserver/bindata_assetfs.go from commit 369637aff 2021-02-11 10:10:21 +00:00
hashicorp-ci 673f4690b7 auto-updated agent/uiserver/bindata_assetfs.go from commit eda38a0ba 2021-02-11 09:54:29 +00:00
R.B. Boyer 6eeccc93ce
connect: update supported envoy point releases to 1.16.2, 1.15.3, 1.14.6, 1.13.7 (#9737) 2021-02-10 13:11:15 -06:00
Pierre Souchay 5bee19ee16 Do not filter tags unless req.TagFilter is set
Send empty array [] instead of [""] in DNS requests when TagFilter is not set

Do not change case sensitivity of services anymore in `getServiceNodes()` since
cache keys are now case insensitive
2021-02-10 10:36:11 +01:00
Daniel Nephin 6b36ab744c streaming: double the cache TTL
10 minutes is the default blocking query timeout. Using the same value results in us hitting
the expired cache entry bug frequently. By extending this TTL we at least mitigate the problem.

The underlying bug still needs to be fixed.
2021-02-09 14:36:26 -05:00
Daniel Nephin 9423c887c4 submatview: do not reset retry waiter when materializer is reset
The materializer is often reset when an error is received. By resetting
the retryWaiter we effectively never wait. The retryWaiter should only
be reset when we get an event without error. This is done in
Materializer.updateView().
2021-02-09 13:56:50 -05:00
Daniel Nephin 0fa51e5ba9 api: Use blocking query for health when near is set
Streaming can not be used for these queries because the near query
paramter indicates a specific sort of the results, and that sort
requires data that is not available to the client from the streaming
API.
2021-02-09 13:55:33 -05:00
Pierre Souchay 6f91085869 Use lower case for serviceName computation of cache keys 2021-02-09 19:19:40 +01:00
Matt Keeler 8dbe342ec9
Stop background refresh of cached data for requests that result in ACL not found errors (#9738) 2021-02-09 10:15:53 -05:00
Freddy 82c269a7c5
Avoid potential proxycfg/xDS deadlock using non-blocking send 2021-02-08 16:14:06 -07:00
R.B. Boyer 39e4ae25ac
connect: connect CA Roots in the primary datacenter should use a SigningKeyID derived from their local intermediate (#9428)
This fixes an issue where leaf certificates issued in primary
datacenters using Vault as a Connect CA would be reissued very
frequently (every ~20 seconds) because the logic meant to detect root
rotation was errantly triggering.

The hash of the rootCA was being compared against a hash of the
intermediateCA and always failing. This doesn't apply to the Consul
built-in CA provider because there is no intermediate in use in the
primary DC.

This is reminiscent of #6513
2021-02-08 13:18:51 -06:00
Pierre Souchay 2fe3ab7db0 [Streaming] Properly filters node-meta queries on health
This wil fix https://github.com/hashicorp/consul/issues/9730
2021-02-08 17:53:18 +01:00
freddygv ec5f75776b Update comments on avoiding proxycfg deadlock 2021-02-08 09:45:45 -07:00
R.B. Boyer 43193a35c6
xds: prevent LDS flaps in mesh gateways due to unstable datacenter lists (#9651)
Also fix a similar issue in Terminating Gateways that was masked by an overzealous test.
2021-02-08 10:19:57 -06:00
Mohammad Banikazemi bcadd341eb Correcting the changed function name in comment
Signed-off-by: Mohammad Banikazemi <mbanikazemi@gmail.com>
2021-02-06 20:23:40 -05:00
freddygv 6e443e5536 Retry send after timer fires, in case no updates occur 2021-02-05 18:00:59 -07:00
Daniel Nephin 30332ffb43 state: Use the tableIndex constant 2021-02-05 18:37:45 -05:00
Daniel Nephin 3ecbeda234 state: Document index table
And move the IndexEntry (which is stored in the table) next to the table
schema definition.
2021-02-05 18:37:45 -05:00
R.B. Boyer adff0c05a7
xds: deduplicate mesh gateway listeners in a stable way (#9650)
In a situation where the mesh gateway is configured to bind to multiple
network interfaces, we use a feature called 'tagged addresses'.
Sometimes an address is duplicated across multiple tags such as 'lan'
and 'lan_ipv4'.

There is code to deduplicate these things when creating envoy listeners,
but that code doesn't ensure that the same tag wins every time. If the
winning tag flaps between xDS discovery requests it will cause the
listener to be drained and replaced.
2021-02-05 16:28:07 -06:00
freddygv de0cb1af7f Make xDS labeling consistent with proxycfg 2021-02-05 15:15:52 -07:00
freddygv 95e7641faa Update proxycfg logging, labels were already attached 2021-02-05 15:14:49 -07:00
Daniel Nephin a4690ac7d9
Merge pull request #9719 from hashicorp/oss/state-store-4
state: remove registerSchema
2021-02-05 14:02:38 -05:00
Daniel Nephin 1c4e0cfa2a
Merge pull request #9718 from hashicorp/oss/dnephin/ent-meta-in-state-store-3
state: convert all table name constants to the new prefix pattern
2021-02-05 14:02:07 -05:00
Daniel Nephin 0814f22715
Merge pull request #9665 from hashicorp/dnephin/state-store-indexes-2
state: move config-entries table definition to config_entries_schema.go
2021-02-05 14:01:08 -05:00
Daniel Nephin 912dbb4cb4
Merge pull request #9664 from hashicorp/dnephin/state-store-indexes
state: move ACL schema and index definitions to acl_schema.go
2021-02-05 13:38:31 -05:00
Daniel Nephin 05d5ec4804 state: remove the need for registerSchema
registerSchema creates some indirection which is not necessary in this
case. newDBSchema can call each of the tables.

Enterprise tables can be added from the existing withEnterpriseSchema
shim.
2021-02-05 12:19:56 -05:00
Daniel Nephin 2cbf8b5fd0 state: rename table name constants to use pattern
the 'table' prefix is shorter, and also reads better in queries.
2021-02-05 12:12:19 -05:00
Daniel Nephin 8ac9d54ccc state: rename connect constants 2021-02-05 12:12:19 -05:00
Daniel Nephin 0c34e474c5 state: rename table name constants to new pattern
Using Apps Hungarian Notation for these constants makes the memdb queries more readable.
2021-02-05 12:12:18 -05:00
Pierre Souchay 7a024ed074 Streaming filter tags + case insensitive lookups for Service Names
Will fix:
 * https://github.com/hashicorp/consul/issues/9695
 * https://github.com/hashicorp/consul/issues/9702
2021-02-04 11:00:51 +01:00
Daniel Nephin 2d5b5afec1 state: Remove unnecessary entMeta arg to EnsureConfigEntry 2021-02-03 18:10:38 -05:00
freddygv 5ba14ad41d Add trace logs to proxycfg state runner and xds srv 2021-02-02 12:26:38 -07:00
freddygv 37190c0d0d Avoid potential deadlock using non-blocking send
Deadlock scenario:
    1. Due to scheduling, the state runner sends one snapshot into
    snapCh and then attempts to send a second. The first send succeeds
    because the channel is buffered, but the second blocks.
    2. Separately, Manager.Watch is called by the xDS server after
    getting a discovery request from Envoy. This function acquires the
    manager lock and then blocks on receiving the CurrentSnapshot from
    the state runner.
    3. Separately, there is a Manager goroutine that reads the snapshots
    from the channel in step 1. These reads are done to notify proxy
    watchers, but they require holding the manager lock. This goroutine
    goes to acquire that lock, but can't because it is held by step 2.

Now, the goroutine from step 3 is waiting on the one from step 2 to
release the lock. The goroutine from step 2 won't release the lock until
the goroutine in step 1 advances. But the goroutine in step 1 is waiting
for the one in step 3. Deadlock.

By making this send non-blocking step 1 above can proceed. The coalesce
timer will be reset and a new valid snapshot will be delivered after it
elapses or when one is requested by xDS.
2021-02-02 11:31:14 -07:00
hashicorp-ci 6fa9a6a1d9 auto-updated agent/uiserver/bindata_assetfs.go from commit e0ff7080a 2021-02-02 10:08:48 +00:00
hashicorp-ci 90917400c6 auto-updated agent/uiserver/bindata_assetfs.go from commit 0b7d676dc 2021-02-01 17:55:03 +00:00
hashicorp-ci a33ff40816 auto-updated agent/uiserver/bindata_assetfs.go from commit 3aef5cde2 2021-02-01 17:35:20 +00:00
Kyle Havlovitz 7dac583863 connect/ca: Allow ForceWithoutCrossSigning for all providers
This allows setting ForceWithoutCrossSigning when reconfiguring the CA
for any provider, in order to forcibly move to a new root in cases where
the old provider isn't reachable or able to cross-sign for whatever
reason.
2021-01-29 13:38:11 -08:00
hashicorp-ci 40c2a3d108 auto-updated agent/uiserver/bindata_assetfs.go from commit 3477b1de7 2021-01-29 16:03:41 +00:00
Daniel Nephin 5b4703f0e4 state: rename config-entries table const to match new pattern 2021-01-28 20:34:34 -05:00
Daniel Nephin cd06b5728c state: move config-entries table to new pattern 2021-01-28 20:34:15 -05:00
Daniel Nephin e8931b868c state: use indexID
this change was already made to enterprise, so backporting it.
2021-01-28 20:30:08 -05:00
Daniel Nephin 1cccdc45c2 state: Move ACL schema indexes to match Ent
and use constants for table and index names.
2021-01-28 20:05:09 -05:00
Daniel Nephin 727a402810
Merge pull request #9302 from hashicorp/dnephin/add-service-3
agent: remove ServiceManager.Start goroutine
2021-01-28 16:59:41 -05:00
Daniel Nephin 1dcafa51a4 config: make config.TestLoad_FullConfig use config.Load
This commit makes a number of changes that should make
TestLoad_FullConfig easier to work with, and make the test more like
real world scenarios.

* use separate files in testdata/ dir to store the config source.
  Separate files are much easier to edit because editors can syntax
  highlight json/hcl, and it makes strings easier to find. Previously
  trying to find strings would match strings used in other tests.
* use the exported config.Load interface instead of internal NewBuilder
  and BuildAndValidate.
* remove the tail config overrides, which are only necessary with
  nonZero works.
2021-01-27 17:51:53 -05:00
Daniel Nephin a0dd45941b config: Unexport Builder and NewBuilder
This type and constructor are implementation details of config loading.
All callers should use config.Load.
2021-01-27 17:41:53 -05:00
Daniel Nephin 32d36d0dd4 config: replace calls to config.NewBuilder with config.Load
This is another incremental change to reduce config loading to a single
small interface. All calls to NewBuilder can be replaced with Load.
2021-01-27 17:34:43 -05:00
Daniel Nephin 97a577502d config: improve the interface of Load
This commit reduces the interface to Load() a bit, in preparation for
unexporting NewBuilder and having everything call Load.

The three arguments are reduced to a single argument by moving the other
two into the options struct.

The three return values are reduced to two by moving the RuntimeConfig
and Warnings into a LoadResult struct.
2021-01-27 17:34:43 -05:00
Daniel Nephin 27df0d9a78
Merge pull request #9252 from hashicorp/dnephin/config-unmethod
config: remove Builder receiver from funcs that dont use it
2021-01-27 17:31:17 -05:00
Matt Keeler f561462064
Upgrade raft-autopilot and wait for autopilot it to stop when revoking leadership (#9644)
Fixes: 9626
2021-01-27 11:14:52 -05:00
hashicorp-ci 49b773233b auto-updated agent/uiserver/bindata_assetfs.go from commit 25f989753 2021-01-27 10:47:58 +00:00
Hans Hasselberg 444cdeb8fb
Add flags to support CA generation for Connect (#9585) 2021-01-27 08:52:15 +01:00
hashicorp-ci 174878eb63 auto-updated agent/uiserver/bindata_assetfs.go from commit 92f0eb3bd 2021-01-26 18:00:09 +00:00
hashicorp-ci 5c7f815469 auto-updated agent/uiserver/bindata_assetfs.go from commit 82a62cd2e 2021-01-26 17:47:18 +00:00
Daniel Nephin f36d77ee4a
Merge pull request #9301 from hashicorp/dnephin/add-service-2
agent: reduce AddService 2
2021-01-26 12:01:34 -05:00
R.B. Boyer c608dc0d60
server: initialize mgw-wanfed to use local gateways more on startup (#9528)
Fixes #9342
2021-01-25 17:30:38 -06:00
Daniel Nephin 8c45f2c1fe agent: use the new lib/mutex for stateLock
Previously the ServiceManager had to run a separate goroutine so that it could block on a channel
send/receive instead of a lock. Using this mutex with TryLock allows us to cancel the lock when
the serviceConfigWatch is stopped.

Without this change removing the ServiceManager.Start goroutine would not be possible because
when AddService is called it acquires the stateLock. While that lock is held, if there are
existing watches for the service, the old watch will be stopped, and the goroutine holding the
lock will attempt to wait for that watcher goroutine to exit.

If the goroutine is handling an update (serviceConfigWatch.handleUpdate) then it can block on
acquiring the stateLock and deadlock the agent.  With this change the context is cancelled as
and the goroutine will exit instead of waiting on the stateLock.
2021-01-25 18:01:47 -05:00
Daniel Nephin 4e42b8f1d4 agent: remove ServiceManager goroutine
The ServiceManager.Start goroutine was used to serialize calls to
agent.addServiceInternal.

All the goroutines which sent events to the channel would block waiting
for a response from that same goroutine, which is effectively the same
as a synchronous call without any channels.

This commit removes the goroutine and channels, and instead calls
addServiceInternal directly. Since all of these goroutines will need to
take the agent.stateLock, the mutex handles the serializing of calls.
2021-01-25 18:01:47 -05:00
Daniel Nephin 45b315060a agent: Minor cosmetic changes in ServiceManager
Also use the non-deprecated func in a test
2021-01-25 18:01:47 -05:00
Daniel Nephin 8b67231e8c agent: update godoc for AddServiceRequest 2021-01-25 18:01:03 -05:00
Daniel Nephin f34703fc63 agent: move checkStateSnapshot
Move the field into the struct for addServiceLocked. Also don't require
setting a default value, so that the callers can leave it as nil if they
don't already have a snapshot.
2021-01-25 18:01:03 -05:00
Daniel Nephin 1495291054 agent: move two fields off of AddServiceRequest 2021-01-25 18:01:03 -05:00
Daniel Nephin 60c6a1c220 agent: Replace two fields on AddServiceRequest with a func field
The two previous fields were mutually exclusive. They can be represented
with a single function which provides the value.
2021-01-25 18:01:03 -05:00
Daniel Nephin d0386ae025 agent: remove serviceRegiration type
Replace with the existing AddServiceRequest struct. These structs are
almost identical. Additionally, the only reason the serviceRegistration
struct existed was to recreate an AddServiceRequest.

By storing and re-using the AddServiceRequest we remove the need to
translate into one type and back to the original type.

We also remove the extra parameters to a function, because those values
are already available from the AddServiceRequest field.

Also a minor optimization to only call tokens.AgentToken() when
necessary. Previous it was being called every time, but the value was
being ignored if the AddServiceRequest had a token.
2021-01-25 18:01:03 -05:00
Daniel Nephin 5f9930a9be agent: remove an a branch in the AddService flow
Handle the decision to use ServiceManager in a single place. Instead of
calling ServiceManager.AddService, then calling back into
addServiceInternal, only call ServiceManager.AddService if we are going
to use it.

This change removes some small duplication and removes a branch from the
AddService flow.
2021-01-25 18:01:03 -05:00
Daniel Nephin 4da0718c57 agent: use fields directly, not temp variables
The temprorary variables make it much harder to trace where and how struct
fields are used. If a field is only used a small number of times than
refer to the field directly.
2021-01-25 17:25:04 -05:00
Daniel Nephin 5b6f806f4f agent: addServiceIternalRequest
Move fields that are only relevant for addServiceInternal onto a new
struct and embed AddServiceRequest.
2021-01-25 17:25:04 -05:00
Daniel Nephin 3d39359bcb agent: move deprecated AddServiceFromSource to a test file
The method is only used in tests, and only exists for legacy calls.

There was one other package which used this method in tests. Export
the AddServiceRequest and a couple of its fields so the new function can
be used in those tests.
2021-01-25 17:25:03 -05:00
Daniel Nephin e44fd1cb92 agent: use a single method for Agent.AddService 2021-01-25 17:25:03 -05:00
Daniel Nephin 6757231b82 agent: rename AddService->AddServiceFromSource
In preparation for extracting a single AddService func that accepts a request struct.
2021-01-25 17:25:01 -05:00
Daniel Nephin 48112e6298
Merge pull request #9420 from hashicorp/dnephin/reduce-duplicate-in-catalog-schema
state: reduce interface for Enterprise schema
2021-01-25 17:04:25 -05:00
Chris Boulton 8a35df81c7
connect: add local_request_timeout_ms to configure local_app http timeouts (#9554) 2021-01-25 13:50:00 -06:00
R.B. Boyer 51e3ca6cbb
server: use the presense of stored federation state data as a sign that we already activated the federation state feature flag (#9519)
This way we only have to wait for the serf barrier to pass once before
we can make use of federation state APIs Without this patch every
restart needs to re-compute the change.
2021-01-25 13:24:32 -06:00
hashicorp-ci 501fc855d3 auto-updated agent/uiserver/bindata_assetfs.go from commit 09d917618 2021-01-25 19:09:57 +00:00
hashicorp-ci 4584771bbf auto-updated agent/uiserver/bindata_assetfs.go from commit 88d5e00f9 2021-01-25 18:52:54 +00:00
hashicorp-ci 8d440c532a auto-updated agent/uiserver/bindata_assetfs.go from commit bb9573832 2021-01-25 18:19:10 +00:00
R.B. Boyer 03790a1f91
server: add OSS stubs supporting validation of source namespaces in service-intentions config entries (#9527) 2021-01-25 11:27:38 -06:00