Commit Graph

938 Commits (5a74bb6d5a1e96ac3e94d55dd44660b2a29e4513)

Author SHA1 Message Date
freddygv 85878685b7 Fixup proxy config test fixtures
- The TestNodeService helper created services with the fixed name "web",
and now that name is overridable.

- The discovery chain snapshot didn't have prepared query endpoints so
the endpoints tests were missing data for prepared queries
2021-08-20 17:38:57 -06:00
Dhia Ayachi 1950ebbe1f
oss portion of ent #1069 (#10883) 2021-08-20 12:57:45 -04:00
R.B. Boyer 097e1645e3
agent: ensure that most agent behavior correctly respects partition configuration (#10880) 2021-08-19 15:09:42 -05:00
R.B. Boyer 310e775a8a
state: partition nodes and coordinates in the state store (#10859)
Additionally:

- partitioned the catalog indexes appropriately for partitioning
- removed a stray reference to a non-existent index named "node.checks"
2021-08-17 13:29:39 -05:00
Daniel Nephin e637cd71f3 acl: use authz consistently as the variable name for an acl.Authorizer
Follow up to https://github.com/hashicorp/consul/pull/10737#discussion_r682147950

Renames all variables for acl.Authorizer to use `authz`. Previously some
places used `rule` which I believe was an old name carried over from the
legacy ACL system.

A couple places also used authorizer.

This commit also removes another couple of authorizer nil checks that
are no longer necessary.
2021-08-17 12:14:10 -04:00
Kyle Havlovitz 073b6c8411 oss: Rename default partition 2021-08-12 14:31:37 -07:00
Daniel Nephin d3325b0253
Merge pull request #10612 from bigmikes/acl-replication-fix
acl: acl replication routine to report the last error message
2021-08-06 18:29:51 -04:00
Daniel Nephin 5b2e5882b4 acl: move check for Intention.DestinationName into Authorizer
Follow up to https://github.com/hashicorp/consul/pull/10737#discussion_r680134445

Move the check for the Intention.DestinationName into the Authorizer to remove the
need to check what kind of Authorizer is being used.

It sounds like this check is only for legacy ACLs, so is probably just a safeguard
.
2021-08-04 18:06:44 -04:00
Daniel Nephin 9cdd823ffc
Merge pull request #10737 from hashicorp/dnephin/remove-authorizer-nil-checks
acl: remove authz == nil checks
2021-08-04 17:39:34 -04:00
Evan Culver 710bd90ef7
checks: Add Interval and Timeout to API response (#10717) 2021-08-03 15:26:49 -07:00
Daniel Nephin f497d5ab30 acl: remove many instances of authz == nil 2021-07-30 13:58:35 -04:00
Evan Culver 727b81a757 Fix intention endpoint test 2021-07-30 12:58:45 -04:00
Chris S. Kim 9c3af1a429
sync enterprise files with oss (#10705) 2021-07-27 17:09:59 -04:00
Chris S. Kim 91c90a672a
agent: update proxy upstreams to inherit namespace from service (#10688) 2021-07-26 17:12:29 -04:00
R.B. Boyer 96b97d6554
replumbing a bunch of api and agent structs for partitions (#10681) 2021-07-22 14:33:22 -05:00
R.B. Boyer fc9b1a277d
sync changes to oss files made in enterprise (#10670) 2021-07-22 13:58:08 -05:00
R.B. Boyer 188e8dc51f
agent/structs: add a bunch more EnterpriseMeta helper functions to help with partitioning (#10669) 2021-07-22 13:20:45 -05:00
Evan Culver 0527dcff57
acls: Show `AuthMethodNamespace` when reading/listing ACL token meta (#10598) 2021-07-15 10:38:52 -07:00
Giulio Micheloni 814ef6b103 acl: fix error type into a string type for serialization issue
acl_endpoint_test.go:507:
        	Error Trace:	acl_endpoint_test.go:507
        	            				retry.go:148
        	            				retry.go:149
        	            				retry.go:103
        	            				acl_endpoint_test.go:504
        	Error:      	Received unexpected error:
        	            	codec.decoder: decodeValue: Cannot decode non-nil codec value into nil error (1 methods)
        	Test:       	TestACLEndpoint_ReplicationStatus
2021-07-15 11:31:44 +02:00
Giulio Micheloni 529fe737ef acl: acl replication routine to report the last error message 2021-07-14 11:50:23 +02:00
Evan Culver 13bd86527b
Add support for returning ACL secret IDs for accessors with acl:write (#10546) 2021-07-08 15:13:08 -07:00
Daniel Nephin 2c4f22a9f0
Merge pull request #10552 from hashicorp/dnephin/ca-remove-rotation-period
ca: remove unused RotationPeriod field
2021-07-06 18:49:33 -04:00
jkirschner-hashicorp 5f73de6fbc
Merge pull request #10560 from jkirschner-hashicorp/change-sane-to-reasonable
Replace use of 'sane' where appropriate
2021-07-06 11:46:04 -04:00
Daniel Nephin 3a045cca8d ca: remove unused RotationPeriod field
This field was never used. Since it is persisted as part of a map[string]interface{} it
is pretty easy to remove it.
2021-07-05 19:15:44 -04:00
Jared Kirschner bd536151e1 Replace use of 'sane' where appropriate
HashiCorp voice, style, and language guidelines recommend avoiding ableist
language unless its reference to ability is accurate in a particular use.
2021-07-02 12:18:46 -04:00
Daniel Nephin 16b21b0864 http: add an X-Consul-Query-Backend header to responses
So that it is easier to detect and test when streaming is being used.
2021-06-28 16:44:58 -04:00
R.B. Boyer ed8a901be7
connect: include optional partition prefixes in SPIFFE identifiers (#10507)
NOTE: this does not include any intentions enforcement changes yet
2021-06-25 16:47:47 -05:00
R.B. Boyer e3835ac6a1
structs: prohibit config entries from referencing more than one partition at a time (#10478)
affected kinds: service-defaults, ingress-gateway, terminating-gateway, service-intentions
2021-06-23 16:44:10 -05:00
R.B. Boyer 8344b7fe2e
structs: prevent service-defaults upstream configs from using wildcard names or namespaces (#10475) 2021-06-23 15:48:54 -05:00
R.B. Boyer ac50db9087
structs: add some missing config entry validation and clean up tests (#10465)
Affects kinds: service-defaults, ingress-gateway, terminating-gateway
2021-06-23 14:11:23 -05:00
Freddy 3ee66b2e9a
Omit empty tproxy config in JSON responses (#10402) 2021-06-15 13:53:35 -06:00
Freddy ffb13f35f1
Rename CatalogDestinationsOnly (#10397)
CatalogDestinationsOnly is a passthrough that would enable dialing
addresses outside of Consul's catalog. However, when this flag is set to
true only _connect_ endpoints for services can be dialed.

This flag is being renamed to signal that non-Connect endpoints can't be
dialed by transparent proxies when the value is set to true.
2021-06-14 14:15:09 -06:00
Freddy 33bd9b5be8
Relax validation for expose.paths config (#10394)
Previously we would return an error if duplicate paths were specified.
This could lead to problems in cases where a user has the same path,
say /healthz, on two different ports.

This validation was added to signal a potential misconfiguration.
Instead we will only check for duplicate listener ports, since that is
what would lead to ambiguity issues when generating xDS config.

In the future we could look into using a single listener and creating
distinct filter chains for each path/port.
2021-06-14 14:04:11 -06:00
Freddy 429f9d8bb8
Add flag for transparent proxies to dial individual instances (#10329) 2021-06-09 14:34:17 -06:00
Daniel Nephin ba15f92a8a structs: fix cache keys
So that requests are cached properly, and the cache does not return the wrong data for a
request.
2021-05-31 17:22:16 -04:00
Daniel Nephin 920ae31598 structs: add two cache completeness tests types that implement cache.Request 2021-05-31 16:54:41 -04:00
Daniel Nephin 46dfdb611f structs: improve the interface of assertCacheInfoKeyIsComplete 2021-05-31 16:54:41 -04:00
Daniel Nephin 7c2957e24d structs: Add more cache key tests 2021-05-31 16:54:40 -04:00
Dhia Ayachi f785c5b332
RPC Timeout/Retries account for blocking requests (#8978) 2021-05-27 17:29:43 -04:00
R.B. Boyer 597448da47
server: ensure that central service config flattening properly resets the state each time (#10239)
The prior solution to call reply.Reset() aged poorly since newer fields
were added to the reply, but not added to Reset() leading serial
blocking query loops on the server to blend replies.

This could manifest as a service-defaults protocol change from
default=>http not reverting back to default after the config entry
reponsible was deleted.
2021-05-14 10:21:44 -05:00
R.B. Boyer 7e1d7803b8
agent: ensure we hash the non-deprecated upstream fields on ServiceConfigRequest (#10240) 2021-05-14 10:15:48 -05:00
Iryna Shustava d7d44f6ae7
Save exposed ports in agent's store and expose them via API (#10173)
* Save exposed HTTP or GRPC ports to the agent's store
* Add those the health checks API so we can retrieve them from the API
* Change redirect-traffic command to also exclude those ports from inbound traffic redirection when expose.checks is set to true.
2021-05-12 13:51:39 -07:00
Mark Anderson 751fe7e314 Fixup more structs_test
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-05-05 10:13:56 -07:00
Mark Anderson 4d0346bc46 Fixup filtering tests
Missed annotations for SocketPath, LocalSocketPath, LocalServiceSocketPath

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-05-05 09:28:17 -07:00
Daniel Nephin 347f3d2128
Merge pull request #10155 from hashicorp/dnephin/config-entry-remove-fields
config-entry: remove Kind and Name field from Mesh config entry
2021-05-04 17:27:56 -04:00
Daniel Nephin c8c85523e1 config-entries: add a test for the API client
Also fixes a bug with listing kind=mesh config entries. ValidateConfigEntryKind was only being used by
the List endpoint, and was yet another place where we have to enumerate all the kinds.

This commit removes ValidateConfigEntryKind and uses MakeConfigEntry instead. This change removes
the need to maintain two separate functions at the cost of creating an instance of the config entry which will be thrown away immediately.
2021-05-04 17:14:21 -04:00
Mark Anderson 8040f91a43 Add support for downstreams
Enhance config by adding SocketPath and LocalSocketPath config values

Supports syntax of the form:
```
services {
  name = "sock_forwarder"
  id = "sock_forwarder.1"
  socket_path = "/tmp/downstream_3.sock"
  connect {
    sidecar_service {
      proxy {
	local_service_socket_path = "/tmp/downstream.sock"
      }
    }
  }
}
```

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-05-04 12:41:43 -07:00
Mark Anderson 583ae65d5b Convert mode to string representation
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-05-04 12:41:43 -07:00
Mark Anderson 06f0f79218 Continue working through proxy and agent
Rework/listeners, rename makeListener

Refactor, tests pass

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-05-04 12:41:43 -07:00
Mark Anderson 8b1217417a First changes for unix domain sockets upstreams
Start making structure changes to support unix domain socket address for upstreams

	upstreams = [
	  {
	    destination_name = "echo-service"
	    local_bind_socket_path = "/tmp/upstream.sock"
	    config {
	      passive_health_check {
		interval = "10s"
		max_failures = 42
	      }
	    }
	  }

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-05-04 12:41:43 -07:00
Freddy ed1082510d
Fixup discovery chain handling in transparent mode (#10168)
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

Previously we would associate the address of a discovery chain target
with the discovery chain's filter chain. This was broken for a few reasons:

- If the upstream is a virtual service, the client proxy has no way of
dialing it because virtual services are not targets of their discovery
chains. The targets are distinct services. This is addressed by watching
the endpoints of all upstream services, not just their discovery chain
targets.

- If multiple discovery chains resolve to the same target, that would
lead to multiple filter chains attempting to match on the target's
virtual IP. This is addressed by only matching on the upstream's virtual
IP.

NOTE: this implementation requires an intention to the redirecting
virtual service and not just to the final destination. This is how
we can know that the virtual service is an upstream to watch.

A later PR will look into traversing discovery chains when computing
upstreams so that intentions are only required to the discovery chain
targets.
2021-05-04 08:45:19 -06:00
Daniel Nephin a07a58a873 config-entry: use custom MarshalJSON for mesh type
So that the Kind field is added to the JSON object.
2021-04-29 17:49:43 -04:00
Daniel Nephin 62efaaab21 config-entry: remove Kind and Name field from Mesh config entry
No config entry needs a Kind field. It is only used to determine the Go type to
target. As we introduce new config entries (like this one) we can remove the kind field
and have the GetKind method return the single supported value.

In this case (similar to proxy-defaults) the Name field is also unnecessary. We always
use the same value. So we can omit the name field entirely.
2021-04-29 17:11:21 -04:00
Freddy 701b49f772
Rename cluster config files to mesh as well (#10148) 2021-04-28 18:15:26 -06:00
Freddy 078c40425f
Rename "cluster" config entry to "mesh" (#10127)
This config entry is being renamed primarily because in k8s the name
cluster could be confusing given that the config entry applies across
federated datacenters.

Additionally, this config entry will only apply to Consul as a service
mesh, so the more generic "cluster" name is not needed.
2021-04-28 16:13:29 -06:00
Daniel Nephin 2e4dc7b705
Merge pull request #10045 from hashicorp/dnephin/state-proxy-defaults
state: remove config-entries kind index
2021-04-20 17:12:50 -04:00
Freddy f265dd635f
Disallow * as service-defaults name (#10069) 2021-04-19 14:23:01 -06:00
R.B. Boyer 70f1ba3a2b
fix test when run in enterprise (#10048) 2021-04-15 17:25:25 -05:00
freddygv a0f3591aee Don't panic on nil UpstreamConfiguration.Clone() 2021-04-15 15:46:21 -06:00
Daniel Nephin b57b3726d2 state: remove unnecessary kind index
The query can be performed using a prefix query on the ID index.

Also backport some enterprise changes to prevent conflicts.
2021-04-15 17:37:28 -04:00
Freddy 3be304be16
Merge pull request #10016 from hashicorp/topology-update 2021-04-15 14:11:23 -06:00
Freddy 439a7fce2d
Split Upstream.Identifier() so non-empty namespace is always prepended in ent (#10031) 2021-04-15 13:54:40 -06:00
R.B. Boyer 4db8b78854
connect: update centralized upstreams representation in service-defaults (#10015) 2021-04-15 14:21:44 -05:00
Matt Keeler bbf5993534
Move static token resolution into the ACLResolver (#10013) 2021-04-14 12:39:35 -04:00
freddygv 8e74eaa684 Update viz endpoint to include topology from intentions 2021-04-14 10:20:15 -06:00
freddygv e1808af729 Fixup tests 2021-04-13 16:08:41 -06:00
freddygv 7cb3f32672 Convert new tproxy structs in api module into ptrs
This way we avoid serializing these when empty. Otherwise users of the
latest version of the api submodule cannot interact with older versions
of Consul, because a new api client would send keys that the older Consul
doesn't recognize yet.
2021-04-13 12:44:25 -06:00
Freddy 8fc60a6ca6
Merge pull request #10000 from hashicorp/remove-upstream-cfg-validation
Remove zero-value validation of upstream cfg structs
2021-04-13 11:00:02 -06:00
freddygv 932fbddd27 Augment intention decision summary with DefaultAllow mode 2021-04-12 19:32:09 -06:00
freddygv b8ed82b808 Fixup bexpr filtering 2021-04-12 10:17:52 -06:00
freddygv d7c43049fa Remove zero-value validation of upstream cfg structs
The zero value of these flags was already being excluded in the xDS
generation of circuit breaker/outlier detection config.

See: makeThresholdsIfNeeded and ToOutlierDetection.
2021-04-12 10:08:57 -06:00
freddygv 7bd51ff536 Replace TransparentProxy bool with ProxyMode
This PR replaces the original boolean used to configure transparent
proxy mode. It was replaced with a string mode that can be set to:

- "": Empty string is the default for when the setting should be
defaulted from other configuration like config entries.
- "direct": Direct mode is how applications originally opted into the
mesh. Proxy listeners need to be dialed directly.
- "transparent": Transparent mode enables configuring Envoy as a
transparent proxy. Traffic must be captured and redirected to the
inbound and outbound listeners.

This PR also adds a struct for transparent proxy specific configuration.
Initially this is not stored as a pointer. Will revisit that decision
before GA.
2021-04-12 09:35:14 -06:00
freddygv 98ba582797 Fixup mesh gateway docs 2021-04-11 15:48:04 -06:00
Tara Tufano 9deb52e868
add http2 ping health checks (#8431)
* add http2 ping checks

* fix test issue

* add h2ping check to config resources

* add new test and docs for h2ping

* fix grammatical inconsistency in H2PING documentation

* resolve rebase conflicts, add test for h2ping tls verification failure

* api documentation for h2ping

* update test config data with H2PING

* add H2PING to protocol buffers and update changelog

* fix typo in changelog entry
2021-04-09 15:12:10 -04:00
freddygv b21224a4c8 PR comments 2021-04-08 11:16:03 -06:00
freddygv ab752c1c86 Avoid sending zero-value upstream defaults from api 2021-04-07 15:03:42 -06:00
freddygv 986bcccbea Pass down upstream defaults to client proxies
This is needed in case the client proxy is in TransparentProxy mode.
Typically they won't have explicit configuration for every upstream, so
this ensures the settings can be applied to all of them when generating
xDS config.
2021-04-07 09:32:47 -06:00
freddygv 77ead5cca9 Prevent wildcard destinations for proxies and upstreams 2021-04-07 09:32:47 -06:00
freddygv 458eb41be1 Prevent synthetic upstreams without addresses from failing duplicate ip/port validation 2021-04-07 09:32:47 -06:00
R.B. Boyer d4c401b350
missed build tag on this file (#9974) 2021-04-06 13:24:11 -05:00
R.B. Boyer 499fee73b3
connect: add toggle to globally disable wildcard outbound network access when transparent proxy is enabled (#9973)
This adds a new config entry kind "cluster" with a single special name "cluster" where this can be controlled.
2021-04-06 13:19:59 -05:00
Daniel Nephin d879fe581d state: Move UpstreamDownstream to state package 2021-03-18 12:08:59 -04:00
Daniel Nephin 65f5b99247 state: add tests for mesh-topology table indexers 2021-03-18 12:08:57 -04:00
Freddy 1c13aa23f1
Merge pull request #9900 from hashicorp/ent-fixes
Fixup enterprise tests from tproxy changes
2021-03-18 08:33:30 -06:00
Freddy 0bab999fe4
Merge pull request #9899 from hashicorp/wildcard-ixn-oss
Add methods to check intention has wildcard src or dst
2021-03-18 08:33:07 -06:00
freddygv 098b9af901 Fixup enterprise tests from tproxy changes 2021-03-17 23:05:00 -06:00
freddygv 9713e3ba38 Add methods to check intention has wildcard src or dst 2021-03-17 22:15:48 -06:00
freddygv 52bf00de8b Split up normalizing from defaulting values for upstream cfg 2021-03-17 21:37:55 -06:00
freddygv f4f45af6d0 Merge master and fix upstream config protocol defaulting 2021-03-17 21:13:40 -06:00
Freddy 8207b832df
Add TransparentProxy option to proxy definitions 2021-03-17 17:01:45 -06:00
Freddy c664938bae
Add per-upstream configuration to service-defaults 2021-03-17 16:59:51 -06:00
freddygv a54d6a9010 Update proxycfg for transparent proxy 2021-03-17 13:40:39 -06:00
Daniel Nephin 9f03e23e44
Merge pull request #9881 from hashicorp/dnephin/state-index-service-check-nodes
state: convert services.node and checks.node indexes
2021-03-17 14:12:02 -04:00
Daniel Nephin bd6332ae25
Merge pull request #9863 from hashicorp/dnephin/config-entry-kind-name
state: move ConfigEntryKindName
2021-03-17 14:09:39 -04:00
Christopher Broglie f0307c73e5 Add support for configuring TLS ServerName for health checks
Some TLS servers require SNI, but the Golang HTTP client doesn't
include it in the ClientHello when connecting to an IP address. This
change adds a new TLSServerName field to health check definitions to
optionally set it. This fixes #9473.
2021-03-16 18:16:44 -04:00
Daniel Nephin 34eb6c01ff state: convert services.node and checks.node indexes
Using NodeIdentity to share the indexes with both.
2021-03-16 13:00:31 -04:00
freddygv 4bdbcff9c0 Fixup upstream test 2021-03-15 17:20:30 -06:00
freddygv 6090cfcf68 PR comments 2021-03-15 16:02:03 -06:00
freddygv 7df846aa24 Pass MeshGateway config in service config request
ResolveServiceConfig is called by service manager before the proxy
registration is in the catalog. Therefore we should pass proxy
registration flags in the request rather than trying to fetch
them from the state store (where they may not exist yet).
2021-03-15 14:32:13 -06:00
freddygv 8b46d8dcbb Restore old Envoy prefix on escape hatches
This is done because after removing ID and NodeName from
ServiceConfigRequest we will no longer know whether a request coming in
is for a Consul client earlier than v1.10.
2021-03-15 14:12:57 -06:00
freddygv 93c3c1780d Only lowercase the protocol when normalizing 2021-03-15 14:12:15 -06:00
freddygv 41b2ba1e58 Add omitempty across the board for UpstreamConfig 2021-03-15 13:23:18 -06:00
freddygv 756ab4c546 Fixup protobufs and tests 2021-03-11 14:58:59 -07:00
freddygv df1f3995f8 Update service manager to store centrally configured upstreams 2021-03-11 11:37:21 -07:00
freddygv 6fd30d0384 Add TransparentProxy opt to proxy definition 2021-03-11 11:37:21 -07:00
freddygv 306ef7d252 Restore old escape hatch alias 2021-03-11 11:36:35 -07:00
freddygv e3dc2a49df Turn Limits and PassiveHealthChecks into pointers 2021-03-11 11:04:40 -07:00
freddygv acec711a6a Update server-side config resolution and client-side merging 2021-03-10 21:05:11 -07:00
freddygv 1710ec87d2 finish moving UpstreamConfig and related fields to structs pkg 2021-03-10 21:04:13 -07:00
Daniel Nephin b06b3dd8f8 state: move ConfigEntryKindName
Previously this type was defined in structs, but unlike the other types in structs this type
is not used by RPC requests. By moving it to state we can better indicate that this is not
an API type, but part of the state implementation.
2021-03-10 12:27:22 -05:00
Daniel Nephin 71b0f0a7a6 structs: remove EnterpriseMeta.GetNamespace
I added this recently without realizing that the method already existed and was named
NamespaceOrEmpty. Replace all calls to GetNamespace with NamespaceOrEmpty or NamespaceOrDefault
as appropriate.
2021-03-09 15:17:26 -05:00
freddygv 87cde19b4c Create new types for service-defaults upstream cfg 2021-03-08 22:10:27 -07:00
Daniel Nephin 5c8a6311b6
Merge pull request #9703 from pierresouchay/streaming_tags_and_case_insensitive
Streaming filter tags + case insensitive lookups for Service Names
2021-02-26 12:06:26 -05:00
John Cowen 5892e75452
ui: Remove any trailing fullstop/period DNS characters from Gateways UI API (#9752)
Previous to this commit, the API response would include Gateway
Addresses in the form `domain.name.:8080`, which due to the addition of
the port is probably not the expected response.

This commit rightTrims any `.` characters from the end of the domain
before formatting the address to include the port resulting in
`domain.name:8080`
2021-02-25 09:34:47 +00:00
Mark Anderson b9d22f48cd
Add fields to the /acl/auth-methods endpoint. (#9741)
* A GET of the /acl/auth-method/:name endpoint returns the fields
MaxTokenTTL and TokenLocality, while a LIST (/acl/auth-methods) does
not.

The list command returns a filtered subset of the full set. This is
somewhat deliberate, so that secrets aren't shown, but the TTL and
Locality fields aren't (IMO) security critical, and it is useful for
the front end to be able to show them.

For consistency these changes mirror the 'omit empty' and string
representation choices made for the GET call.

This includes changes to the gRPC and API code in the client.

The new output looks similar to this
curl 'http://localhost:8500/v1/acl/auth-methods' | jq '.'

  {
    "MaxTokenTTL": "8m20s",
    "Name": "minikube-ttl-local2",
    "Type": "kubernetes",
    "Description": "minikube auth method",
    "TokenLocality": "local",
    "CreateIndex": 530,
    "ModifyIndex": 530,
    "Namespace": "default"
  }
]

Signed-off-by: Mark Anderson <manderson@hashicorp.com>

* Add changelog

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-02-17 08:16:57 -08:00
Daniel Nephin d1772ae305 structs: rename EnterpriseMeta constructor
To match the Go convention.
2021-02-16 14:45:43 -05:00
Pierre Souchay 6f91085869 Use lower case for serviceName computation of cache keys 2021-02-09 19:19:40 +01:00
R.B. Boyer 03790a1f91
server: add OSS stubs supporting validation of source namespaces in service-intentions config entries (#9527) 2021-01-25 11:27:38 -06:00
Daniel Nephin 29ce5ec575 structs: fix caching of ServiceSpecificRequest when ingress=true
The field was not being included in the cache info key. This would result in a DNS request for
web.service.consul returning the same result as web.ingress.consul, when those results should
not be the same.
2021-01-14 17:01:40 -05:00
kevinkengne 2e7e78999d
add completeness test for types with CacheInfo method (#9480)
include all fields when fuzzing in tests
split tests by struct type

Ensure the new value for the field is different

fuzzer.Fuzz could produce the same value again in some cases.

Use a custom fuzz function for QueryOptions. That type is an embedded struct in the request types
but only one of the fields is important to include in the cache key.

Move enterpriseMetaField to an oss file so that we can change it in enterprise.
2021-01-12 19:45:46 -05:00
Daniel Nephin 2187808803 structs: add tests for String() methods
To show that printing one of these IDs works properly now that the String() method
receiver is no longer a pointer.
2021-01-07 18:47:38 -05:00
Daniel Nephin d113f0e690 structs: Fix printing of IDs
These types are used as values (not pointers) in other structs. Using a pointer receiver causes
problems when the value is printed. fmt will not call the String method if it is passed a value
and the String method has a pointer receiver. By using a value receiver the correct string is printed.

Also remove some unused methods.
2021-01-07 18:47:38 -05:00
Matt Keeler 85e5da53d5
Special case the error returned when we have a Raft leader but are not tracking it in the ServerLookup (#9487)
This can happen when one other node in the cluster such as a client is unable to communicate with the leader server and sees it as failed. When that happens its failing status eventually gets propagated to the other servers in the cluster and eventually this can result in RPCs returning “No cluster leader” error.

That error is misleading and unhelpful for determing the root cause of the issue as its not raft stability but rather and client -> server networking issue. Therefore this commit will add a new error that will be returned in that case to differentiate between the two cases.
2021-01-04 14:05:23 -05:00
R.B. Boyer d921690bfe
acl: global tokens created by auth methods now correctly replicate to secondary datacenters (#9351)
Previously the tokens would fail to insert into the secondary's state
store because the AuthMethod field of the ACLToken did not point to a
known auth method from the primary.
2020-12-09 15:22:29 -06:00
Kyle Havlovitz c4eff420be
Merge pull request #9009 from hashicorp/update-secondary-ca
connect: Fix an issue with updating CA config in a secondary datacenter
2020-11-30 14:49:28 -08:00
Kyle Havlovitz 0bfda4481f Add CA server delegate interface for testing 2020-11-19 20:08:06 -08:00
Kyle Havlovitz 9be7c6401c connect: update some function comments in CA manager 2020-11-17 16:00:19 -08:00
Matt Keeler 66fd23d67f
Refactor to call non-voting servers read replicas (#9191)
Co-authored-by: Kit Patella <kit@jepsen.io>
2020-11-17 10:53:57 -05:00
R.B. Boyer c7233ba871
server: remove config entry CAS in legacy intention API bridge code (#9151)
Change so line-item intention edits via the API are handled via the state store instead of via CAS operations.

Fixes #9143
2020-11-13 14:42:21 -06:00
R.B. Boyer c003871c54
server: break up Intention.Apply monolithic method (#9007)
The Intention.Apply RPC is quite large, so this PR attempts to break it down into smaller functions and dissolves the pre-config-entry approach to the breakdown as it only confused things.
2020-11-13 09:15:39 -06:00
Matt Keeler c048e86bb2
Switch to using the external autopilot module 2020-11-09 09:22:11 -05:00
Daniel Nephin b532e092dc structs: add a namespace test for CheckServiceNode.CanRead 2020-10-30 15:07:04 -04:00
Daniel Nephin c42fe5ae43 subscribe: set the request namespace 2020-10-30 14:34:04 -04:00
Daniel Nephin a5dd2001cf stream: remove Event.Key
Makes Payload a type with FilterByKey so that Payloads can implement
filtering by key. With this approach we don't need to expose a Namespace
field on Event, and we don't need to invest micro formats or require a
bunch of code to be aware of exactly how the key field is encoded.
2020-10-28 16:48:04 -04:00
R.B. Boyer 58387fef0a
server: config entry replication now correctly uses namespaces in comparisons (#9024)
Previously config entries sharing a kind & name but in different
namespaces could occasionally cause "stuck states" in replication
because the namespace fields were ignored during the differential
comparison phase.

Example:

Two config entries written to the primary:

    kind=A,name=web,namespace=bar
    kind=A,name=web,namespace=foo

Under the covers these both get saved to memdb, so they are sorted by
all 3 components (kind,name,namespace) during natural iteration. This
means that before the replication code does it's own incomplete sort,
the underlying data IS sorted by namespace ascending (bar comes before
foo).

After one pass of replication the primary and secondary datacenters have
the same set of config entries present. If
"kind=A,name=web,namespace=bar" were to be deleted, then things get
weird. Before replication the two sides look like:

primary: [
    kind=A,name=web,namespace=foo
]
secondary: [
    kind=A,name=web,namespace=bar
    kind=A,name=web,namespace=foo
]

The differential comparison phase walks these two lists in sorted order
and first compares "kind=A,name=web,namespace=foo" vs
"kind=A,name=web,namespace=bar" and falsely determines they are the SAME
and are thus cause an update of "kind=A,name=web,namespace=foo". Then it
compares "<nothing>" with "kind=A,name=web,namespace=foo" and falsely
determines that the latter should be DELETED.

During reconciliation the deletes are processed before updates, and so
for a brief moment in the secondary "kind=A,name=web,namespace=foo" is
erroneously deleted and then immediately restored.

Unfortunately after this replication phase the final state is identical
to the initial state, so when it loops around again (rate limited) it
repeats the same set of operations indefinitely.
2020-10-23 13:41:54 -05:00
Freddy 9c04cbc40f
Add HasExact to topology endpoint (#9010) 2020-10-23 10:45:41 -06:00
Daniel Nephin 8b601fdcac
Merge pull request #8771 from amenzhinsky/fix-grpc-use-tls-mapping
Fix GRPCUseTLS flag HTTP API mapping
2020-10-21 18:37:11 -04:00
s-christoff 9bb348c6c7
Enhance the output of consul snapshot inspect (#8787) 2020-10-09 14:57:29 -05:00
Freddy 13df5d5bf8
Add protocol to the topology endpoint response (#8868) 2020-10-08 17:31:54 -06:00
Daniel Nephin da6400192b
Merge pull request #8768 from hashicorp/streaming/add-subscribe-service
subscribe: add subscribe service for streaming change events
2020-10-07 21:24:03 -04:00
Freddy da91e999f6
Return intention info in svc topology endpoint (#8853) 2020-10-07 18:35:34 -06:00
Daniel Nephin 21c21191f4 structs: add CheckServiceNode.CanRead
And use it from the subscribe endpoint.
2020-10-07 18:15:13 -04:00
R.B. Boyer 1b413b0444
connect: support defining intentions using layer 7 criteria (#8839)
Extend Consul’s intentions model to allow for request-based access control enforcement for HTTP-like protocols in addition to the existing connection-based enforcement for unspecified protocols (e.g. tcp).
2020-10-06 17:09:13 -05:00
R.B. Boyer a2a8e9c783
connect: intentions are now managed as a new config entry kind "service-intentions" (#8834)
- Upgrade the ConfigEntry.ListAll RPC to be kind-aware so that older
copies of consul will not see new config entries it doesn't understand
replicate down.

- Add shim conversion code so that the old API/CLI method of interacting
with intentions will continue to work so long as none of these are
edited via config entry endpoints. Almost all of the read-only APIs will
continue to function indefinitely.

- Add new APIs that operate on individual intentions without IDs so that
the UI doesn't need to implement CAS operations.

- Add a new serf feature flag indicating support for
intentions-as-config-entries.

- The old line-item intentions way of interacting with the state store
will transparently flip between the legacy memdb table and the config
entry representations so that readers will never see a hiccup during
migration where the results are incomplete. It uses a piece of system
metadata to control the flip.

- The primary datacenter will begin migrating intentions into config
entries on startup once all servers in the datacenter are on a version
of Consul with the intentions-as-config-entries feature flag. When it is
complete the old state store representations will be cleared. We also
record a piece of system metadata indicating this has occurred. We use
this metadata to skip ALL of this code the next time the leader starts
up.

- The secondary datacenters continue to run the old intentions
replicator until all servers in the secondary DC and primary DC support
intentions-as-config-entries (via serf flag). Once this condition it met
the old intentions replicator ceases.

- The secondary datacenters replicate the new config entries as they are
migrated in the primary. When they detect that the primary has zeroed
it's old state store table it waits until all config entries up to that
point are replicated and then zeroes its own copy of the old state store
table. We also record a piece of system metadata indicating this has
occurred. We use this metadata to skip ALL of this code the next time
the leader starts up.
2020-10-06 13:24:05 -05:00
R.B. Boyer 4998a08c56
server: create new memdb table for storing system metadata (#8703)
This adds a new very tiny memdb table and corresponding raft operation
for updating a very small effective map[string]string collection of
"system metadata". This can persistently record a fact about the Consul
state machine itself.

The first use of this feature will come in a later PR.
2020-10-06 10:08:37 -05:00
freddygv 413a894a1a Do not evaluate discovery chain for topology upstreams 2020-10-05 10:24:50 -06:00
freddygv cf7b7fcdd6 Single DB txn for ServiceTopology and other PR comments 2020-10-05 10:24:50 -06:00
freddygv f906b94351 Add func to combine up+downstream queries 2020-10-05 10:24:50 -06:00
freddygv b012d8374e support querying upstreams/downstreams from registrations 2020-10-05 10:24:50 -06:00
Aliaksandr Mianzhynski 74cfba7065 Fix GRPCUseTLS flag HTTP API mapping 2020-09-29 18:29:56 +03:00
freddygv 9fa1b13df9 Resolve conflicts 2020-09-29 08:59:18 -06:00
R.B. Boyer 0fb088aac3
agent: make the json/hcl decoding of ConnectProxyConfig fully work with CamelCase and snake_case (#8741)
Fixes #7418
2020-09-24 13:58:52 -05:00
Paul Banks 7d58901ae8
Fix bad int -> string conversions caught by go vet changes in 1.15 (#8739) 2020-09-24 11:14:07 +01:00
Kyle Havlovitz 35bb09f85c
Merge pull request #8646 from hashicorp/common-intermediate-ttl
Move IntermediateCertTTL to common CA config
2020-09-15 12:03:29 -07:00
freddygv 7fd518ff1d Merge master 2020-09-14 16:17:43 -06:00
freddygv 7b9d1b41d5 Resolve conflicts against master 2020-09-11 18:41:58 -06:00
freddygv 768dbaa68d Add session flag to cookie config 2020-09-11 18:34:03 -06:00
freddygv eab90ea9fa Revert EnvoyConfig nesting 2020-09-11 09:21:43 -06:00
Kyle Havlovitz 2f7210bde2 Move IntermediateCertTTL to common CA config 2020-09-10 00:23:22 -07:00
freddygv cd4cf5161f Update resolver defaulting 2020-09-03 13:08:44 -06:00
freddygv eaa250cc80 Ensure resolver node with LB isn't considered default 2020-09-03 08:55:57 -06:00
freddygv ef877449ce Move valid policies to pkg level 2020-09-02 15:49:03 -06:00
freddygv f81fe6a1a1 Remove LB infix and move injection to xds 2020-09-02 15:13:50 -06:00
R.B. Boyer 119e945c3e
connect: all config entries pick up a meta field (#8596)
Fixes #8595
2020-09-02 14:10:25 -05:00
R.B. Boyer d0f74cd1e8
connect: fix bug in preventing some namespaced config entry modifications (#8601)
Whenever an upsert/deletion of a config entry happens, within the open
state store transaction we speculatively test compile all discovery
chains that may be affected by the pending modification to verify that
the write would not create an erroneous scenario (such as splitting
traffic to a subset that did not exist).

If a single discovery chain evaluation references two config entries
with the same kind and name in different namespaces then sometimes the
upsert/deletion would be falsely rejected. It does not appear as though
this bug would've let invalid writes through to the state store so the
correction does not require a cleanup phase.
2020-09-02 10:47:19 -05:00
freddygv 63f79e5f9b Restructure structs and other PR comments 2020-09-02 09:10:50 -06:00
freddygv 0236e169bb Add documentation for resolver LB cfg 2020-08-28 14:46:13 -06:00
freddygv 81115b6eaa Compile down LB policy to disco chain nodes 2020-08-28 13:11:04 -06:00
freddygv ff56a64b08 Add LB policy to service-resolver 2020-08-27 19:44:02 -06:00
Jack 9e1c6727f9
Add http2 and grpc support to ingress gateways (#8458) 2020-08-27 15:34:08 -06:00
R.B. Boyer 74d5df7c7a
xds: use envoy's rbac filter to handle intentions entirely within envoy (#8569) 2020-08-27 12:20:58 -05:00
Daniel Nephin 07ad662131 Fix panic when decoding 'Connect: null'
Surprisingly the json Unmarshal updates the aux pointer to a nil.
2020-08-20 17:52:14 -04:00
Hans Hasselberg a932aafc91
add primary keys to list keyring (#8522)
During gossip encryption key rotation it would be nice to be able to see if all nodes are using the same key. This PR adds another field to the json response from `GET v1/operator/keyring` which lists the primary keys in use per dc. That way an operator can tell when a key was successfully setup as primary key.

Based on https://github.com/hashicorp/serf/pull/611 to add primary key to list keyring output:

```json
[
  {
    "WAN": true,
    "Datacenter": "dc2",
    "Segment": "",
    "Keys": {
      "0OuM4oC3Os18OblWiBbZUaHA7Hk+tNs/6nhNYtaNduM=": 6,
      "SINm887hKTzmMWeBNKTJReaTLX3mBEJKriDyt88Ad+g=": 6
    },
    "PrimaryKeys": {
      "SINm887hKTzmMWeBNKTJReaTLX3mBEJKriDyt88Ad+g=": 6
    },
    "NumNodes": 6
  },
  {
    "WAN": false,
    "Datacenter": "dc2",
    "Segment": "",
    "Keys": {
      "0OuM4oC3Os18OblWiBbZUaHA7Hk+tNs/6nhNYtaNduM=": 8,
      "SINm887hKTzmMWeBNKTJReaTLX3mBEJKriDyt88Ad+g=": 8
    },
    "PrimaryKeys": {
      "SINm887hKTzmMWeBNKTJReaTLX3mBEJKriDyt88Ad+g=": 8
    },
    "NumNodes": 8
  },
  {
    "WAN": false,
    "Datacenter": "dc1",
    "Segment": "",
    "Keys": {
      "0OuM4oC3Os18OblWiBbZUaHA7Hk+tNs/6nhNYtaNduM=": 3,
      "SINm887hKTzmMWeBNKTJReaTLX3mBEJKriDyt88Ad+g=": 8
    },
    "PrimaryKeys": {
      "SINm887hKTzmMWeBNKTJReaTLX3mBEJKriDyt88Ad+g=": 8
    },
    "NumNodes": 8
  }
]
```

I intentionally did not change the CLI output because I didn't find a good way of displaying this information. There are a couple of options that we could implement later:
* add a flag to show the primary keys
* add a flag to show json output

Fixes #3393.
2020-08-18 09:50:24 +02:00
R.B. Boyer e3cd4a8539
connect: use stronger validation that ingress gateways have compatible protocols defined for their upstreams (#8470)
Fixes #8466

Since Consul 1.8.0 there was a bug in how ingress gateway protocol
compatibility was enforced. At the point in time that an ingress-gateway
config entry was modified the discovery chain for each upstream was
checked to ensure the ingress gateway protocol matched. Unfortunately
future modifications of other config entries were not validated against
existing ingress-gateway definitions, such as:

1. create tcp ingress-gateway pointing to 'api' (ok)
2. create service-defaults for 'api' setting protocol=http (worked, but not ok)
3. create service-splitter or service-router for 'api' (worked, but caused an agent panic)

If you were to do these in a different order, it would fail without a
crash:

1. create service-defaults for 'api' setting protocol=http (ok)
2. create service-splitter or service-router for 'api' (ok)
3. create tcp ingress-gateway pointing to 'api' (fail with message about
   protocol mismatch)

This PR introduces the missing validation. The two new behaviors are:

1. create tcp ingress-gateway pointing to 'api' (ok)
2. (NEW) create service-defaults for 'api' setting protocol=http ("ok" for back compat)
3. (NEW) create service-splitter or service-router for 'api' (fail with
   message about protocol mismatch)

In consideration for any existing users that may be inadvertently be
falling into item (2) above, that is now officiall a valid configuration
to be in. For anyone falling into item (3) above while you cannot use
the API to manufacture that scenario anymore, anyone that has old (now
bad) data will still be able to have the agent use them just enough to
generate a new agent/proxycfg error message rather than a panic.
Unfortunately we just don't have enough information to properly fix the
config entries.
2020-08-12 11:19:20 -05:00
Kyle Havlovitz c39a275666 fsm: Fix snapshot bug with restoring node/service/check indexes 2020-08-11 11:49:52 -07:00
freddygv aa6c59dbfc end to end changes to pass gatewayservices to /ui/services/ 2020-07-30 10:21:11 -06:00
Matt Keeler 2ee9fe0a4d
Move generation of the CA Configuration from the agent code into a method on the RuntimeConfig (#8363)
This allows this to be reused elsewhere.
2020-07-23 16:05:28 -04:00
Matt Keeler 2713c0e682
Refactor the agentpb package (#8362)
First move the whole thing to the top-level proto package name.

Secondly change some things around internally to have sub-packages.
2020-07-23 11:24:20 -04:00
Daniel Nephin 5d36f98710 agent/consul: Add support for NotModified to two endpoints
A query made with AllowNotModifiedResponse and a MinIndex, where the
result has the same Index as MinIndex, will return an empty response
with QueryMeta.NotModified set to true.

Co-authored-by: Pierre Souchay <pierresouchay@users.noreply.github.com>
2020-07-02 17:05:46 -04:00
Yury Evtikhov 10361dd210 DNS: add IsErrQueryNotFound function for easier error evaluation 2020-07-01 03:41:44 +01:00
Yury Evtikhov 8d18422f19 DNS: fix agent returning SERVFAIL where NXDOMAIN should be returned 2020-07-01 01:51:21 +01:00
R.B. Boyer 462f0f37ed
connect: various changes to make namespaces for intentions work more like for other subsystems (#8194)
Highlights:

- add new endpoint to query for intentions by exact match

- using this endpoint from the CLI instead of the dump+filter approach

- enforcing that OSS can only read/write intentions with a SourceNS or
  DestinationNS field of "default".

- preexisting OSS intentions with now-invalid namespace fields will
  delete those intentions on initial election or for wildcard namespaces
  an attempt will be made to downgrade them to "default" unless one
  exists.

- also allow the '-namespace' CLI arg on all of the intention subcommands

- update lots of docs
2020-06-26 16:59:15 -05:00
Freddy 10d6e9c458
Split up unused key validation for oss/ent (#8189)
Split up unused key validation in config entry decode for oss/ent.

This is needed so that we can return an informative error in OSS if namespaces are provided.
2020-06-25 13:58:29 -06:00
Chris Piraino df48db0abd
Merge pull request #7932 from hashicorp/ingress/internal-ui-endpoint-multiple-ports
Update gateway-services-nodes API endpoint to allow multiple addresses
2020-06-24 17:11:01 -05:00
Chris Piraino f213d3592a remove obsolete comments about test parallelization 2020-06-24 16:36:13 -05:00
Chris Piraino b3db907bdf Update gateway-services-nodes API endpoint to allow multiple addresses
Previously, we were only returning a single ListenerPort for a single
service. However, we actually allow a single service to be serviced over
multiple ports, as well as allow users to define what hostnames they
expect their services to be contacted over. When no hosts are defined,
we return the default ingress domain for any configured DNS domain.

To show this in the UI, we modify the gateway-services-nodes API to
return a GatewayConfig.Addresses field, which is a list of addresses
over which the specific service can be contacted.
2020-06-24 16:35:23 -05:00
freddygv c791fbc79c Update namespaces subject-verb agreement 2020-06-23 10:57:30 -06:00
freddygv 044d027ff8 Remove break 2020-06-22 19:59:04 -06:00
freddygv 70810b0602 Let users know namespaces are ent only in config entry decode 2020-06-22 19:59:04 -06:00
Daniel Nephin 068b43df90 Enable gofmt simplify
Code changes done automatically with 'gofmt -s -w'
2020-06-16 13:21:11 -04:00
Matt Keeler d3881dd754
ACL Node Identities (#7970)
A Node Identity is very similar to a service identity. Its main targeted use is to allow creating tokens for use by Consul agents that will grant the necessary permissions for all the typical agent operations (node registration, coordinate updates, anti-entropy).

Half of this commit is for golden file based tests of the acl token and role cli output. Another big updates was to refactor many of the tests in agent/consul/acl_endpoint_test.go to use the same style of tests and the same helpers. Besides being less boiler plate in the tests it also uses a common way of starting a test server with ACLs that should operate without any warnings regarding deprecated non-uuid master tokens etc.
2020-06-16 12:54:27 -04:00
freddygv cd927eed5e Remove unused method and fixup docs ref 2020-06-12 13:47:43 -06:00
freddygv 19e3954603 Move compound service names to use ServiceName type 2020-06-12 13:47:43 -06:00
Chris Piraino 6fa48c9512
Allow users to set hosts to the wildcard specifier when TLS is disabled (#8083)
This allows easier demoing/testing of ingress gateways, while still
preserving the validation we have for DNSSANs
2020-06-11 10:03:06 -05:00
Chris Piraino 496e683360
Merge pull request #8064 from hashicorp/ingress/health-query-param
Add API query parameter ?ingress to allow users to find ingress gateways associated to a service
2020-06-09 16:08:28 -05:00
Chris Piraino c1d329c5dd Remove TODO note about ingress API, it is done! 2020-06-09 14:58:30 -05:00
Daniel Nephin 08f1ed16b4
Merge pull request #7900 from hashicorp/dnephin/add-linter-staticcheck-2
intentions: fix a bug in Intention.SetHash
2020-06-09 15:40:20 -04:00
Hans Hasselberg 242994a016
acl: do not resolve local tokens from remote dcs (#8068) 2020-06-09 21:13:09 +02:00
Daniel Nephin c66c533d73
Merge pull request #7964 from hashicorp/dnephin/remove-patch-slice-of-maps-forward-compat
config: Use HookWeakDecodeFromSlice in place of PatchSliceOfMaps
2020-06-08 19:53:04 -04:00
Daniel Nephin 75cbbe2702 config: add HookWeakDecodeFromSlice
Currently opaque config blocks (config entries, and CA provider config) are
modified by PatchSliceOfMaps, making it impossible for these opaque
config sections to contain slices of maps.

In order to fix this problem, any lazy-decoding of these blocks needs to support
weak decoding of []map[string]interface{} to a struct type before
PatchSliceOfMaps is replaces. This is necessary because these config
blobs are persisted, and during an upgrade an older version of Consul
could read one of the new configuration values, which would cause an error.

To support the upgrade path, this commit first introduces the new hooks
for weak decoding of []map[string]interface{} and uses them only in the
lazy-decode paths. That way, in a future release, new style
configuration will be supported by the older version of Consul.

This decode hook has a number of advantages:

1. It no longer panics. It allows mapstructure to report the error
2. It no longer requires the user to declare which fields are slices of
   structs. It can deduce that information from the 'to' value.
3. It will make it possible to preserve opaque configuration, allowing
   for structured opaque config.
2020-06-08 17:05:09 -04:00
Hans Hasselberg 98eea08d3b
Tokens converted from legacy ACLs get their Hash computed (#8047)
* Fixes #5606: Tokens converted from legacy ACLs get their Hash computed

This allows new style token replication to work for legacy tokens as well when they change.

* tests: fix timestamp comparison

Co-authored-by: Matt Keeler <mjkeeler7@gmail.com>
2020-06-08 21:44:06 +02:00
Daniel Nephin ce6cc094a1 intentions: fix a bug in Intention.SetHash
Found using staticcheck.

binary.Write does not accept int types without a size. The error from binary.Write was ignored, so we never saw this error. Casting the data to uint64 produces a correct hash.

Also deprecate the Default{Addr,Port} fields, and prevent them from being encoded. These fields will always be empty and are not used.
Removing these would break backwards compatibility, so they are left in place for now.

Co-authored-by: Hans Hasselberg <me@hans.io>
2020-06-05 14:51:43 -04:00
Daniel Nephin 99eb583ebc
Replace goe/verify.Values with testify/require.Equal (#7993)
* testing: replace most goe/verify.Values with require.Equal

One difference between these two comparisons is that go/verify considers
nil slices/maps to be equal to empty slices/maps, where as testify/require
does not, and does not appear to provide any way to enable that behaviour.

Because of this difference some expected values were changed from empty
slices to nil slices, and some calls to verify.Values were left.

* Remove github.com/pascaldekloe/goe/verify

Reduce the number of assertion packages we use from 2 to 1
2020-06-02 12:41:25 -04:00
R.B. Boyer 833211c14c
acl: allow auth methods created in the primary datacenter to optionally create global tokens (#7899) 2020-06-01 11:44:47 -05:00
Jono Sosulska c554ba9e10
Replace whitelist/blacklist terminology with allowlist/denylist (#7971)
* Replace whitelist/blacklist terminology with allowlist/denylist
2020-05-29 14:19:16 -04:00
Daniel Nephin c88fae0aac ci: Add staticcheck and fix most errors
Three of the checks are temporarily disabled to limit the size of the
diff, and allow us to enable all the other checks in CI.

In a follow up we can fix the issues reported by the other checks one
at a time, and enable them.
2020-05-28 11:59:58 -04:00
Daniel Nephin 4f2bff174d
Merge pull request #7963 from hashicorp/dnephin/replace-lib-translate-keys
Replace lib.TranslateKeys with a mapstructure decode hook
2020-05-27 16:51:26 -04:00
Daniel Nephin 6a2d7d77c0 config: use the new HookTranslateKeys instead of lib.TranslateKeys
With the exception of CA provider config, which will be migrated at some
later time.
2020-05-27 16:24:47 -04:00
Daniel Nephin 8ced4300c8 Add alias struct tags for new decode hook 2020-05-27 16:24:47 -04:00
R.B. Boyer 77f2e54618
create lib/stringslice package (#7934) 2020-05-27 11:47:32 -05:00
Daniel Nephin 600645b5f9 Add unconvert linter
To find unnecessary type convertions
2020-05-12 13:47:25 -04:00
Daniel Nephin 47238a693d
Merge pull request #7819 from hashicorp/dnephin/remove-t.Parallel-1
test: Remove t.Parallel() from agent/structs tests
2020-05-12 12:11:57 -04:00
Freddy b3ec383d04
Gateway Services Nodes UI Endpoint (#7685)
The endpoint supports queries for both Ingress Gateways and Terminating Gateways. Used to display a gateway's linked services in the UI.
2020-05-11 11:35:17 -06:00
Kyle Havlovitz 136549205c
Merge pull request #7759 from hashicorp/ingress/tls-hosts
Add TLS option for Ingress Gateway listeners
2020-05-11 09:18:43 -07:00
Kyle Havlovitz 8d140ce9af Disallow the blanket wildcard prefix from being used as custom host 2020-05-08 20:24:18 -07:00
Daniel Nephin e60bb9f102 test: Remove t.Parallel() from agent/structs tests
go test will only run tests in parallel within a single package. In this case the package test run time is exactly the same with or without t.Parallel() (~0.7s).

In generally we should avoid t.Parallel() as it causes a number of problems with `go test` not reporting failure messages correctly. I encountered one of these problems, which is what prompted this change.  Since `t.Parallel` is not providing any benefit in this package, this commit removes it.

The change was automated with:

    git grep -l 't.Parallel' | xargs sed -i -e '/t.Parallel/d'
2020-05-08 14:06:10 -04:00
Freddy c32a4f1ece
Fix up enterprise compatibility for gateways (#7813) 2020-05-08 09:44:34 -06:00
Chris Piraino 0c22eacca8 Add TLS field to ingress API structs
- Adds test in api and command/config/write packages
2020-05-06 15:12:02 -05:00
Chris Piraino 0b9ba9660d Validate hosts input in ingress gateway config entry
We can only allow host names that are valid domain names because we put
these hosts into a DNSSAN. In addition, we validate that the wildcard
specifier '*' is only present as the leftmost label to allow for a
wildcard DNSSAN and associated wildcard Host routing in the ingress
gateway proxy.
2020-05-06 15:12:02 -05:00
Kyle Havlovitz f14c54e25e Add TLS option and DNS SAN support to ingress config
xds: Only set TLS context for ingress listener when requested
2020-05-06 15:12:02 -05:00
Chris Piraino d8517bd6fd Better document wildcard specifier interactions 2020-05-06 15:06:14 -05:00
Kyle Havlovitz f9672f9bf1 Make sure IngressHosts isn't parsed during JSON decode 2020-05-06 15:06:14 -05:00
Chris Piraino f40833d094 Allow Hosts field to be set on an ingress config entry
- Validate that this cannot be set on a 'tcp' listener nor on a wildcard
service.
- Add Hosts field to api and test in consul config write CLI
- xds: Configure envoy with user-provided hosts from ingress gateways
2020-05-06 15:06:13 -05:00
Chris Piraino b73a13fc9e Remove service_subset field from ingress config entry
We decided that this was not a useful MVP feature, and just added
unnecessary complexity
2020-05-06 15:06:13 -05:00
Kyle Havlovitz 247f9eaf13 Allow ingress gateways to route traffic based on Host header
This commit adds the necessary changes to allow an ingress gateway to
route traffic from a single defined port to multiple different upstream
services in the Consul mesh.

To do this, we now require all HTTP requests coming into the ingress
gateway to specify a Host header that matches "<service-name>.*" in
order to correctly route traffic to the correct service.

- Differentiate multiple listener's route names by port
- Adds a case in xds for allowing default discovery chains to create a
  route configuration when on an ingress gateway. This allows default
  services to easily use host header routing
- ingress-gateways have a single route config for each listener
  that utilizes domain matching to route to different services.
2020-05-06 15:06:13 -05:00
R.B. Boyer a854e4d9c5
acl: oss plumbing to support auth method namespace rules in enterprise (#7794)
This includes website docs updates.
2020-05-06 13:48:04 -05:00
R.B. Boyer 22eb016153
acl: add MaxTokenTTL field to auth methods (#7779)
When set to a non zero value it will limit the ExpirationTime of all
tokens created via the auth method.
2020-05-04 17:02:57 -05:00
R.B. Boyer ca52ba7068
acl: add DisplayName field to auth methods (#7769)
Also add a few missing acl fields in the api.
2020-05-04 15:18:25 -05:00
R.B. Boyer b282268408
sdk: extracting testutil.RequireErrorContains from various places it was duplicated (#7753) 2020-05-01 11:56:34 -05:00
Freddy 137a2c32c6
TLS Origination for Terminating Gateways (#7671) 2020-04-27 16:25:37 -06:00
freddygv 915db10903 Avoid deleting mappings for services linked to other gateways on dereg 2020-04-27 11:08:41 -06:00
freddygv c9385129ae Require service:read to read terminating-gateway config 2020-04-27 11:07:06 -06:00
Chris Piraino 115d2d5db5
Expect default enterprise metadata in gateway tests (#7664)
This makes it so that both OSS and enterprise tests pass correctly

In the api tests, explicitly set namespace to empty string so that tests
can be shared.
2020-04-20 09:02:35 -05:00
Kyle Havlovitz e9e8c0e730
Ingress Gateways for TCP services (#7509)
* Implements a simple, tcp ingress gateway workflow

This adds a new type of gateway for allowing Ingress traffic into Connect from external services.

Co-authored-by: Chris Piraino <cpiraino@hashicorp.com>
2020-04-16 14:00:48 -07:00
Daniel Nephin f46d1b5c94 agent/structs: Remove ServiceID.Init and CheckID.Init
The Init method provided the same functionality as the New constructor.
The constructor is both more widely used, and more idiomatic, so remove
the Init method.

This change is in preparation for fixing printing of these IDs.
2020-04-15 12:09:56 -04:00
Freddy 9eb1867fbb
Terminating gateway discovery (#7571)
* Enable discovering terminating gateways

* Add TerminatingGatewayServices to state store

* Use GatewayServices RPC endpoint for ingress/terminating
2020-04-08 12:37:24 -06:00
Freddy aae14b3951
Add decode rules for Expose cfg in service-defaults (#7611) 2020-04-07 19:37:47 -06:00
Matt Keeler 0e7d3d93b3
Enable filtering language support for the v1/connect/intentions… (#7593)
* Enable filtering language support for the v1/connect/intentions listing API

* Update website for filtering of Intentions

* Update website/source/api/connect/intentions.html.md
2020-04-07 11:48:44 -04:00
Freddy 90576060bc
Add config entry for terminating gateways (#7545)
This config entry will be used to configure terminating gateways.

It accepts the name of the gateway and a list of services the gateway will represent.

For each service users will be able to specify: its name, namespace, and additional options for TLS origination.

Co-authored-by: Kyle Havlovitz <kylehav@gmail.com>
Co-authored-by: Chris Piraino <cpiraino@hashicorp.com>
2020-03-31 13:27:32 -06:00
Kyle Havlovitz c911174327
Add config entry/state for Ingress Gateways (#7483)
* Add Ingress gateway config entry and other relevant structs

* Add api package tests for ingress gateways

* Embed EnterpriseMeta into ingress service struct

* Add namespace fields to api module and test consul config write decoding

* Don't require a port for ingress gateways

* Add snakeJSON and camelJSON cases in command test

* Run Normalize on service's ent metadata

Sadly cannot think of a way to test this in OSS.

* Every protocol requires at least 1 service

* Validate ingress protocols

* Update agent/structs/config_entry_gateways.go

Co-authored-by: Chris Piraino <cpiraino@hashicorp.com>
Co-authored-by: Freddy <freddygv@users.noreply.github.com>
2020-03-31 11:59:10 -05:00
Freddy 18d356899c
Enable CLI to register terminating gateways (#7500)
* Enable CLI to register terminating gateways

* Centralize gateway proxy configuration
2020-03-26 10:20:56 -06:00
Alejandro Baez bafa69bb69
Add PolicyReadByName for API (#6615) 2020-03-25 10:34:24 -04:00
R.B. Boyer 85a08bf8ed
server: strip local ACL tokens from RPCs during forwarding if crossing datacenters (#7419)
Fixes #7414
2020-03-10 11:15:22 -05:00
R.B. Boyer 6adad71125
wan federation via mesh gateways (#6884)
This is like a Möbius strip of code due to the fact that low-level components (serf/memberlist) are connected to high-level components (the catalog and mesh-gateways) in a twisty maze of references which make it hard to dive into. With that in mind here's a high level summary of what you'll find in the patch:

There are several distinct chunks of code that are affected:

* new flags and config options for the server

* retry join WAN is slightly different

* retry join code is shared to discover primary mesh gateways from secondary datacenters

* because retry join logic runs in the *agent* and the results of that
  operation for primary mesh gateways are needed in the *server* there are
  some methods like `RefreshPrimaryGatewayFallbackAddresses` that must occur
  at multiple layers of abstraction just to pass the data down to the right
  layer.

* new cache type `FederationStateListMeshGatewaysName` for use in `proxycfg/xds` layers

* the function signature for RPC dialing picked up a new required field (the
  node name of the destination)

* several new RPCs for manipulating a FederationState object:
  `FederationState:{Apply,Get,List,ListMeshGateways}`

* 3 read-only internal APIs for debugging use to invoke those RPCs from curl

* raft and fsm changes to persist these FederationStates

* replication for FederationStates as they are canonically stored in the
  Primary and replicated to the Secondaries.

* a special derivative of anti-entropy that runs in secondaries to snapshot
  their local mesh gateway `CheckServiceNodes` and sync them into their upstream
  FederationState in the primary (this works in conjunction with the
  replication to distribute addresses for all mesh gateways in all DCs to all
  other DCs)

* a "gateway locator" convenience object to make use of this data to choose
  the addresses of gateways to use for any given RPC or gossip operation to a
  remote DC. This gets data from the "retry join" logic in the agent and also
  directly calls into the FSM.

* RPC (`:8300`) on the server sniffs the first byte of a new connection to
  determine if it's actually doing native TLS. If so it checks the ALPN header
  for protocol determination (just like how the existing system uses the
  type-byte marker).

* 2 new kinds of protocols are exclusively decoded via this native TLS
  mechanism: one for ferrying "packet" operations (udp-like) from the gossip
  layer and one for "stream" operations (tcp-like). The packet operations
  re-use sockets (using length-prefixing) to cut down on TLS re-negotiation
  overhead.

* the server instances specially wrap the `memberlist.NetTransport` when running
  with gateway federation enabled (in a `wanfed.Transport`). The general gist is
  that if it tries to dial a node in the SAME datacenter (deduced by looking
  at the suffix of the node name) there is no change. If dialing a DIFFERENT
  datacenter it is wrapped up in a TLS+ALPN blob and sent through some mesh
  gateways to eventually end up in a server's :8300 port.

* a new flag when launching a mesh gateway via `consul connect envoy` to
  indicate that the servers are to be exposed. This sets a special service
  meta when registering the gateway into the catalog.

* `proxycfg/xds` notice this metadata blob to activate additional watches for
  the FederationState objects as well as the location of all of the consul
  servers in that datacenter.

* `xds:` if the extra metadata is in place additional clusters are defined in a
  DC to bulk sink all traffic to another DC's gateways. For the current
  datacenter we listen on a wildcard name (`server.<dc>.consul`) that load
  balances all servers as well as one mini-cluster per node
  (`<node>.server.<dc>.consul`)

* the `consul tls cert create` command got a new flag (`-node`) to help create
  an additional SAN in certs that can be used with this flavor of federation.
2020-03-09 15:59:02 -05:00
Matt Keeler 7584dfe8c8 Fix session backwards incompatibility with 1.6.x and earlier. 2020-03-05 15:34:55 -05:00
Matt Keeler e231d62bc9
Make the config entry and leaf cert cache types ns aware (#7256) 2020-02-10 19:26:01 -05:00
Hans Hasselberg 6739fe6e83
connect: add validations around intermediate cert ttl (#7213) 2020-02-11 00:05:49 +01:00
Akshay Ganeshen 8beb716414
feat: support sending body in HTTP checks (#6602) 2020-02-10 09:27:12 -07:00
Matt Keeler d0cd092e3b
Catalog + Namespace OSS changes. (#7219)
* Various Prepared Query + Namespace things

* Last round of OSS changes for a namespaced catalog
2020-02-10 10:40:44 -05:00
R.B. Boyer 8c596953b0
agent: ensure that we always use the same settings for msgpack (#7245)
We set RawToString=true so that []uint8 => string when decoding an interface{}.
We set the MapType so that map[interface{}]interface{} decodes to map[string]interface{}.

Add tests to ensure that this doesn't break existing usages.

Fixes #7223
2020-02-07 15:50:24 -06:00
Matt Keeler 9e5fd7f925
OSS Changes for various config entry namespacing bugs (#7226) 2020-02-06 10:52:25 -05:00
Matt Keeler dfb0177dbc
Testing updates to support namespaced testing of the agent/xds… (#7185)
* Various testing updates to support namespaced testing of the agent/xds package

* agent/proxycfg package updates to support better namespace testing
2020-02-03 09:26:47 -05:00
Matt Keeler 6855a778c2
Updates to the Txn API for namespaces (#7172)
* Updates to the Txn API for namespaces

* Update agent/consul/txn_endpoint.go

Co-Authored-By: R.B. Boyer <rb@hashicorp.com>

Co-authored-by: R.B. Boyer <public@richardboyer.net>
2020-01-30 13:12:26 -05:00
Chris Piraino 401221de58
Allow users to configure either unstructured or JSON logging (#7130)
* hclog Allow users to choose between unstructured and JSON logging
2020-01-28 17:50:41 -06:00
Matt Keeler c09693e545
Updates to Config Entries and Connect for Namespaces (#7116) 2020-01-24 10:04:58 -05:00
Hans Hasselberg 804eb17094
connect: check if intermediate cert needs to be renewed. (#6835)
Currently when using the built-in CA provider for Connect, root certificates are valid for 10 years, however secondary DCs get intermediates that are valid for only 1 year. There is no mechanism currently short of rotating the root in the primary that will cause the secondary DCs to renew their intermediates.
This PR adds a check that renews the cert if it is half way through its validity period.

In order to be able to test these changes, a new configuration option was added: IntermediateCertTTL which is set extremely low in the tests.
2020-01-17 23:27:13 +01:00
Aestek ba8fd8296f Add support for dual stack IPv4/IPv6 network (#6640)
* Use consts for well known tagged adress keys

* Add ipv4 and ipv6 tagged addresses for node lan and wan

* Add ipv4 and ipv6 tagged addresses for service lan and wan

* Use IPv4 and IPv6 address in DNS
2020-01-17 09:54:17 -05:00
Matt Keeler 663cf1e9a8
AuthMethod updates to support alternate namespace logins (#7029) 2020-01-14 10:09:29 -05:00
Matt Keeler 8bd34e126f
Intentions ACL enforcement updates (#7028)
* Renamed structs.IntentionWildcard to structs.WildcardSpecifier

* Refactor ACL Config

Get rid of remnants of enterprise only renaming.

Add a WildcardName field for specifying what string should be used to indicate a wildcard.

* Add wildcard support in the ACL package

For read operations they can call anyAllowed to determine if any read access to the given resource would be granted.

For write operations they can call allAllowed to ensure that write access is granted to everything.

* Make v1/agent/connect/authorize namespace aware

* Update intention ACL enforcement

This also changes how intention:read is granted. Before the Intention.List RPC would allow viewing an intention if the token had intention:read on the destination. However Intention.Match allowed viewing if access was allowed for either the source or dest side. Now Intention.List and Intention.Get fall in line with Intention.Matches previous behavior.

Due to this being done a few different places ACL enforcement for a singular intention is now done with the CanRead and CanWrite methods on the intention itself.

* Refactor Intention.Apply to make things easier to follow.
2020-01-13 15:51:40 -05:00
R.B. Boyer 10f04a8c4a connect: derive connect certificate serial numbers from a memdb index instead of the provider table max index (#7011) 2020-01-09 16:32:19 +01:00
Matt Keeler fa2003d7cb
Move Session.CheckIDs into OSS only code. (#6993) 2020-01-03 15:51:19 -05:00
Matt Keeler a78f7d7a34
OSS changes for implementing token based namespace inferencing
remove debug log
2019-12-18 14:07:08 -05:00
Matt Keeler 80d13d500b
Miscellaneous acl package cleanup
• Renamed EnterpriseACLConfig to just Config
• Removed chained_authorizer_oss.go as it was empty
• Renamed acl.go to errors.go to more closely describe its contents
2019-12-18 13:44:32 -05:00
Matt Keeler 0b346616e9
Rename EnterpriseAuthorizerContext -> AuthorizerContext 2019-12-18 13:43:24 -05:00
Matt Keeler 5934f803bf
Sync of OSS changes to support namespaces (#6909) 2019-12-09 21:26:41 -05:00
rerorero 34649b8820 [ci] fix: go-fmt fails on master branch (#6906) 2019-12-08 20:30:46 -05:00
Matt Keeler 2343413bf0
Fix the TestAPI_CatalogRegistration test 2019-12-06 15:47:41 -05:00
Matt Keeler 8f0ab0129e
Miscellaneous Fixes (#6896)
Ensure we close the Sentinel Evaluator so as not to leak go routines

Fix a bunch of test logging so that various warnings when starting a test agent go to the ltest logger and not straight to stdout.

Various canned ent meta types always return a valid pointer (no more nils). This allows us to blindly deref + assign in various places.

Update ACL index tracking to ensure oss -> ent upgrades will work as expected.

Update ent meta parsing to include function to disallow wildcarding.
2019-12-06 14:01:34 -05:00
Matt Keeler a704ebe639
Add Namespace support to the API module and the CLI commands (#6874)
Also update the Docs and fixup the HTTP API to return proper errors when someone attempts to use Namespaces with an OSS agent.

Add Namespace HTTP API docs

Make all API endpoints disallow unknown fields
2019-12-06 11:14:56 -05:00
Matt Keeler deb91f3d3c
[Feature] API: Add a internal endpoint to query for ACL authori… (#6888)
* Implement endpoint to query whether the given token is authorized for a set of operations

* Updates to allow for remote ACL authorization via RPC

This is only used when making an authorization request to a different datacenter.
2019-12-06 09:25:26 -05:00
Matt Keeler b069d6777b
OSS KV Modifications to Support Namespaces 2019-11-25 12:57:35 -05:00
Matt Keeler 7b471f6bf8
OSS Modifications necessary for sessions namespacing 2019-11-25 12:07:04 -05:00
Paul Banks cd1b613352
connect: Add AWS PCA provider (#6795)
* Update AWS SDK to use PCA features.

* Add AWS PCA provider

* Add plumbing for config, config validation tests, add test for inheriting existing CA resources created by user

* Unparallel the tests so we don't exhaust PCA limits

* Merge updates

* More aggressive polling; rate limit pass through on sign; Timeout on Sign and CA create

* Add AWS PCA docs

* Fix Vault doc typo too

* Doc typo

* Apply suggestions from code review

Co-Authored-By: R.B. Boyer <rb@hashicorp.com>
Co-Authored-By: kaitlincarter-hc <43049322+kaitlincarter-hc@users.noreply.github.com>

* Doc fixes; tests for erroring if State is modified via API

* More review cleanup

* Uncomment tests!

* Minor suggested clean ups
2019-11-21 17:40:29 +00:00
Paul Banks b621910618
Support Connect CAs that can't cross sign (#6726)
* Support Connect CAs that can't cross sign

* revert spurios mod changes from make tools

* Add log warning when forcing CA rotation

* Fixup SupportsCrossSigning to report errors and work with Plugin interface (fixes tests)

* Fix failing snake_case test

* Remove misleading comment

* Revert "Remove misleading comment"

This reverts commit bc4db9cabed8ad5d0e39b30e1fe79196d248349c.

* Remove misleading comment

* Regen proto files messed up by rebase
2019-11-11 21:36:22 +00:00
Paul Banks 45d57ca601
connect: Allow CA Providers to store small amount of state (#6751)
* pass logger through to provider

* test for proper operation of NeedsLogger

* remove public testServer function

* Ooops actually set the logger in all the places we need it - CA config set wasn't and causing segfault

* Fix all the other places in tests where we set the logger

* Allow CA Providers to persist some state

* Update CA provider plugin interface

* Fix plugin stubs to match provider changes

* Update agent/connect/ca/provider.go

Co-Authored-By: R.B. Boyer <rb@hashicorp.com>

* Cleanup review comments
2019-11-11 20:57:16 +00:00
Matt Keeler ab5a05f71d
Fix type name (#6728) 2019-11-01 16:58:00 -04:00
Matt Keeler 825e19bc5f
Add DirEntry method to fill enterprise authz context 2019-11-01 16:48:44 -04:00
Paul Banks 87699eca2f
Fix support for RSA CA keys in Connect. (#6638)
* Allow RSA CA certs for consul and vault providers to correctly sign EC leaf certs.

* Ensure key type ad bits are populated from CA cert and clean up tests

* Add integration test and fix error when initializing secondary CA with RSA key.

* Add more tests, fix review feedback

* Update docs with key type config and output

* Apply suggestions from code review

Co-Authored-By: R.B. Boyer <rb@hashicorp.com>
2019-11-01 13:20:26 +00:00
Matt Keeler 5d687ce6a9
Fix the Synthetic Policy Tests (#6715) 2019-10-30 15:15:14 -04:00
Sarah Adams 78ad8203a4
Use encoding/json as JSON decoder instead of mapstructure (#6680)
Fixes #6147
2019-10-29 11:13:36 -07:00
Matt Keeler 79f78632e1
Update the ACL Resolver to allow for Consul Enterprise specific hooks. (#6687) 2019-10-25 11:06:16 -04:00
Matt Keeler e4ea9b0a96
Updates to allow for Namespacing ACL resources in Consul Enterp… (#6675)
Main Changes:

• method signature updates everywhere to account for passing around enterprise meta.
• populate the EnterpriseAuthorizerContext for all ACL related authorizations.
• ACL resource listings now operate like the catalog or kv listings in that the returned entries are filtered down to what the token is allowed to see. With Namespaces its no longer all or nothing.
• Modified the acl.Policy parsing to abstract away basic decoding so that enterprise can do it slightly differently. Also updated method signatures so that when parsing a policy it can take extra ent metadata to use during rules validation and policy creation.

Secondary Changes:

• Moved protobuf encoding functions out of the agentpb package to eliminate circular dependencies.
• Added custom JSON unmarshalers for a few ACL resource types (to support snake case and to get rid of mapstructure)
• AuthMethod validator cache is now an interface as these will be cached per-namespace for Consul Enterprise.
• Added checks for policy/role link existence at the RPC API so we don’t push the request through raft to have it fail internally.
• Forward ACL token delete request to the primary datacenter when the secondary DC doesn’t have the token.
• Added a bunch of ACL test helpers for inserting ACL resource test data.
2019-10-24 14:38:09 -04:00
Freddy 60f6ec0c2f
Store check type in catalog (#6561) 2019-10-17 20:33:11 +02:00
Matt Keeler 973341a592
ACL Authorizer overhaul (#6620)
* ACL Authorizer overhaul

To account for upcoming features every Authorization function can now take an extra *acl.EnterpriseAuthorizerContext. These are unused in OSS and will always be nil.

Additionally the acl package has received some thorough refactoring to enable all of the extra Consul Enterprise specific authorizations including moving sentinel enforcement into the stubbed structs. The Authorizer funcs now return an acl.EnforcementDecision instead of a boolean. This improves the overall interface as it makes multiple Authorizers easily chainable as they now indicate whether they had an authoritative decision or should use some other defaults. A ChainedAuthorizer was added to handle this Authorizer enforcement chain and will never itself return a non-authoritative decision.

* Include stub for extra enterprise rules in the global management policy

* Allow for an upgrade of the global-management policy
2019-10-15 16:58:50 -04:00
PHBourquin 039615641e Checks to passing/critical only after reaching a consecutive success/failure threshold (#5739)
A check may be set to become passing/critical only if a specified number of successive
checks return passing/critical in a row. Status will stay identical as before until
the threshold is reached.
This feature is available for HTTP, TCP, gRPC, Docker & Monitor checks.
2019-10-14 21:49:49 +01:00
R.B. Boyer c4b92d5534
connect: connect CA Roots in secondary datacenters should use a SigningKeyID derived from their local intermediate (#6513)
This fixes an issue where leaf certificates issued in secondary
datacenters would be reissued very frequently (every ~20 seconds)
because the logic meant to detect root rotation was errantly triggering
because a hash of the ultimate root (in the primary) was being compared
against a hash of the local intermediate root (in the secondary) and
always failing.
2019-09-26 11:54:14 -05:00
Matt Keeler 76cf54068b
Expand the QueryOptions and QueryMeta interfaces (#6545)
In a previous PR I made it so that we had interfaces that would work enough to allow blockingQueries to work. However to complete this we need all fields to be settable and gettable.

Notes:
   • If Go ever gets contracts/generics then we could get rid of all the Getters/Setters
   • protoc / protoc-gen-gogo are going to generate all the getters for us.
   • I copied all the getters/setters from the protobuf funcs into agent/structs/protobuf_compat.go
   • Also added JSON marshaling funcs that use jsonpb for protobuf types.
2019-09-26 09:55:02 -04:00
Freddy fdd10dd8b8
Expose HTTP-based paths through Connect proxy (#6446)
Fixes: #5396

This PR adds a proxy configuration stanza called expose. These flags register
listeners in Connect sidecar proxies to allow requests to specific HTTP paths from outside of the node. This allows services to protect themselves by only
listening on the loopback interface, while still accepting traffic from non
Connect-enabled services.

Under expose there is a boolean checks flag that would automatically expose all
registered HTTP and gRPC check paths.

This stanza also accepts a paths list to expose individual paths. The primary
use case for this functionality would be to expose paths for third parties like
Prometheus or the kubelet.

Listeners for requests to exposed paths are be configured dynamically at run
time. Any time a proxy, or check can be registered, a listener can also be
created.

In this initial implementation requests to these paths are not
authenticated/encrypted.
2019-09-25 20:55:52 -06:00
Matt Keeler 51dcd126b7
Add support for implementing new requests with protobufs instea… (#6502)
* Add build system support for protobuf generation

This is done generically so that we don’t have to keep updating the makefile to add another proto generation.

Note: anything not in the vendor directory and with a .proto extension will be run through protoc if the corresponding namespace.pb.go file is not up to date.

If you want to rebuild just a single proto file you can do so with: make proto-rebuild PROTOFILES=<list of proto files to rebuild>

Providing the PROTOFILES var will override the default behavior of finding all the .proto files.

* Start adding types to the agent/proto package

These will be needed for some other work and are by no means comprehensive.

* Add ability to resolve/fixup the agentpb.ACLLinks structure in the state store.

* Use protobuf marshalling of raft requests instead of msgpack for protoc generated types.

This does not change any encoding of existing types.

* Removed structs package automatically encoding with protobuf marshalling

Instead the caller of raftApply that wants to opt-in to protobuf encoding will have to call `raftApplyProtobuf`

* Run update-vendor to fixup modules.txt

Nothing changed as far as dependencies go but the ordering of modules in that file depends on the time they are first seen and its not alphabetical.

* Rename some things and implement the structs.RPCInfo interface bits

agentpb.QueryOptions and agentpb.WriteRequest implement 3 of the 4 RPCInfo funcs and the new TargetDatacenter message type implements the fourth.

* Use the right encoding function.

* Renamed agent/proto package to agent/agentpb to prevent package name conflicts

* Update modules.txt to fix ordering

* Change blockingQuery to take in interfaces for the query options and meta

* Add %T to error output.

* Add/Update some comments
2019-09-20 14:37:22 -04:00
Pierre Souchay be50400c62 Distinguish between DC not existing and not being available (#6399) 2019-09-03 09:46:24 -06:00
R.B. Boyer fd1c62ee8b
connect: ensure time.Duration fields retain their human readable forms in the API (#6348)
This applies for both config entries and the compiled discovery chain.

Also omit some other config entries fields when empty.
2019-08-19 15:31:05 -05:00
R.B. Boyer 561b2fe606
connect: generate the full SNI names for discovery targets in the compiler rather than in the xds package (#6340) 2019-08-19 13:03:03 -05:00
R.B. Boyer ae79cdab1b
connect: introduce ExternalSNI field on service-defaults (#6324)
Compiling this will set an optional SNI field on each DiscoveryTarget.
When set this value should be used for TLS connections to the instances
of the target. If not set the default should be used.

Setting ExternalSNI will disable mesh gateway use for that target. It also 
disables several service-resolver features that do not make sense for an 
external service.
2019-08-19 12:19:44 -05:00
R.B. Boyer 1a485011d0
connect: updating a service-defaults config entry should leave an unset protocol alone (#6342)
If the entry is updated for reasons other than protocol it is surprising
that the value is explicitly persisted as 'tcp' rather than leaving it
empty and letting it fall back dynamically on the proxy-defaults value.
2019-08-19 10:44:06 -05:00
R.B. Boyer 3975cb89bf
agent: blocking central config RPCs iterations should not interfere with each other (#6316) 2019-08-14 09:08:46 -05:00
hashicorp-ci 5919c7c184 Merge Consul OSS branch 'master' at commit 8f7586b339 2019-08-13 02:00:43 +00:00
Sarah Adams 8ff1f481fe
add flag to allow /operator/keyring requests to only hit local servers (#6279)
Add parameter local-only to operator keyring list requests to force queries to only hit local servers (no WAN traffic).

HTTP API: GET /operator/keyring?local-only=true
CLI: consul keyring -list --local-only

Sending the local-only flag with any non-GET/list request will result in an error.
2019-08-12 11:11:11 -07:00
Mike Morris 65be58703c
connect: remove managed proxies (#6220)
* connect: remove managed proxies implementation and all supporting config options and structs

* connect: remove deprecated ProxyDestination

* command: remove CONNECT_PROXY_TOKEN env var

* agent: remove entire proxyprocess proxy manager

* test: remove all managed proxy tests

* test: remove irrelevant managed proxy note from TestService_ServerTLSConfig

* test: update ContentHash to reflect managed proxy removal

* test: remove deprecated ProxyDestination test

* telemetry: remove managed proxy note

* http: remove /v1/agent/connect/proxy endpoint

* ci: remove deprecated test exclusion

* website: update managed proxies deprecation page to note removal

* website: remove managed proxy configuration API docs

* website: remove managed proxy note from built-in proxy config

* website: add note on removing proxy subdirectory of data_dir
2019-08-09 15:19:30 -04:00
R.B. Boyer 8e22d80e35
connect: fix failover through a mesh gateway to a remote datacenter (#6259)
Failover is pushed entirely down to the data plane by creating envoy
clusters and putting each successive destination in a different load
assignment priority band. For example this shows that normally requests
go to 1.2.3.4:8080 but when that fails they go to 6.7.8.9:8080:

- name: foo
  load_assignment:
    cluster_name: foo
    policy:
      overprovisioning_factor: 100000
    endpoints:
    - priority: 0
      lb_endpoints:
      - endpoint:
          address:
            socket_address:
              address: 1.2.3.4
              port_value: 8080
    - priority: 1
      lb_endpoints:
      - endpoint:
          address:
            socket_address:
              address: 6.7.8.9
              port_value: 8080

Mesh gateways route requests based solely on the SNI header tacked onto
the TLS layer. Envoy currently only lets you configure the outbound SNI
header at the cluster layer.

If you try to failover through a mesh gateway you ideally would
configure the SNI value per endpoint, but that's not possible in envoy
today.

This PR introduces a simpler way around the problem for now:

1. We identify any target of failover that will use mesh gateway mode local or
   remote and then further isolate any resolver node in the compiled discovery
   chain that has a failover destination set to one of those targets.

2. For each of these resolvers we will perform a small measurement of
   comparative healths of the endpoints that come back from the health API for the
   set of primary target and serial failover targets. We walk the list of targets
   in order and if any endpoint is healthy we return that target, otherwise we
   move on to the next target.

3. The CDS and EDS endpoints both perform the measurements in (2) for the
   affected resolver nodes.

4. For CDS this measurement selects which TLS SNI field to use for the cluster
   (note the cluster is always going to be named for the primary target)

5. For EDS this measurement selects which set of endpoints will populate the
   cluster. Priority tiered failover is ignored.

One of the big downsides to this approach to failover is that the failover
detection and correction is going to be controlled by consul rather than
deferring that entirely to the data plane as with the prior version. This also
means that we are bound to only failover using official health signals and
cannot make use of data plane signals like outlier detection to affect
failover.

In this specific scenario the lack of data plane signals is ok because the
effectiveness is already muted by the fact that the ultimate destination
endpoints will have their data plane signals scrambled when they pass through
the mesh gateway wrapper anyway so we're not losing much.

Another related fix is that we now use the endpoint health from the
underlying service, not the health of the gateway (regardless of
failover mode).
2019-08-05 13:30:35 -05:00
R.B. Boyer c395affc93
connect: expose an API endpoint to compile the discovery chain (#6248)
In addition to exposing compilation over the API cleaned up the structures that would be exchanged to be cleaner and easier to support and understand.

Also removed ability to configure the envoy OverprovisioningFactor.
2019-08-02 15:34:54 -05:00
R.B. Boyer dcb609af83
connect: detect and prevent circular discovery chain references (#6246) 2019-08-02 09:18:45 -05:00
R.B. Boyer f02924fafe
connect: simplify the compiled discovery chain data structures (#6242)
This should make them better for sending over RPC or the API.

Instead of a chain implemented explicitly like a linked list (nodes
holding pointers to other nodes) instead switch to a flat map of named
nodes with nodes linking other other nodes by name. The shipped
structure is just a map and a string to indicate which key to start
from.

Other changes:

* inline the compiler option InferDefaults as true

* introduce compiled target config to avoid needing to send back
  additional maps of Resolvers; future target-specific compiled state
  can go here

* move compiled MeshGateway out of the Resolver and into the
  TargetConfig where it makes more sense.
2019-08-01 22:44:05 -05:00
R.B. Boyer 6393edba53
connect: reconcile how upstream configuration works with discovery chains (#6225)
* connect: reconcile how upstream configuration works with discovery chains

The following upstream config fields for connect sidecars sanely
integrate into discovery chain resolution:

- Destination Namespace/Datacenter: Compilation occurs locally but using
different default values for namespaces and datacenters. The xDS
clusters that are created are named as they normally would be.

- Mesh Gateway Mode (single upstream): If set this value overrides any
value computed for any resolver for the entire discovery chain. The xDS
clusters that are created may be named differently (see below).

- Mesh Gateway Mode (whole sidecar): If set this value overrides any
value computed for any resolver for the entire discovery chain. If this
is specifically overridden for a single upstream this value is ignored
in that case. The xDS clusters that are created may be named differently
(see below).

- Protocol (in opaque config): If set this value overrides the value
computed when evaluating the entire discovery chain. If the normal chain
would be TCP or if this override is set to TCP then the result is that
we explicitly disable L7 Routing and Splitting. The xDS clusters that
are created may be named differently (see below).

- Connect Timeout (in opaque config): If set this value overrides the
value for any resolver in the entire discovery chain. The xDS clusters
that are created may be named differently (see below).

If any of the above overrides affect the actual result of compiling the
discovery chain (i.e. "tcp" becomes "grpc" instead of being a no-op
override to "tcp") then the relevant parameters are hashed and provided
to the xDS layer as a prefix for use in naming the Clusters. This is to
ensure that if one Upstream discovery chain has no overrides and
tangentially needs a cluster named "api.default.XXX", and another
Upstream does have overrides for "api.default.XXX" that they won't
cross-pollinate against the operator's wishes.

Fixes #6159
2019-08-01 22:03:34 -05:00
R.B. Boyer 8564b6bb38
connect: validate upstreams and prevent duplicates (#6224)
* connect: validate upstreams and prevent duplicates

* Actually run Upstream.Validate() instead of ignoring it as dead code.

* Prevent two upstreams from declaring the same bind address and port.
  It wouldn't work anyway.

* Prevent two upstreams from being declared that use the same
  type+name+namespace+datacenter. Due to how the Upstream.Identity()
  function worked this ended up mostly being enforced in xDS at use-time,
  but it should be enforced more clearly at register-time.
2019-08-01 13:26:02 -05:00
Paul Banks e87cef2bb8 Revert "connect: support AWS PCA as a CA provider" (#6251)
This reverts commit 3497b7c00d.
2019-07-31 09:08:10 -04:00
Todd Radel 3497b7c00d
connect: support AWS PCA as a CA provider (#6189)
Port AWS PCA provider from consul-ent
2019-07-30 22:57:51 -04:00
Todd Radel 2552f4a11a
connect: Support RSA keys in addition to ECDSA (#6055)
Support RSA keys in addition to ECDSA
2019-07-30 17:47:39 -04:00
R.B. Boyer c6c4a2251a Merge Consul OSS branch master at commit b3541c4f34 2019-07-26 10:34:24 -05:00
Jeff Mitchell 94c73d0c92 Chunking support (#6172)
* Initial chunk support

This uses the go-raft-middleware library to allow for chunked commits to the KV
2019-07-24 17:06:39 -04:00
Matt Keeler 3053342198
Envoy Mesh Gateway integration tests (#6187)
* Allow setting the mesh gateway mode for an upstream in config files

* Add envoy integration test for mesh gateways

This necessitated many supporting changes in most of the other test cases.

Add remote mode mesh gateways integration test
2019-07-24 17:01:42 -04:00
R.B. Boyer ad9e7b6ae9
connect: allow L7 routers to match on http methods (#6164)
Fixes #6158
2019-07-23 20:56:39 -05:00
R.B. Boyer 85cf2706e6
connect: change router syntax for matching query parameters to resemble the syntax for matching paths and headers for consistency. (#6163)
This is a breaking change, but only in the context of the beta series.
2019-07-23 20:55:26 -05:00
R.B. Boyer 1dbd92e091
connect: validate and test more of the L7 config entries (#6156) 2019-07-23 20:50:23 -05:00
R.B. Boyer e039dfd7f8
connect: rework how the service resolver subset OnlyPassing flag works (#6173)
The main change is that we no longer filter service instances by health,
preferring instead to render all results down into EDS endpoints in
envoy and merely label the endpoints as HEALTHY or UNHEALTHY.

When OnlyPassing is set to true we will force consul checks in a
'warning' state to render as UNHEALTHY in envoy.

Fixes #6171
2019-07-23 20:20:24 -05:00
Matt Keeler d7fe8befa9
Update go-bexpr (#6190)
* Update go-bexpr to v0.1.1

This brings in:

• `in`/`not in` operators to do substring matching
• `matches` / `not matches` operators to perform regex string matching.

* Add the capability to auto-generate the filtering selector ops tables for our docs
2019-07-23 14:45:20 -04:00
Matt Keeler 4728329aeb
Various Gateway Fixes (#6093)
* Ensure the mesh gateway configuration comes back in the api within each upstream

* Add a test for the MeshGatewayConfig in the ToAPI functions

* Ensure we don’t use gateways for dc local connections

* Update the svc kind index for deletions

* Replace the proxycfg.state cache with an interface for testing

Also start implementing proxycfg state testing.

* Update the state tests to verify some gateway watches for upstream-targets of a discovery chain.
2019-07-12 17:19:37 -04:00
R.B. Boyer bcd2de3a2e
implement some missing service-router features and add more xDS testing (#6065)
- also implement OnlyPassing filters for non-gateway clusters
2019-07-12 14:16:21 -05:00
R.B. Boyer 9138a97054
Fix bug in service-resolver redirects if the destination uses a default resolver. (#6122)
Also:
- add back an internal http endpoint to dump a compiled discovery chain for debugging purposes

Before the CompiledDiscoveryChain.IsDefault() method would test:

- is this chain just one resolver step?
- is that resolver step just the default?

But what I forgot to test:

- is that resolver step for the same service that the chain represents?

This last point is important because if you configured just one config
entry:

    kind = "service-resolver"
    name = "web"
    redirect {
      service = "other"
    }

and requested the chain for "web" you'd get back a **default** resolver
for "other".  In the xDS code the IsDefault() method is used to
determine if this chain is "empty". If it is then we use the
pre-discovery-chain logic that just uses data embedded in the Upstream
object (and still lets the escape hatches function).

In the example above that means certain parts of the xDS code were going
to try referencing a cluster named "web..." despite the other parts of
the xDS code maintaining clusters named "other...".
2019-07-12 12:21:25 -05:00
R.B. Boyer 67a36e3452
handle structs.ConfigEntry decoding similarly to api.ConfigEntry decoding (#6106)
Both 'consul config write' and server bootstrap config entries take a
decoding detour through mapstructure on the way from HCL to an actual
struct. They both may take in snake_case or CamelCase (for consistency)
so need very similar handling.

Unfortunately since they are operating on mirror universes of structs
(api.* vs structs.*) the code cannot be identitical, so try to share the
kind-configuration and duplicate the rest for now.
2019-07-12 12:20:30 -05:00
Matt Keeler 6e65811db2
Envoy CLI bind addresses (#6107)
* Ensure we MapWalk the proxy config in the NodeService and ServiceNode structs

This gets rid of some json encoder errors in the catalog endpoints

* Allow passing explicit bind addresses to envoy

* Move map walking to the ConnectProxyConfig struct

Any place where this struct gets JSON encoded will benefit as opposed to having to implement it everywhere.

* Fail when a non-empty address is provided and not bindable

* camel case

* Update command/connect/envoy/envoy.go

Co-Authored-By: Paul Banks <banks@banksco.de>
2019-07-12 12:57:31 -04:00
Matt Keeler 3eb3ee5a15
Merge pull request #6053 from hashicorp/gateways_and_resolvers
Integrate Mesh Gateways with ServiceResolverSubsets
2019-07-02 12:05:08 -04:00
R.B. Boyer 43770b9391
digest the proxy-defaults protocol into the graph (#6050) 2019-07-02 11:01:17 -05:00
Matt Keeler 3b6d5e382a Implement caching for config entry lists
Update agent/cache-types/config_entry.go

Co-Authored-By: R.B. Boyer <public@richardboyer.net>
2019-07-02 10:11:19 -04:00
R.B. Boyer 4bdb690a25
activate most discovery chain features in xDS for envoy (#6024) 2019-07-01 22:10:51 -05:00
Matt Keeler bdebe62fd0
Fix some tests that I broke when refactoring the ConfigSnapshot (#6051)
* Fix some tests that I broke when refactoring the ConfigSnapshot

* Make sure the MeshGateway config is added to all the right api structs

* Fix some more tests
2019-07-01 19:47:58 -04:00
Matt Keeler 8d953f5840 Implement Mesh Gateways
This includes both ingress and egress functionality.
2019-07-01 16:28:30 -04:00
Matt Keeler 4bc1277315 Include a content hash of the intention for use during replication 2019-07-01 16:28:30 -04:00
Matt Keeler 3943e38133 Implement Kind based ServiceDump and caching of the ServiceDump RPC 2019-07-01 16:28:30 -04:00
R.B. Boyer 2ad516aeaf
do some initial config entry graph validation during writes (#6047) 2019-07-01 15:23:36 -05:00
hashicorp-ci 43bda6fb76 Merge Consul OSS branch 'master' at commit e91f73f592 2019-06-30 02:00:31 +00:00
Hans Hasselberg 33a7df3330
tls: auto_encrypt enables automatic RPC cert provisioning for consul clients (#5597) 2019-06-27 22:22:07 +02:00
R.B. Boyer 6a52f9f9fb
initial version of L7 config entry compiler (#5994)
With this you should be able to fetch all of the relevant discovery
chain config entries from the state store in one query and then feed
them into the compiler outside of a transaction.

There are a lot of TODOs scattered through here, but they're mostly
around handling fun edge cases and can be deferred until more of the
plumbing works completely.
2019-06-27 13:38:21 -05:00
R.B. Boyer ceef44bbc9
adding new config entries for L7 discovery chain (unused) (#5987) 2019-06-27 12:37:43 -05:00
hashicorp-ci f4304e2e5b Merge Consul OSS branch 'master' at commit 4eb73973b6 2019-06-27 02:00:41 +00:00
Pierre Souchay 0e907f5aa8 Support for maximum size for Output of checks (#5233)
* Support for maximum size for Output of checks

This PR allows users to limit the size of output produced by checks at the agent 
and check level.

When set at the agent level, it will limit the output for all checks monitored
by the agent.

When set at the check level, it can override the agent max for a specific check but
only if it is lower than the agent max.

Default value is 4k, and input must be at least 1.
2019-06-26 09:43:25 -06:00
Matt Keeler 43c5ba0304
New Cache Types (#5995)
* Add a cache type for the Catalog.ListServices endpoint

* Add a cache type for the Catalog.ListDatacenters endpoint
2019-06-24 14:11:34 -04:00
Aestek b839f52195 kv: do not trigger watches when setting the same value (#5885)
If a KVSet is performed but does not update the entry, do not trigger
watches for this key.
This avoids releasing blocking queries for KV values that did not
actually changed.
2019-06-18 15:06:29 +02:00
Matt Keeler f3d9b999ee
Add tagged addresses for services (#5965)
This allows addresses to be tagged at the service level similar to what we allow for nodes already. The address translation that can be enabled with the `translate_wan_addrs` config was updated to take these new addresses into account as well.
2019-06-17 10:51:50 -04:00
R.B. Boyer 40336fd353
agent: fix several data races and bugs related to node-local alias checks (#5876)
The observed bug was that a full restart of a consul datacenter (servers
and clients) in conjunction with a restart of a connect-flavored
application with bring-your-own-service-registration logic would very
frequently cause the envoy sidecar service check to never reflect the
aliased service.

Over the course of investigation several bugs and unfortunate
interactions were corrected:

(1)

local.CheckState objects were only shallow copied, but the key piece of
data that gets read and updated is one of the things not copied (the
underlying Check with a Status field). When the stock code was run with
the race detector enabled this highly-relevant-to-the-test-scenario field
was found to be racy.

Changes:

 a) update the existing Clone method to include the Check field
 b) copy-on-write when those fields need to change rather than
    incrementally updating them in place.

This made the observed behavior occur slightly less often.

(2)

If anything about how the runLocal method for node-local alias check
logic was ever flawed, there was no fallback option. Those checks are
purely edge-triggered and failure to properly notice a single edge
transition would leave the alias check incorrect until the next flap of
the aliased check.

The change was to introduce a fallback timer to act as a control loop to
double check the alias check matches the aliased check every minute
(borrowing the duration from the non-local alias check logic body).

This made the observed behavior eventually go away when it did occur.

(3)

Originally I thought there were two main actions involved in the data race:

A. The act of adding the original check (from disk recovery) and its
   first health evaluation.

B. The act of the HTTP API requests coming in and resetting the local
   state when re-registering the same services and checks.

It took awhile for me to realize that there's a third action at work:

C. The goroutines associated with the original check and the later
   checks.

The actual sequence of actions that was causing the bad behavior was
that the API actions result in the original check to be removed and
re-added _without waiting for the original goroutine to terminate_. This
means for brief windows of time during check definition edits there are
two goroutines that can be sending updates for the alias check status.

In extremely unlikely scenarios the original goroutine sees the aliased
check start up in `critical` before being removed but does not get the
notification about the nearly immediate update of that check to
`passing`.

This is interlaced wit the new goroutine coming up, initializing its
base case to `passing` from the current state and then listening for new
notifications of edge triggers.

If the original goroutine "finishes" its update, it then commits one
more write into the local state of `critical` and exits leaving the
alias check no longer reflecting the underlying check.

The correction here is to enforce that the old goroutines must terminate
before spawning the new one for alias checks.
2019-05-24 13:36:56 -05:00
R.B. Boyer 20eefeea11
acl: a role binding rule for a role that does not exist should be ignored (#5778)
I wrote the docs under this assumption but completely forgot to actually
enforce it.
2019-05-03 14:22:44 -05:00
R.B. Boyer b4371bcccd
acl: enforce that you cannot persist tokens and roles with missing links except during replication (#5779) 2019-05-02 15:02:21 -05:00
Matt Keeler 42d32db817
Fix ConfigEntryResponse binary marshaller and ensure we watch the chan in ConfigEntry.Get even when no entry exists. (#5773) 2019-05-02 15:25:29 -04:00
Paul Banks 8f5b16ebaf
Fix uint8 conversion issues for service config response maps. 2019-05-02 14:11:33 +01:00
Paul Banks 0cfb6051ea Add integration test for central config; fix central config WIP (#5752)
* Add integration test for central config; fix central config WIP

* Add integration test for central config; fix central config WIP

* Set proxy protocol correctly and begin adding upstream support

* Add upstreams to service config cache key and start new notify watcher if they change.

This doesn't update the tests to pass though.

* Fix some merging logic get things working manually with a hack (TODO fix properly)

* Simplification to not allow enabling sidecars centrally - it makes no sense without upstreams anyway

* Test compile again and obvious ones pass. Lots of failures locally not debugged yet but may be flakes. Pushing up to see what CI does

* Fix up service manageer and API test failures

* Remove the enable command since it no longer makes much sense without being able to turn on sidecar proxies centrally

* Remove version.go hack - will make integration test fail until release

* Remove unused code from commands and upstream merge

* Re-bump version to 1.5.0
2019-05-01 16:39:31 -07:00
Matt Keeler 69f902608c
Update to use a consulent build tag instead of just ent (#5759) 2019-05-01 11:11:27 -04:00
Matt Keeler d0f410cd84
Make a few config entry endpoints return 404s and allow for snake_case and lowercase key names. (#5748) 2019-04-30 18:19:19 -04:00
Matt Keeler 4daa1585b0
ACL Token ID Initialization (#5307) 2019-04-30 11:45:36 -04:00
Kyle Havlovitz aba54cec55 Add HTTP endpoints for config entry management (#5718) 2019-04-29 18:08:09 -04:00
Paul Banks 421ecd32fc
Connect: allow configuring Envoy for L7 Observability (#5558)
* Add support for HTTP proxy listeners

* Add customizable bootstrap configuration options

* Debug logging for xDS AuthZ

* Add Envoy Integration test suite with basic test coverage

* Add envoy command tests to cover new cases

* Add tracing integration test

* Add gRPC support WIP

* Merged changes from master Docker. get CI integration to work with same Dockerfile now

* Make docker build optional for integration

* Enable integration tests again!

* http2 and grpc integration tests and fixes

* Fix up command config tests

* Store all container logs as artifacts in circle on fail

* Add retries to outer part of stats measurements as we keep missing them in CI

* Only dump logs on failing cases

* Fix typos from code review

* Review tidying and make tests pass again

* Add debug logs to exec test.

* Fix legit test failure caused by upstream rename in envoy config

* Attempt to reduce cases of bad TLS handshake in CI integration tests

* bring up the right service

* Add prometheus integration test

* Add test for denied AuthZ both HTTP and TCP

* Try ANSI term for Circle
2019-04-29 17:27:57 +01:00
R.B. Boyer e47d7eeddb acl: adding support for kubernetes auth provider login (#5600)
* auth providers
* binding rules
* auth provider for kubernetes
* login/logout
2019-04-26 14:49:25 -05:00
R.B. Boyer cc1aa3f973 acl: adding Roles to Tokens (#5514)
Roles are named and can express the same bundle of permissions that can
currently be assigned to a Token (lists of Policies and Service
Identities). The difference with a Role is that it not itself a bearer
token, but just another entity that can be tied to a Token.

This lets an operator potentially curate a set of smaller reusable
Policies and compose them together into reusable Roles, rather than
always exploding that same list of Policies on any Token that needs
similar permissions.

This also refactors the acl replication code to be semi-generic to avoid
3x copypasta.
2019-04-26 14:49:12 -05:00
R.B. Boyer 7928305279 making ACLToken.ExpirationTime a *time.Time value instead of time.Time (#5663)
This is mainly to avoid having the API return "0001-01-01T00:00:00Z" as
a value for the ExpirationTime field when it is not set. Unfortunately
time.Time doesn't respect the json marshalling "omitempty" directive.
2019-04-26 14:48:16 -05:00
R.B. Boyer db43fc3a20 acl: ACL Tokens can now be assigned an optional set of service identities (#5390)
These act like a special cased version of a Policy Template for granting
a token the privileges necessary to register a service and its connect
proxy, and read upstreams from the catalog.
2019-04-26 14:48:04 -05:00
R.B. Boyer 2144bd7fbd acl: tokens can be created with an optional expiration time (#5353) 2019-04-26 14:47:51 -05:00
Matt Keeler 5befe0f5d5
Implement config entry replication (#5706) 2019-04-26 13:38:39 -04:00
Kyle Havlovitz c269369760 Make central service config opt-in and rework the initial registration 2019-04-24 06:11:08 -07:00
Kyle Havlovitz 88e1d8ce03 Fill out the service manager functionality and fix tests 2019-04-23 00:17:28 -07:00
Kyle Havlovitz 7c25869e67 Add the service registration manager to the agent 2019-04-23 00:17:27 -07:00
Kyle Havlovitz b186c3020c
Merge pull request #5615 from hashicorp/config-entry-rpc
Add RPC endpoints for config entry operations
2019-04-23 00:16:54 -07:00
Kyle Havlovitz fed7595d45 Rename config entry ACL methods 2019-04-22 23:55:11 -07:00
Matt Keeler afa1cc98d1
Implement data filtering of some endpoints (#5579)
Fixes: #4222 

# Data Filtering

This PR will implement filtering for the following endpoints:

## Supported HTTP Endpoints

- `/agent/checks`
- `/agent/services`
- `/catalog/nodes`
- `/catalog/service/:service`
- `/catalog/connect/:service`
- `/catalog/node/:node`
- `/health/node/:node`
- `/health/checks/:service`
- `/health/service/:service`
- `/health/connect/:service`
- `/health/state/:state`
- `/internal/ui/nodes`
- `/internal/ui/services`

More can be added going forward and any endpoint which is used to list some data is a good candidate.

## Usage

When using the HTTP API a `filter` query parameter can be used to pass a filter expression to Consul. Filter Expressions take the general form of:

```
<selector> == <value>
<selector> != <value>
<value> in <selector>
<value> not in <selector>
<selector> contains <value>
<selector> not contains <value>
<selector> is empty
<selector> is not empty
not <other expression>
<expression 1> and <expression 2>
<expression 1> or <expression 2>
```

Normal boolean logic and precedence is supported. All of the actual filtering and evaluation logic is coming from the [go-bexpr](https://github.com/hashicorp/go-bexpr) library

## Other changes

Adding the `Internal.ServiceDump` RPC endpoint. This will allow the UI to filter services better.
2019-04-16 12:00:15 -04:00
Kyle Havlovitz 690e9dd2c0 Move the ACL logic into the ConfigEntry interface 2019-04-10 14:27:28 -07:00
Kyle Havlovitz f2ed482680 Add RPC endpoints for config entry operations 2019-04-06 23:38:08 -07:00
Kyle Havlovitz a2fa9a0019 Cleaned up some error handling/comments around config entries 2019-04-02 15:42:12 -07:00
Kyle Havlovitz d16be2e269 Encode config entry FSM messages in a generic type 2019-03-28 00:06:56 -07:00
Kyle Havlovitz f6df5c9b3b Clean up service config state store methods 2019-03-27 16:52:38 -07:00
Kyle Havlovitz e199c37ee4
Add some basic normalize/validation logic for config entries 2019-03-22 09:25:37 -07:00
Kyle Havlovitz d92577c16b Fix fsm serialization and add snapshot/restore 2019-03-20 16:13:13 -07:00
Kyle Havlovitz 17aa6a5a34 Fill out state store/FSM functions and add tests 2019-03-19 15:56:17 -07:00
Kyle Havlovitz 9d07add047 Add config types and state store table 2019-03-19 10:06:46 -07:00
R.B. Boyer f4a3b9d518
fix typos reported by golangci-lint:misspell (#5434) 2019-03-06 11:13:28 -06:00
Matt Keeler 90040f8bff Fixes for CVE-2019-8336
Fix error in detecting raft replication errors.

Detect redacted token secrets and prevent attempting to insert.

Add a Redacted field to the TokenBatchRead and TokenRead RPC endpoints

This will indicate whether token secrets have been redacted.

Ensure any token with a redacted secret in secondary datacenters is removed.

Test that redacted tokens cannot be replicated.
2019-03-04 19:13:24 +00:00
Aestek f1cdfbe40e Allow DNS interface to use agent cache (#5300)
Adds two new configuration parameters "dns_config.use_cache" and
"dns_config.cache_max_age" controlling how DNS requests use the agent
cache when querying servers.
2019-02-25 14:06:01 -05:00
R.B. Boyer 324ba5df17
update TestStateStore_ACLBootstrap to not rely upon request mutation (#5335) 2019-02-12 16:09:26 -06:00
Matt Keeler acfd87c673
Improve Connect with Prepared Queries (#5291)
Given a query like:

```
{
   "Name": "tagged-connect-query",
   "Service": {
      "Service": "foo",
      "Tags": ["tag"],
      "Connect": true
   }
}
```

And a Consul configuration like:

```
{
   "services": [
      "name": "foo",
      "port": 8080,
      "connect": { "sidecar_service": {} },
      "tags": ["tag"]
   ]
}
```

If you executed the query it would always turn up with 0 results. This was because the sidecar service was being created without any tags. You could instead make your config look like:

```
{
   "services": [
      "name": "foo",
      "port": 8080,
      "connect": { "sidecar_service": {
         "tags": ["tag"]
      } },
      "tags": ["tag"]
   ]
}
```

However that is a bit redundant for most cases. This PR ensures that the tags and service meta of the parent service get copied to the sidecar service. If there are any tags or service meta set in the sidecar service definition then this copying does not take place. After the changes, the query will now return the expected results.

A second change was made to prepared queries in this PR which is to allow filtering on ServiceMeta just like we allow for filtering on NodeMeta.
2019-02-04 09:36:51 -05:00
Hans Hasselberg 552e150536 correct name 2019-01-25 11:00:56 +01:00
Hans Hasselberg aebb50d47d simpler fix 2019-01-24 17:12:08 +01:00
Hans Hasselberg 5db185a7e4 do not export that type 2019-01-24 17:05:57 +01:00
Hans Hasselberg 7f44100101 fix marshalling 2019-01-24 17:03:26 +01:00
Hans Hasselberg d4790b2827 demo nomad problem 2019-01-24 16:45:54 +01:00
Matt Keeler d5a3ba6cda
Disregard rules when set on a management token (#5261)
* Disregard rules when set on a management token

* Add unit test for legacy mgmt token with rules
2019-01-23 15:48:38 -05:00
Kyle Havlovitz 5bdf130767
Merge pull request #4869 from hashicorp/txn-checks
Add node/service/check operations to transaction api
2019-01-22 11:16:09 -08:00
Paul Banks ef9f27cbc8
connect: tame thundering herd of CSRs on CA rotation (#5228)
* Support rate limiting and concurrency limiting CSR requests on servers; handle CA rotations gracefully with jitter and backoff-on-rate-limit in client

* Add CSR rate limiting docs

* Fix config naming and add tests for new CA configs
2019-01-22 17:19:36 +00:00
Kyle Havlovitz 21380021af txn: update existing txn api docs with new operations 2019-01-15 16:54:07 -08:00
Matt Keeler 1ec5f2a27f
Store leaf cert indexes in raft and use for the ModifyIndex on the returned certs (#5211)
* Store leaf cert indexes in raft and use for the ModifyIndex on the returned certs

This ensures that future certificate signings will have a strictly greater ModifyIndex than any previous certs signed.
2019-01-11 16:04:57 -05:00
Paul Banks 0638e09b6e
connect: agent leaf cert caching improvements (#5091)
* Add State storage and LastResult argument into Cache so that cache.Types can safely store additional data that is eventually expired.

* New Leaf cache type working and basic tests passing. TODO: more extensive testing for the Root change jitter across blocking requests, test concurrent fetches for different leaves interact nicely with rootsWatcher.

* Add multi-client and delayed rotation tests.

* Typos and cleanup error handling in roots watch

* Add comment about how the FetchResult can be used and change ca leaf state to use a non-pointer state.

* Plumb test override of root CA jitter through TestAgent so that tests are deterministic again!

* Fix failing config test
2019-01-10 12:46:11 +00:00
Hans Hasselberg 067027230b
connect: add tls config for vault connect ca provider (#5125)
* add tlsconfig for vault connect ca provider.
* add options to the docs
* add tests for new configuration
2019-01-08 17:09:22 +01:00
Paul Banks b29bc906ee
bugfix: use ServiceTags to generate cache key hash (#4987)
* bugfix: use ServiceTags to generate cahce key hash

* update unit test

* update

* remote print log

* Update .gitignore

* Completely deprecate ServiceTag field internally for clarity

* Add explicit test for CacheInfo cases
2019-01-07 21:30:47 +00:00
Grégoire Seux 4f62a3b528 Implement /v1/agent/health/service/<service name> endpoint (#3551)
This endpoint aggregates all checks related to <service id> on the agent
and return an appropriate http code + the string describing the worst
check.

This allows to cleanly expose service status to other component, hiding
complexity of multiple checks.
This is especially useful to use consul to feed a load balancer which
would delegate health checking to consul agent.

Exposing this endpoint on the agent is necessary to avoid a hit on
consul servers and avoid decreasing resiliency (this endpoint will work
even if there is no consul leader in the cluster).
2019-01-07 09:39:23 -05:00
Kyle Havlovitz 67bac7a815 api: add support for new txn operations 2018-12-12 10:54:09 -08:00
Kyle Havlovitz 7759e9ea8b txn: add service operations 2018-12-12 10:04:10 -08:00
Kyle Havlovitz ab58986ac3 txn: add node operations 2018-12-12 10:04:10 -08:00
Kyle Havlovitz b371ea8783 Add check operations to transaction api 2018-12-12 10:04:10 -08:00
R.B. Boyer 934fae659f
acl: add stub hooks to support some plumbing in enterprise (#4951) 2018-11-13 15:35:54 -06:00
Paul Banks 54c2ff6aca
connect: remove additional trust-domain validation (#4934)
* connct: Remove additional trust-domain validation

* Comment typos

* Update connect_ca.go
2018-11-12 20:20:12 +00:00
Kyle Havlovitz 4a73a59d70
Merge pull request #4917 from hashicorp/replication-token-cleanup
Use acl replication_token for connect
2018-11-12 09:12:54 -08:00
Kyle Havlovitz e8dd89359a
agent: fix formatting 2018-11-07 02:16:03 -08:00
R.B. Boyer 9211d2701d
fix comment typos (#4890) 2018-11-02 12:00:39 -05:00
Matt Keeler f9cf0eb36e Remaining ACL Unit Tests (#4852)
* Add leader token upgrade test and fix various ACL enablement bugs

* Update the leader ACL initialization tests.

* Add a StateStore ACL tests for ACLTokenSet and ACLTokenGetBy* functions

* Advertise the agents acl support status with the agent/self endpoint.

* Make batch token upsert CAS’able to prevent consistency issues with token auto-upgrade

* Finish up the ACL state store token tests

* Finish the ACL state store unit tests

Also rename some things to make them more consistent.

* Do as much ACL replication testing as I can.
2018-10-31 13:00:46 -07:00
Kyle Havlovitz 819566f6b7 fsm: add Intention operations to transactions for internal use 2018-10-19 10:02:28 -07:00
Matt Keeler 18b29c45c4
New ACLs (#4791)
This PR is almost a complete rewrite of the ACL system within Consul. It brings the features more in line with other HashiCorp products. Obviously there is quite a bit left to do here but most of it is related docs, testing and finishing the last few commands in the CLI. I will update the PR description and check off the todos as I finish them over the next few days/week.
Description

At a high level this PR is mainly to split ACL tokens from Policies and to split the concepts of Authorization from Identities. A lot of this PR is mostly just to support CRUD operations on ACLTokens and ACLPolicies. These in and of themselves are not particularly interesting. The bigger conceptual changes are in how tokens get resolved, how backwards compatibility is handled and the separation of policy from identity which could lead the way to allowing for alternative identity providers.

On the surface and with a new cluster the ACL system will look very similar to that of Nomads. Both have tokens and policies. Both have local tokens. The ACL management APIs for both are very similar. I even ripped off Nomad's ACL bootstrap resetting procedure. There are a few key differences though.

    Nomad requires token and policy replication where Consul only requires policy replication with token replication being opt-in. In Consul local tokens only work with token replication being enabled though.
    All policies in Nomad are globally applicable. In Consul all policies are stored and replicated globally but can be scoped to a subset of the datacenters. This allows for more granular access management.
    Unlike Nomad, Consul has legacy baggage in the form of the original ACL system. The ramifications of this are:
        A server running the new system must still support other clients using the legacy system.
        A client running the new system must be able to use the legacy RPCs when the servers in its datacenter are running the legacy system.
        The primary ACL DC's servers running in legacy mode needs to be a gate that keeps everything else in the entire multi-DC cluster running in legacy mode.

So not only does this PR implement the new ACL system but has a legacy mode built in for when the cluster isn't ready for new ACLs. Also detecting that new ACLs can be used is automatic and requires no configuration on the part of administrators. This process is detailed more in the "Transitioning from Legacy to New ACL Mode" section below.
2018-10-19 12:04:07 -04:00
Kyle Havlovitz c617326470 re-add Connect multi-dc config changes
This reverts commit 8bcfbaffb6.
2018-10-19 08:41:03 -07:00
Jack Pearkes 8bcfbaffb6 Revert "Connect multi-dc config" (#4784) 2018-10-11 17:32:45 +01:00
Rebecca Zanzig 34e5516834 Support multiple tags for health and catalog http api endpoints (#4717)
* Support multiple tags for health and catalog api endpoints

Fixes #1781.

Adds a `ServiceTags` field to the ServiceSpecificRequest to support
multiple tags, updates the filter logic in the catalog store, and
propagates these change through to the health and catalog endpoints.

Note: Leaves `ServiceTag` in the struct, since it is being used as
part of the DNS lookup, which in turn uses the health check.

* Update the api package to support multiple tags

Includes additional tests.

* Update new tests to use the `require` library

* Update HealthConnect check after a bad merge
2018-10-11 12:50:05 +01:00
Pierre Souchay 51b33ef015 [Performance On Large clusters] Reduce updates on large services (#4720)
* [Performance On Large clusters] Checks do update services/nodes only when really modified to avoid too many updates on very large clusters

In a large cluster, when having a few thousands of nodes, the anti-entropy
mechanism performs lots of changes (several per seconds) while
there is no real change. This patch wants to improve this in order
to increase Consul scalability when using many blocking requests on
health for instance.

* [Performance for large clusters] Only updates index of service if service is really modified

* [Performance for large clusters] Only updates index of nodes if node is really modified

* Added comments / ensure IsSame() has clear semantics

* Avoid having modified boolean, return nil directly if stutures are Same

* Fixed unstable unit tests TestLeader_ChangeServerID

* Rewrite TestNode_IsSame() for better readability as suggested by @banks

* Rename ServiceNode.IsSame() into IsSameService() + added unit tests

* Do not duplicate TestStructs_ServiceNode_Conversions() and increase test coverage of IsSameService

* Clearer documentation in IsSameService

* Take into account ServiceProxy into ServiceNode.IsSameService()

* Fixed IsSameService() with all new structures
2018-10-11 12:42:39 +01:00
Kyle Havlovitz 98d95cfa80 connect: add ExternalTrustDomain to CARoot fields 2018-10-10 12:16:47 -07:00
Paul Banks c9217c958e merge feedback: fix typos; actually use deliverLatest added previously but not plumbed in 2018-10-10 16:55:34 +01:00
Paul Banks 0f27ffd163 Proxy Config Manager (#4729)
* Proxy Config Manager

This component watches for local state changes on the agent and ensures that each service registered locally with Kind == connect-proxy has it's state being actively populated in the cache.

This serves two purposes:
 1. For the built-in proxy, it ensures that the state needed to accept connections is available in RAM shortly after registration and likely before the proxy actually starts accepting traffic.
 2. For (future - next PR) xDS server and other possible future proxies that require _push_ based config discovery, this provides a mechanism to subscribe and be notified about updates to a proxy instance's config including upstream service discovery results.

* Address review comments

* Better comments; Better delivery of latest snapshot for slow watchers; Embed Config

* Comment typos

* Add upstream Stringer for funsies
2018-10-10 16:55:34 +01:00
Paul Banks e812f5516a Add -sidecar-for and new /agent/service/:service_id endpoint (#4691)
- A new endpoint `/v1/agent/service/:service_id` which is a generic way to look up the service for a single instance. The primary value here is that it:
   - **supports hash-based blocking** and so;
   - **replaces `/agent/connect/proxy/:proxy_id`** as the mechanism the built-in proxy uses to read its config.
   - It's not proxy specific and so works for any service.
   - It has a temporary shim to call through to the existing endpoint to preserve current managed proxy config defaulting behaviour until that is removed entirely (tested).
 - The built-in proxy now uses the new endpoint exclusively for it's config
 - The built-in proxy now has a `-sidecar-for` flag that allows the service ID of the _target_ service to be specified, on the condition that there is exactly one "sidecar" proxy (that is one that has `Proxy.DestinationServiceID` set) for the service registered.
 - Several fixes for edge cases for SidecarService
 - A fix for `Alias` checks - when running locally they didn't update their state until some external thing updated the target. If the target service has no checks registered as below, then the alias never made it past critical.
2018-10-10 16:55:34 +01:00
Paul Banks 1e7eace066 Add SidecarService Syntax sugar to Service Definition (#4686)
* Added new Config for SidecarService in ServiceDefinitions.

* WIP: all the code needed for SidecarService is written... none of it is tested other than config :). Need API updates too.

* Test coverage for the new sidecarServiceFromNodeService method.

* Test API registratrion with SidecarService

* Recursive Key Translation 🤦

* Add tests for nested sidecar defintion arrays to ensure they are translated correctly

* Use dedicated internal state rather than Service Meta for tracking sidecars for deregistration.

Add tests for deregistration.

* API struct for agent register. No other endpoint should be affected yet.

* Additional test cases to cover updates to API registrations
2018-10-10 16:55:34 +01:00
Paul Banks b83bbf248c Add Proxy Upstreams to Service Definition (#4639)
* Refactor Service Definition ProxyDestination.

This includes:
 - Refactoring all internal structs used
 - Updated tests for both deprecated and new input for:
   - Agent Services endpoint response
   - Agent Service endpoint response
   - Agent Register endpoint
     - Unmanaged deprecated field
     - Unmanaged new fields
     - Managed deprecated upstreams
     - Managed new
   - Catalog Register
     - Unmanaged deprecated field
     - Unmanaged new fields
     - Managed deprecated upstreams
     - Managed new
   - Catalog Services endpoint response
   - Catalog Node endpoint response
   - Catalog Service endpoint response
 - Updated API tests for all of the above too (both deprecated and new forms of register)

TODO:
 - config package changes for on-disk service definitions
 - proxy config endpoint
 - built-in proxy support for new fields

* Agent proxy config endpoint updated with upstreams

* Config file changes for upstreams.

* Add upstream opaque config and update all tests to ensure it works everywhere.

* Built in proxy working with new Upstreams config

* Command fixes and deprecations

* Fix key translation, upstream type defaults and a spate of other subtele bugs found with ned to end test scripts...

TODO: tests still failing on one case that needs a fix. I think it's key translation for upstreams nested in Managed proxy struct.

* Fix translated keys in API registration.
≈

* Fixes from docs
 - omit some empty undocumented fields in API
 - Bring back ServiceProxyDestination in Catalog responses to not break backwards compat - this was removed assuming it was only used internally.

* Documentation updates for Upstreams in service definition

* Fixes for tests broken by many refactors.

* Enable travis on f-connect branch in this branch too.

* Add consistent Deprecation comments to ProxyDestination uses

* Update version number on deprecation notices, and correct upstream datacenter field with explanation in docs
2018-10-10 16:55:34 +01:00
Paul Banks 88388d760d Support Agent Caching for Service Discovery Results (#4541)
* Add cache types for catalog/services and health/services and basic test that caching works

* Support non-blocking cache types with Cache-Control semantics.

* Update API docs to include caching info for every endpoint.

* Comment updates per PR feedback.

* Add note on caching to the 10,000 foot view on the architecture page to make the new data path more clear.

* Document prepared query staleness quirk and force all background requests to AllowStale so we can spread service discovery load across servers.
2018-10-10 16:55:34 +01:00
Kyle Havlovitz d515d25856
Merge pull request #4644 from hashicorp/ca-refactor
connect/ca: rework initialization/root generation in providers
2018-09-13 13:08:34 -07:00
Paul Banks 74f2a80a42
Fix CA pruning when CA config uses string durations. (#4669)
* Fix CA pruning when CA config uses string durations.

The tl;dr here is:

 - Configuring LeafCertTTL with a string like "72h" is how we do it by default and should be supported
 - Most of our tests managed to escape this by defining them as time.Duration directly
 - Out actual default value is a string
 - Since this is stored in a map[string]interface{} config, when it is written to Raft it goes through a msgpack encode/decode cycle (even though it's written from server not over RPC).
 - msgpack decode leaves the string as a `[]uint8`
 - Some of our parsers required string and failed
 - So after 1 hour, a default configured server would throw an error about pruning old CAs
 - If a new CA was configured that set LeafCertTTL as a time.Duration, things might be OK after that, but if a new CA was just configured from config file, intialization would cause same issue but always fail still so would never prune the old CA.
 - Mostly this is just a janky error that got passed tests due to many levels of complicated encoding/decoding.

tl;dr of the tl;dr: Yay for type safety. Map[string]interface{} combined with msgpack always goes wrong but we somehow get bitten every time in a new way :D

We already fixed this once! The main CA config had the same problem so @kyhavlov already wrote the mapstructure DecodeHook that fixes it. It wasn't used in several places it needed to be and one of those is notw in `structs` which caused a dependency cycle so I've moved them.

This adds a whole new test thta explicitly tests the case that broke here. It also adds tests that would have failed in other places before (Consul and Vaul provider parsing functions). I'm not sure if they would ever be affected as it is now as we've not seen things broken with them but it seems better to explicitly test that and support it to not be bitten a third time!

* Typo fix

* Fix bad Uint8 usage
2018-09-13 15:43:00 +01:00
Kyle Havlovitz c112a72880
connect/ca: some cleanup and reorganizing of the new methods 2018-09-11 16:43:04 -07:00
Pierre Souchay eddcf228ea Implementation of Weights Data structures (#4468)
* Implementation of Weights Data structures

Adding this datastructure will allow us to resolve the
issues #1088 and #4198

This new structure defaults to values:
```
   { Passing: 1, Warning: 0 }
```

Which means, use weight of 0 for a Service in Warning State
while use Weight 1 for a Healthy Service.
Thus it remains compatible with previous Consul versions.

* Implemented weights for DNS SRV Records

* DNS properly support agents with weight support while server does not (backwards compatibility)

* Use Warning value of Weights of 1 by default

When using DNS interface with only_passing = false, all nodes
with non-Critical healthcheck used to have a weight value of 1.
While having weight.Warning = 0 as default value, this is probably
a bad idea as it breaks ascending compatibility.

Thus, we put a default value of 1 to be consistent with existing behaviour.

* Added documentation for new weight field in service description

* Better documentation about weights as suggested by @banks

* Return weight = 1 for unknown Check states as suggested by @banks

* Fixed typo (of -> or) in error message as requested by @mkeeler

* Fixed unstable unit test TestRetryJoin

* Fixed unstable tests

* Fixed wrong Fatalf format in `testrpc/wait.go`

* Added notes regarding DNS SRV lookup limitations regarding number of instances

* Documentation fixes and clarification regarding SRV records with weights as requested by @banks

* Rephrase docs
2018-09-07 15:30:47 +01:00
Kyle Havlovitz 546bdf8663
connect/ca: add Configure/GenerateRoot to provider interface 2018-09-06 19:18:59 -07:00
Kyle Havlovitz b51d76f469
fsm: add missing CA config to snapshot/restore logic 2018-08-16 11:58:50 -07:00
Kyle Havlovitz ed87949385
Merge pull request #4400 from hashicorp/leaf-cert-ttl
Add configurable leaf cert TTL to Connect CA
2018-07-25 17:53:25 -07:00
Kyle Havlovitz ce10de036e
connect/ca: check LeafCertTTL when rotating expired roots 2018-07-20 16:04:04 -07:00
Kyle Havlovitz d6ca015a42
connect/ca: add configurable leaf cert TTL 2018-07-16 13:33:37 -07:00
Mitchell Hashimoto 5bc27feb0b
agent/structs: check is alias if node is empty 2018-07-12 09:36:11 -07:00
Mitchell Hashimoto f0658a0ede
agent/config: support configuring alias check 2018-07-12 09:36:10 -07:00
Kyle Havlovitz 4e5fb6bc19
connect: add provider state to snapshots 2018-07-11 11:34:49 -07:00
Kyle Havlovitz 401b206a2e
Store the time CARoot is rotated out instead of when to prune 2018-07-06 16:05:25 -07:00
Kyle Havlovitz 1492243e0a
connect/ca: add logic for pruning old stale RootCA entries 2018-07-02 10:35:05 -07:00
Matt Keeler 163fe11101 Make sure we omit the Kind value in JSON if empty 2018-06-25 12:26:10 -07:00
Kyle Havlovitz 1a8ac686b2 connect/ca: add the Vault CA provider 2018-06-25 12:25:41 -07:00
Mitchell Hashimoto 7cbbac43a3 agent: clarify comment 2018-06-25 12:25:14 -07:00
Paul Banks 2c21ead80e More test tweaks 2018-06-25 12:25:13 -07:00
Paul Banks 4a54f8f7e3 Fix some tests failures caused by the sorting change and some cuased by previous UpdatePrecedence() change 2018-06-25 12:25:13 -07:00
Mitchell Hashimoto 3c17144fb5 agent/structs: JSON marshal the configuration for a managed proxy 2018-06-25 12:25:12 -07:00
Mitchell Hashimoto 028aa78e83 agent/consul: set precedence value on struct itself 2018-06-25 12:24:16 -07:00
Mitchell Hashimoto daf46c9cfa agent/consul: support a Connect option on prepared query request 2018-06-25 12:24:12 -07:00
Mitchell Hashimoto 440b1b2d97 agent/consul: prepared query supports "Connect" field 2018-06-25 12:24:11 -07:00
Mitchell Hashimoto 1830c6b308 agent: switch ConnectNative to an embedded struct 2018-06-25 12:24:10 -07:00
Mitchell Hashimoto eb3fcb39b3 agent/consul/state: support querying by Connect native 2018-06-25 12:24:08 -07:00
Mitchell Hashimoto 424272361d agent: agent service registration supports Connect native services 2018-06-25 12:24:08 -07:00
Mitchell Hashimoto d6a823ad0d agent/consul: support catalog registration with Connect native 2018-06-25 12:24:07 -07:00
Mitchell Hashimoto 0accfc1628
agent: rename test to check 2018-06-14 09:42:18 -07:00
Mitchell Hashimoto d68462fca6
agent/consul: implement Intention.Test endpoint 2018-06-14 09:42:17 -07:00
Paul Banks c1f2025d96
Return TrustDomain from CARoots RPC 2018-06-14 09:42:15 -07:00
Kyle Havlovitz 6e9f1f8acb
Add more metadata to structs.CARoot 2018-06-14 09:42:15 -07:00
Kyle Havlovitz 627aa80d5a
Use provider state table for a global serial index 2018-06-14 09:42:15 -07:00
Mitchell Hashimoto 965a902474
agent/structs: validate service definitions, port required for proxy 2018-06-14 09:42:13 -07:00
Mitchell Hashimoto 171bf8d599
agent: clean up defaulting of proxy configuration
This cleans up and unifies how proxy settings defaults are applied.
2018-06-14 09:42:10 -07:00
Mitchell Hashimoto 1a2b28602c
agent: start proxy manager 2018-06-14 09:42:09 -07:00
Mitchell Hashimoto fcd2ab2338
agent/proxy: manager and basic tests, not great coverage yet coming soon 2018-06-14 09:42:08 -07:00
Mitchell Hashimoto 476ea7b04a
agent: start/stop proxies 2018-06-14 09:42:08 -07:00
Mitchell Hashimoto aaa2431350
agent: change connect command paths to be slices, not strings
This matches other executable configuration and allows us to cleanly
separate executable from arguments without trying to emulate shell
parsing.
2018-06-14 09:42:08 -07:00