Commit Graph

5054 Commits (a152e0ee7c440db8459c79e83af0a16f2cb1b007)

Author SHA1 Message Date
Kyle Havlovitz 2904d0a431
Pull virtual IPs for filter chains from discovery chains (#17375) 2023-05-17 11:18:39 -07:00
R.B. Boyer 21c6e0e8e6
fix two typos (#17389) 2023-05-17 08:50:26 -07:00
Connor 0789661ce5
Rename hcp-metrics-collector to consul-telemetry-collector (#17327)
* Rename hcp-metrics-collector to consul-telemetry-collector

* Fix docs

* Fix doc comment

---------

Co-authored-by: Ashvitha Sridharan <ashvitha.sridharan@hashicorp.com>
2023-05-16 14:36:05 -04:00
Dan Bond 8dee353492
agent: don't write server metadata in dev mode (#17383)
Signed-off-by: Dan Bond <danbond@protonmail.com>
2023-05-16 02:50:27 -07:00
wangxinyi7 70ed184c2b
counterpart of the ent in oss (#17367) 2023-05-15 10:49:43 -07:00
Semir Patel abeccb4c76
Support update resource with change in GroupVersion (#17330) 2023-05-15 09:42:01 -05:00
Matt Keeler d37572bd44
Add a Node health controller (#17214)
This will aggregate all HealthStatus objects owned by the Node and update the status of the Node with an overall health.
2023-05-15 09:55:03 -04:00
Dan Upton 0a38fc1a2a
resource: handle `ErrWatchClosed` in `WatchList` endpoint (#17289) 2023-05-15 12:35:10 +01:00
Dan Bond 95f462d5f1
agent: prevent very old servers re-joining a cluster with stale data (#17171)
* agent: configure server lastseen timestamp

Signed-off-by: Dan Bond <danbond@protonmail.com>

* use correct config

Signed-off-by: Dan Bond <danbond@protonmail.com>

* add comments

Signed-off-by: Dan Bond <danbond@protonmail.com>

* use default age in test golden data

Signed-off-by: Dan Bond <danbond@protonmail.com>

* add changelog

Signed-off-by: Dan Bond <danbond@protonmail.com>

* fix runtime test

Signed-off-by: Dan Bond <danbond@protonmail.com>

* agent: add server_metadata

Signed-off-by: Dan Bond <danbond@protonmail.com>

* update comments

Signed-off-by: Dan Bond <danbond@protonmail.com>

* correctly check if metadata file does not exist

Signed-off-by: Dan Bond <danbond@protonmail.com>

* follow instructions for adding new config

Signed-off-by: Dan Bond <danbond@protonmail.com>

* add comments

Signed-off-by: Dan Bond <danbond@protonmail.com>

* update comments

Signed-off-by: Dan Bond <danbond@protonmail.com>

* Update agent/agent.go

Co-authored-by: Dan Upton <daniel@floppy.co>

* agent/config: add validation for duration with min

Signed-off-by: Dan Bond <danbond@protonmail.com>

* docs: add new server_rejoin_age_max config definition

Signed-off-by: Dan Bond <danbond@protonmail.com>

* agent: add unit test for checking server last seen

Signed-off-by: Dan Bond <danbond@protonmail.com>

* agent: log continually for 60s before erroring

Signed-off-by: Dan Bond <danbond@protonmail.com>

* pr comments

Signed-off-by: Dan Bond <danbond@protonmail.com>

* remove unneeded todo

* agent: fix error message

Signed-off-by: Dan Bond <danbond@protonmail.com>

---------

Signed-off-by: Dan Bond <danbond@protonmail.com>
Co-authored-by: Dan Upton <daniel@floppy.co>
2023-05-15 04:05:47 -07:00
Hans Hasselberg b6097a99b8
Add new fields to HCP bootstrap config request and push state request
To support linking cluster, HCP needs to know the datacenter and if ACLs are enabled. Otherwise hosted Consul Core UI won't work properly.
2023-05-12 21:01:56 -06:00
Eric Haberkorn 8bb16567cd
sidecar-proxy refactor (#17328) 2023-05-12 16:49:42 -04:00
Chris Thain b9102c295d
Add Network Filter Support for Envoy Extensions (#17325) 2023-05-12 09:52:50 -07:00
Kyle Havlovitz 81d8332524
Attach service virtual IP info to compiled discovery chain (#17295)
* Add v1/internal/service-virtual-ip for manually setting service VIPs

* Attach service virtual IP info to compiled discovery chain

* Separate auto-assigned and manual VIPs in response
2023-05-12 02:28:16 +00:00
Kyle Havlovitz bd0eb07ed3
Add /v1/internal/service-virtual-ip for manually setting service VIPs (#17294) 2023-05-12 00:38:52 +00:00
R.B. Boyer cd80ea18ff
grpc: ensure grpc resolver correctly uses lan/wan addresses on servers (#17270)
The grpc resolver implementation is fed from changes to the
router.Router. Within the router there is a map of various areas storing
the addressing information for servers in those areas. All map entries
are of the WAN variety except a single special entry for the LAN.

Addressing information in the LAN "area" are local addresses intended
for use when making a client-to-server or server-to-server request.

The client agent correctly updates this LAN area when receiving lan serf
events, so by extension the grpc resolver works fine in that scenario.

The server agent only initially populates a single entry in the LAN area
(for itself) on startup, and then never mutates that area map again.
For normal RPCs a different structure is used for LAN routing.

Additionally when selecting a server to contact in the local datacenter
it will randomly select addresses from either the LAN or WAN addressed
entries in the map.

Unfortunately this means that the grpc resolver stack as it exists on
server agents is either broken or only accidentally functions by having
servers dial each other over the WAN-accessible address. If the operator
disables the serf wan port completely likely this incidental functioning
would break.

This PR enforces that local requests for servers (both for stale reads
or leader forwarded requests) exclusively use the LAN "area" information
and also fixes it so that servers keep that area up to date in the
router.

A test for the grpc resolver logic was added, as well as a higher level
full-stack test to ensure the externally perceived bug does not return.
2023-05-11 11:08:57 -05:00
Dan Upton 5030101cdb
resource: add missing validation to the `List` and `WatchList` endpoints (#17213) 2023-05-10 10:38:48 +01:00
Derek Menteer 5ecab506a6
Fix ent bug caused by #17241. (#17278)
Fix ent bug caused by #17241

All tests passed in OSS, but not ENT. This is a patch to resolve
the problem for both.
2023-05-09 16:36:29 -05:00
cskh 48f7d99305
snapshot: some improvments to the snapshot process (#17236)
* snapshot: some improvments to the snapshot process

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
2023-05-09 15:28:52 -04:00
Semir Patel 40eefaba18
Reaper controller for cascading deletes of owner resources (#17256) 2023-05-09 13:57:40 -05:00
Freddy 7c3e9cd862
Hash namespace+proxy ID when creating socket path (#17204)
UNIX domain socket paths are limited to 104-108 characters, depending on
the OS. This limit was quite easy to exceed when testing the feature on
Kubernetes, due to how proxy IDs encode the Pod ID eg:
metrics-collector-59467bcb9b-fkkzl-hcp-metrics-collector-sidecar-proxy

To ensure we stay under that character limit this commit makes a
couple changes:
- Use a b64 encoded SHA1 hash of the namespace + proxy ID to create a
  short and deterministic socket file name.
- Add validation to proxy registrations and proxy-defaults to enforce a
  limit on the socket directory length.
2023-05-09 12:20:26 -06:00
Dan Upton d53a1d4a27
resource: add helpers for more efficiently comparing IDs etc (#17224) 2023-05-09 19:02:24 +01:00
Derek Menteer 4f6da20fe5
Fix multiple issues related to proxycfg health queries. (#17241)
Fix multiple issues related to proxycfg health queries.

1. The datacenter was not being provided to a proxycfg query, which resulted in
bypassing agentless query optimizations and using the normal API instead.

2. The health rpc endpoint would return a zero index when insufficient ACLs were
detected. This would result in the agent cache performing an infinite loop of
queries in rapid succession without backoff.
2023-05-09 12:37:58 -05:00
Dan Upton 972998203e
controller: deduplicate items in queue (#17168) 2023-05-09 18:14:20 +01:00
Dan Upton 6e1bc57469
Controller Runtime 2023-05-09 15:25:55 +01:00
Matt Keeler 34915670f2
Register new catalog & mesh protobuf types with the resource registry (#17225) 2023-05-08 15:36:35 -04:00
Derek Menteer 50ef6a697e
Fix issue with peer stream node cleanup. (#17235)
Fix issue with peer stream node cleanup.

This commit encompasses a few problems that are closely related due to their
proximity in the code.

1. The peerstream utilizes node IDs in several locations to determine which
nodes / services / checks should be cleaned up or created. While VM deployments
with agents will likely always have a node ID, agentless uses synthetic nodes
and does not populate the field. This means that for consul-k8s deployments, all
services were likely bundled together into the same synthetic node in some code
paths (but not all), resulting in strange behavior. The Node.Node field should
be used instead as a unique identifier, as it should always be populated.

2. The peerstream cleanup process for unused nodes uses an incorrect query for
node deregistration. This query is NOT namespace aware and results in the node
(and corresponding services) being deregistered prematurely whenever it has zero
default-namespace services and 1+ non-default-namespace services registered on
it. This issue is tricky to find due to the incorrect logic mentioned in #1,
combined with the fact that the affected services must be co-located on the same
node as the currently deregistering service for this to be encountered.

3. The stream tracker did not understand differences between services in
different namespaces and could therefore report incorrect numbers. It was
updated to utilize the full service name to avoid conflicts and return proper
results.
2023-05-08 13:13:25 -05:00
Semir Patel 991a002fcc
resource: List resources by owner (#17190) 2023-05-08 12:26:19 -05:00
Dan Upton 917afcf3c6
controller: make the `WorkQueue` generic (#16982) 2023-05-05 15:38:22 +01:00
John Eikenberry bd76fdeaeb
enable auto-tidy expired issuers in vault (as CA)
When using vault as a CA and generating the local signing cert, try to
enable the PKI endpoint's auto-tidy feature with it set to tidy expired
issuers.
2023-05-03 20:30:37 +00:00
Nathan Coleman bdef22354b
Use auth context when evaluating service read permissions (#17207)
Co-authored-by: Blake Covarrubias <1812+blake@users.noreply.github.com>
2023-05-02 16:23:42 -04:00
Poonam Jadhav ef5d54fd4c
feat: add no-op reporting background routine (#17178) 2023-04-28 20:07:03 -04:00
Eric Haberkorn 2c0da88ce7
fix panic in `injectSANMatcher` when `tlsContext` is `nil` (#17185) 2023-04-28 16:27:57 -04:00
Paul Glass e4a341c88a
Permissive mTLS: Config entry filtering and CLI warnings (#17183)
This adds filtering for service-defaults: consul config list -filter 'MutualTLSMode == "permissive"'.

It adds CLI warnings when the CLI writes a config entry and sees that either service-defaults or proxy-defaults contains MutualTLSMode=permissive, or sees that the mesh config entry contains AllowEnablingPermissiveMutualTLSMode=true.
2023-04-28 12:51:36 -05:00
R.B. Boyer 6b4986907d
peering: ensure that merged central configs of peered upstreams for partitioned downstreams work (#17179)
Partitioned downstreams with peered upstreams could not properly merge central config info (i.e. proxy-defaults and service-defaults things like mesh gateway modes) if the upstream had an empty DestinationPartition field in Enterprise.

Due to data flow, if this setup is done using Consul client agents the field is never empty and thus does not experience the bug.

When a service is registered directly to the catalog as is the case for consul-dataplane use this field may be empty and and the internal machinery of the merging function doesn't handle this well.

This PR ensures the internal machinery of that function is referentially self-consistent.
2023-04-28 12:36:08 -05:00
Semir Patel 1037bf7f69
Sync .golangci.yml from ENT (#17180) 2023-04-28 17:14:37 +00:00
John Landa eded58b62a
Remove artificial ACLTokenMaxTTL limit for configuring acl token expiry (#17066)
* Remove artificial ACLTokenMaxTTL limit for configuring acl token expiry

* Add changelog

* Remove test on default MaxTokenTTL

* Change to imperitive tense for changelog entry
2023-04-28 10:57:30 -05:00
Semir Patel 9fef1c7f17
Create tombstone on resource `Delete` (#17108) 2023-04-28 10:49:08 -05:00
Dan Upton eff5dd1812
resource: owner references must include a uid (#17169) 2023-04-28 11:22:42 +01:00
Freddy e02ef16f02
Update HCP bootstrapping to support existing clusters (#16916)
* Persist HCP management token from server config

We want to move away from injecting an initial management token into
Consul clusters linked to HCP. The reasoning is that by using a separate
class of token we can have more flexibility in terms of allowing HCP's
token to co-exist with the user's management token.

Down the line we can also more easily adjust the permissions attached to
HCP's token to limit it's scope.

With these changes, the cloud management token is like the initial
management token in that iit has the same global management policy and
if it is created it effectively bootstraps the ACL system.

* Update SDK and mock HCP server

The HCP management token will now be sent in a special field rather than
as Consul's "initial management" token configuration.

This commit also updates the mock HCP server to more accurately reflect
the behavior of the CCM backend.

* Refactor HCP bootstrapping logic and add tests

We want to allow users to link Consul clusters that already exist to
HCP. Existing clusters need care when bootstrapped by HCP, since we do
not want to do things like change ACL/TLS settings for a running
cluster.

Additional changes:

* Deconstruct MaybeBootstrap so that it can be tested. The HCP Go SDK
  requires HTTPS to fetch a token from the Auth URL, even if the backend
  server is mocked. By pulling the hcp.Client creation out we can modify
  its TLS configuration in tests while keeping the secure behavior in
  production code.

* Add light validation for data received/loaded.

* Sanitize initial_management token from received config, since HCP will
  only ever use the CloudConfig.MangementToken.

* Add changelog entry
2023-04-27 22:27:39 +02:00
John Maguire 391ed069c4
APIGW: Update how status conditions for certificates are handled (#17115)
* Move status condition for invalid certifcate to reference the listener
that is using the certificate

* Fix where we set the condition status for listeners and certificate
refs, added tests

* Add changelog
2023-04-27 15:54:44 +00:00
Semir Patel 5eaeb7b8e5
Support Envoy's MaxEjectionPercent and BaseEjectionTime config entries for passive health checks (#15979)
* Add MaxEjectionPercent to config entry

* Add BaseEjectionTime to config entry

* Add MaxEjectionPercent and BaseEjectionTime to protobufs

* Add MaxEjectionPercent and BaseEjectionTime to api

* Fix integration test breakage

* Verify MaxEjectionPercent and BaseEjectionTime in integration test upstream confings

* Website docs for MaxEjectionPercent and BaseEjection time

* Add `make docs` to browse docs at http://localhost:3000

* Changelog entry

* so that is the difference between consul-docker and dev-docker

* blah

* update proto funcs

* update proto

---------

Co-authored-by: Maliz <maliheh.monshizadeh@hashicorp.com>
2023-04-26 15:59:48 -07:00
Michael Wilkerson 80b1dbcc7d
fixed aliases for sameness group (sameness_group) (#17161) 2023-04-26 14:53:23 -07:00
Eric Haberkorn a87115c598
add acl filter logs (#17143) 2023-04-26 10:57:35 -04:00
Dan Upton faae7bb5f2
testing: `RunResourceService` helper (#17068) 2023-04-26 11:57:10 +01:00
Semir Patel e7bb8fdf15
Fix or disable pipeline breaking changes that made it into main in last day or so (#17130)
* Fix straggler from renaming Register->RegisterTypes

* somehow a lint failure got through previously

* Fix lint-consul-retry errors

* adding in fix for success jobs getting skipped. (#17132)

* Temporarily disable inmem backend conformance test to get green pipeline

* Another test needs disabling

---------

Co-authored-by: John Murret <john.murret@hashicorp.com>
2023-04-25 15:17:48 -05:00
Dan Upton b9c485dcb8
Controller Supervision (#17016) 2023-04-25 12:52:35 +01:00
John Maguire e47f3216e5
APIGW Normalize Status Conditions (#16994)
* normalize status conditions for gateways and routes

* Added tests for checking condition status and panic conditions for
validating combinations, added dummy code for fsm store

* get rid of unneeded gateway condition generator struct

* Remove unused file

* run go mod tidy

* Update tests, add conflicted gateway status

* put back removed status for test

* Fix linting violation, remove custom conflicted status

* Update fsm commands oss

* Fix incorrect combination of type/condition/status

* cleaning up from PR review

* Change "invalidCertificate" to be of accepted status

* Move status condition enums into api package

* Update gateways controller and generated code

* Update conditions in fsm oss tests

* run go mod tidy on consul-container module to fix linting

* Fix type for gateway endpoint test

* go mod tidy from changes to api

* go mod tidy on troubleshoot

* Fix route conflicted reason

* fix route conflict reason rename

* Fix text for gateway conflicted status

* Add valid certificate ref condition setting

* Revert change to resolved refs to be handled in future PR
2023-04-24 16:22:55 -04:00
Michael Wilkerson 001d540afc
Add sameness group field to prepared queries (#17089)
* added method for converting SamenessGroupConfigEntry
- added new method `ToQueryFailoverTargets` for converting a SamenessGroupConfigEntry's members to a list of QueryFailoverTargets
- renamed `ToFailoverTargets` ToServiceResolverFailoverTargets to distinguish it from `ToQueryFailoverTargets`

* Added SamenessGroup to PreparedQuery
- exposed Service.Partition to API when defining a prepared query
- added a method for determining if a QueryFailoverOptions is empty
- This will be useful for validation
- added unit tests

* added method for retrieving a SamenessGroup to state store

* added logic for using PQ with SamenessGroup
- added branching path for SamenessGroup handling in execute. It will be handled separate from the normal PQ case
- added a new interface so that the `GetSamenessGroupFailoverTargets` can be properly tested
- separated the execute logic into a `targetSelector` function so that it can be used for both failover and sameness group PQs
- split OSS only methods into new PQ OSS files
- added validation that `samenessGroup` is an enterprise only feature

* added documentation for PQ SamenessGroup
2023-04-24 13:21:28 -07:00
Derek Menteer a33b224a55
Fix virtual services being included in intention topology as downstreams. (#17099) 2023-04-24 12:03:26 -05:00
Semir Patel 46816071df
De-scope tenenacy requirements to OSS only for now. (#17087)
Partition and namespace must be "default"
Peername must be "local"
2023-04-24 08:14:51 -05:00