Commit Graph

4995 Commits (77b49df6a3fd0ffc74114a8b4ac81122c98ec5cc)

Author SHA1 Message Date
hc-github-team-consul-core bc5a3576cf
Backport of bump testcontainers-go from 0.22.0 and remove pinned go version in in… into release/1.15.x (#18407)
* bump version

* remove duplicate test file

---------


Co-authored-by: Hui Kang <hui.kang@hashicorp.com>
2023-08-08 21:53:11 -04:00
hc-github-team-consul-core 6ba25f7678
Backport of docs: fix the description of client rpc into release/1.15.x (#18212)
backport of commit 4e8cdddb9f

Co-authored-by: cskh <hui.kang@hashicorp.com>
2023-08-03 22:54:33 +00:00
hc-github-team-consul-core fae68445cb
Backport of Fix policy lookup to allow for slashes into release/1.15.x (#18371)
* backport of commit 1a9cded960

* backport of commit cfec746d8b

* backport of commit 8a9db4cffc

* backport of commit ac13bf16d6

---------

Co-authored-by: Jeremy Jacobson <jeremy.jacobson@hashicorp.com>
2023-08-03 20:45:15 +00:00
Ashvitha 1fcd7747d2
Backport of [HCP Telemetry] Periodic Refresh for Dynamic Telemetry Configuration into release/1.15.x (#18360)
[HCP Telemetry] Periodic Refresh for Dynamic Telemetry Configuration (#18168)

* OTElExporter now uses an EndpointProvider to discover the endpoint

* OTELSink uses a ConfigProvider to obtain filters and labels configuration

* improve tests for otel_sink

* Regex logic is moved into client for a method on the TelemetryConfig object

* Create a telemetry_config_provider and update deps to use it

* Fix conversion

* fix import newline

* Add logger to hcp client and move telemetry_config out of the client.go file

* Add a telemetry_config.go to refactor client.go

* Update deps

* update hcp deps test

* Modify telemetry_config_providers

* Check for nil filters

* PR review updates

* Fix comments and move around pieces

* Fix comments

* Remove context from client struct

* Moved ctx out of sink struct and fixed filters, added a test

* Remove named imports, use errors.New if not fformatting

* Remove HCP dependencies in telemetry package

* Add success metric and move lock only to grab the t.cfgHahs

* Update hash

* fix nits

* Create an equals method and add tests

* Improve telemetry_config_provider.go tests

* Add race test

* Add missing godoc

* Remove mock for MetricsClient

* Avoid goroutine test panics

* trying to kick CI lint issues by upgrading mod

* imprve test code and add hasher for testing

* Use structure logging for filters, fix error constants, and default to allow all regex

* removed hashin and modify logic to simplify

* Improve race test and fix PR feedback by removing hash equals and avoid testing the timer.Ticker logic, and instead unit test

* Ran make go-mod-tidy

* Use errtypes in the test

* Add changelog

* add safety check for exporter endpoint

* remove require.Contains by using error types, fix structure logging, and fix success metric typo in exporter

* Fixed race test to have changing config values

* Send success metric before modifying config

* Avoid the defer and move the success metric under
2023-08-02 13:38:00 -04:00
hc-github-team-consul-core 30a16232e9
Backport of [CC-5719] Add support for builtin global-read-only policy into release/1.15.x (#18344)
[CC-5719] Add support for builtin global-read-only policy (#18319)

* [CC-5719] Add support for builtin global-read-only policy

* Add changelog

* Add read-only to docs

* Fix some minor issues.

* Change from ReplaceAll to Sprintf

* Change IsValidPolicy name to return an error instead of bool

* Fix PolicyList test

* Fix other tests

* Apply suggestions from code review



* Fix state store test for policy list.

* Fix naming issues

* Update acl/validation.go



* Update agent/consul/acl_endpoint.go

---------

Co-authored-by: Jeremy Jacobson <jjacobson93@users.noreply.github.com>
Co-authored-by: Paul Glass <pglass@hashicorp.com>
Co-authored-by: Chris Thain <32781396+cthain@users.noreply.github.com>
2023-08-01 21:42:12 +00:00
Jeremy Jacobson 1cdb1d5b61
Backport of [CC-5718] Remove HCP token requirement during bootstrap (#18288)
[CC-5718] Remove HCP token requirement during bootstrap
2023-07-27 09:28:25 -07:00
hc-github-team-consul-core 3980a286a0
Backport of NET-5049: subscribe to bound-api-gateway only after receiving api-gateway into release/1.15.x (#18294)
* backport of commit 972b20aee7

* backport of commit 2756817142

* backport of commit 72c8f1e4db

---------

Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>
2023-07-26 22:26:12 +00:00
cskh e9615bfa59
Backport of member cli: add -filter expression to flags into release/1.15.x (#18279)
member cli: add -filter expression to flags
2023-07-25 16:23:22 -04:00
hc-github-team-consul-core 816e9116d3
Backport of Explicitly enable WebSocket upgrades into release/1.15.x (#18210)
This PR explicitly enables WebSocket upgrades in Envoy's UpgradeConfig for all
proxy types. (API Gateway, Ingress, and Sidecar.)

Fixes #8283

Co-authored-by: Blake Covarrubias <blake@covarrubi.as>
2023-07-20 14:57:48 -07:00
Dhia Ayachi c940dac2a3
Fix a bug that wrongly trims domains when there is an overlap with DC name 1.15.x (#18163)
Fix a bug that wrongly trims domains when there is an overlap with DC name (#17160)

* Fix a bug that wrongly trims domains when there is an overlap with DC name

Before this change, when DC name and domain/alt-domain overlap, the domain name incorrectly trimmed from the query.

Example:

Given: datacenter = dc-test, alt-domain = test.consul.
Querying for "test-node.node.dc-test.consul" will faile, because the
code was trimming "test.consul" instead of just ".consul"

This change, fixes the issue by adding dot (.) before trimming

* trimDomain: ensure domain trimmed without modyfing original domains

* update changelog

---------

Co-authored-by: Alex Simenduev <shamil.si@gmail.com>
2023-07-18 10:03:31 -04:00
hc-github-team-consul-core c8a6121eba
Backport of Passes configured role name to Vault for AWS auth in Connect CA into release/1.15.x (#18098)
* backport of commit 4034bb2b3e

* backport of commit 9c4c3c50f0

* backport of commit 7282078993

---------

Co-authored-by: Tom Davies <thomas.23.davies@bt.com>
2023-07-12 16:36:28 +00:00
hc-github-team-consul-core 9c4861bb93
Backport of [OSS] Fix initial_fetch_timeout to wait for all xDS resources into release/1.15.x (#18064)
* fix(connect): set initial_fetch_time to wait indefinitely

* changelog

* PR feedback 1

---------

Co-authored-by: DanStough <dan.stough@hashicorp.com>
2023-07-10 17:43:45 -04:00
Derek Menteer 77014793b4
Manual backport of #17894. (#18008) 2023-07-05 09:49:00 -05:00
hc-github-team-consul-core 0a465dd0d2
Backport of watch: support -filter for consul watch: checks, services, nodes, service into release/1.15.x (#17986)
* backport to 1.15.x

---------

Co-authored-by: cskh <hui.kang@hashicorp.com>
2023-06-30 13:26:31 -07:00
hc-github-team-consul-core 20b8427f8c
Backport of feature - [NET - 4005] - [Supportability] Reloadable Configuration - enable_debug into release/1.15.x (#17968)
* backport of commit 10f500e895

* merge conf resolved

* removed experiments

---------

Co-authored-by: Ashesh Vidyut <ashesh.vidyut@hashicorp.com>
Co-authored-by: Ashesh Vidyut <134911583+absolutelightning@users.noreply.github.com>
2023-06-30 14:07:17 +00:00
hc-github-team-consul-core 7c3f4528ba
Backport of Fixes Secondary ConnectCA update into release/1.15.x (#17953)
* backport of commit afa1f42cc7

* backport of commit e0970025d4

* backport of commit 2f2aad545b

* backport of commit 4a5c9c181f

---------

Co-authored-by: Ranjandas <thejranjan@gmail.com>
Co-authored-by: Chris S. Kim <kisunji92@gmail.com>
2023-06-29 14:53:15 +00:00
hc-github-team-consul-core 65cc60e87f
Backport of APIGW Normalize Status Conditions into release/1.15.x (#17844)
* APIGW Normalize Status Conditions (#16994)

* normalize status conditions for gateways and routes

* Added tests for checking condition status and panic conditions for
validating combinations, added dummy code for fsm store

* get rid of unneeded gateway condition generator struct

* Remove unused file

* run go mod tidy

* Update tests, add conflicted gateway status

* put back removed status for test

* Fix linting violation, remove custom conflicted status

* Update fsm commands oss

* Fix incorrect combination of type/condition/status

* cleaning up from PR review

* Change "invalidCertificate" to be of accepted status

* Move status condition enums into api package

* Update gateways controller and generated code

* Update conditions in fsm oss tests

* run go mod tidy on consul-container module to fix linting

* Fix type for gateway endpoint test

* go mod tidy from changes to api

* go mod tidy on troubleshoot

* Fix route conflicted reason

* fix route conflict reason rename

* Fix text for gateway conflicted status

* Add valid certificate ref condition setting

* Revert change to resolved refs to be handled in future PR

* Resolve sneaky merge conflicts

---------

Co-authored-by: John Maguire <john.maguire@hashicorp.com>
Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>
2023-06-23 14:05:11 -04:00
hc-github-team-consul-core 1291786d76
Backport of APIGW: Update how status conditions for certificates are handled into release/1.15.x (#17170)
* APIGW: Update how status conditions for certificates are handled (#17115)

* Move status condition for invalid certifcate to reference the listener
that is using the certificate

* Fix where we set the condition status for listeners and certificate
refs, added tests

* Add changelog

* Remove unused code from backport

* Revert go mod changes

---------

Co-authored-by: John Maguire <john.maguire@hashicorp.com>
Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>
2023-06-22 16:58:06 -05:00
hc-github-team-consul-core 75c396bf00
Backport of fixes #17732 - AccessorID in request body should be optional when updating ACL token into release/1.15.x (#17832)
* backport of commit 31d96f5fb2

* backport of commit 78dbcfbeee

* backport of commit de3dceed99

* backport of commit 2c436e6f59

---------

Co-authored-by: gbolo <george.bolo@gmail.com>
2023-06-21 18:50:45 +00:00
hc-github-team-consul-core 6a833cd2cb
Backport of [NET-3865] [Supportability] Additional Information in the output of 'consul operator raft list-peers' into release/1.15.x (#17737)
* backport of commit 2f94024b35

* backport of commit 7626d0992d

* backport of commit 79aabc958a

* backport of commit 44eee4168b

* merge commit fa40654

* fix merge conf

* [NET-3865] [Supportability] Additional Information in the output of 'consul operator raft list-peers' (#17582)

* init

* fix tests

* added -detailed in docs

* added change log

* fix doc

* checking for entry in map

* fix tests

* removed detailed flag

* removed detailed flag

* revert unwanted changes

* removed unwanted changes

* updated change log

* pr review comment changes

* pr comment changes single API instead of two

* fix change log

* fix tests

* fix tests

* fix test operator raft endpoint test

* Update .changelog/17582.txt

Co-authored-by: Semir Patel <semir.patel@hashicorp.com>

* nits

* updated docs

---------

Co-authored-by: Semir Patel <semir.patel@hashicorp.com>

* [NET-3865] [Supportability] Additional Information in the output of 'consul operator raft list-peers' (#17582)

* init

* fix tests

* added -detailed in docs

* added change log

* fix doc

* checking for entry in map

* fix tests

* removed detailed flag

* removed detailed flag

* revert unwanted changes

* removed unwanted changes

* updated change log

* pr review comment changes

* pr comment changes single API instead of two

* fix change log

* fix tests

* fix tests

* fix test operator raft endpoint test

* Update .changelog/17582.txt

Co-authored-by: Semir Patel <semir.patel@hashicorp.com>

* nits

* updated docs

---------

Co-authored-by: Semir Patel <semir.patel@hashicorp.com>

* resolve diff

* fix merge conf

* fix merge conf

* removed detailed flag

---------

Co-authored-by: Ashesh Vidyut <ashesh.vidyut@hashicorp.com>
Co-authored-by: Ashesh Vidyut <134911583+absolutelightning@users.noreply.github.com>
Co-authored-by: Semir Patel <semir.patel@hashicorp.com>
2023-06-18 04:29:38 +00:00
hc-github-team-consul-core 9ff0f1be6a
HCP Add node id/name to config (#17750) (#17795)
Co-authored-by: chappie <6537530+chapmanc@users.noreply.github.com>
2023-06-16 19:23:39 +00:00
hc-github-team-consul-core 9b469bcbf5
backport of commit 48889f33ec (#17745)
Co-authored-by: Chris Chapman <cchapman@hashicorp.com>
2023-06-14 18:33:35 +00:00
hc-github-team-consul-core b8ffb1d305
backport of commit fc71ecc87e (#17572)
Co-authored-by: Maliz <maliheh.monshizadeh@hashicorp.com>
2023-06-13 18:37:19 +00:00
hc-github-team-consul-core 713b888d22
backport of commit ca0754dedc (#17556)
Co-authored-by: cskh <hui.kang@hashicorp.com>
2023-06-13 12:17:35 -04:00
hc-github-team-consul-core 91f0d5ba6c
backport of commit 38a42074ef (#17701)
Co-authored-by: DanStough <dan.stough@hashicorp.com>
2023-06-13 14:44:18 +00:00
R.B. Boyer 75451c1490
cache: fix a few minor goroutine leaks in leaf certs and the agent cache (#17636) 2023-06-09 11:27:04 -05:00
hc-github-team-consul-core 08a106c400
Backport of [API Gateway] Fix rate limiting for API gateways into release/1.15.x (#17634)
* backport of commit fb2f3b6100

* backport of commit 178abb8495

* backport of commit 77b3998774

* backport of commit a245b326ac

---------

Co-authored-by: Andrew Stucki <andrew.stucki@hashicorp.com>
2023-06-09 12:43:20 +00:00
hc-github-team-consul-core de037e570a
Backport of [API Gateway] Fix trust domain for external peered services in synthesis code into release/1.15.x (#17615)
* backport of commit 63e8743472

* backport of commit 896f11789b

---------

Co-authored-by: Andrew Stucki <andrew.stucki@hashicorp.com>
2023-06-08 16:40:16 +00:00
hc-github-team-consul-core 692bdb6694
fix some testing.T retry.R mixups (#17600) (#17603)
Fix some linter warnings before updating the lint-consul-retry code in hashicorp/lint-consul-retry#4

backport of m #17600 onto release/1.15.x

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2023-06-07 14:39:35 -05:00
hc-github-team-consul-core 47582d4f45
Backport of Fix subscribing/fetching objects not in the default partition into release/1.15.x (#17583)
* backport of commit 7c2c22129f

* backport of commit fc0053f666

---------

Co-authored-by: Andrew Stucki <andrew.stucki@hashicorp.com>
2023-06-06 13:32:05 +00:00
hc-github-team-consul-core db584696fd
Backport of Fix up case where subscription is terminated due to ACLs changing or a snapshot restore occurring into release/1.15.x (#17567)
* backport of commit 82e7d4fe18

* backport of commit 45008e27c3

* backport of commit bdee9e3b98

---------

Co-authored-by: Andrew Stucki <andrew.stucki@hashicorp.com>
2023-06-05 17:32:17 +00:00
Michael Zalimeni aca09d2f65
Manual backport 1.15.x of Avoid panic applying TProxy Envoy extensions (#17539)
backport of commit 8c30455761
2023-06-01 13:09:15 -04:00
hc-github-team-consul-core 7d94518014
Backport of [API Gateway] Fix use of virtual resolvers in HTTPRoutes into release/1.15.x (#17536)
* backport of commit ee9fd909fc

* backport of commit 6bc25b5bcc

---------

Co-authored-by: Andrew Stucki <andrew.stucki@hashicorp.com>
2023-05-31 21:19:56 +00:00
hc-github-team-consul-core f9d3f8a065
Backport of hoststats: add package for collecting host statistics including cpu memory and disk usage into release/1.15.x (#17532)
* backport of commit fafb68b28d

* backport of commit baffef45bd

* backport of commit 369619ba3a

* backport of commit b7f5032136

* backport of commit ac6071721c

* backport of commit 2a3dd5066a

* backport of commit 50621fa098

* backport of commit 6e3ceb0347

* backport of commit bf69a829f2

---------

Co-authored-by: Nick Ethier <nethier@hashicorp.com>
2023-05-31 18:09:13 +00:00
hc-github-team-consul-core fe5a963c62
backport of commit 94998bec4b (#17518)
Co-authored-by: Ashvitha Sridharan <ashvitha.sridharan@hashicorp.com>
2023-05-30 18:42:12 +00:00
Ashvitha 4bd3ed96c8
HCP Telemetry Feature (#17460) (#17502)
* Move hcp client to subpackage hcpclient (#16800)

* [HCP Observability] New MetricsClient (#17100)

* Client configured with TLS using HCP config and retry/throttle

* Add tests and godoc for metrics client

* close body after request

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* remove clone

* Extract CloudConfig and mock for future PR

* Switch to hclog.FromContext

* [HCP Observability] OTELExporter (#17128)

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* [HCP Observability] OTELSink (#17159)

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Initialize OTELSink with sync.Map for all the instrument stores.

* Moved PeriodicReader init to NewOtelReader function. This allows us to use a ManualReader for tests.

* Switch to mutex instead of sync.Map to avoid type assertion

* Add gauge store

* Clarify comments

* return concrete sink type

* Fix lint errors

* Move gauge store to be within sink

* Use context.TODO,rebase and clenaup opts handling

* Rebase onto otl exporter to downgrade metrics API to v1.15.0-rc.1

* Fix imports

* Update to latest stable version by rebasing on cc-4933, fix import, remove mutex init, fix opts error messages and use logger from ctx

* Add lots of documentation to the OTELSink

* Fix gauge store comment and check ok

* Add select and ctx.Done() check to gauge callback

* use require.Equal for attributes

* Fixed import naming

* Remove float64 calls and add a NewGaugeStore method

* Change name Store to Set in gaugeStore, add concurrency tests in both OTELSink and gauge store

* Generate 100 gauge operations

* Seperate the labels into goroutines in sink test

* Generate kv store for the test case keys to avoid using uuid

* Added a race test with 300 samples for OTELSink

* Do not pass in waitgroup and use error channel instead.

* Using SHA 7dea2225a218872e86d2f580e82c089b321617b0 to avoid build failures in otel

* Fix nits

* [HCP Observability] Init OTELSink in Telemetry (#17162)

* Move hcp client to subpackage hcpclient (#16800)

* [HCP Observability] New MetricsClient (#17100)

* Client configured with TLS using HCP config and retry/throttle

* Add tests and godoc for metrics client

* close body after request

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* remove clone

* Extract CloudConfig and mock for future PR

* Switch to hclog.FromContext

* [HCP Observability] New MetricsClient (#17100)

* Client configured with TLS using HCP config and retry/throttle

* Add tests and godoc for metrics client

* close body after request

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* remove clone

* Extract CloudConfig and mock for future PR

* Switch to hclog.FromContext

* [HCP Observability] New MetricsClient (#17100)

* Client configured with TLS using HCP config and retry/throttle

* Add tests and godoc for metrics client

* close body after request

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* remove clone

* Extract CloudConfig and mock for future PR

* Switch to hclog.FromContext

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Initialize OTELSink with sync.Map for all the instrument stores.

* Moved PeriodicReader init to NewOtelReader function. This allows us to use a ManualReader for tests.

* Switch to mutex instead of sync.Map to avoid type assertion

* Add gauge store

* Clarify comments

* return concrete sink type

* Fix lint errors

* Move gauge store to be within sink

* Use context.TODO,rebase and clenaup opts handling

* Rebase onto otl exporter to downgrade metrics API to v1.15.0-rc.1

* Fix imports

* Update to latest stable version by rebasing on cc-4933, fix import, remove mutex init, fix opts error messages and use logger from ctx

* Add lots of documentation to the OTELSink

* Fix gauge store comment and check ok

* Add select and ctx.Done() check to gauge callback

* use require.Equal for attributes

* Fixed import naming

* Remove float64 calls and add a NewGaugeStore method

* Change name Store to Set in gaugeStore, add concurrency tests in both OTELSink and gauge store

* Generate 100 gauge operations

* Seperate the labels into goroutines in sink test

* Generate kv store for the test case keys to avoid using uuid

* Added a race test with 300 samples for OTELSink

* [HCP Observability] OTELExporter (#17128)

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* Do not pass in waitgroup and use error channel instead.

* Using SHA 7dea2225a218872e86d2f580e82c089b321617b0 to avoid build failures in otel

* Rebase onto otl exporter to downgrade metrics API to v1.15.0-rc.1

* Initialize OTELSink with sync.Map for all the instrument stores.

* Added telemetry agent to client and init sink in deps

* Fixed client

* Initalize sink in deps

* init sink in telemetry library

* Init deps before telemetry

* Use concrete telemetry.OtelSink type

* add /v1/metrics

* Avoid returning err for telemetry init

* move sink init within the IsCloudEnabled()

* Use HCPSinkOpts in deps instead

* update golden test for configuration file

* Switch to using extra sinks in the telemetry library

* keep name MetricsConfig

* fix log in verifyCCMRegistration

* Set logger in context

* pass around MetricSink in deps

* Fix imports

* Rebased onto otel sink pr

* Fix URL in test

* [HCP Observability] OTELSink (#17159)

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Initialize OTELSink with sync.Map for all the instrument stores.

* Moved PeriodicReader init to NewOtelReader function. This allows us to use a ManualReader for tests.

* Switch to mutex instead of sync.Map to avoid type assertion

* Add gauge store

* Clarify comments

* return concrete sink type

* Fix lint errors

* Move gauge store to be within sink

* Use context.TODO,rebase and clenaup opts handling

* Rebase onto otl exporter to downgrade metrics API to v1.15.0-rc.1

* Fix imports

* Update to latest stable version by rebasing on cc-4933, fix import, remove mutex init, fix opts error messages and use logger from ctx

* Add lots of documentation to the OTELSink

* Fix gauge store comment and check ok

* Add select and ctx.Done() check to gauge callback

* use require.Equal for attributes

* Fixed import naming

* Remove float64 calls and add a NewGaugeStore method

* Change name Store to Set in gaugeStore, add concurrency tests in both OTELSink and gauge store

* Generate 100 gauge operations

* Seperate the labels into goroutines in sink test

* Generate kv store for the test case keys to avoid using uuid

* Added a race test with 300 samples for OTELSink

* Do not pass in waitgroup and use error channel instead.

* Using SHA 7dea2225a218872e86d2f580e82c089b321617b0 to avoid build failures in otel

* Fix nits

* pass extraSinks as function param instead

* Add default interval as package export

* remove verifyCCM func

* Add clusterID

* Fix import and add t.Parallel() for missing tests

* Kick Vercel CI

* Remove scheme from endpoint path, and fix error logging

* return metrics.MetricSink for sink method

* Update SDK

* [HCP Observability] Metrics filtering and Labels in Go Metrics sink (#17184)

* Move hcp client to subpackage hcpclient (#16800)

* [HCP Observability] New MetricsClient (#17100)

* Client configured with TLS using HCP config and retry/throttle

* Add tests and godoc for metrics client

* close body after request

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* remove clone

* Extract CloudConfig and mock for future PR

* Switch to hclog.FromContext

* [HCP Observability] New MetricsClient (#17100)

* Client configured with TLS using HCP config and retry/throttle

* Add tests and godoc for metrics client

* close body after request

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* remove clone

* Extract CloudConfig and mock for future PR

* Switch to hclog.FromContext

* [HCP Observability] New MetricsClient (#17100)

* Client configured with TLS using HCP config and retry/throttle

* Add tests and godoc for metrics client

* close body after request

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* remove clone

* Extract CloudConfig and mock for future PR

* Switch to hclog.FromContext

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Initialize OTELSink with sync.Map for all the instrument stores.

* Moved PeriodicReader init to NewOtelReader function. This allows us to use a ManualReader for tests.

* Switch to mutex instead of sync.Map to avoid type assertion

* Add gauge store

* Clarify comments

* return concrete sink type

* Fix lint errors

* Move gauge store to be within sink

* Use context.TODO,rebase and clenaup opts handling

* Rebase onto otl exporter to downgrade metrics API to v1.15.0-rc.1

* Fix imports

* Update to latest stable version by rebasing on cc-4933, fix import, remove mutex init, fix opts error messages and use logger from ctx

* Add lots of documentation to the OTELSink

* Fix gauge store comment and check ok

* Add select and ctx.Done() check to gauge callback

* use require.Equal for attributes

* Fixed import naming

* Remove float64 calls and add a NewGaugeStore method

* Change name Store to Set in gaugeStore, add concurrency tests in both OTELSink and gauge store

* Generate 100 gauge operations

* Seperate the labels into goroutines in sink test

* Generate kv store for the test case keys to avoid using uuid

* Added a race test with 300 samples for OTELSink

* [HCP Observability] OTELExporter (#17128)

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* Do not pass in waitgroup and use error channel instead.

* Using SHA 7dea2225a218872e86d2f580e82c089b321617b0 to avoid build failures in otel

* Rebase onto otl exporter to downgrade metrics API to v1.15.0-rc.1

* Initialize OTELSink with sync.Map for all the instrument stores.

* Added telemetry agent to client and init sink in deps

* Fixed client

* Initalize sink in deps

* init sink in telemetry library

* Init deps before telemetry

* Use concrete telemetry.OtelSink type

* add /v1/metrics

* Avoid returning err for telemetry init

* move sink init within the IsCloudEnabled()

* Use HCPSinkOpts in deps instead

* update golden test for configuration file

* Switch to using extra sinks in the telemetry library

* keep name MetricsConfig

* fix log in verifyCCMRegistration

* Set logger in context

* pass around MetricSink in deps

* Fix imports

* Rebased onto otel sink pr

* Fix URL in test

* [HCP Observability] OTELSink (#17159)

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Create new OTELExporter which uses the MetricsClient
Add transform because the conversion is in an /internal package

* Fix lint error

* early return when there are no metrics

* Add NewOTELExporter() function

* Downgrade to metrics SDK version: v1.15.0-rc.1

* Fix imports

* fix small nits with comments and url.URL

* Fix tests by asserting actual error for context cancellation, fix parallel, and make mock more versatile

* Cleanup error handling and clarify empty metrics case

* Fix input/expected naming in otel_transform_test.go

* add comment for metric tracking

* Add a general isEmpty method

* Add clear error types

* update to latest version 1.15.0 of OTEL

* Client configured with TLS using HCP config and retry/throttle

* run go mod tidy

* Remove one abstraction to use the config from deps

* Address PR feedback

* Initialize OTELSink with sync.Map for all the instrument stores.

* Moved PeriodicReader init to NewOtelReader function. This allows us to use a ManualReader for tests.

* Switch to mutex instead of sync.Map to avoid type assertion

* Add gauge store

* Clarify comments

* return concrete sink type

* Fix lint errors

* Move gauge store to be within sink

* Use context.TODO,rebase and clenaup opts handling

* Rebase onto otl exporter to downgrade metrics API to v1.15.0-rc.1

* Fix imports

* Update to latest stable version by rebasing on cc-4933, fix import, remove mutex init, fix opts error messages and use logger from ctx

* Add lots of documentation to the OTELSink

* Fix gauge store comment and check ok

* Add select and ctx.Done() check to gauge callback

* use require.Equal for attributes

* Fixed import naming

* Remove float64 calls and add a NewGaugeStore method

* Change name Store to Set in gaugeStore, add concurrency tests in both OTELSink and gauge store

* Generate 100 gauge operations

* Seperate the labels into goroutines in sink test

* Generate kv store for the test case keys to avoid using uuid

* Added a race test with 300 samples for OTELSink

* Do not pass in waitgroup and use error channel instead.

* Using SHA 7dea2225a218872e86d2f580e82c089b321617b0 to avoid build failures in otel

* Fix nits

* pass extraSinks as function param instead

* Add default interval as package export

* remove verifyCCM func

* Add clusterID

* Fix import and add t.Parallel() for missing tests

* Kick Vercel CI

* Remove scheme from endpoint path, and fix error logging

* return metrics.MetricSink for sink method

* Update SDK

* Added telemetry agent to client and init sink in deps

* Add node_id and __replica__ default labels

* add function for default labels and set x-hcp-resource-id

* Fix labels tests

* Commit suggestion for getDefaultLabels



* Fixed server.id, and t.Parallel()

* Make defaultLabels a method on the TelemetryConfig object

* Rename FilterList to lowercase filterList

* Cleanup filter implemetation by combining regex into a single one, and making the type lowercase

* Fix append

* use regex directly for filters

* Fix x-resource-id test to use mocked value

* Fix log.Error formats

* Forgot the len(opts.Label) optimization)

* Use cfg.NodeID instead

---------



* remove replic tag (#17484)

* [HCP Observability] Add custom metrics for OTEL sink, improve logging, upgrade modules and cleanup metrics client (#17455)

* Add custom metrics for Exporter and transform operations

* Improve deps logging

Run go mod tidy

* Upgrade SDK and OTEL

* Remove the partial success implemetation and check for HTTP status code in metrics client

* Add x-channel

* cleanup logs in deps.go based on PR feedback

* Change to debug log and lowercase

* address test operation feedback

* use GetHumanVersion on version

* Fix error wrapping

* Fix metric names

* [HCP Observability] Turn off retries for now until dynamically configurable (#17496)

* Remove retries for now until dynamic configuration is possible

* Clarify comment

* Update changelog

* improve changelog

---------

Co-authored-by: Joshua Timmons <joshua.timmons1@gmail.com>
2023-05-29 17:02:52 -04:00
hc-github-team-consul-core d60e96f590
Backport of perf: Remove expensive reflection from raft/mesh hot path into release/1.15.x (#17493)
* backport of commit 04b6a90934

* backport of commit 13e40572bd

---------

Co-authored-by: Lincoln Stoll <me@lds.li>
Co-authored-by: John Murret <john.murret@hashicorp.com>
2023-05-26 18:04:50 +00:00
Michael Zalimeni 66f7d7b207
Manual backport 1.15.x of Disable remote proxy patching except AWS Lambda (#17432)
backport of commit ba18381ab2
2023-05-24 22:30:25 -04:00
hc-github-team-consul-core 6c7c0cf66d
Backport of Fix namespaced peer service updates / deletes. into release/1.15.x (#17458)
* backport of commit dc995c18fd

* backport of commit 169a3279b6

---------

Co-authored-by: Derek Menteer <derek.menteer@hashicorp.com>
2023-05-24 21:55:16 +00:00
hc-github-team-consul-core a5960414a1
Backport of [OSS] gRPC Blocking Queries into release/1.15.x (#17441)
* backport of feat: initial grpc blocking queries (11b1dd2)

* backport of changelog and docs update (5ceef20)

---------

Co-authored-by: DanStough <dan.stough@hashicorp.com>
2023-05-24 11:35:45 -04:00
hc-github-team-consul-core 1cc9457e29
Backport of Only synthesize anonymous token in primary DC into release/1.15.x (#17431)
* backport of commit 1e48592aaa

* Add integration test for wan fed issue

* Add changelog

---------

Co-authored-by: Paul Glass <pglass@hashicorp.com>
2023-05-23 15:30:09 +00:00
R.B. Boyer 16334439b0
[1.15.x] prototest: fix early return condition in AssertElementsMatch (#17417)
manual backport of #17416 to 1.15.x
2023-05-22 13:49:55 -05:00
Ashvitha 87d3f45ab5
Rename hcp-metrics-collector to consul-telemetry-collector (#17327) (#17412)
* Rename hcp-metrics-collector to consul-telemetry-collector

* Fix docs

* Fix doc comment

---------

Co-authored-by: Connor <connor.kelly@hashicorp.com>
2023-05-19 11:33:36 -04:00
hc-github-team-consul-core 7dccf73d21
backport of commit ccf91c7b7f (#17384)
Co-authored-by: Dan Bond <danbond@protonmail.com>
2023-05-16 10:17:02 +00:00
malizz 6a2c87739e
fix test on 1.15.x branch (#17374) 2023-05-15 12:15:21 -07:00
Eric Haberkorn ef072a84b8
sidecar-proxy refactor (#17336) 2023-05-15 10:50:29 -04:00
Dan Bond 71651953ef
agent: prevent very old servers re-joining a cluster with stale data (#17357)
Signed-off-by: Dan Bond <danbond@protonmail.com>
2023-05-15 13:14:13 +00:00
hc-github-team-consul-core 64c29e59fb
Backport of grpc: fix data race in balancer registration into release/1.15.x (#17351)
Registering gRPC balancers is thread-unsafe because they are stored in a
global map variable that is accessed without holding a lock. Therefore,
it's expected that balancers are registered _once_ at the beginning of
your program (e.g. in a package `init` function) and certainly not after
you've started dialing connections, etc.

> NOTE: this function must only be called during initialization time
> (i.e. in an init() function), and is not thread-safe.

While this is fine for us in production, it's challenging for tests that
spin up multiple agents in-memory. We currently register a balancer per-
agent which holds agent-specific state that cannot safely be shared.

This commit introduces our own registry that _is_ thread-safe, and
implements the Builder interface such that we can call gRPC's `Register`
method once, on start-up. It uses the same pattern as our resolver
registry where we use the dial target's host (aka "authority"), which is
unique per-agent, to determine which builder to use.
2023-05-15 12:34:17 +01:00
Hans Hasselberg 85fbc59e29
manual backport of #17296 to release/1.15.x (#17343)
* update hcp-sdk-go

* add version, datacenter and acl info

* fewer changes

* go mod tidy and lint

* less code

* remove duplicated dep

* fmt

* trigger ci
2023-05-13 23:18:46 +02:00
Hans Hasselberg 42eec33b0e
Update HCP bootstrapping to support existing clusters 1.15.x version (#17305)
* Persist HCP management token from server config

We want to move away from injecting an initial management token into
Consul clusters linked to HCP. The reasoning is that by using a separate
class of token we can have more flexibility in terms of allowing HCP's
token to co-exist with the user's management token.

Down the line we can also more easily adjust the permissions attached to
HCP's token to limit it's scope.

With these changes, the cloud management token is like the initial
management token in that iit has the same global management policy and
if it is created it effectively bootstraps the ACL system.

* Update SDK and mock HCP server

The HCP management token will now be sent in a special field rather than
as Consul's "initial management" token configuration.

This commit also updates the mock HCP server to more accurately reflect
the behavior of the CCM backend.

* Refactor HCP bootstrapping logic and add tests

We want to allow users to link Consul clusters that already exist to
HCP. Existing clusters need care when bootstrapped by HCP, since we do
not want to do things like change ACL/TLS settings for a running
cluster.

Additional changes:

* Deconstruct MaybeBootstrap so that it can be tested. The HCP Go SDK
  requires HTTPS to fetch a token from the Auth URL, even if the backend
  server is mocked. By pulling the hcp.Client creation out we can modify
  its TLS configuration in tests while keeping the secure behavior in
  production code.

* Add light validation for data received/loaded.

* Sanitize initial_management token from received config, since HCP will
  only ever use the CloudConfig.MangementToken.

* Add changelog entry

---------

Co-authored-by: freddygv <freddy@hashicorp.com>
Co-authored-by: John Murret <john.murret@hashicorp.com>
2023-05-12 23:01:18 +02:00