Commit Graph

760 Commits (e1bfdd22571d874e1f9154d1a35ce3a8569ded36)

Author SHA1 Message Date
Ayoub Mrini af2a1cb10c
Merge pull request #15227 from aniketnk/i15185_1
Run discovery/kubernetes tests in parallel
2024-12-05 10:48:26 +01:00
machine424 c9f3d9b47f
doc(nomad): adjust sections about nomad_sd_config's server
test(nomad): extend TestConfiguredService with more valid/invalid servers configs

fixes https://github.com/prometheus/prometheus/issues/12306

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2024-12-03 19:41:45 +01:00
hongmengning 2a1b940ae4 discovery: fix some function names in comment
Signed-off-by: hongmengning <go@before.tech>
2024-11-25 17:33:04 +08:00
Aniket Kaulavkar f7685caf0d Parallelize discovery/kubernetes tests using t.Parallel()
Signed-off-by: Aniket Kaulavkar <aniket.kaulavkar@gmail.com>
2024-11-14 10:44:03 +05:30
Matthieu MOREL af1a19fc78 enable errorf rule from perfsprint linter
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2024-11-06 16:50:36 +01:00
Giedrius Statkevičius 58fedb6b61 discovery/kubernetes: optimize more gets
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
2024-10-28 17:17:37 +02:00
Giedrius Statkevičius 716fd5b11f discovery/kubernetes: use namespacedName
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
2024-10-28 16:19:56 +02:00
Giedrius Statkevičius e452308e37 discovery/kubernetes: optimize resolvePodRef
resolvePodRef is in a hot path:

```
ROUTINE ======================== github.com/prometheus/prometheus/discovery/kubernetes.(*Endpoints).resolvePodRef in discovery/kubernetes/endpoints.go
    2.50TB     2.66TB (flat, cum) 22.28% of Total
         .          .    447:func (e *Endpoints) resolvePodRef(ref *apiv1.ObjectReference) *apiv1.Pod {
         .          .    448:   if ref == nil || ref.Kind != "Pod" {
         .          .    449:           return nil
         .          .    450:   }
    2.50TB     2.50TB    451:   p := &apiv1.Pod{}
         .          .    452:   p.Namespace = ref.Namespace
         .          .    453:   p.Name = ref.Name
         .          .    454:
         .   156.31GB    455:   obj, exists, err := e.podStore.Get(p)
         .          .    456:   if err != nil {
         .          .    457:           level.Error(e.logger).Log("msg", "resolving pod ref failed", "err", err)
         .          .    458:           return nil
         .          .    459:   }
         .          .    460:   if !exists {
```

This is some low hanging fruit that we can easily optimize. The key of
an object has format "namespace/name" so generate that inside of
Prometheus itself and use pooling.

```
goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/discovery/kubernetes
cpu: Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz
                 │   olddisc    │               newdisc               │
                 │    sec/op    │   sec/op     vs base                │
ResolvePodRef-16   516.3n ± 17%   289.5n ± 7%  -43.92% (p=0.000 n=10)

                 │   olddisc    │              newdisc               │
                 │     B/op     │    B/op     vs base                │
ResolvePodRef-16   1168.00 ± 0%   24.00 ± 0%  -97.95% (p=0.000 n=10)

                 │  olddisc   │            newdisc             │
                 │ allocs/op  │ allocs/op   vs base            │
ResolvePodRef-16   2.000 ± 0%   2.000 ± 0%  ~ (p=1.000 n=10) ¹
¹ all samples are equal
```

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
2024-10-28 12:12:40 +02:00
3Juhwan 685d6d169f refactor: reorder fields in defaultSDConfig initialization for consistency
Signed-off-by: 3Juhwan <13selfesteem91@naver.com>
2024-10-28 10:40:49 +01:00
Ayoub Mrini 98dcd28b1a
Merge pull request #15170 from machine424/awldi
fix(discovery): Handle cache.DeletedFinalStateUnknown in node informers' DeleteFunc
2024-10-18 17:33:08 +02:00
akunszt 08a7162502
discovery: aws/ec2 unit tests (#14364)
* discovery: add aws/ec2 unit tests 

* discovery: initial skeleton for aws/ec2 unit tests

This is a - very likely - not too useful unit test for the AWS SD. It is
commited so other people can check the basic logic and the
implementation.

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>

* discovery: fix linter complains about ec2_test.go

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>

* discovery: add basic unit test for aws

This tests only the basic labelling, not including the VPC related
information.

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>

* discovery: fix linter complains about ec2_test.go

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>

* discovery: other linter fixes in aws/ec2_test.go

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>

* discovery: implement remaining tests for aws/ec2

The coverage is not 100% but I think it is a good starting point if
someone wants to improve that.

Currently it covers all the AWS API calls.

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>

* discovery: make linter happy in aws/ec2_test.go

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>

* discovery: make utility funtcions private

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>

* discover: no global variable in the aws/ec2 test

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>

* discovery: common body for some tests in ec2

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>

* discovery: try to make golangci-lint happy

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>

* discovery: make every non-test function private

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>

* discovery: test for errors first in TestRefresh

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>

* discovery: move refresh tests into the function

This way people can find both the test cases and the execution of the
test at the same place.

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>

* discovery: fix copyright date

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>

* discovery: remove misleading comment

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>

* discovery: rename test for easier identification

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>

* discovery: use static values for the test cases

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>

* discover: try to make the linter happy

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>

* discovery: drop redundant data from ec2 and use common ptr functions

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>

* discovery: use Error instead of Equal

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>

* discovery: merge refreshAZIDs tests into one

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>

---------

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>
2024-10-16 14:36:37 +02:00
machine424 b1c356beea
fix(discovery): Handle cache.DeletedFinalStateUnknown in node informers' DeleteFunc
Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2024-10-16 10:20:37 +02:00
M Viswanath Sai 16bba78f15
discovery: Improve Azure test coverage to 50% (#14586)
* azure sd: separate refresh and refreshAzure
* azure sd: create a client with mocked servers for tests
* add test for refresh function

---------

Signed-off-by: mviswanathsai <mviswanath.sai.met21@itbhu.ac.in>
2024-10-13 10:24:51 +02:00
Bryan Boreham b87b88ddc2
Merge branch 'main' into consul-catalog-filter-support
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-10-08 12:20:31 +01:00
TJ Hoplock 6ebfbd2d54 chore!: adopt log/slog, remove go-kit/log
For: #14355

This commit updates Prometheus to adopt stdlib's log/slog package in
favor of go-kit/log. As part of converting to use slog, several other
related changes are required to get prometheus working, including:
- removed unused logging util func `RateLimit()`
- forward ported the util/logging/Deduper logging by implementing a small custom slog.Handler that does the deduping before chaining log calls to the underlying real slog.Logger
- move some of the json file logging functionality to use prom/common package functionality
- refactored some of the new json file logging for scraping
- changes to promql.QueryLogger interface to swap out logging methods for relevant slog sugar wrappers
- updated lots of tests that used/replicated custom logging functionality, attempting to keep the logical goal of the tests consistent after the transition
- added a healthy amount of `if logger == nil { $makeLogger }` type conditional checks amongst various functions where none were provided -- old code that used the go-kit/log.Logger interface had several places where there were nil references when trying to use functions like `With()` to add keyvals on the new *slog.Logger type

Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>
2024-10-07 15:58:50 -04:00
Matthieu MOREL ab64966e9d
fix: use "ErrorContains" or "EqualError" instead of "Contains(t, err.Error()" and "Equal(t, err.Error()" (#15094)
* fix: use "ErrorContains" or "EqualError" instead of "Contains(t, err.Error()" and "Equal(t, err.Error()"

---------

Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-10-06 16:35:29 +00:00
bas smit 73997289c3 tests: update discovery tests with new labael
Previous commit added the pod_container_init label to discovery, so all
the tests need to reflect that.

Signed-off-by: bas smit <bsmit@bol.com>
2024-10-01 10:26:58 +02:00
bas smit a10dc9298e sd k8s: support sidecar containers in endpoint discovery
Sidecar containers are a newish feature in k8s. They're implemented
similar to init containers but actually stay running and allow you to
delay startup of your application pod until the sidecar started (like
init containers always do).

This adds the ports of the sidecar container to the list of discovered
endpoint(slice), allowing you to target those containers as well.
The implementation is a copy of that of Pod discovery

fixes: #14927

Signed-off-by: bas smit <bsmit@bol.com>
2024-10-01 10:26:58 +02:00
bas smit 7a90d73fa6 sd k8s: test for sidecar container support in endpoints
This test is expected to fail, the followup will add the feature

Signed-off-by: bas smit <bsmit@bol.com>
2024-10-01 10:26:58 +02:00
machine424 b5569c4070 fix(discovery): adjust how type is retrieved in Configs' MarshalYAML/UnmarshalYAML
Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2024-09-30 12:33:07 +02:00
machine424 97f3219157 test(discovery): add a Configs test showing that the custom unmarshalling/marshalling is broken.
This went under the radar because the utils are never called directly.

We usually marshall/unmarshal Configs as embeded in a struct using UnmarshalYAMLWithInlineConfigs/MarshalYAMLWithInlineConfigs
which bypasses Configs' custom UnmarshalYAML/MarshalYAML

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2024-09-30 12:33:07 +02:00
Nathan Baulch 50cd453c8f
chore: Fix typos (#14868)
* Fix typos

---------

Signed-off-by: Nathan Baulch <nathan.baulch@gmail.com>
2024-09-10 22:32:03 +02:00
machine424 d18fa62ae9
chore(discovery): enable new-service-discovery-manager by default and drop legacymanager package
Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2024-09-05 12:46:03 +02:00
Jan Fajerski fe4289b502 Merge branch 'main' into HEAD 2024-09-04 18:50:00 +02:00
Jan Fajerski 00315ce15e Merge branch 'main' into 3.0-main-sync-24-08-30
using -Xours

Signed-off-by: Jan Fajerski <jfajersk@redhat.com>
2024-09-02 11:27:18 +02:00
machine424 d23d196db5 fix(discovery): prevent the manager from storing stale targetGroups
Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2024-08-30 14:39:31 +02:00
machine424 c586c15ae6 fix(discovery): make discovery manager notify consumers of dropped targets for still defined jobs
scrape/manager_test.go: add a test to check that the manager gets notified
for targets that got dropped by discovery to reproduce: https://github.com/prometheus/prometheus/issues/12858#issuecomment-1732318102

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2024-08-28 17:39:02 +02:00
Bryan Boreham 4202be5e79 Merge branch 'release-2.54' into merge-2.54.1-into-main 2024-08-27 12:04:48 +01:00
beorn7 0f760f63dd lint: Revamp our linting rules, mostly around doc comments
Several things done here:

- Set `max-issues-per-linter` to 0 so that we actually see all linter
  warnings and not just 50 per linter. (As we also set
  `max-same-issues` to 0, I assume this was the intention from the
  beginning.)

- Stop using the golangci-lint default excludes (by setting
  `exclude-use-default: false`. Those are too generous and don't match
  our style conventions. (I have re-added some of the excludes
  explicitly in this commit. See below.)

- Re-add the `errcheck` exclusion we have used so far via the
  defaults.

- Exclude the signature requirement `govet` has for `Seek` methods
  because we use non-standard `Seek` methods a lot. (But we keep other
  requirements, while the default excludes completely disabled the
  check for common method segnatures.)

- Exclude warnings about missing doc comments on exported symbols. (We
  used to be pretty adamant about doc comments, but stopped that at
  some point in the past. By now, we have about 500 missing doc
  comments. We may consider reintroducing this check, but that's
  outside of the scope of this commit. The default excludes of
  golangci-lint essentially ignore doc comments completely.)

- By stop using the default excludes, we now get warnings back on
  malformed doc comments. That's the most impactful change in this
  commit. It does not enforce doc comments (again), but _if_ there is
  a doc comment, it has to have the recommended form. (Most of the
  changes in this commit are fixing this form.)

- Improve wording/spelling of some comments in .golangci.yml, and
  remove an outdated comment.

- Leave `package-comments` inactive, but add a TODO asking if we
  should change that.

- Add a new sub-linter `comment-spacings` (and fix corresponding
  comments), which avoids missing spaces after the leading `//`.

Signed-off-by: beorn7 <beorn@grafana.com>
2024-08-22 17:36:11 +02:00
Jan Fajerski 5138922b0d Merge branch 'main' into 3.0-main-sync-24-08-21 2024-08-21 09:09:36 +02:00
ouyang1204@gmail.com 89dee48cc8 fix the issue of failing to match the first network when the container is reconnected to a new network
Signed-off-by: ouyang1204@gmail.com <ouyang1204@gmail.com>
2024-08-19 21:26:25 +08:00
Arve Knudsen 3a78e76282 Upgrade golangci-lint to v1.60.1
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-08-18 12:13:25 +02:00
Julien Pivotto 7711cd5ab5 Remove deprecated storage.tsdb.retention flag
Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>
Signed-off-by: Julien <roidelapluie@o11y.eu>
2024-08-16 13:53:09 +02:00
cuiweiyuan 1800af54f0 chore: fix some function names
Signed-off-by: cuiweiyuan <cuiweiyuan@aliyun.com>
2024-08-15 13:57:21 +08:00
Björn Rabenstein 3f16a2e7de
Merge pull request #14543 from jan--f/3.0-main-sync-24-08-01
3.0 main sync 24 08 01
2024-08-13 15:54:13 +02:00
Julien 3933cba052
Merge pull request #14365 from simonpasquier/fix-12884
discovery(k8s): remove support for API versions no longer served
2024-08-09 12:48:54 +02:00
Bryan Boreham 79a0ba9d64
Merge pull request #13503 from tylitianrui/chore/remove_redundance
remove redundant code
2024-07-30 12:44:03 +01:00
Bryan Boreham ce3bd4abea Update for Docker deprecation
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-07-17 17:03:32 +01:00
Simon Pasquier 145988d48f
discovery(k8s): remove support for API versions no longer served
This commit removes support for the following API versions:
* `discovery.k8s.io/v1beta1` API version of EndpointSlice (no longer
  served as of v1.25).
* `networking.k8s.io/v1beta1` API version of Ingress (no longer served
  as of v1.22).

Closes #12884

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2024-07-04 14:54:27 +02:00
Paulo Dias f4b1fcb73e
discovery: add support for gathering flavor name in Openstack discovery (#14312)
* feat: add support for gathering flavor name in Openstack discovery

Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>

* Update instance.go

Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com>
Signed-off-by: Paulo Dias <44772900+paulojmdias@users.noreply.github.com>

* Update configuration.md

Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com>
Signed-off-by: Paulo Dias <44772900+paulojmdias@users.noreply.github.com>

* fix: fix linting

Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>

* fix: fix instance type

Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>

* Update docs/configuration/configuration.md

Co-authored-by: Simon Pasquier <spasquie@redhat.com>
Signed-off-by: Paulo Dias <44772900+paulojmdias@users.noreply.github.com>

---------

Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>
Signed-off-by: Paulo Dias <44772900+paulojmdias@users.noreply.github.com>
Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com>
Co-authored-by: Simon Pasquier <spasquie@redhat.com>
2024-06-30 19:18:18 +02:00
Bryan Boreham c5040c5ea9
Merge pull request #10490 from DrAuYueng/fix-docker-sd-service-missing
[ENHANCEMENT] Docker SD: add MatchFirstNetwork for containers with multiple networks

Fixes docker sd service misssing in shared mode and deduplicate targets by network
2024-06-26 12:33:50 +01:00
Arve Knudsen d902116b41 Fix various linting errors
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-06-24 16:11:53 -07:00
unknown 0d25931049 rebase main and adjust the configuration
Signed-off-by: ouyang1204@gmail.com <ouyang1204@gmail.com>
2024-06-21 19:10:18 +08:00
akunszt 2aaf99dd0a
discovery: aws: expose Primary IPv6 addresses as label, partially fixes #7406 (#14156)
* discovery: aws: expose Primary IPv6 addresses as label

Add __meta_ec2_primary_ipv6_addresses label. This label contains the
Primary IPv6 address for every ENI attached to the EC2 instance. It is
ordered by the DeviceIndex and the missing elements (interface without
Primary IPv6 address) are kept in the list.

---------

Signed-off-by: Arpad Kunszt <akunszt@hiya.com>
Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com>
2024-06-20 14:36:20 +01:00
machine424 94d28cd6cf chore(notifier): add a reproducer for https://github.com/prometheus/prometheus/issues/13676
to show "targets groups update" starvation when the notifications queue is full and an Alertmanager
is down.

The existing `TestHangingNotifier` that was added in https://github.com/prometheus/prometheus/pull/10948 doesn't really reflect the reality as the SD changes are manually fed into `syncCh` in a continuous way, whereas in reality, updates are only resent every `updatert`.

The test added here sets up an SD manager and links it to the notifier. The SD changes will be triggered by that manager as it's done in reality.

Signed-off-by: machine424 <ayoubmrini424@gmail.com>

Co-authored-by: Ethan Hunter <ehunter@hudson-trading.com>
2024-06-19 09:43:52 +02:00
György Krajcsovits b18e2d3ef6 Format linter pragma correctly
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-06-05 16:50:09 +02:00
György Krajcsovits 180059e6fa Update scaleway-sdk-go and silence deprecation warnings
This reverts commit 777daea86e.
2024-06-05 16:37:58 +02:00
György Krajcsovits 0ee4cf8d94 Update Go dependencies for 2.53
Ran "make update-all-go-deps" , but do not bump go version and remove
toolchain clause from go.mod


Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-06-04 18:02:21 +02:00
Arthur Silva Sens d73537ccc0
Merge branch 'main' into resolveconflicts-2.52.1 2024-05-31 09:36:17 -03:00
David Andruczyk 851f68d1cc BUGFIX: Need seperate listOptions structs since linodego writes into them for pagination
Signed-off-by: David Andruczyk <dandrucz@akamai.com>
2024-05-27 17:19:58 +00:00