Commit Graph

14695 Commits (ee18650636ded348348f648dc3edd90633dd0480)

Author SHA1 Message Date
Bartlomiej Plotka dd4eb4590d
Merge pull request #15222 from jmichalek132/jm-add-tcp-events-to-tracing
tracing: add tcp events to remote store span
2024-11-01 14:26:15 +01:00
Arthur Silva Sens ccaff318ff
Merge pull request #15249 from prometheus/otlp-fix-sanitiza
bugfix: Fix otlp translator for foreign characters
2024-10-31 19:05:37 -03:00
Jan Fajerski 284135a905
Merge pull request #15223 from jan--f/release-3.0.0-rc.0
Prepare release 3.0.0 rc.0
2024-10-31 20:33:37 +01:00
Jan Fajerski 5fdd382ec3 CHANGELOG: remove duplicate entry
Signed-off-by: Jan Fajerski <jfajersk@redhat.com>
2024-10-31 16:22:59 +01:00
Ben Ye b7aca45de7 fix round function ignoring enableDelayedNameRemoval feature flag
Signed-off-by: Ben Ye <benye@amazon.com>
2024-10-31 00:30:22 -07:00
Arthur Silva Sens ea06f1a1d1
bugfix: Fix otlp translation of foreign characters
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
2024-10-30 19:29:31 -03:00
Arthur Silva Sens 8588289c24
otlp translator: Add test showing bugs
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
2024-10-30 19:28:53 -03:00
Julius Volz 76ca7d08d9 Fixup: re-add erroneously removed lines
Signed-off-by: Julius Volz <julius.volz@gmail.com>
2024-10-30 16:43:10 +01:00
Juraj Michalek 76ff12b32a chore: only create span events
Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com>
2024-10-30 09:41:38 +01:00
Juraj Michalek 7ecdb55b02 chore: use otelhttptrace instead
Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com>
2024-10-30 09:41:38 +01:00
Juraj Michalek 3c1ffbb2fd chore: added idleTimeMs and fixed linting issues
Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com>
2024-10-30 09:41:38 +01:00
Juraj Michalek 2e7c739d44 chore: add tcp events to remote store span
Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com>
2024-10-30 09:41:38 +01:00
Julius Volz c861b31b72 Support UTF-8 metric names and labels in web UI
Fixes most of https://github.com/prometheus/prometheus/issues/15202

This should address all areas of the UI except for the autocompletion in the
codemirror-promql text editor. The strategy here is that any time we print or
internally serialize (like for the PromLens tree view) either a metric name or
a label name as part of a selector or in other relevant parts of PromQL, we
check whether it contains characters beyond what was previously supported, and
if so, quote and escape it. In the case of metric names, we also have to move
them from the beginning of the selector into the curly braces.

Signed-off-by: Julius Volz <julius.volz@gmail.com>
2024-10-29 20:22:52 +01:00
Ganesh Vernekar d0eecb1223
Merge pull request #15239 from colega/revert-14975-process-mempostings-delete-with-gomaxprocs-workers
Revert "Process `MemPostings.Delete()` with `GOMAXPROCS` workers"
2024-10-29 12:27:43 -04:00
Oleg Zaytsev ba11a55df4
Revert "Process `MemPostings.Delete()` with `GOMAXPROCS` workers"
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2024-10-29 17:13:40 +01:00
Jan Fajerski e2f55c34c9 fix CHANGELOG formatting and add entry for #14694
Signed-off-by: Jan Fajerski <jfajersk@redhat.com>
2024-10-29 14:41:50 +01:00
Nicolas Takashi b6c538972c
[REFACTORY] simplify appender commit (#15112)
* [REFACTOR] simplify appender commit

Signed-off-by: Nicolas Takashi <nicolas.tcs@hotmail.com>
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com>
2024-10-29 12:34:02 +00:00
Ayoub Mrini 350e0d5bc1
Merge pull request #15228 from shenpengfeng/main
chore: fix function name in comment
2024-10-29 12:12:59 +01:00
shenpengfeng a44db5f784 chore: fix function name in comment
Signed-off-by: shenpengfeng <xinhangzhou@icloud.com>
2024-10-29 17:58:44 +08:00
Arve Knudsen 706dcfeecf
tsdb.CircularExemplarStorage: Avoid racing (#15231)
* tsdb.CircularExemplarStorage: Avoid racing

---------

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-10-29 10:40:46 +01:00
Ayoub Mrini 559722dc68
Merge pull request #15206 from multani/fix-doc-formatting
doc: fix formatting
2024-10-28 22:53:55 +01:00
Ayoub Mrini 99c0e68013
Merge pull request #15230 from GiedriusS/optimize_resolvepodref
discovery/kubernetes: optimize resolvePodRef
2024-10-28 22:53:13 +01:00
Bryan Boreham 5571c7dc98 FastRegexMatcher: use stack memory for lowercase copy of string
Up to 32-byte values this saves garbage, runs faster.
For prefixes, only `toLower` the part we need for the map lookup.

Split toNormalisedLower into fast and slow paths, to avoid a penalty
for the `copy` call in the case where no allocations are done.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-10-28 16:28:58 +00:00
Jan Fajerski 5408184452 react-app: bump version in lock file
Signed-off-by: Jan Fajerski <jfajersk@redhat.com>
2024-10-28 17:10:34 +01:00
Jack Westbrook 7cda23ba32
fix(lezer-promql): fix missing types export in package.json (#15161)
Signed-off-by: Jack Westbrook <jack.westbrook@gmail.com>
2024-10-28 17:05:10 +01:00
Giedrius Statkevičius 58fedb6b61 discovery/kubernetes: optimize more gets
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
2024-10-28 17:17:37 +02:00
Giedrius Statkevičius 716fd5b11f discovery/kubernetes: use namespacedName
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
2024-10-28 16:19:56 +02:00
Giedrius Statkevičius e452308e37 discovery/kubernetes: optimize resolvePodRef
resolvePodRef is in a hot path:

```
ROUTINE ======================== github.com/prometheus/prometheus/discovery/kubernetes.(*Endpoints).resolvePodRef in discovery/kubernetes/endpoints.go
    2.50TB     2.66TB (flat, cum) 22.28% of Total
         .          .    447:func (e *Endpoints) resolvePodRef(ref *apiv1.ObjectReference) *apiv1.Pod {
         .          .    448:   if ref == nil || ref.Kind != "Pod" {
         .          .    449:           return nil
         .          .    450:   }
    2.50TB     2.50TB    451:   p := &apiv1.Pod{}
         .          .    452:   p.Namespace = ref.Namespace
         .          .    453:   p.Name = ref.Name
         .          .    454:
         .   156.31GB    455:   obj, exists, err := e.podStore.Get(p)
         .          .    456:   if err != nil {
         .          .    457:           level.Error(e.logger).Log("msg", "resolving pod ref failed", "err", err)
         .          .    458:           return nil
         .          .    459:   }
         .          .    460:   if !exists {
```

This is some low hanging fruit that we can easily optimize. The key of
an object has format "namespace/name" so generate that inside of
Prometheus itself and use pooling.

```
goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/discovery/kubernetes
cpu: Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz
                 │   olddisc    │               newdisc               │
                 │    sec/op    │   sec/op     vs base                │
ResolvePodRef-16   516.3n ± 17%   289.5n ± 7%  -43.92% (p=0.000 n=10)

                 │   olddisc    │              newdisc               │
                 │     B/op     │    B/op     vs base                │
ResolvePodRef-16   1168.00 ± 0%   24.00 ± 0%  -97.95% (p=0.000 n=10)

                 │  olddisc   │            newdisc             │
                 │ allocs/op  │ allocs/op   vs base            │
ResolvePodRef-16   2.000 ± 0%   2.000 ± 0%  ~ (p=1.000 n=10) ¹
¹ all samples are equal
```

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
2024-10-28 12:12:40 +02:00
3Juhwan 685d6d169f refactor: reorder fields in defaultSDConfig initialization for consistency
Signed-off-by: 3Juhwan <13selfesteem91@naver.com>
2024-10-28 10:40:49 +01:00
György Krajcsovits eafe72a0d0 perf(nhcb): optimize away most allocations in convertnhcb
In general aim for the happy case when the exposer lists the buckets
in ascending order.

Use Compact(2) to compact the result of nhcb convert.

This is more in line with how client_golang optimizes spans vs
buckets.
aef8aedb4b/prometheus/histogram.go (L1485)

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-10-28 08:34:54 +01:00
George Krajcsovits eb3b349024
fix(nhcb): created timestamp fails when keeping classic histograms (#15218)
The wrong source was used to return the created timestamp, leading to
index out of bound panic. One line fix.

Refactor the requirement test to be generic and be able to
test OpenMetrics and Prom parsers as well.
There are some differencies in what the parsers support, the Prom
parser doesn't have created timestamp.

The protobuf parser uses different formatting to identify the metric
for the scrape loop.
Each parser represents the sample timestamp differently.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-10-28 08:31:43 +01:00
Pedro Tanaka bab587b9dc
Agent: allow for ingestion of CT samples (#15124)
* Remove unused option from HeadOptions

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Improve docs for appendable() method in head appender

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Ingest CT (float) samples in Agent DB

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* allow for ingestion of CT native histogram

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* adding some verification for ct ts

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Validating CT histogram before append and add newly created series to pending series

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* checking the wal for written samples

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Checking for samples in test

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* adding case for validations

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* fixing comparison when dedupelabels is enabled

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* unite tests, use table testing

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Implement CT related methods in timestampTracker for write storage

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* adding error case to test

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* removing unused fields

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Updating lastTs for series when adding CT to invalidate duplicates

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* making sure that updating the lastTS wont cause OOO later on in Commit();

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

---------

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
2024-10-27 01:06:34 +01:00
Jan Fajerski 3acb3144fe update CHANGELOG
Signed-off-by: Jan Fajerski <jfajersk@redhat.com>
2024-10-26 09:03:10 +02:00
gopi 372b83d7b8
Documented that WAL can still be written after memory-snapshot-on-shutdown (#15179)
Documented that WAL can still be written after memory-snapshot-on-shutdown - #10824

Co-authored-by: Björn Rabenstein <github@rabenste.in>
Signed-off-by: gopi <gopi.singaravelan.k@gmail.com>

---------

Signed-off-by: Gopi-eng2202 <gopi.singaravelan.k@gmail.com>
Signed-off-by: gopi <gopi.singaravelan.k@gmail.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
2024-10-25 21:40:15 +02:00
Ayoub Mrini 93db81dd3d
Merge pull request #14983 from machine424/dopp
fix(storage/mergeQuerier): fix a data race
2024-10-25 18:34:51 +02:00
Ayoub Mrini 42a30a0c47
Merge pull request #15217 from CharlieTLe/support-int-exemplars
fix(otlptranslator/prometheusremotewrite): support int exemplar value type
2024-10-25 18:29:35 +02:00
Charlie Le d87f7440ca support int exemplar value type
When the exemplar type is an int, it incorrectly gets converted to a 0
when DoubleValue() is called on the exemplar. This adds a check to
ensure that the value is converted properly based on the type.

Signed-off-by: Charlie Le <charlie_le@apple.com>
2024-10-25 08:47:54 -07:00
Jan Fajerski 38fd48e6b5 v2.55.0
-----BEGIN SSH SIGNATURE-----
 U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgX42TrpDUXJbbi9yZ3hs6cDg+kz
 G6d3nAlAb2XQInrEgAAAADZ2l0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5
 AAAAQGoSEKIFT/BfavtG2qW9n7NYonNQk/9r6gCLvxln9elt1hiY0ZGcwRhm1QNx6FotxJ
 Y3LB9dt4s5akB3fOPkYwc=
 -----END SSH SIGNATURE-----

Merge tag 'v2.55.0' into release-3.0.0-rc.0

v2.55.0
2024-10-25 14:16:22 +02:00
Jan Fajerski f131cdd4c5
3.0 migration guide (#15099)
* docs: 2 to 3 migration guide

Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

* docs/stability: add 3.0 section

Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

* docs/migration: details on enabling legacy name validation

Signed-off-by: Owen Williams <owen.williams@grafana.com>\

* migration: add log format and `le` normalization

Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

* migration: add new enable_http2 default for remote write

Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

---------

Signed-off-by: Jan Fajerski <jfajersk@redhat.com>
Signed-off-by: Owen Williams <owen.williams@grafana.com>
Co-authored-by: Owen Williams <owen.williams@grafana.com>
2024-10-25 12:30:13 +02:00
Jan Fajerski 24a10528ac
Merge pull request #15205 from tjhop/chore/slog-fixes
slog: various fixes
2024-10-25 11:11:46 +02:00
Jan Fajerski 07d01a9e0c
Merge pull request #15219 from jan--f/rw-default-http2-off
[CHANGE] Remote-write: default enable_http2 to false
2024-10-25 11:07:55 +02:00
TJ Hoplock 4f9e4dc016 ref: remove unused deduper log wrapper methods
I used these wrapper methods during initial development of the custom
handler that the deduper now implements. Since the deduper implements
slog.Handler and can be used directly as a logger, these wrapper methods
are no longer needed.

Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>
2024-10-24 22:31:37 -04:00
TJ Hoplock b602393473 fix: avoid data race in log deduper
This change should have been included in the initial prometheus slog
conversion, but I must've lost track of it in all the rebases involved
in that PR.

This changes the dedupe logger so that the only method that needs to use
the lock is the `Handle()` method that actually interacts with the
deduplication map.

Ex:
```
==================
WARNING: DATA RACE
Write at 0x00c000518bc0 by goroutine 29481:
  github.com/prometheus/prometheus/util/logging.(*Deduper).WithAttrs()
      /home/tjhop/go/src/github.com/prometheus/prometheus/util/logging/dedupe.go:89 +0xef
  log/slog.(*Logger).With()
      /home/tjhop/.asdf/installs/golang/1.23.1/go/src/log/slog/logger.go:132 +0x106
  github.com/prometheus/prometheus/storage/remote.NewQueueManager()
      /home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/queue_manager.go:483 +0x7a9
  github.com/prometheus/prometheus/storage/remote.(*WriteStorage).ApplyConfig()
      /home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/write.go:201 +0x102c
  github.com/prometheus/prometheus/storage/remote.(*Storage).ApplyConfig()
      /home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/storage.go:92 +0xfd
  github.com/prometheus/prometheus/storage/remote.TestWriteStorageApplyConfigsDuringCommit.func1()
      /home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/storage_test.go:172 +0x3e4
  github.com/prometheus/prometheus/storage/remote.TestWriteStorageApplyConfigsDuringCommit.gowrap1()
      /home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/storage_test.go:174 +0x41

Previous read at 0x00c000518bc0 by goroutine 31261:
  github.com/prometheus/prometheus/util/logging.(*Deduper).Handle()
      /home/tjhop/go/src/github.com/prometheus/prometheus/util/logging/dedupe.go:82 +0x2b1
  log/slog.(*Logger).log()
      /home/tjhop/.asdf/installs/golang/1.23.1/go/src/log/slog/logger.go:257 +0x228
  log/slog.(*Logger).Error()
      /home/tjhop/.asdf/installs/golang/1.23.1/go/src/log/slog/logger.go:230 +0x3d4
  github.com/prometheus/prometheus/tsdb/wlog.(*Watcher).loop()
      /home/tjhop/go/src/github.com/prometheus/prometheus/tsdb/wlog/watcher.go:254 +0x2db
  github.com/prometheus/prometheus/tsdb/wlog.(*Watcher).Start.gowrap1()
      /home/tjhop/go/src/github.com/prometheus/prometheus/tsdb/wlog/watcher.go:227 +0x33

Goroutine 29481 (running) created at:
  github.com/prometheus/prometheus/storage/remote.TestWriteStorageApplyConfigsDuringCommit()
      /home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/storage_test.go:164 +0xe4
  testing.tRunner()
      /home/tjhop/.asdf/installs/golang/1.23.1/go/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /home/tjhop/.asdf/installs/golang/1.23.1/go/src/testing/testing.go:1743 +0x44

Goroutine 31261 (running) created at:
  github.com/prometheus/prometheus/tsdb/wlog.(*Watcher).Start()
      /home/tjhop/go/src/github.com/prometheus/prometheus/tsdb/wlog/watcher.go:227 +0x177
  github.com/prometheus/prometheus/storage/remote.(*QueueManager).Start()
      /home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/queue_manager.go:934 +0x304
  github.com/prometheus/prometheus/storage/remote.(*WriteStorage).ApplyConfig()
      /home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/write.go:232 +0x151b
  github.com/prometheus/prometheus/storage/remote.(*Storage).ApplyConfig()
      /home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/storage.go:92 +0xfd
  github.com/prometheus/prometheus/storage/remote.TestWriteStorageApplyConfigsDuringCommit.func1()
      /home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/storage_test.go:172 +0x3e4
  github.com/prometheus/prometheus/storage/remote.TestWriteStorageApplyConfigsDuringCommit.gowrap1()
      /home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/storage_test.go:174 +0x41
==================
--- FAIL: TestWriteStorageApplyConfigsDuringCommit (2.26s)
    testing.go:1399: race detected during execution of test
FAIL
FAIL    github.com/prometheus/prometheus/storage/remote 68.321s
```

Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>
2024-10-24 22:30:38 -04:00
Jan Fajerski 7939eab77a remote-write: change test default expected to http2 disabled
Signed-off-by: Jan Fajerski <jfajersk@redhat.com>
2024-10-24 22:32:08 +02:00
Bryan Boreham 20fdc8f541 [CHANGE] Remote-write: default enable_http2 to false
Remote-write creates several shards to parallelise sending, each with
its own http connection. We do not want them all combined onto one
socket by http2.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-10-24 22:27:06 +02:00
Ben Ye 99882eec3b log last series labelset when hitting OOO series labels during compaction
Signed-off-by: Ben Ye <benye@amazon.com>
2024-10-24 09:27:15 -07:00
Jesus Vazquez 3cb09acb21
Docs: Remove experimental note on out of order feature (#15215)
Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com>
2024-10-24 18:18:21 +02:00
George Krajcsovits 469573b13b
fix(nhcb): do not return nhcb from parse if exponential is present (#15209)
From: https://github.com/prometheus/prometheus/pull/14978#discussion_r1800755481
Also encode the requirement table set in #13532

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-10-24 18:14:05 +02:00
Jonathan Ballet 7ca90e5729 doc: fix formatting
Signed-off-by: Jonathan Ballet <jon@multani.info>
2024-10-24 08:55:23 +02:00
George Krajcsovits 2182b83271
feat(nhcb): implement created timestamp handling (#15198)
Call through to the underlaying parser if we are not in a histogram
and the entry is a series or exponential native histogram. Otherwise store
and retrieve CT for NHCB.

* fix(omparser): losing exemplars when CT is parsed

Fixes: #15137
Ignore exemplars while peeking ahead during CT parsing.
Simplify state reset with defer().

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-10-24 07:38:58 +02:00