Commit Graph

1245 Commits (00146b69fc9486413332a230dbed407127952807)

Author SHA1 Message Date
Oleg Zaytsev cd1f8ac129
MemPostings: keep a map of label values slices (#15426)
While investigating lock contention on `MemPostings`, we saw that lots
of locking is happening in `LabelValues` and
`PostingsForLabelsMatching`, both copying the label values slices while
holding the mutex.

This adds an extra map that holds an append-only label values slice for
each one of the label names. Since the slice is append-only, it can be
copied without holding the mutex.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2024-11-29 12:52:56 +01:00
Charles Korn 96adc410ba
tsdb/chunkenc: don't reuse custom value slices between histograms
Signed-off-by: Charles Korn <charles.korn@grafana.com>
2024-11-29 16:28:09 +11:00
Oleg Zaytsev 9ad93ba8df
Optimize l=~".+" matcher (#15474)
Since dot is matching newline now, `l=~".+"` is "any non empty label
value", and #14144 added a specific method in the index for that so we
don't need to run the matcher on each one of the label values.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2024-11-27 12:33:20 +01:00
Bryan Boreham ca3119bd24 TSDB: eliminate one yolostring
When the only use of a []byte->string conversion is as a map key, Go
doesn't allocate.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-11-26 17:21:55 +00:00
Bryan Boreham e98c19c1ce [PERF] TSDB: Cache all symbols for compaction
Trade a bit more memory for a lot less CPU spent looking up symbols.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-11-26 17:21:55 +00:00
Oleg Zaytsev 9aa6e041d3
MemPostings: allocate ListPostings once in PFALV (#15465)
Same as #15427 but for the new method added in #14144

Instead of allocating each ListPostings one by one, allocate them all in
one go.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2024-11-26 16:03:45 +01:00
DC d535d501d1
[DOCS] Improve description of WAL record format (#14936)
Signed-off-by: DC <413331538@qq.com>
2024-11-26 11:48:17 +00:00
Bryan Boreham dd0252a774
Merge pull request #15380 from bboreham/improve-loadwbl
[BUGFIX] TSDB: Apply fixes from loadWAL to loadWBL
2024-11-25 17:31:49 +00:00
Bryan Boreham 7996a13fdd
Merge pull request #15403 from bboreham/fix-rw-benchmark-startup
[TESTS] Remote-Write: Fix BenchmarkStartup
2024-11-25 17:31:24 +00:00
Oleg Zaytsev cc390aab64
MemPostings: allocate ListPostings once in PFLM (#15427)
Instead of allocating ListPostings pointers one by one, allocate a slice
and take pointers from that. It's faster, and also generates less
garbage (NewListPostings is one of the top offenders in number of
allocations).

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2024-11-20 17:52:20 +01:00
Arve Knudsen 89bbb885e5
Upgrade to golangci-lint v1.62.0 (#15424)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-11-20 17:22:20 +01:00
Björn Rabenstein 384c5951ef
Merge pull request #14489 from harry671003/implement_metadata_limit
storage: Implement limit in mergeGenericQuerier
2024-11-19 17:32:16 +01:00
Arve Knudsen 06d54fcc6c
[PERF] TSDB: Optimize inverse matching (#14144)
Simple follow-up to #13620. Modify `tsdb.PostingsForMatchers` to use the optimized tsdb.IndexReader.PostingsForLabelMatching method also for inverse matching.

Introduce method `PostingsForAllLabelValues`, to avoid changing the existing method.

The performance is much improved for a subset of the cases; there are up to
~60% CPU gains and ~12.5% reduction in memory usage. 

Remove `TestReader_InversePostingsForMatcherHonorsContextCancel` since
`inversePostingsForMatcher` only passes `ctx` to `IndexReader` implementations now.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-11-19 15:49:01 +00:00
Bryan Boreham 0ef0b75a4f [TESTS] Remote-Write: Fix BenchmarkStartup
It was crashing due to uninitialized metrics, and not terminating due to
incorrectly reading segment names.

We need to export `SetMetrics` to avoid the first problem.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-11-15 11:22:07 +00:00
Fiona Liao c599d37668
Always return unknown hint for first sample in non-gauge histogram chunk (#15343)
Always return unknown hint for first sample in non-gauge histogram chunk

---------

Signed-off-by: Fiona Liao <fiona.liao@grafana.com>
Co-authored-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-11-12 15:14:06 +01:00
Bryan Boreham 5450e6d368 [BUGFIX] TSDB: Apply fixes from loadWAL to loadWBL
Move a couple of variables inside the scope of a goroutine, to avoid
data races.

Use `zeropool` to reduce garbage and avoid some lint warnings.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-11-11 18:41:33 +00:00
Ben Ye 140f4aa9ae
feat: Allow customizing TSDB postings decoder (#13567)
* allow customizing TSDB postings decoder

---------

Signed-off-by: Ben Ye <benye@amazon.com>
2024-11-11 07:59:24 +01:00
Ben Ye f9057544cb
Fix AllPostings added twice (#13893)
* handle all postings added twice

---------

Signed-off-by: Ben Ye <benye@amazon.com>
2024-11-10 18:17:21 +01:00
🌲 Harry 🌊 John 🏔 f9bc50b247 storage: Implement limit in mergeGenericQuerier
Signed-off-by: 🌲 Harry 🌊 John 🏔 <johrry@amazon.com>
2024-11-07 09:08:23 -08:00
Bryan Boreham f42b37ff2f
[BUGFIX] TSDB: Fix race on stale values in headAppender (#15322)
* [BUGFIX] TSDB: Fix race on stale values in headAppender

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* Simplify

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

---------

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-11-06 16:51:39 +01:00
Matthieu MOREL af1a19fc78 enable errorf rule from perfsprint linter
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2024-11-06 16:50:36 +01:00
Bryan Boreham 02aa6d1de6
Merge pull request #15338 from bboreham/cosmetic-tsdb
[COMMENT] Remove duplicate line
2024-11-05 12:03:04 +00:00
Oleg Zaytsev b1e4052682
MemPostings.Delete(): make pauses to unlock and let the readers read (#15242)
This introduces back some unlocking that was removed in #13286 but in a
more balanced way, as suggested by @pracucci.

For TSDBs with a lot of churn, Delete() can take a couple of seconds,
and while it's holding the mutex, reads and writes are blocked waiting
for that mutex, increasing the number of connections handled and memory
usage.

This implementation pauses every 4K labels processed (note that also
compared to #13286 we're not processing all the label-values anymore,
but only the affected ones, because of #14307), makes sure that it's
possible to get the read lock, and waits for a few milliseconds more.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Marco Pracucci <marco@pracucci.com>
2024-11-05 12:59:57 +01:00
Bryan Boreham 541c7fd9fe [COMMENT] Remove duplicate line
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-11-05 11:03:40 +00:00
Alban Hurtaud 4b56af7eb8
Add hidden flag for the delayed compaction random time window (#14919)
* Add hidden flag for the delayed compaction random time window

Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com>

* Update cmd/prometheus/main.go

Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com>
Signed-off-by: Alban Hurtaud <alban.hurtaud@amadeus.com>

* Update cmd/prometheus/main.go

Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com>
Signed-off-by: Alban Hurtaud <alban.hurtaud@amadeus.com>

* Update tsdb/db.go

Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com>
Signed-off-by: Alban Hurtaud <alban.hurtaud@amadeus.com>

* Fix flag name according to review - add test for delay

Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com>

* Fix afer main rebase

Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com>

* Implement review comments

Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com>

* Update generatedelaytest to try with limit values

Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com>

---------

Signed-off-by: Alban HURTAUD <alban.hurtaud@amadeus.com>
Signed-off-by: Alban Hurtaud <alban.hurtaud@amadeus.com>
Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com>
2024-11-04 08:26:26 +01:00
Bryan Boreham 2fbbfc3da8 Revert "Fix `MemPostings.Add` and `MemPostings.Get` data race (#15141)"
This reverts commit 50ef0dc954.

Memory allocation goes so high in Prombench that the system is unusable.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-11-03 12:30:34 +00:00
Bryan Boreham e2e01c1cff
Merge pull request #15216 from yeya24/log-last-series-labels
log last series labelset when hitting OOO series labels
2024-11-01 14:15:39 +00:00
Oleg Zaytsev ba11a55df4
Revert "Process `MemPostings.Delete()` with `GOMAXPROCS` workers"
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2024-10-29 17:13:40 +01:00
Nicolas Takashi b6c538972c
[REFACTORY] simplify appender commit (#15112)
* [REFACTOR] simplify appender commit

Signed-off-by: Nicolas Takashi <nicolas.tcs@hotmail.com>
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com>
2024-10-29 12:34:02 +00:00
Arve Knudsen 706dcfeecf
tsdb.CircularExemplarStorage: Avoid racing (#15231)
* tsdb.CircularExemplarStorage: Avoid racing

---------

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-10-29 10:40:46 +01:00
Pedro Tanaka bab587b9dc
Agent: allow for ingestion of CT samples (#15124)
* Remove unused option from HeadOptions

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Improve docs for appendable() method in head appender

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Ingest CT (float) samples in Agent DB

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* allow for ingestion of CT native histogram

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* adding some verification for ct ts

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Validating CT histogram before append and add newly created series to pending series

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* checking the wal for written samples

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Checking for samples in test

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* adding case for validations

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* fixing comparison when dedupelabels is enabled

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* unite tests, use table testing

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Implement CT related methods in timestampTracker for write storage

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* adding error case to test

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* removing unused fields

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Updating lastTs for series when adding CT to invalidate duplicates

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* making sure that updating the lastTS wont cause OOO later on in Commit();

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

---------

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
2024-10-27 01:06:34 +01:00
Ayoub Mrini 93db81dd3d
Merge pull request #14983 from machine424/dopp
fix(storage/mergeQuerier): fix a data race
2024-10-25 18:34:51 +02:00
Ben Ye 99882eec3b log last series labelset when hitting OOO series labels during compaction
Signed-off-by: Ben Ye <benye@amazon.com>
2024-10-24 09:27:15 -07:00
Vanshika cccbe72514
TSDB: Fix some edge cases when OOO is enabled (#14710)
Fix some edge cases when OOO is enabled

Signed-off-by: Vanshikav123 <vanshikav928@gmail.com>
Signed-off-by: Vanshika <102902652+Vanshikav123@users.noreply.github.com>
Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com>
Co-authored-by: Jesus Vazquez <jesusvzpg@gmail.com>
2024-10-23 17:34:28 +02:00
machine424 cebcdce78a
fix(storage/mergeQuerier): copy the matcjers slice before passing it to queriers as
some of them may alter it.

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2024-10-22 14:08:47 +02:00
machine424 eb523a6b29
fix(storage/mergeQuerier): add a reproducer for data race that occurs when one of the queriers alters the passed matchers and propose a fix
Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2024-10-22 14:08:46 +02:00
György Krajcsovits a4083f14e8 Fix populateWithDelChunkSeriesIterator corrupting chunk meta
When handling recoded histogram chunks the min time of the chunk is
updated by mistake. It should only update when the chunk is completely new.
Otherwise the ongoing chunk's meta will be later than the previously
written samples in it.

Same bug as https://github.com/prometheus/prometheus/pull/14629

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-10-18 10:34:22 +02:00
György Krajcsovits e6a682f046 Reproduce populateWithDelChunkSeriesIterator corrupting chunk meta
When handling recoded histogram chunks the min time of the chunk is
updated by mistake. It should only update when the chunk is completely
new.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-10-18 10:34:22 +02:00
machine424 ab2475c426
test(tsdb): add a reproducer for https://github.com/prometheus/prometheus/issues/14422
Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2024-10-15 20:39:25 +02:00
Bryan Boreham 1e1f6ab9df
Merge pull request #15120 from bboreham/floor-ino-mint
[BUGFIX] TSDB: Don't read in-order chunks from before head MinTime
2024-10-15 10:27:38 +01:00
George Krajcsovits b8867f8ead
Merge pull request #15142 from krajorama/fix-appendhistogram-race
bugfix: data race in head.Appender.AppendHistogram and Commit
2024-10-14 08:13:39 +02:00
Oleg Zaytsev 50ef0dc954
Fix `MemPostings.Add` and `MemPostings.Get` data race (#15141)
* Tests for Mempostings.{Add,Get} data race
* Fix MemPostings.{Add,Get} data race

We can't modify the postings list that are held in MemPostings as they
might already be in use by some readers.

* Modify BenchmarkHeadStripeSeriesCreate to have common labels

If there are no common labels on the series, we don't excercise the
ordering part of MemSeries, as we're just creating slices of one element
for each label value.

---------

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2024-10-11 15:21:15 +02:00
György Krajcsovits bb70370d72 TSDB head: fix race between AppendHistogram and Commit
Move writing memSeries lastHistogramValue and lastFloatHistogramValue
after series creation under lock.

The resulting code isn't totally correct in the sense that we're setting
these values before Commit() , so they might be overwritten/rolled back
later.

Also Append of stale sample checks the values without lock, so there's
still a potential race.

The correct solution would be to set these only in Commit() which we
actually do, but then Commit() would also need to process samples in
order and not floats first, then histograms, then float histograms - which
leads to not knowing what stale marker to write for histograms.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-10-10 16:59:15 +02:00
György Krajcsovits 631fadc4ca Unit test for data race in head.Appender.AppendHistogram
Two Appenders race when creating a series with a native histogram
as the memSeries will be common and the lastHistogram field is written
without lock.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-10-10 14:10:07 +02:00
beorn7 12c39d5421 docs: Some nitpicking in chunks.md
- `float histogram` → `floathistogram`, as it is used in the code.
- Actual link encodings to the code (to find the actual numerical values).
- `<bytes>` → `<data>` for consistency.

Signed-off-by: beorn7 <beorn@grafana.com>
2024-10-09 14:32:12 +02:00
beorn7 a4cb52ff15 docs: Update chunk layot for NHCB
Signed-off-by: beorn7 <beorn@grafana.com>
2024-10-09 14:19:20 +02:00
Björn Rabenstein 02d0de9987
Merge pull request #14997 from fionaliao/fl/update-format-docs
Update chunk format docs with native histograms and OOO
2024-10-09 13:29:01 +02:00
TJ Hoplock 6ebfbd2d54 chore!: adopt log/slog, remove go-kit/log
For: #14355

This commit updates Prometheus to adopt stdlib's log/slog package in
favor of go-kit/log. As part of converting to use slog, several other
related changes are required to get prometheus working, including:
- removed unused logging util func `RateLimit()`
- forward ported the util/logging/Deduper logging by implementing a small custom slog.Handler that does the deduping before chaining log calls to the underlying real slog.Logger
- move some of the json file logging functionality to use prom/common package functionality
- refactored some of the new json file logging for scraping
- changes to promql.QueryLogger interface to swap out logging methods for relevant slog sugar wrappers
- updated lots of tests that used/replicated custom logging functionality, attempting to keep the logical goal of the tests consistent after the transition
- added a healthy amount of `if logger == nil { $makeLogger }` type conditional checks amongst various functions where none were provided -- old code that used the go-kit/log.Logger interface had several places where there were nil references when trying to use functions like `With()` to add keyvals on the new *slog.Logger type

Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>
2024-10-07 15:58:50 -04:00
Bryan Boreham 91de19fbef [BUGFIX] TSDB: Don't read in-order chunks from before head MinTime
Because we are reimplementing the `IndexReader` to fetch in-order and
out-of-order chunks together, we must reproduce the behaviour of
`Head.indexRange()`, which floors the minimum time queried at `head.MinTime()`.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-10-07 13:50:03 +01:00
Matthieu MOREL ab64966e9d
fix: use "ErrorContains" or "EqualError" instead of "Contains(t, err.Error()" and "Equal(t, err.Error()" (#15094)
* fix: use "ErrorContains" or "EqualError" instead of "Contains(t, err.Error()" and "Equal(t, err.Error()"

---------

Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-10-06 16:35:29 +00:00