Commit Graph

1245 Commits (4aeed2c4b1684f6e696defbfc975156fcb7f1bf6)

Author SHA1 Message Date
Jeanette Tan 14f8dded39 Merge branch 'main' into nhcb
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2024-06-07 19:17:14 +08:00
Jeanette Tan 9adc1699c3 fix according to code review
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2024-06-07 18:50:59 +08:00
Ben Ye 8a08f452b6
tsdb: Allow passing a custom compactor to override the default one (#14113)
* expose hook in tsdb to allow customizing compactor

Signed-off-by: Ben Ye <benye@amazon.com>

* address comment

Signed-off-by: Ben Ye <benye@amazon.com>

---------

Signed-off-by: Ben Ye <benye@amazon.com>
2024-06-04 19:11:36 -04:00
Bryan Boreham 42b546a43d
tsdb: add details to duplicate sample error (#13277)
Now the error will include the timestamp and the existing and new values.
When you are trying to track down the source of this error, it can be
useful to see that the values are close, or alternating, or something
else.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-06-04 08:54:09 +01:00
Arve Knudsen b8b9015e38 tsdb/index: Fix TestReader_PostingsForLabelMatchingHonorsContextCancel
Fix number of series in
TestReader_PostingsForLabelMatchingHonorsContextCancel (off by one).

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-06-03 17:29:06 +02:00
Bryan Boreham 3ee52abb53 [ENHANCEMENT] TSDB: Save map lookup on validation
Goes faster.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-05-30 09:17:11 +01:00
Bryan Boreham 7d98487447 [ENHANCEMENT] TSDB: let Resize re-use buffer
This saves having to zero the buffer every time.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-05-30 09:17:11 +01:00
Bryan Boreham c0bb156eca [ENHANCEMENT] TSDB: Eliminate pointer when storing exemplars
Saves memory and effort.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-05-30 09:17:11 +01:00
Bryan Boreham 3eb5581877 [ENHANCEMENT] TSDB: Reduce map lookups on exemplar index
In many cases we already have a pointer to the entry.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-05-30 09:17:11 +01:00
Bryan Boreham f0c50b5a66 [Test] TSDB: BenchmarkResizeExemplar multiple per series
One exemplar per series is not a typical workload. Make it the same as
`BenchmarkAddExemplar`.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-05-30 09:17:11 +01:00
Bryan Boreham 929fbf860e [Test] TSDB: let BenchmarkAddExemplar reuse slots
Test with different amounts of capacity and exemplars, so that sometimes
new exemplars are evicting older exemplars.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-05-30 09:16:30 +01:00
Ben Ye 6683895620
optimize regex matching for empty label values in posting match (#14075)
Also update tests.

Signed-off-by: Ben Ye <benye@amazon.com>
2024-05-29 16:03:33 +01:00
Arve Knudsen 694f717dc4 Watcher.readSegment: Only consider unknown rec types failures
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-05-28 15:23:50 +02:00
Arve Knudsen a4942ffa8c Merge remote-tracking branch 'prometheus/main' into arve/wlog-histograms 2024-05-28 12:06:29 +02:00
Arve Knudsen b2396c0c8f Upgrade to golangci-lint v1.59.0
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-05-27 22:38:48 +02:00
Arve Knudsen 90145fa884 Merge remote-tracking branch 'prometheus/main' into arve/wlog-histograms 2024-05-27 15:19:33 +02:00
Alan Protasio 8894d65cd6
Fix head stats and hooks when replaying a corrupted snapshot (#14079)
* Fixing head stats and hooks when replaying a corrupted snapshot

Signed-off-by: alanprot <alanprot@gmail.com>

* Fixing create/removed series metrics

Signed-off-by: alanprot <alanprot@gmail.com>

* Refactoring to have common code between gc and flush method

Signed-off-by: alanprot <alanprot@gmail.com>

* Update tsdb/head.go

Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com>
Signed-off-by: Alan Protasio <alanprot@gmail.com>

* refactor

Signed-off-by: alanprot <alanprot@gmail.com>

* Update tsdb/head_test.go

Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
Signed-off-by: Alan Protasio <alanprot@gmail.com>

* Update tsdb/head_test.go

Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
Signed-off-by: Alan Protasio <alanprot@gmail.com>

---------

Signed-off-by: alanprot <alanprot@gmail.com>
Signed-off-by: Alan Protasio <alanprot@gmail.com>
Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
2024-05-24 22:43:21 -04:00
Björn Rabenstein 3119b8a055
Merge pull request #13218 from machine424/ro-promtool
Make DBReadOnly more RO
2024-05-21 13:27:40 +02:00
Oleg Zaytsev fe9cb5a803
Check context every 128 labels instead of 100 (#14118)
Follow up on https://github.com/prometheus/prometheus/pull/14096

As promised, I bring a benchmark, which shows a very small improvement
if context is checked every 128 iterations of label instead of every
100.

It's much easier for a computer to check modulo 128 than modulo 100.
This is a very small 0-2% improvement but I'd say this is one of the
hottest paths of the app so this is still relevant.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2024-05-21 11:30:43 +02:00
Arve Knudsen 5ca56eeb6b
tsdb/index: Refactor Reader tests (#14071)
tsdb/index: Refactor Reader tests

Co-authored-by: Björn Rabenstein <github@rabenste.in>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

---------

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
2024-05-16 11:51:46 +02:00
Oleksandr Redko f10c3454e9 Enable perfsprint linter and fix up code
Signed-off-by: Oleksandr Redko <oleksandr.red+github@gmail.com>
2024-05-15 17:51:05 +03:00
György Krajcsovits b215a41be4 tsdb/index/postings: fix missing lock unlock
Followup to #14096

Unfortunately the previous PR introduced this bug by not releasing the
lock before returning.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-05-15 14:02:39 +02:00
George Krajcsovits fdaafdb041
tsdb: check for context cancel before regex matching postings (#14096)
* tsdb: check for context cancel before regex matching postings

Regex matching can be heavy if the regex takes a lot of cycles to
evaluate and we can get stuck evaluating postings for a long time
without this fix. The constant checkContextEveryNIterations=100
may be changed later.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-05-15 06:26:19 +02:00
Jeanette Tan f028496133 Merge branch 'main' into nhcb
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2024-05-14 16:20:15 +08:00
Arve Knudsen f094dd50e0 Merge remote-tracking branch 'prometheus/main' into arve/wlog-histograms 2024-05-09 11:58:37 +02:00
Arve Knudsen 5c4310aa37
[ENHANCEMENT] TSDB: Optimize querying with regexp matchers
Add method `PostingsForLabelMatching` to `tsdb.IndexReader`, to obtain postings for labels with a certain name and values accepted by a provided callback, and use it from `tsdb.PostingsForMatchers`.
The intention is to optimize regexp matcher paths, especially not having to load all label values before matching on them.

Plus tests, and refactor some `tsdb/index.Reader` methods.

Benchmarking shows memory reduction up to ~100%, and speedup of up to ~50%.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
2024-05-09 10:55:30 +01:00
Arve Knudsen 90f08de96a Merge remote-tracking branch 'prometheus/main' into arve/wlog-histograms 2024-05-08 18:14:30 +02:00
Arve Knudsen d699dc3c77
Fix language in docs and comments (#14041)
Fix language in docs and comments

---------

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
2024-05-08 17:57:09 +02:00
Arve Knudsen f1e4c33943 Merge remote-tracking branch 'prometheus/main' into arve/wlog-histograms 2024-05-08 15:06:41 +02:00
Arve Knudsen 108a6bc9f6 tsdb/chunkenc.Pool: Refactor Get and Put
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-05-08 13:37:25 +02:00
Jeanette Tan 796b1bbfde Merge branch 'main' into nhcb
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2024-05-08 19:11:39 +08:00
Arve Knudsen fb6a45f06b tsdb/wlog: Only treat unknown record types as failure
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-05-06 11:58:02 +02:00
Alan Protasio d15869af32
Avoid creating new slices for labels values on postings for matchers (#13958)
* Avoid creating new slices for labels values on postings for matchers

Signed-off-by: alanprot <alanprot@gmail.com>

* refactor

Signed-off-by: alanprot <alanprot@gmail.com>

---------

Signed-off-by: alanprot <alanprot@gmail.com>
2024-04-24 16:41:33 +02:00
György Krajcsovits bcafa5f1f9 Merge remote-tracking branch 'upstream/main' into update-nhcb 2024-04-24 11:06:59 +02:00
Arthur Silva Sens b5b5e1e5ae
Merge pull request #13919 from GiedriusS/dont_forget_to_unregister
tsdb/wlog: unregister metrics on WL close
2024-04-18 16:44:03 -03:00
Giedrius Statkevičius bdf490726a tsdb/wlog: add test for metrics unregistering
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
2024-04-18 11:11:37 +03:00
machine424 c5a1cc9148
chore(tsdb): add a sandboxDir to DBReadOnly, the directory can be used for transient file writes.
use it in loadDataAsQueryable to make sure the RO Head doesn't truncate or cut new chunks in data/chunks_head/.

add a -sandbox-dir-root flag to "promtool tsdb dump/dump-openmetrics" to control the root of that sandbox dirrectory.

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2024-04-15 17:00:25 +02:00
Giedrius Statkevičius 3b8fe00767 tsdb/wlog: unregister metrics on WL close
Thanos can create and destroy TSDBs dynamically, and once a TSDB
disappears its files are deleted. Calculating the size of the
WAL then fails with errors like:

```
msg: "Failed to calculate size of "wal" dir", "err": "lstat
/tsdbdir/wal: no such file or directory", "caller": "wlog.go:271"
```

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
2024-04-11 11:30:05 +03:00
Matthieu MOREL 6f595c6762
golangci-lint: enable whitespace linter (#13905)
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2024-04-11 09:27:54 +01:00
Jonathan Halterman 633224886a
Write out of order hint when initially creating meta file (#13894)
Signed-off-by: Jonathan Halterman <jonathan@grafana.com>
Signed-off-by: Jonathan Halterman <jhalterman@gmail.com>
Co-authored-by: Jesus Vazquez <jesusvazquez@users.noreply.github.com>
2024-04-08 17:34:14 +02:00
Łukasz Mierzwa 277f04f0c4
Stop compactions if there's a block to write (#13754)
* Stop compactions if there's a block to write

db.Compact() checks if there's a block to write with HEAD chunks before calling db.compactBlocks().
This is to ensure that if we need to write a block then it happens ASAP, otherwise memory usage might keep growing.

But what can also happen is that we don't need to write any block, we start db.compactBlocks(),
compaction takes hours, and in the meantime HEAD needs to write out chunks to a block.

This can be especially problematic if, for example, you run Thanos sidecar that's uploading block,
which requires that compactions are disabled. Then you disable Thanos sidecar and re-enable compactions.
When db.compactBlocks() is finally called it might have a huge number of blocks to compact, which might
take a very long time, during which HEAD cannot write out chunks to a new block.
In such case memory usage will keep growing until either:
- compactions are finally finished and HEAD can write a block
- we run out of memory and Prometheus gets OOM-killed

This change adds a check for pending HEAD block writes inside db.compactBlocks(), so that
we bail out early if there are still compactions to run, but we also need to write a new
block.

Also add a test for compactBlocks.

---------

Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>
Signed-off-by: Lukasz Mierzwa <lukasz@cloudflare.com>
2024-04-07 18:28:28 +01:00
Bryan Boreham fc567a1df8
Merge pull request #13889 from komisan19/refactor/utilize_standard_functions_max/min
refactor: utilize standard functions max/min in promtool and tsdb
2024-04-06 10:23:18 +01:00
Julien 8eb9228af8
Merge pull request #13864 from yeya24/expose-compactor-metrics
Expose compactor metrics
2024-04-05 11:24:41 +02:00
Jonathan Halterman 113938aeb8
Log out of order when writing a block (#13888)
Signed-off-by: Jonathan Halterman <jonathan@grafana.com>
2024-04-04 14:26:13 +02:00
komisan19 0249e080b4 refactor: utilize standard functions max/min
Signed-off-by: komisan19 <18901496+komisan19@users.noreply.github.com>
2024-04-04 03:15:38 +09:00
Nicolas Takashi 8125634086
[refactor] moving mergedOOOChunks Iterator (#13881)
Signed-off-by: Nicolas Takashi <nicolas.tcs@hotmail.com>
2024-04-03 10:14:34 +02:00
carehabit a672662073
all: fix some typos (#13863)
Signed-off-by: carehabit <shenyuting@outlook.com>
2024-04-01 18:06:05 +02:00
Ben Ye ded35ef20d expose compactor metrics
Signed-off-by: Ben Ye <benye@amazon.com>
2024-03-31 15:10:29 -07:00
Nicolas Takashi 0b762db154
[refactor] moving mergedOOOChunks to ooo_head_read
Signed-off-by: Nicolas Takashi <nicolas.tcs@hotmail.com>
2024-03-29 23:33:15 +00:00
György Krajcsovits 2a4aa085d2 Merge branch 'main' into nhcb 2024-03-27 18:42:10 +01:00
George Krajcsovits 4eab18abd6
[nhcb branch] Use single bit to differentiate between optimized bounds and floats (#13828)
* Use single bit to differentiate between optimized bounds and floats

Use one bit to decide what kind of data to read/write.
This reduces storage need of floats from 72 bits to 65 bits and makes the
integers store in 5 to 32 bits instead of 16.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>
Co-authored-by: Jeanette Tan <jeanette.tan@grafana.com>
2024-03-27 18:40:59 +01:00
Arve Knudsen 35aab01de0 tsdb/wlog.Checkpoint: Handle also float histograms
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-03-26 15:27:32 +01:00
Bartlomiej Plotka de05d5e11b
Merge pull request #13776 from aknuds1/arve/checkpoint-histogram-samples
tsdb/wlog.Checkpoint: Fix counting of histogram samples
2024-03-26 12:33:48 +01:00
Bartlomiej Plotka 25578f2b22
[test] Merge pull request #13790 from aknuds1/arve/retention-commit
tsdb.BeyondTimeRetention: Fix comment and test at retention duration
2024-03-26 12:26:32 +01:00
Nick Pillitteri 481f14e1c0
TSDB: Don't rely on integer overflow in head compaction check (#13755)
* TSDB: Don't compact the head block when empty

Don't compact the Head block if there have not yet been any samples
appended.

Previously, the logic for determining if the head should be compacted
relied on the default values for min and max time and integer overflow
when they were checked in `Head.compactable()`. The check in
`Head.compactable()` effectively did `math.MinInt64 - math.MaxInt64`
which overflowed and wrapped to `1`. Since `1` is less than `1.5`
times the chunk range, compaction did not happen. This was the correct
behavior but relying on overflow wrapping is surprising.

This change add a method for checking if the min and max time for the
head is unset and uses it to short-circuit compaction in that case.
It also replaces several explicit checks for the default value to
determine if the head has not yet had any samples added.

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
2024-03-26 12:17:38 +01:00
Ben Ye ceca6c4716
[ENHANCEMENT] TSDB: Log more statistics during startup (#13838)
* log chunk snapshot and mmap chunks replay duration together with total replay duration

Signed-off-by: Ben Ye <benye@amazon.com>
2024-03-26 11:16:27 +00:00
Bryan Boreham 78c0fd2f4d
Merge pull request #13799 from machine424/wbl
chore(tsdb): set the wbl to nil as well in DBReadOnly.loadDataAsQuery…
2024-03-25 15:03:56 +01:00
Bryan Boreham 080d440bf8 Merge remote-tracking branch 'origin/main' into pr/13461 2024-03-25 12:14:26 +00:00
György Krajcsovits a3d1a46eda Merge branch 'main' into nhcb 2024-03-22 14:51:48 +01:00
zenador 4acbb7dea6
Add custom buckets to native histogram chunks encoding (#13706)
* add custom bounds to chunks encoding
* change custom buckets schema number
* rename custom bounds to custom values

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2024-03-22 14:36:39 +01:00
machine424 2a2e2ed28b
chore(tsdb): set the wbl to nil as well in DBReadOnly.loadDataAsQueryable
Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2024-03-20 17:13:39 +01:00
Arve Knudsen 07332f7427 TestTimeRetention: Split into two sub-tests
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-03-20 15:06:36 +01:00
Arve Knudsen af694dc295 Merge TestDB_BeyondTimeRetention into TestTimeRetention
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-03-20 09:07:16 +01:00
Arve Knudsen 9c7a734063 tsdb.BeyondTimeRetention: Fix comment and test at retention duration
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-03-19 09:10:21 +01:00
Bryan Boreham a0e93e403e
Merge pull request #13764 from bboreham/remove-deprecated-wal
[Cleanup] TSDB: Remove old deprecated WAL implementation

Deprecated since 2018.
2024-03-17 09:34:57 +00:00
Darshan Chaudhary b7047f7fcb
Fix retention boundary so 2h retention deletes blocks right at the 2h boundary (#9633)
Signed-off-by: darshanime <deathbullet@gmail.com>
2024-03-15 19:35:16 +01:00
Arve Knudsen cef1025ea8 tsdb/wlog.Checkpoint: Fix counting of histogram samples
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-03-15 10:23:59 +01:00
Bryan Boreham d45b5deb75 TSDB: move function only used in tests
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-03-15 08:54:47 +00:00
Bryan Boreham 3274cac0d3 TSDB: remove unused function
Was only used in old WAL implementation.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-03-15 08:51:57 +00:00
Arve Knudsen 1de49d5b69
Remove unused function tsdb/chunks.PopulatedChunk (#13763)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-03-14 11:15:17 +01:00
Bryan Boreham 87edf1f960 [Cleanup] TSDB: Remove old deprecated WAL implementation
Deprecated since 2018.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-03-13 15:57:23 +00:00
Bryan Boreham d08f054950
[ENHANCEMENT] TSDB: Check CRC without allocating (#13742)
Use the existing utility function which does this.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-03-12 12:24:27 +01:00
carrychair 856f6e49c8 fix function and struct name
Signed-off-by: carrychair <linghuchong404@gmail.com>
2024-03-09 17:53:17 +08:00
Björn Rabenstein 9acae57937
Merge pull request #13681 from krajorama/native-latency-histograms
Add native histograms to latency/duration metrics
2024-03-07 20:46:43 +01:00
Bryan Boreham bbe39af99f
tsdb: zero out Labels and memSeries pointers in pool (#13712)
* tsdb: zero out Labels and memSeries pointers in pool

So that the garbage-collector doesn't see this memory as still in use.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

---------

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Signed-off-by: Björn Rabenstein <github@rabenste.in>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
2024-03-06 13:36:34 +01:00
György Krajcsovits 4d4d822c36 Add native histograms to latency/duration metrics
Dogfood native histograms.
Allow dependent projects to migrate to native histograms.

I took the defaults from client_golang.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-03-01 14:44:38 +01:00
machine424 f477e0539a
Move from golang.org/x/exp/slices into slices now that we only support Go >= 1.21
Prevent adding back golang.org/x/exp/slices.

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2024-02-28 14:54:53 +01:00
Bryan Boreham 925134e6de tsdb tests: make work with labels SymbolTable
Need to initialize decoders with SymbolTable.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-26 11:45:25 +00:00
Bryan Boreham 93b72ec5dd tsdb: create SymbolTables for labels as required
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-26 11:45:25 +00:00
Bryan Boreham 4d6bb2e0e4
Merge pull request #13628 from bboreham/cleanup-13583
tsdb/wlog: small cleanup of WAL watcher after #13583
2024-02-23 12:39:17 +00:00
George Krajcsovits c6c8f63516
Merge pull request #13607 from fionaliao/ooo-samples-appended-type
Add sample type label to outOfOrderSamplesAppended metric
2024-02-23 09:41:45 +01:00
Bryan Boreham 6ed56c9f04 WAL watcher: improve comments
Clarify in the first comment that it is `watch()` that waits, and reduce
verbiage.

The second comment was slightly contradictory to the first and otherwise
didn't seem to add much, since `currentSegment` was incremented just a
few lines later.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-22 09:32:46 +00:00
Bryan Boreham a975a83079 tsdb: clean up Watcher debug messages
Print lastSegment after it gets initialized.

Move variable declaration to first use.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-22 09:19:18 +00:00
Bryan Boreham 78f46bccca tsdb/wlog tests: remove unnecessary sleep check
Sleep() is documented to return immediately on negative or zero input.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-22 09:14:52 +00:00
Callum Styan 0c71230784
fix bug that would cause us to endlessly fall behind (#13583)
* fix bug that would cause us to only read from the WAL on the 15s
fallback timer if remote write had fallen behind and is no longer
reading from the WAL segment that is currently being written to

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* remove unintended logging, fix lint, plus allow test to take slightly
longer because cloud CI

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* address review feedback

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* fix watcher sleeps in test, flu brain is smooth

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* increase timeout, unfortunately cloud CI can require a longer timeout

Signed-off-by: Callum Styan <callumstyan@gmail.com>

---------

Signed-off-by: Callum Styan <callumstyan@gmail.com>
2024-02-21 17:09:07 -08:00
Fiona Liao 841a133514 Move histogramsAppended to be more consistent
Signed-off-by: Fiona Liao <fiona.liao@grafana.com>
2024-02-21 11:15:04 +00:00
Fiona Liao 52389647b2 Add type label to outOfOrderSamplesAppended metric
Signed-off-by: Fiona Liao <fiona.liao@grafana.com>
2024-02-19 15:24:39 +00:00
Bryan Boreham c0e36e6bb3 Standardise exemplar label as "trace_id"
This is consistent with the OpenTelemetry standard, and an example in OpenMetrics.

https://github.com/open-telemetry/opentelemetry-specification/blob/89aa01348139/specification/metrics/data-model.md#exemplars
https://github.com/OpenObservability/OpenMetrics/blob/138654493130/specification/OpenMetrics.md#exemplars-1

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-15 14:20:08 +00:00
Bryan Boreham 12cac5bd5c tsdb tests: use go-cmp instead of DeepEquals
Also one simpler call checking nil.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-08 19:32:33 +00:00
Bryan Boreham 17f48f2b3b Tests: use replacement DeepEquals in more places
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-08 19:32:33 +00:00
Bryan Boreham 39af788dbd Tests: use replacement DeepEquals using go-cmp
Use DeepEqual replacement using go-cmp, which is more flexible.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-08 19:30:20 +00:00
Peter Štibraný e2b9cfeeeb
Enforce chunks ordering when writing index. (#8085)
Document conditions on chunks. Add check on chunk time ordering.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
2024-02-04 16:31:49 +01:00
Bryan Boreham 98c4889029
Merge pull request #9298 from Creatone/creatone/use-testify
tests: Move from t.Errorf and others.
2024-02-04 16:27:57 +01:00
Mikhail Fesenko 419dd265cc
Fix strange code, add messages to code brought in #8106 (#13509)
Signed-off-by: Mikhail Fesenko <proggga@gmail.com>
2024-02-02 10:00:38 +01:00
Bryan Boreham 16e68c01e4 tests: remove err from message when testify prints it already
For instance `require.NoError` will print the unexpected error; we don't
need to include it in the message.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-01 14:18:01 +00:00
Mikhail Fesenko 5f2c3a5d3e
Small improvements, add const, remove copypasta (#8106)
Signed-off-by: Mikhail Fesenko <proggga@gmail.com>
Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com>
2024-02-01 14:30:50 +01:00
Paweł Szulik 5961f78186 Refactor tsdb tests to use testify.
Signed-off-by: Paweł Szulik <paul.szulik@gmail.com>
2024-01-31 16:03:17 +00:00
Bryan Boreham 34230bb172 tsdb/wlog: close segment files sooner
'defer' runs at the end of the whole function; we should close each
segment file as soon as we finished reading it.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-01-31 12:12:19 +00:00
Bryan Boreham cd4562d3a6
Merge pull request #13473 from bboreham/pure-mutex
tsdb: use cheaper Mutex on series
2024-01-30 09:57:08 +00:00
Marco Pracucci 501bc6419e
Add ShardedPostings() support to TSDB (#10421)
This PR is a reference implementation of the proposal described in #10420.

In addition to what described in #10420, in this PR I've introduced labels.StableHash(). The idea is to offer an hashing function which doesn't change over time, and that's used by query sharding in order to get a stable behaviour over time. The implementation of labels.StableHash() is the hashing function used by Prometheus before stringlabels, and what's used by Grafana Mimir for query sharding (because built before stringlabels was a thing).

Follow up work
As mentioned in #10420, if this PR is accepted I'm also open to upload another foundamental piece used by Grafana Mimir query sharding to accelerate the query execution: an optional, configurable and fast in-memory cache for the series hashes.

Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-01-29 11:57:27 +00:00
Bryan Boreham 66237c1996 tsdb: use cheaper Mutex on series
Mutex is 8 bytes; RWMutex is 24 bytes and much more complicated. Since
`RLock` is only used in two places, `UpdateMetadata` and `Delete`,
neither of which are hotspots, we should use the cheaper one.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-01-26 11:01:39 +00:00
Marco Pracucci ec9cada56e
Remove unused isRegexMetaCharacter()
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-01-26 06:35:02 +01:00
Marco Pracucci 515890ec53
Use Matcher.SetMatches()
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-01-26 06:26:52 +01:00
Marco Pracucci a1a45990a2
Fix TestPostingsForMatcher
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-01-25 14:59:39 +01:00
Bryan Boreham b9eab6e4b8
tsdb: simplify internal series delete function (#13261)
Lifting an optimisation from Agent code, `seriesHashmap.del` can use
the unique series reference, doesn't need to check Labels.
Also streamline the logic for deleting from `unique` and `conflicts` maps,
and add some comments to help the next person.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-01-25 11:57:54 +01:00
Bryan Boreham 3f30ad3cc2
Merge pull request #13015 from bboreham/smaller-txring
tsdb: make transaction isolation data structures smaller
2024-01-25 10:48:15 +00:00
Arve Knudsen ba7012ec6a
TestHeadLabelValuesWithMatchers: Add test case (#13414)
Add test case to TestHeadLabelValuesWithMatchers, while fixing a couple
of typos in other test cases. Also enclosing some implicit sub-tests in a
`t.Run` call to make them explicitly sub-tests.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-01-24 10:47:56 +01:00
Filip Petkovski 583f3e587c
Optimize histogram iterators (#13340)
Optimize histogram iterators

Histogram iterators allocate new objects in the AtHistogram and
AtFloatHistogram methods, which makes calculating rates over long
ranges expensive.

In #13215 we allowed an existing object to be reused
when converting an integer histogram to a float histogram. This commit follows
the same idea and allows injecting an existing object in the AtHistogram and
AtFloatHistogram methods. When the injected value is nil, iterators allocate
new histograms, otherwise they populate and return the injected object.

The commit also adds a CopyTo method to Histogram and FloatHistogram which
is used in the BufferedIterator to overwrite items in the ring instead of making
new copies.

Note that a specialized HPoint pool is needed for all of this to work 
(`matrixSelectorHPool`).

---------

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
2024-01-23 17:02:14 +01:00
Oleg Zaytsev ed172a6667
Optimize label values with matchers by taking shortcuts (#13426)
Don't calculate postings beforehand: we may not need them. If all
matchers are for the requested label, we can just filter its values.

Also, if there are no values at all, no need to run any kind of
logic.

Also add more labelValuesWithMatchers benchmarks

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2024-01-23 11:40:21 +01:00
Julien Pivotto f52605b584
Merge pull request #13415 from aknuds1/arve/test-label-values-with-matchers-one-more
TestLabelValuesWithMatchers: Add test case
2024-01-18 11:57:12 +01:00
tyltr f97fa2736c remove obsolete build tag
Signed-off-by: tyltr <tylitianrui@126.com>
2024-01-17 22:26:32 +08:00
Arve Knudsen 8598150f48 TestLabelValuesWithMatchers: Add test case
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-01-17 12:51:56 +01:00
Marco Pracucci 7852a7c516
Fix regressions introduced by #13242
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-01-16 12:00:53 +01:00
Giedrius Statkevičius b695e069b8
tsdb/main: wire "EnableOverlappingCompaction" to tsdb.Options (#13398)
This added the https://github.com/prometheus/prometheus/pull/13393
"EnableOverlappingCompaction" parameter to the compactor code but not to
the tsdb.Options. I forgot about that. Add it to `tsdb.Options` too and
set it to `true` in Prometheus.

Copy/paste the description from
https://github.com/prometheus/prometheus/pull/13393#issuecomment-1891787986

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
2024-01-15 16:42:40 +01:00
Ben Kochie 17920623e7
Merge pull request #13391 from GiedriusS/compact_merge_func
tsdb/compact: fix passing merge func
2024-01-15 09:43:06 +01:00
Giedrius Statkevičius 3a48adc54f tsdb: add enable overlapping compaction
This functionality is needed in downstream projects because they have a
separate component that does compaction.

Upstreaming
7c8e9a2a76/tsdb/compact.go (L323-L325).

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
2024-01-12 11:19:41 +02:00
Giedrius Statkevičius 9b759135d1 tsdb/compact: fix passing merge func
Fixing a very small logical problem I've introduced :(.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
2024-01-11 12:07:54 +02:00
Giedrius Statkevičius 61b4080a14
tsdb/{index,compact}: allow using custom postings encoding format (#13242)
* tsdb/{index,compact}: allow using custom postings encoding format

We would like to experiment with a different postings encoding format in
Thanos so in this change I am proposing adding another argument to
`NewWriter` which would allow users to change the format if needed.
Also, wire the leveled compactor so that it would be possible to change
the format there too.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* tsdb/compact: use a struct for leveled compactor options

As discussed on Slack, let's use a struct for the options in leveled
compactor.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* tsdb: make changes after Bryan's review

- Make changes less intrusive
- Turn the postings encoder type into a function
- Add NewWriterWithEncoder()

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

---------

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
2024-01-08 09:48:27 +00:00
Bryan Boreham bad3f23f23 agent: add BenchmarkCreateSeries
Based on the one in tsdb/head_test.go.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-31 10:23:43 +00:00
Bryan Boreham e64d7d8928 agent: make the global hash lookup table smaller
This is the same change made in #13040, plus subsequent improvements,
applied to agent-mode code.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-31 10:23:43 +00:00
Bryan Boreham 252031c86f Revert "Adding small test update for temp dir using t.TempDir (#13293)"
This reverts commit 2ddb3596ef.

Various tests are failing in CI after this change; reverting to free up
other work.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-30 19:17:30 +00:00
Mile Druzijanic 2ddb3596ef
Adding small test update for temp dir using t.TempDir (#13293)
* Adding small test update for temp dir using t.TempDir

Signed-off-by: Mile Druzijanic <miledruz@gmail.com>
Signed-off-by: Mile Druzijanic <zedsprogramms@gmail.com>

* removing not required cleanup

Signed-off-by: Mile Druzijanic <zedsprogramms@gmail.com>

---------

Signed-off-by: Mile Druzijanic <miledruz@gmail.com>
Signed-off-by: Mile Druzijanic <zedsprogramms@gmail.com>
2023-12-28 21:49:57 +01:00
Björn Rabenstein 6b8e945388
Merge pull request #13289 from fpetkovski/fix-histogram-reuse
Fix reusing float histograms
2023-12-25 22:45:03 +01:00
Bryan Boreham 8065bef172 Move metric type definitions to common/model
They are used in multiple repos, so common is a better place for them.
Several packages now don't depend on `model/textparse`, e.g.
`storage/remote`.

Also remove `metadata` struct from `api.go`, since it was identical to
a struct in the `metadata` package.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-19 18:56:54 +00:00
Filip Petkovski 1f69dcfa6b
Fix reusing float histograms
In https://github.com/prometheus/prometheus/pull/13276 we started reusing float histogram objects to reduce allocations in PromQL.
That PR introduces a bug where histogram pointers gets copied to the beginning of the histograms slice,
but are still kept in the end of the slice. When a new histogram is read into the last element,
it can overwrite a previous element because the pointer is the same.

This commit fixes the issue by moving outdated points to the end of the slice
so that we don't end up with duplicate pointers in the same buffer. In other words,
the slice gets rotated so that old objects can get reused.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2023-12-14 11:53:58 +01:00
Bryan Boreham d0c2d9c0b9
Merge pull request #12878 from bboreham/loser-tree
postings: use Loser Tree for merge
2023-12-12 21:38:30 +00:00
Björn Rabenstein 928d07e3bd
Merge branch 'main' into arve/typos
Signed-off-by: Björn Rabenstein <beorn@grafana.com>
2023-12-12 12:02:03 +01:00
Giedrius Statkevičius f36b56a62c
tsdb: remove unused option (#13282)
Digging around the TSDB code and I've found that this flag is unused so
let's remove it.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
2023-12-12 09:58:54 +00:00
Matthieu MOREL 8f6cf3aabb tsdb: use Go standard errors
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-12-11 12:18:54 +00:00
Arthur Silva Sens 5082655392
Append Created Timestamps (#12733)
* Append created timestamps.

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

* Log when created timestamps are ignored

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

* Proposed changes to Append CT PR.

Changes:

* Changed textparse Parser interface for consistency and robustness.
* Changed CT interface to be more explicit and handle validation.
* Simplified test, change scrapeManager to allow testability.
* Added TODOs.

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Updates.

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Addressed comments.

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Refactor head_appender test

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

* Fix linter issues

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

* Use model.Sample in head appender test

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

---------

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>
Signed-off-by: bwplotka <bwplotka@gmail.com>
Co-authored-by: bwplotka <bwplotka@gmail.com>
2023-12-11 08:43:42 +00:00
Bryan Boreham ab3a47b489 postings: use Loser Tree for merge
It's faster.

Note change to test - instead of requiring that the data structure is
identical to `EmptyPostings()`, check that calling `Next()` returns
false, which implies it was empty.

Also the check for context cancellation during initialization was
removed. Initialization should be a small portion of the work done
during merge, so it's not worth plumbing a context argument through.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-08 16:07:25 +00:00
Bryan Boreham ee700151a3 tsdb/index: add benchmark for Postings.Merge
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-08 16:00:22 +00:00
Filip Petkovski 10a82f87fd
Enable reusing memory when converting between histogram types
The 'ToFloat' method on integer histograms currently allocates new memory
each time it is called.

This commit adds an optional *FloatHistogram parameter that can be used
to reuse span and bucket slices. It is up to the caller to make sure the
input float histogram is not used anymore after the call.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2023-12-08 10:22:59 +01:00
Matthieu MOREL 9c4782f1cc
golangci-lint: enable testifylint linter (#13254)
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-12-07 11:35:01 +00:00
Arve Knudsen 237bfea46b `chunks.Reader`: Fix typo in ChunkOrIterable doc string.
Also fix comment typo in `FloatHistogram.Sub`.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-12-07 08:28:45 +01:00
Matthieu MOREL 998fafe679
tsdb/wlog: use Go standard errors (#13144)
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-12-04 17:08:43 +00:00
Julien Levesy e4ec263bcc
fix(wlog/watcher): read segment synchronously when not tailing (#13224)
Signed-off-by: Julien Levesy <jlevesy@gmail.com>
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Co-authored-by: Callum Styan <callumstyan@gmail.com>
2023-12-01 14:26:38 -08:00
Julien Levesy 501f514389
feat(tsdb/agent): notify remote storage when commit happens (#13223)
Signed-off-by: Julien Levesy <jlevesy@gmail.com>
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Co-authored-by: Callum Styan <callumstyan@gmail.com>
2023-12-01 14:00:26 -08:00
Oleksandr Redko 2a75604f8e
Enable default revive rules (#13068)
Signed-off-by: Oleksandr Redko <Oleksandr_Redko@epam.com>
2023-11-29 17:23:34 +00:00
Fiona Liao b8bcaef14d
Fix histogram append errors (#13201)
* Fix histogram append errors

We should check counterReset condition rather than okToAppend because if
there's a counter reset, okToAppend is always set to false.

Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>
2023-11-29 11:39:12 +01:00
Fiona Liao ce126230e7
Fix chunks iterator bug when tombstone covers a whole chunk (#13209)
When no samples are returned in a chunk because all the samples have
been deleted, the chunk iterator then stops without iterating through
any remaining chunks.

Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>
2023-11-29 11:24:04 +01:00
Xiaochao Dong 28d8f1650c
tsdb: Make sure the cache for postings cardinality properly honors the label name (#12653)
Add a string remembering which label and limit the cache corresponds to.

Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
2023-11-28 13:54:37 +00:00
Arve Knudsen 1200c89d0c
Fix tsdb.stripeSeries.gc so it handles conflicts properly (#13195)
* Fix tsdb.stripeSeries.gc so it handles conflicts properly

tsdb.stripeSeries.gc needs to prune seriesHashmap.conflicts first,
otherwise seriesHashmap replaces the unique field with the first among
the conflicts. Also add regression test.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* TestStripeSeries_gc: Support stringlabels, don't use internals

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

---------

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-11-28 14:43:35 +01:00
Fiona Liao 5bee0cfce2
Change `ChunkReader.Chunk()` to `ChunkOrIterable()`
The ChunkReader interface's Chunk() has been changed to ChunkOrIterable(). 

This is a precursor to OOO native histogram support - with OOO native histograms, the chunks.Meta passed to Chunk() can result in multiple chunks being returned rather than just a single chunk (e.g. if oooMergedChunk has a counter reset in the middle). 

To support this, ChunkOrIterable() requires either a single chunk or an iterable to be returned. If an iterable is returned, the caller has the responsibility of converting the samples from the iterable into possibly multiple chunks. The OOOHeadChunkReader now returns an iterable rather than a chunk to prepare for the native histograms case. Also as a beneficial side effect, oooMergedChunk and boundedChunk has been simplified as they only need to implement the Iterable interface now, not the full Chunk interface.

---------

Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
2023-11-28 11:14:29 +01:00
Arve Knudsen ecc37588b0
tsdb: seriesHashmap.set by making receiver a pointer (#13193)
* Fix tsdb.seriesHashmap.set by making receiver a pointer

The method tsdb.seriesHashmap.set currently doesn't set the conflicts
field properly, due to the receiver being a non-pointer. Fix by turning
the receiver into a pointer, and add a corresponding regression test.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-11-27 15:40:30 +00:00
Charles Korn 59844498f7
Fix issue where queries can fail or omit OOO samples if OOO head compaction occurs between creating a querier and reading chunks (#13115)
* Add failing test.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Don't run OOO head garbage collection while reads are running.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Add further test cases for different order of operations.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Ensure all queriers are closed if `DB.blockChunkQuerierForRange()` fails.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Ensure all queriers are closed if `DB.Querier()` fails.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Invert error handling in `DB.Querier()` and `DB.blockChunkQuerierForRange()` to make it clearer

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Ensure that queries that touch OOO data can't block OOO head garbage collection forever.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Address PR feedback: fix parameter name in comment

Co-authored-by: Jesus Vazquez <jesusvazquez@users.noreply.github.com>
Signed-off-by: Charles Korn <charleskorn@users.noreply.github.com>

* Address PR feedback: use `lastGarbageCollectedMmapRef`

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Address PR feedback: ensure pending reads are cleaned up if creating an OOO querier fails

Signed-off-by: Charles Korn <charles.korn@grafana.com>

---------

Signed-off-by: Charles Korn <charles.korn@grafana.com>
Signed-off-by: Charles Korn <charleskorn@users.noreply.github.com>
Co-authored-by: Jesus Vazquez <jesusvazquez@users.noreply.github.com>
2023-11-24 12:38:38 +01:00
Bryan Boreham f13bc1a5c9
Merge pull request #13040 from bboreham/smaller-stripeseries
TSDB: make the global hash lookup table smaller
2023-11-20 12:12:09 +00:00
Oleg Zaytsev f997c72f29
Make head block ULIDs descriptive (#13100)
* Make head block ULIDs descriptive

As far as I understand, these ULIDs aren't persisted anywhere, so it
should be safe to change them.

When debugging an issue, seeing an ULID like
`2ZBXFNYVVFDXFPGSB1CHFNYQTZ` or `33DXR7JA39CHDKMQ9C40H6YVVF` isn't very
helpful, so I propose to make them readable in their ULID string
version.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* Set a different ULID for RangeHead

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

---------

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2023-11-17 12:29:36 +01:00
Julien Pivotto 1b84c01b76
Merge pull request #13143 from mmorel-35/patch-3
tsdb/tsdbutil: use Go standard errors
2023-11-17 10:23:04 +01:00
Julien Pivotto 9cb96ad2ea
Merge pull request #13142 from mmorel-35/patch-2
tsdb/fileutil: use Go standard errors
2023-11-17 10:20:40 +01:00
Julien Pivotto 58eca19ac0
Merge pull request #13141 from mmorel-35/patch-1
tsdb/errors: fix errorlint linter
2023-11-17 10:20:00 +01:00
zenador 32ee1b15de
Fix error on ingesting out-of-order exemplars (#13021)
Fix and improve ingesting exemplars for native histograms.

See code comment for a detailed explanation of the algorithm.

Note that this changes the current behavior for all kind of samples slightly: We now allow exemplars with the same timestamp as during the last scrape if the value or the labels have changed.

Also note that we now do not ingest exemplars without timestamps for native histograms anymore.

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>

---------

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: zenador <zenador@users.noreply.github.com>
Co-authored-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
2023-11-16 15:07:37 +01:00
Matthieu MOREL d7c3bc4cb0
tsdb/tsdbutil: use Go standard errors
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-14 20:46:36 +01:00
Matthieu MOREL e60a508dd8 tsdb/errors: fix errorlint linter
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-14 19:16:12 +00:00
Matthieu MOREL e3041740e4 tsdb/fileutil: use Go standard errors
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-14 19:05:21 +00:00
Matthieu MOREL dd8871379a remplace errors.Errorf by fmt.Errorf
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-14 13:04:31 +00:00
Bryan Boreham 1bfb3ed062
Labels: reduce allocations when creating from TSDB WAL (#13044)
* Labels: reduce allocations when creating from TSDB

When reading the WAL, by passing references into the buffer we can avoid
copying strings under `-tags stringlabels`.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-14 11:36:35 +00:00
Julien Pivotto b718dca70b
Merge pull request #13108 from mmorel-35/patch-2
tsdb/chunkenc: use Go standard errors package
2023-11-14 00:54:35 +01:00
Julien Pivotto 3173feb5ef
Merge pull request #13107 from mmorel-35/patch-1
tsdb/agent: use Go standard errors package
2023-11-14 00:54:11 +01:00
Julien Pivotto 90ed7b08dc
Merge pull request #13124 from mmorel-35/patch-5
tsdb/index: use Go standard errors package
2023-11-14 00:53:49 +01:00
Julien Pivotto afc57eb306
Merge pull request #13109 from mmorel-35/patch-3
tsdb/chunks: use Go standard errors package
2023-11-14 00:52:59 +01:00
Julien Pivotto b6274ee747
Merge pull request #13114 from mmorel-35/patch-4
tsdb/wlog: use Go standard errors package
2023-11-14 00:52:21 +01:00
Julien Pivotto 7bcae56b6e
Merge pull request #13130 from mmorel-35/patch-6
tsdb/encoding: use Go standard errors package
2023-11-14 00:51:47 +01:00
Julien Pivotto f53af5c849
Merge pull request #13131 from mmorel-35/patch-7
tsdb/tombstones: use Go standard errors package
2023-11-14 00:51:31 +01:00
Julien Pivotto 17fe6bc296
Merge pull request #13133 from mmorel-35/patch-8
tsdb/record: use Go standard errors package
2023-11-14 00:51:16 +01:00
Bryan Boreham 65a443e6e3 TSDB: initialize conflicts map only when we need it.
Suggested by @songjiayang.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-13 16:20:31 +00:00
Bryan Boreham 37573d083b
Merge pull request #13056 from songjiayang/symbolCacheEntry-filed-alignment
TSDB block index writer: reduce memory used by symbol cache

Improve struct field alignment for symbolCacheEntry
2023-11-13 14:37:43 +00:00
George Krajcsovits acc114fe55
Fix panic during tsdb Commit (#13092)
* Fix panic during tsdb Commit

Fixes the following

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x19deb45]

goroutine 651118930 [running]:
github.com/prometheus/prometheus/tsdb.(*headAppender).Commit(0xc19100f7c0)
	/drone/src/vendor/github.com/prometheus/prometheus/tsdb/head_append.go:855 +0x245
github.com/prometheus/prometheus/tsdb.dbAppender.Commit({{0x35bd6f0?, 0xc19100f7c0?}, 0xc000fa4c00?})
	/drone/src/vendor/github.com/prometheus/prometheus/tsdb/db.go:1159 +0x2f

We theorize that the panic happened due the the series referenced by the
exemplar being removed between AppendExemplar and Commit due to being idle.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-11-12 14:51:37 +00:00
Matthieu MOREL 469e415d09
Update record.go
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-11 21:01:24 +01:00
Matthieu MOREL 69c07ec6ae
Update record_test.go
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-11 20:57:42 +01:00
Matthieu MOREL 63691d82a5
tsdb/record: use Go standard errors package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-11 20:52:49 +01:00
Matthieu MOREL c74b7ad4fb
Update tombstones.go
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-11 19:22:06 +01:00
Matthieu MOREL 118460a64f
tsdb/tombstones: use Go standard errors package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-11 19:19:50 +01:00
Matthieu MOREL 4d6d3c1715
tsdb/encoding: use Go standard errors package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-11 19:01:11 +01:00
George Krajcsovits 39a35d92bc
tsdb/head: wlog exemplars after samples (#13113)
When samples are committed in the head, they are also written to the WAL.
The order of WAL records should be sample then exemplar, but this was
not the case for native histogram samples. This PR fixes that.

The problem with the wrong order is that remote write reads the WAL and
sends the recorded timeseries in the WAL order, which means exemplars
arrived before histogram samples. If the receiving side is Prometheus
TSDB and the series has not existed before then the exemplar does not
currently create the series. Which means the exemplar is rejected and lost.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-11-11 17:30:16 +01:00
Matthieu MOREL 2972cc5e8f tsdb/index: use Go standard errors package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-09 21:37:41 +00:00
Bryan Boreham e6c0f69f98 TSDB: Only pay for hash collisions when they happen
Instead of a map of slices of `*memSeries`, ready for any of them to
hold series where hash values collide, split into a map of `*memSeries`
and a map of slices which is usually empty, since hash collisions are
a one-in-a-billion thing.

The `del` method gets more complicated, to maintain the invariant that
a series is only in one of the two maps.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-09 07:44:39 -06:00
Bryan Boreham ce4e757704 TSDB: refine variable naming in chunk gc
Slight further refactor.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-09 07:44:39 -06:00
Bryan Boreham 071d5732af TSDB: refactor cleanup of chunks and series
Extract the middle of the loop into a function, so it will be
easier to modify the `seriesHashmap` data structure.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-09 07:44:39 -06:00
machine424 a32fbc3658
head.go: Remove an unneeded snapshot trigger that was moved in https://github.com/prometheus/prometheus/pull/9328
and brougt back by mistake in 095f572d4a as part of https://github.com/prometheus/prometheus/pull/11447

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2023-11-09 11:46:46 +01:00
Matthieu MOREL fb48a351f0 tsdb/wlog: use Go standard errors package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-08 21:41:58 +00:00
songjiayang 443867f1aa symbolCacheEntry field type alignment, thus saving 8 bytes.
Signed-off-by: songjiayang <songjiayang1@gmail.com>
2023-11-09 00:43:27 +08:00
Arve Knudsen ae9221e152
tsdb/index.Symbols: Drop context argument from Lookup method (#13058)
Drop context argument from tsdb/index.Symbols.Lookup since lookup
should be fast and the context checking is a performance hit.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-11-08 13:08:33 +01:00
Matthieu MOREL ece8286305 tsdb/chunk: use Go standard errors package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-08 09:19:44 +00:00
Matthieu MOREL b60f9f801e tsdb/chunkenc: use Go standard errors package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-08 08:38:57 +00:00
Matthieu MOREL 724737006d
tsdb/agent: use Go standard errors package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>

Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-08 09:22:31 +01:00
Goutham Veeramachaneni b80617a248
Merge pull request #12881 from dimitarvdimitrov/dimitar/TestQuerierIndexQueriesRace-less-synchronisation
Improve sensitivity of TestQuerierIndexQueriesRace
2023-11-07 12:16:43 +01:00
Linas Medziunas ebed7d0612 Change Validate to be a method on histogram structs
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-03 16:47:59 +02:00
Linas Medziunas 1f8aea11d6 Move histogram validation code to model/histogram
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-03 16:17:24 +02:00
Linas Medziunas 1cd6c1cde5 ValidateHistogram: strict Count check in absence of NaNs
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-03 16:17:24 +02:00
beorn7 5dca994f64 Merge branch 'release-2.48' into beorn7/release 2023-11-02 19:58:33 +01:00
Jeanette Tan 52eb303031 Refactor assigning MinTime in histogram chunks
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-11-02 21:23:05 +08:00
Jeanette Tan 3ccaaa40ba Fix according to code review
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-11-02 13:37:07 +08:00
György Krajcsovits 4296ecbd14 tsdb/compact_test.go: test mixed typed series with PopulateBlock
Add testcase and update test so that it can test native histograms as well.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-11-02 13:33:42 +08:00
Jeanette Tan 27abf09e7f Fix missing MinTime in histogram chunks
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-11-02 13:33:39 +08:00
Jeanette Tan 2f7060bd5a Expand TestPopulateWithTombSeriesIterators to test earlier deletion intervals for histogram chunks as well as time-overlapping chunks
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-11-02 13:33:35 +08:00
Jeanette Tan 7a4a1127b7 Expand TestPopulateWithTombSeriesIterators to test min max times of chunks, including mixed chunks
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-11-02 13:33:33 +08:00
Jeanette Tan 04aabdd7cc Refactor TestPopulateWithDelSeriesIterator unit tests to reuse more code
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-11-02 13:33:30 +08:00
Jeanette Tan 46be85f2dc Make TestPopulateWithDelSeriesIterator tests cover histogram types and check MinTime
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-11-02 13:33:26 +08:00
Björn Rabenstein a43669e611
Merge pull request #12928 from alexandear/ci-enable-godot
ci(lint): enable godot; append dot at the end of comments
2023-11-01 17:15:41 +01:00
Julien Pivotto f568221610
Merge pull request #13057 from prometheus/release-2.48
Merge release-2.48 back into main
2023-10-31 15:24:39 -04:00
Oleksandr Redko fa90ca46e5 ci(lint): enable godot; append dot at the end of comments
Signed-off-by: Oleksandr Redko <Oleksandr_Redko@epam.com>
2023-10-31 19:53:38 +02:00
Oleksandr Redko 8e5f0387a2
ci(lint): enable nolintlint and remove redundant comments (#12926)
Signed-off-by: Oleksandr Redko <Oleksandr_Redko@epam.com>
2023-10-31 12:35:13 +01:00
zenador 80e977aae6
Remove `NewPossibleNonCounterInfo` and minimise creating empty annotations (#13012)
* Remove NewPossibleNonCounterInfo until it can be made more efficient, and avoid creating empty annotations as much as possible

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-10-24 17:36:07 +01:00
Márcio Carôso dff1c395f6
Expose --storage.tsdb.retention.time in metric prometheus_tsdb_retention_limit_seconds (#12986)
* Expose --storage.tsdb.retention.time in a metric

Signed-off-by: Marcio Caroso <msscaroso@gmail.com>

---------

Signed-off-by: Marcio Caroso <msscaroso@gmail.com>
2023-10-24 13:34:42 +02:00
Björn Rabenstein 059f7f0738
Merge pull request #12997 from prometheus/wal-samples-size
TSDB: Pre-size buffer to read samples from WAL
2023-10-24 13:26:06 +02:00
Bryan Boreham 90e98e0235 tsdb: create isolation transaction slice on demand
When Prometheus restarts it creates every series read in from the WAL,
but many of those series will be finished, and never receive any more
samples. By defering allocation of the txRing slice to when it is first
needed, we save 32 bytes per stale series.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-10-21 13:45:47 +00:00
Bryan Boreham 6fe8217ce4 tsdb: shrink txRing with smaller integers
4 billion active transactions ought to be enough for anyone.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-10-21 12:44:34 +00:00
Jeanette Tan 71a36d2396 Very minor refactor of the integer overflow fix
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-10-19 13:17:46 +08:00
Bryan Boreham 26fa2e8356 TSDB: Pre-size buffer to read samples from WAL
When reading the WAL this method is called with buffers from a pool, on
multiple goroutines. Pre-allocating sufficient size avoids slow growth
and many reallocations in `append`.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-10-17 17:31:26 +00:00
George Krajcsovits 7d7b9eacff
Fix int32 overflow issues (#12978)
On a 32 bit architecture the size of int is 32 bits. Thus converting from
int64, uint64 can overflow it and flip the sign.

Try for yourself in playground:
package main

import "fmt"

func main() {
	x := int64(0x1F0000001)
	y := int64(1)
	z := int32(x - y) // numerically this is 0x1F0000000
	fmt.Printf("%v\n", z)
}

Prints -268435456 as if x was smaller.

Followup to #12650

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-10-16 16:23:26 +02:00
Paschalis Tsilias 42b8f2f5fc tsdb/agent: allow ingestion of OOO samples (#12897)
Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com>
Signed-off-by: Levi Harrison <git@leviharrison.dev>
2023-10-15 13:47:42 -04:00
Ganesh Vernekar 4df2f2432b Additionally wrap WBL replay error (#12406)
* Additionally wrap WBL replay error

Although WBL replay is already wrapped with errLoadWbl,
there are other errors that can happen during a WBL replay.
We should not try to repair WAL in those cases.

This commit additionally wraps the final error in Head.Init again
with errLoadWbl so that WBL replay errors can be identified properly.

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com>
Co-authored-by: Jesus Vazquez <jesusvzpg@gmail.com>
Signed-off-by: Levi Harrison <git@leviharrison.dev>
2023-10-15 13:47:42 -04:00
Paschalis Tsilias afab845e65
tsdb/agent: allow ingestion of OOO samples (#12897)
Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com>
2023-10-13 16:33:09 +02:00
Ganesh Vernekar f5913266a1
Additionally wrap WBL replay error (#12406)
* Additionally wrap WBL replay error

Although WBL replay is already wrapped with errLoadWbl,
there are other errors that can happen during a WBL replay.
We should not try to repair WAL in those cases.

This commit additionally wraps the final error in Head.Init again
with errLoadWbl so that WBL replay errors can be identified properly.

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com>
Co-authored-by: Jesus Vazquez <jesusvzpg@gmail.com>
2023-10-13 14:21:35 +02:00
Oleg Zaytsev fe90dcccff
Revert ListPostings change (#12955)
Reverts change from https://github.com/prometheus/prometheus/pull/12906

The benchmarks show that it's slower when intersecting, which is a
common usage for ListPostings (when intersecting matchers from Head)

(old is before #12906, new is #12906):

                           │     old     │                 new                 │
                           │   sec/op    │   sec/op     vs base                │
Intersect/LongPostings1-16   20.54µ ± 1%   21.11µ ± 1%   +2.76% (p=0.000 n=20)
Intersect/LongPostings2-16   51.03m ± 1%   52.40m ± 2%   +2.69% (p=0.000 n=20)
Intersect/ManyPostings-16    194.2m ± 3%   332.1m ± 1%  +71.00% (p=0.000 n=20)
geomean                      5.882m        7.161m       +21.74%

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2023-10-09 17:25:18 +02:00
Oleg Zaytsev 5bd8c8c561
Clarify Postings.At() contract (#12921)
It's implicit, but should be explicit. It is invalid to call At() after
a failed call to Next() or Seek().

Following up on https://github.com/prometheus/prometheus/pull/12906

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2023-10-09 16:15:06 +02:00
Oleg Zaytsev 1492031ef2
Optimize ListPostings Next() (#12906)
The Next() call of ListPostings() was updating two values, while we can
just update the position. This is up to 30% faster for high number of
Postings.

goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/tsdb/index
cpu: 11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz
                              │     old     │                 new                 │
                              │   sec/op    │   sec/op     vs base                │
ListPostings/count=100-16       819.2n ± 0%   732.6n ± 0%  -10.58% (p=0.000 n=20)
ListPostings/count=1000-16      2.685µ ± 1%   2.017µ ± 0%  -24.88% (p=0.000 n=20)
ListPostings/count=10000-16     21.43µ ± 1%   14.81µ ± 0%  -30.91% (p=0.000 n=20)
ListPostings/count=100000-16    209.4µ ± 1%   143.3µ ± 0%  -31.55% (p=0.000 n=20)
ListPostings/count=1000000-16   2.086m ± 1%   1.436m ± 1%  -31.18% (p=0.000 n=20)
geomean                         29.02µ        21.41µ       -26.22%

We're talking about microseconds here, but they just keep adding.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2023-10-02 16:24:25 +02:00
Arve Knudsen de7e057d3c
tsdb: Tighten up sub-benchmark scope in BenchmarkQuerier (#12718)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-10-02 12:16:37 +02:00
Björn Rabenstein 0de7f39e6a
Merge pull request #12894 from linasm/linasm/test-case-for-ValidateHistogram
Additional test case for ValidateHistogram
2023-09-27 14:16:57 +02:00
Linas Medziunas 1aad4004c3 Additional test case for ValidateHistogram
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-09-27 09:34:43 +03:00
Bryan Boreham 6dcbd653e9
tsdb: register metrics after Head is initialized (#12876)
This avoids situations where metrics are scraped before the data they
are trying to look at is initialized.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-09-25 21:57:08 +01:00
Goutham Veeramachaneni 86729d4d7b
Update exp package (#12650) 2023-09-21 22:53:51 +02:00
Björn Rabenstein f8dd8770ac
Merge pull request #12757 from bboreham/reuse-bufiter
TSDB: re-use iterator when moving between series
2023-09-21 14:08:53 +02:00
Dimitar Dimitrov 1155d736b6
Improve sensitivity of TestQuerierIndexQueriesRace
Currently, the two goroutines race against each other and it's possible that the main test goroutine finishes way earlier than appendSeries has had a chance to run at all.

I tested this change by breaking the code that X fixed and running the race test 100 times. Without the additional time.Sleep the test failed 11 times. With the sleep it failed 65 out of the 100 runs. Which is still not ideal, but it's a step forward.

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
2023-09-21 12:30:08 +02:00
Dimitar Dimitrov 6f1284ac93 Fix exit condition of TestQuerierIndexQueriesRace
The test was introduced in # but was changed during the code review and not reran with the faulty code since then.

Closes #

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
2023-09-20 20:22:26 +01:00
Björn Rabenstein 864da019cd
Merge pull request #12874 from krajorama/outof-order-chunks
Fix duplicate sample detection at chunk size limit
2023-09-20 18:01:21 +02:00
Björn Rabenstein 9071913fd9
Merge pull request #12831 from aknuds1/arve/posting-context
Add context argument to `tsdb.PostingsForMatchers`
2023-09-20 17:15:15 +02:00
György Krajcsovits 9dbd100a5e Refactor solution to not repeat code
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-09-20 15:54:00 +02:00
György Krajcsovits 96d03b6f46 Fix duplicate sample detection at chunks size limit
Before cutting a new XOR chunk in case the chunk goes over the size
limit, check that the timestamp is in order and not equal or older
than the latest sample in the old chunk.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-09-20 14:49:56 +02:00
György Krajcsovits 56b3a015b6 Add regression test for duplicate detection at chunk size limit
TestHeadDetectsDuplcateSampleAtSizeLimit tests a regression where a
 duplicate sample,is appended to the head, right when the head chunk is
 at the size limit. The test adds all samples as duplicate, thus
 expecting that the result has exactly half of the samples.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-09-20 14:32:20 +02:00
Björn Rabenstein 83891135c6
Merge pull request #12838 from krajorama/fix-disappearing-span-panic
Fix counterResetInAnyBucket panic
2023-09-19 17:10:27 +02:00
George Krajcsovits 3512b2d678
storage: make histogram reset handling consistent in chainSampleIterator (#12779)
storage: make histogram reset handling consistent in chainSampleIterator

---------

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-09-19 17:06:46 +02:00
Alan Protasio 959c98441b Add context argument to tsdb.PostingsForMatchers
Signed-off-by: Alan Protasio <alanprot@gmail.com>
2023-09-16 18:13:32 +02:00
zenador 69edd8709b
Add warnings (and annotations) to PromQL query results (#12152)
Return annotations (warnings and infos) from PromQL queries

This generalizes the warnings we have already used before (but only for problems with remote read) as "annotations".

Annotations can be warnings or infos (the latter could be false positives). We do not treat them different in the API for now and return them all as "warnings". It would be easy to distinguish them and return infos separately, should that appear useful in the future.

The new annotations are then used to create a lot of warnings or infos during PromQL evaluations. Partially these are things we have wanted for a long time (e.g. inform the user that they have applied `rate` to a metric that doesn't look like a counter), but the new native histograms have created even more needs for those annotations (e.g. if a query tries to aggregate float numbers with histograms).

The annotations added here are not yet complete. A prominent example would be a warning about a range too short for a rate calculation. But such a warnings is more tricky to create with good fidelity and we will tackle it later.

Another TODO is to take annotations into account when evaluating recording rules.

---------

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-09-14 18:57:31 +02:00
Arve Knudsen 156222cc50
Add context argument to LabelQuerier.LabelValues (#12665)
Add context argument to LabelQuerier.LabelValues and
LabelQuerier.SortedLabelValues.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-09-14 16:02:04 +02:00
Arve Knudsen a964349e97
Add context argument to LabelQuerier.LabelNames (#12666)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-09-14 10:39:51 +02:00
Arve Knudsen 4451ba10b4
Add context argument to IndexReader.Postings (#12667)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-09-13 17:45:06 +02:00
Arve Knudsen 6ef9ed0bc3
Add context argument to DB.Delete (#12834)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-09-13 15:43:06 +02:00
György Krajcsovits b2fa4d910a Fix more counterResetInAnyBucket edgecases
Case a) empty span is at the beginning of the spans.
Case b) two consequtive empty spans with positive offsets.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-09-13 15:18:40 +02:00
Fiona Liao 4419399e4e
Do WBL mmap marker replay concurrently (#12801)
* Benchmark WBL

Extended WAL benchmark test with WBL parts too - added basic cases for
OOO handling - a percentage of series have a percentage of samples set
as OOO ones.

Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>
2023-09-12 21:31:10 +02:00
Shirley d3a1044354
WBL loading: don't send empty buffers over chan (#12808)
Signed-off-by: Shirley Leu <4163034+fridgepoet@users.noreply.github.com>
Co-authored-by: Fiona Liao <fiona.y.liao@gmail.com>
2023-09-12 16:26:02 +02:00
Arve Knudsen 6daee89e5f
Add context argument to Querier.Select (#12660)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-09-12 12:37:38 +02:00
Bryan Boreham f711d71aa8
Merge pull request #12798 from fionaliao/remove-duplicated-max-time
Remove duplicated ms.mmMaxTime check in processWALSamples
2023-09-06 09:17:42 +01:00
Fiona Liao f211fcd92d
Remove duplicated ms.mmMaxTime check in WAL
Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>
2023-09-05 15:23:03 +01:00
George Krajcsovits b6f903b5f9
Fix handling of explicit counter reset header in histograms. (#12772)
* Fix handling of explicit counter reset header in histograms.

Explicit counter reset were being ignored.
Also there was no unit test coverage.
Add test case for the first sample in a chunk.
Add test case for non first sample in chunk.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

---------

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-09-01 23:39:15 +02:00
Dimitar Dimitrov b40865833d
PostingsForMatchers race with creating new series (#12558)
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
2023-08-29 11:03:27 +02:00
Bryan Boreham c5671c6d97
Merge pull request #12755 from bboreham/rangequery-benchmark-mmap
promql: force mmap of head chunks in BenchmarkRangeQuery
2023-08-26 15:56:52 +01:00
Bryan Boreham bdc7983956 TSDB: re-use iterator when moving between series
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-08-26 14:01:44 +00:00
Bryan Boreham 5d22d422ab
Merge pull request #12690 from michalbiesek/feat-go-bump
Update Go version to 1.21
2023-08-26 14:36:15 +01:00