Commit Graph

1195 Commits (e0444d9384c00ba10e84febb23557854170ae15f)

Author SHA1 Message Date
George Krajcsovits 4eab18abd6
[nhcb branch] Use single bit to differentiate between optimized bounds and floats (#13828)
* Use single bit to differentiate between optimized bounds and floats

Use one bit to decide what kind of data to read/write.
This reduces storage need of floats from 72 bits to 65 bits and makes the
integers store in 5 to 32 bits instead of 16.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>
Co-authored-by: Jeanette Tan <jeanette.tan@grafana.com>
2024-03-27 18:40:59 +01:00
Arve Knudsen 35aab01de0 tsdb/wlog.Checkpoint: Handle also float histograms
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-03-26 15:27:32 +01:00
Bartlomiej Plotka de05d5e11b
Merge pull request #13776 from aknuds1/arve/checkpoint-histogram-samples
tsdb/wlog.Checkpoint: Fix counting of histogram samples
2024-03-26 12:33:48 +01:00
Bartlomiej Plotka 25578f2b22
[test] Merge pull request #13790 from aknuds1/arve/retention-commit
tsdb.BeyondTimeRetention: Fix comment and test at retention duration
2024-03-26 12:26:32 +01:00
Nick Pillitteri 481f14e1c0
TSDB: Don't rely on integer overflow in head compaction check (#13755)
* TSDB: Don't compact the head block when empty

Don't compact the Head block if there have not yet been any samples
appended.

Previously, the logic for determining if the head should be compacted
relied on the default values for min and max time and integer overflow
when they were checked in `Head.compactable()`. The check in
`Head.compactable()` effectively did `math.MinInt64 - math.MaxInt64`
which overflowed and wrapped to `1`. Since `1` is less than `1.5`
times the chunk range, compaction did not happen. This was the correct
behavior but relying on overflow wrapping is surprising.

This change add a method for checking if the min and max time for the
head is unset and uses it to short-circuit compaction in that case.
It also replaces several explicit checks for the default value to
determine if the head has not yet had any samples added.

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
2024-03-26 12:17:38 +01:00
Ben Ye ceca6c4716
[ENHANCEMENT] TSDB: Log more statistics during startup (#13838)
* log chunk snapshot and mmap chunks replay duration together with total replay duration

Signed-off-by: Ben Ye <benye@amazon.com>
2024-03-26 11:16:27 +00:00
Bryan Boreham 78c0fd2f4d
Merge pull request #13799 from machine424/wbl
chore(tsdb): set the wbl to nil as well in DBReadOnly.loadDataAsQuery…
2024-03-25 15:03:56 +01:00
Bryan Boreham 080d440bf8 Merge remote-tracking branch 'origin/main' into pr/13461 2024-03-25 12:14:26 +00:00
György Krajcsovits a3d1a46eda Merge branch 'main' into nhcb 2024-03-22 14:51:48 +01:00
zenador 4acbb7dea6
Add custom buckets to native histogram chunks encoding (#13706)
* add custom bounds to chunks encoding
* change custom buckets schema number
* rename custom bounds to custom values

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2024-03-22 14:36:39 +01:00
machine424 2a2e2ed28b
chore(tsdb): set the wbl to nil as well in DBReadOnly.loadDataAsQueryable
Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2024-03-20 17:13:39 +01:00
Arve Knudsen 07332f7427 TestTimeRetention: Split into two sub-tests
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-03-20 15:06:36 +01:00
Arve Knudsen af694dc295 Merge TestDB_BeyondTimeRetention into TestTimeRetention
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-03-20 09:07:16 +01:00
Arve Knudsen 9c7a734063 tsdb.BeyondTimeRetention: Fix comment and test at retention duration
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-03-19 09:10:21 +01:00
Bryan Boreham a0e93e403e
Merge pull request #13764 from bboreham/remove-deprecated-wal
[Cleanup] TSDB: Remove old deprecated WAL implementation

Deprecated since 2018.
2024-03-17 09:34:57 +00:00
Darshan Chaudhary b7047f7fcb
Fix retention boundary so 2h retention deletes blocks right at the 2h boundary (#9633)
Signed-off-by: darshanime <deathbullet@gmail.com>
2024-03-15 19:35:16 +01:00
Arve Knudsen cef1025ea8 tsdb/wlog.Checkpoint: Fix counting of histogram samples
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-03-15 10:23:59 +01:00
Bryan Boreham d45b5deb75 TSDB: move function only used in tests
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-03-15 08:54:47 +00:00
Bryan Boreham 3274cac0d3 TSDB: remove unused function
Was only used in old WAL implementation.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-03-15 08:51:57 +00:00
Arve Knudsen 1de49d5b69
Remove unused function tsdb/chunks.PopulatedChunk (#13763)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-03-14 11:15:17 +01:00
Bryan Boreham 87edf1f960 [Cleanup] TSDB: Remove old deprecated WAL implementation
Deprecated since 2018.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-03-13 15:57:23 +00:00
Bryan Boreham d08f054950
[ENHANCEMENT] TSDB: Check CRC without allocating (#13742)
Use the existing utility function which does this.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-03-12 12:24:27 +01:00
carrychair 856f6e49c8 fix function and struct name
Signed-off-by: carrychair <linghuchong404@gmail.com>
2024-03-09 17:53:17 +08:00
Björn Rabenstein 9acae57937
Merge pull request #13681 from krajorama/native-latency-histograms
Add native histograms to latency/duration metrics
2024-03-07 20:46:43 +01:00
Bryan Boreham bbe39af99f
tsdb: zero out Labels and memSeries pointers in pool (#13712)
* tsdb: zero out Labels and memSeries pointers in pool

So that the garbage-collector doesn't see this memory as still in use.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

---------

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Signed-off-by: Björn Rabenstein <github@rabenste.in>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
2024-03-06 13:36:34 +01:00
György Krajcsovits 4d4d822c36 Add native histograms to latency/duration metrics
Dogfood native histograms.
Allow dependent projects to migrate to native histograms.

I took the defaults from client_golang.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-03-01 14:44:38 +01:00
machine424 f477e0539a
Move from golang.org/x/exp/slices into slices now that we only support Go >= 1.21
Prevent adding back golang.org/x/exp/slices.

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2024-02-28 14:54:53 +01:00
Bryan Boreham 925134e6de tsdb tests: make work with labels SymbolTable
Need to initialize decoders with SymbolTable.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-26 11:45:25 +00:00
Bryan Boreham 93b72ec5dd tsdb: create SymbolTables for labels as required
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-26 11:45:25 +00:00
Bryan Boreham 4d6bb2e0e4
Merge pull request #13628 from bboreham/cleanup-13583
tsdb/wlog: small cleanup of WAL watcher after #13583
2024-02-23 12:39:17 +00:00
George Krajcsovits c6c8f63516
Merge pull request #13607 from fionaliao/ooo-samples-appended-type
Add sample type label to outOfOrderSamplesAppended metric
2024-02-23 09:41:45 +01:00
Bryan Boreham 6ed56c9f04 WAL watcher: improve comments
Clarify in the first comment that it is `watch()` that waits, and reduce
verbiage.

The second comment was slightly contradictory to the first and otherwise
didn't seem to add much, since `currentSegment` was incremented just a
few lines later.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-22 09:32:46 +00:00
Bryan Boreham a975a83079 tsdb: clean up Watcher debug messages
Print lastSegment after it gets initialized.

Move variable declaration to first use.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-22 09:19:18 +00:00
Bryan Boreham 78f46bccca tsdb/wlog tests: remove unnecessary sleep check
Sleep() is documented to return immediately on negative or zero input.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-22 09:14:52 +00:00
Callum Styan 0c71230784
fix bug that would cause us to endlessly fall behind (#13583)
* fix bug that would cause us to only read from the WAL on the 15s
fallback timer if remote write had fallen behind and is no longer
reading from the WAL segment that is currently being written to

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* remove unintended logging, fix lint, plus allow test to take slightly
longer because cloud CI

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* address review feedback

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* fix watcher sleeps in test, flu brain is smooth

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* increase timeout, unfortunately cloud CI can require a longer timeout

Signed-off-by: Callum Styan <callumstyan@gmail.com>

---------

Signed-off-by: Callum Styan <callumstyan@gmail.com>
2024-02-21 17:09:07 -08:00
Fiona Liao 841a133514 Move histogramsAppended to be more consistent
Signed-off-by: Fiona Liao <fiona.liao@grafana.com>
2024-02-21 11:15:04 +00:00
Fiona Liao 52389647b2 Add type label to outOfOrderSamplesAppended metric
Signed-off-by: Fiona Liao <fiona.liao@grafana.com>
2024-02-19 15:24:39 +00:00
Bryan Boreham c0e36e6bb3 Standardise exemplar label as "trace_id"
This is consistent with the OpenTelemetry standard, and an example in OpenMetrics.

https://github.com/open-telemetry/opentelemetry-specification/blob/89aa01348139/specification/metrics/data-model.md#exemplars
https://github.com/OpenObservability/OpenMetrics/blob/138654493130/specification/OpenMetrics.md#exemplars-1

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-15 14:20:08 +00:00
Bryan Boreham 12cac5bd5c tsdb tests: use go-cmp instead of DeepEquals
Also one simpler call checking nil.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-08 19:32:33 +00:00
Bryan Boreham 17f48f2b3b Tests: use replacement DeepEquals in more places
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-08 19:32:33 +00:00
Bryan Boreham 39af788dbd Tests: use replacement DeepEquals using go-cmp
Use DeepEqual replacement using go-cmp, which is more flexible.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-08 19:30:20 +00:00
Peter Štibraný e2b9cfeeeb
Enforce chunks ordering when writing index. (#8085)
Document conditions on chunks. Add check on chunk time ordering.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
2024-02-04 16:31:49 +01:00
Bryan Boreham 98c4889029
Merge pull request #9298 from Creatone/creatone/use-testify
tests: Move from t.Errorf and others.
2024-02-04 16:27:57 +01:00
Mikhail Fesenko 419dd265cc
Fix strange code, add messages to code brought in #8106 (#13509)
Signed-off-by: Mikhail Fesenko <proggga@gmail.com>
2024-02-02 10:00:38 +01:00
Bryan Boreham 16e68c01e4 tests: remove err from message when testify prints it already
For instance `require.NoError` will print the unexpected error; we don't
need to include it in the message.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-02-01 14:18:01 +00:00
Mikhail Fesenko 5f2c3a5d3e
Small improvements, add const, remove copypasta (#8106)
Signed-off-by: Mikhail Fesenko <proggga@gmail.com>
Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com>
2024-02-01 14:30:50 +01:00
Paweł Szulik 5961f78186 Refactor tsdb tests to use testify.
Signed-off-by: Paweł Szulik <paul.szulik@gmail.com>
2024-01-31 16:03:17 +00:00
Bryan Boreham 34230bb172 tsdb/wlog: close segment files sooner
'defer' runs at the end of the whole function; we should close each
segment file as soon as we finished reading it.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-01-31 12:12:19 +00:00
Bryan Boreham cd4562d3a6
Merge pull request #13473 from bboreham/pure-mutex
tsdb: use cheaper Mutex on series
2024-01-30 09:57:08 +00:00
Marco Pracucci 501bc6419e
Add ShardedPostings() support to TSDB (#10421)
This PR is a reference implementation of the proposal described in #10420.

In addition to what described in #10420, in this PR I've introduced labels.StableHash(). The idea is to offer an hashing function which doesn't change over time, and that's used by query sharding in order to get a stable behaviour over time. The implementation of labels.StableHash() is the hashing function used by Prometheus before stringlabels, and what's used by Grafana Mimir for query sharding (because built before stringlabels was a thing).

Follow up work
As mentioned in #10420, if this PR is accepted I'm also open to upload another foundamental piece used by Grafana Mimir query sharding to accelerate the query execution: an optional, configurable and fast in-memory cache for the series hashes.

Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-01-29 11:57:27 +00:00
Bryan Boreham 66237c1996 tsdb: use cheaper Mutex on series
Mutex is 8 bytes; RWMutex is 24 bytes and much more complicated. Since
`RLock` is only used in two places, `UpdateMetadata` and `Delete`,
neither of which are hotspots, we should use the cheaper one.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-01-26 11:01:39 +00:00
Marco Pracucci ec9cada56e
Remove unused isRegexMetaCharacter()
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-01-26 06:35:02 +01:00
Marco Pracucci 515890ec53
Use Matcher.SetMatches()
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-01-26 06:26:52 +01:00
Marco Pracucci a1a45990a2
Fix TestPostingsForMatcher
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-01-25 14:59:39 +01:00
Bryan Boreham b9eab6e4b8
tsdb: simplify internal series delete function (#13261)
Lifting an optimisation from Agent code, `seriesHashmap.del` can use
the unique series reference, doesn't need to check Labels.
Also streamline the logic for deleting from `unique` and `conflicts` maps,
and add some comments to help the next person.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-01-25 11:57:54 +01:00
Bryan Boreham 3f30ad3cc2
Merge pull request #13015 from bboreham/smaller-txring
tsdb: make transaction isolation data structures smaller
2024-01-25 10:48:15 +00:00
Arve Knudsen ba7012ec6a
TestHeadLabelValuesWithMatchers: Add test case (#13414)
Add test case to TestHeadLabelValuesWithMatchers, while fixing a couple
of typos in other test cases. Also enclosing some implicit sub-tests in a
`t.Run` call to make them explicitly sub-tests.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-01-24 10:47:56 +01:00
Filip Petkovski 583f3e587c
Optimize histogram iterators (#13340)
Optimize histogram iterators

Histogram iterators allocate new objects in the AtHistogram and
AtFloatHistogram methods, which makes calculating rates over long
ranges expensive.

In #13215 we allowed an existing object to be reused
when converting an integer histogram to a float histogram. This commit follows
the same idea and allows injecting an existing object in the AtHistogram and
AtFloatHistogram methods. When the injected value is nil, iterators allocate
new histograms, otherwise they populate and return the injected object.

The commit also adds a CopyTo method to Histogram and FloatHistogram which
is used in the BufferedIterator to overwrite items in the ring instead of making
new copies.

Note that a specialized HPoint pool is needed for all of this to work 
(`matrixSelectorHPool`).

---------

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
2024-01-23 17:02:14 +01:00
Oleg Zaytsev ed172a6667
Optimize label values with matchers by taking shortcuts (#13426)
Don't calculate postings beforehand: we may not need them. If all
matchers are for the requested label, we can just filter its values.

Also, if there are no values at all, no need to run any kind of
logic.

Also add more labelValuesWithMatchers benchmarks

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2024-01-23 11:40:21 +01:00
Julien Pivotto f52605b584
Merge pull request #13415 from aknuds1/arve/test-label-values-with-matchers-one-more
TestLabelValuesWithMatchers: Add test case
2024-01-18 11:57:12 +01:00
tyltr f97fa2736c remove obsolete build tag
Signed-off-by: tyltr <tylitianrui@126.com>
2024-01-17 22:26:32 +08:00
Arve Knudsen 8598150f48 TestLabelValuesWithMatchers: Add test case
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-01-17 12:51:56 +01:00
Marco Pracucci 7852a7c516
Fix regressions introduced by #13242
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2024-01-16 12:00:53 +01:00
Giedrius Statkevičius b695e069b8
tsdb/main: wire "EnableOverlappingCompaction" to tsdb.Options (#13398)
This added the https://github.com/prometheus/prometheus/pull/13393
"EnableOverlappingCompaction" parameter to the compactor code but not to
the tsdb.Options. I forgot about that. Add it to `tsdb.Options` too and
set it to `true` in Prometheus.

Copy/paste the description from
https://github.com/prometheus/prometheus/pull/13393#issuecomment-1891787986

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
2024-01-15 16:42:40 +01:00
Ben Kochie 17920623e7
Merge pull request #13391 from GiedriusS/compact_merge_func
tsdb/compact: fix passing merge func
2024-01-15 09:43:06 +01:00
Giedrius Statkevičius 3a48adc54f tsdb: add enable overlapping compaction
This functionality is needed in downstream projects because they have a
separate component that does compaction.

Upstreaming
7c8e9a2a76/tsdb/compact.go (L323-L325).

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
2024-01-12 11:19:41 +02:00
Giedrius Statkevičius 9b759135d1 tsdb/compact: fix passing merge func
Fixing a very small logical problem I've introduced :(.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
2024-01-11 12:07:54 +02:00
Giedrius Statkevičius 61b4080a14
tsdb/{index,compact}: allow using custom postings encoding format (#13242)
* tsdb/{index,compact}: allow using custom postings encoding format

We would like to experiment with a different postings encoding format in
Thanos so in this change I am proposing adding another argument to
`NewWriter` which would allow users to change the format if needed.
Also, wire the leveled compactor so that it would be possible to change
the format there too.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* tsdb/compact: use a struct for leveled compactor options

As discussed on Slack, let's use a struct for the options in leveled
compactor.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* tsdb: make changes after Bryan's review

- Make changes less intrusive
- Turn the postings encoder type into a function
- Add NewWriterWithEncoder()

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

---------

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
2024-01-08 09:48:27 +00:00
Bryan Boreham bad3f23f23 agent: add BenchmarkCreateSeries
Based on the one in tsdb/head_test.go.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-31 10:23:43 +00:00
Bryan Boreham e64d7d8928 agent: make the global hash lookup table smaller
This is the same change made in #13040, plus subsequent improvements,
applied to agent-mode code.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-31 10:23:43 +00:00
Bryan Boreham 252031c86f Revert "Adding small test update for temp dir using t.TempDir (#13293)"
This reverts commit 2ddb3596ef.

Various tests are failing in CI after this change; reverting to free up
other work.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-30 19:17:30 +00:00
Mile Druzijanic 2ddb3596ef
Adding small test update for temp dir using t.TempDir (#13293)
* Adding small test update for temp dir using t.TempDir

Signed-off-by: Mile Druzijanic <miledruz@gmail.com>
Signed-off-by: Mile Druzijanic <zedsprogramms@gmail.com>

* removing not required cleanup

Signed-off-by: Mile Druzijanic <zedsprogramms@gmail.com>

---------

Signed-off-by: Mile Druzijanic <miledruz@gmail.com>
Signed-off-by: Mile Druzijanic <zedsprogramms@gmail.com>
2023-12-28 21:49:57 +01:00
Björn Rabenstein 6b8e945388
Merge pull request #13289 from fpetkovski/fix-histogram-reuse
Fix reusing float histograms
2023-12-25 22:45:03 +01:00
Bryan Boreham 8065bef172 Move metric type definitions to common/model
They are used in multiple repos, so common is a better place for them.
Several packages now don't depend on `model/textparse`, e.g.
`storage/remote`.

Also remove `metadata` struct from `api.go`, since it was identical to
a struct in the `metadata` package.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-19 18:56:54 +00:00
Filip Petkovski 1f69dcfa6b
Fix reusing float histograms
In https://github.com/prometheus/prometheus/pull/13276 we started reusing float histogram objects to reduce allocations in PromQL.
That PR introduces a bug where histogram pointers gets copied to the beginning of the histograms slice,
but are still kept in the end of the slice. When a new histogram is read into the last element,
it can overwrite a previous element because the pointer is the same.

This commit fixes the issue by moving outdated points to the end of the slice
so that we don't end up with duplicate pointers in the same buffer. In other words,
the slice gets rotated so that old objects can get reused.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2023-12-14 11:53:58 +01:00
Bryan Boreham d0c2d9c0b9
Merge pull request #12878 from bboreham/loser-tree
postings: use Loser Tree for merge
2023-12-12 21:38:30 +00:00
Björn Rabenstein 928d07e3bd
Merge branch 'main' into arve/typos
Signed-off-by: Björn Rabenstein <beorn@grafana.com>
2023-12-12 12:02:03 +01:00
Giedrius Statkevičius f36b56a62c
tsdb: remove unused option (#13282)
Digging around the TSDB code and I've found that this flag is unused so
let's remove it.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
2023-12-12 09:58:54 +00:00
Matthieu MOREL 8f6cf3aabb tsdb: use Go standard errors
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-12-11 12:18:54 +00:00
Arthur Silva Sens 5082655392
Append Created Timestamps (#12733)
* Append created timestamps.

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

* Log when created timestamps are ignored

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

* Proposed changes to Append CT PR.

Changes:

* Changed textparse Parser interface for consistency and robustness.
* Changed CT interface to be more explicit and handle validation.
* Simplified test, change scrapeManager to allow testability.
* Added TODOs.

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Updates.

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Addressed comments.

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Refactor head_appender test

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

* Fix linter issues

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

* Use model.Sample in head appender test

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>

---------

Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>
Signed-off-by: bwplotka <bwplotka@gmail.com>
Co-authored-by: bwplotka <bwplotka@gmail.com>
2023-12-11 08:43:42 +00:00
Bryan Boreham ab3a47b489 postings: use Loser Tree for merge
It's faster.

Note change to test - instead of requiring that the data structure is
identical to `EmptyPostings()`, check that calling `Next()` returns
false, which implies it was empty.

Also the check for context cancellation during initialization was
removed. Initialization should be a small portion of the work done
during merge, so it's not worth plumbing a context argument through.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-08 16:07:25 +00:00
Bryan Boreham ee700151a3 tsdb/index: add benchmark for Postings.Merge
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-12-08 16:00:22 +00:00
Filip Petkovski 10a82f87fd
Enable reusing memory when converting between histogram types
The 'ToFloat' method on integer histograms currently allocates new memory
each time it is called.

This commit adds an optional *FloatHistogram parameter that can be used
to reuse span and bucket slices. It is up to the caller to make sure the
input float histogram is not used anymore after the call.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2023-12-08 10:22:59 +01:00
Matthieu MOREL 9c4782f1cc
golangci-lint: enable testifylint linter (#13254)
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-12-07 11:35:01 +00:00
Arve Knudsen 237bfea46b `chunks.Reader`: Fix typo in ChunkOrIterable doc string.
Also fix comment typo in `FloatHistogram.Sub`.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-12-07 08:28:45 +01:00
Matthieu MOREL 998fafe679
tsdb/wlog: use Go standard errors (#13144)
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-12-04 17:08:43 +00:00
Julien Levesy e4ec263bcc
fix(wlog/watcher): read segment synchronously when not tailing (#13224)
Signed-off-by: Julien Levesy <jlevesy@gmail.com>
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Co-authored-by: Callum Styan <callumstyan@gmail.com>
2023-12-01 14:26:38 -08:00
Julien Levesy 501f514389
feat(tsdb/agent): notify remote storage when commit happens (#13223)
Signed-off-by: Julien Levesy <jlevesy@gmail.com>
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Co-authored-by: Callum Styan <callumstyan@gmail.com>
2023-12-01 14:00:26 -08:00
Oleksandr Redko 2a75604f8e
Enable default revive rules (#13068)
Signed-off-by: Oleksandr Redko <Oleksandr_Redko@epam.com>
2023-11-29 17:23:34 +00:00
Fiona Liao b8bcaef14d
Fix histogram append errors (#13201)
* Fix histogram append errors

We should check counterReset condition rather than okToAppend because if
there's a counter reset, okToAppend is always set to false.

Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>
2023-11-29 11:39:12 +01:00
Fiona Liao ce126230e7
Fix chunks iterator bug when tombstone covers a whole chunk (#13209)
When no samples are returned in a chunk because all the samples have
been deleted, the chunk iterator then stops without iterating through
any remaining chunks.

Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>
2023-11-29 11:24:04 +01:00
Xiaochao Dong 28d8f1650c
tsdb: Make sure the cache for postings cardinality properly honors the label name (#12653)
Add a string remembering which label and limit the cache corresponds to.

Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
2023-11-28 13:54:37 +00:00
Arve Knudsen 1200c89d0c
Fix tsdb.stripeSeries.gc so it handles conflicts properly (#13195)
* Fix tsdb.stripeSeries.gc so it handles conflicts properly

tsdb.stripeSeries.gc needs to prune seriesHashmap.conflicts first,
otherwise seriesHashmap replaces the unique field with the first among
the conflicts. Also add regression test.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* TestStripeSeries_gc: Support stringlabels, don't use internals

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

---------

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-11-28 14:43:35 +01:00
Fiona Liao 5bee0cfce2
Change `ChunkReader.Chunk()` to `ChunkOrIterable()`
The ChunkReader interface's Chunk() has been changed to ChunkOrIterable(). 

This is a precursor to OOO native histogram support - with OOO native histograms, the chunks.Meta passed to Chunk() can result in multiple chunks being returned rather than just a single chunk (e.g. if oooMergedChunk has a counter reset in the middle). 

To support this, ChunkOrIterable() requires either a single chunk or an iterable to be returned. If an iterable is returned, the caller has the responsibility of converting the samples from the iterable into possibly multiple chunks. The OOOHeadChunkReader now returns an iterable rather than a chunk to prepare for the native histograms case. Also as a beneficial side effect, oooMergedChunk and boundedChunk has been simplified as they only need to implement the Iterable interface now, not the full Chunk interface.

---------

Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
2023-11-28 11:14:29 +01:00
Arve Knudsen ecc37588b0
tsdb: seriesHashmap.set by making receiver a pointer (#13193)
* Fix tsdb.seriesHashmap.set by making receiver a pointer

The method tsdb.seriesHashmap.set currently doesn't set the conflicts
field properly, due to the receiver being a non-pointer. Fix by turning
the receiver into a pointer, and add a corresponding regression test.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-11-27 15:40:30 +00:00
Charles Korn 59844498f7
Fix issue where queries can fail or omit OOO samples if OOO head compaction occurs between creating a querier and reading chunks (#13115)
* Add failing test.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Don't run OOO head garbage collection while reads are running.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Add further test cases for different order of operations.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Ensure all queriers are closed if `DB.blockChunkQuerierForRange()` fails.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Ensure all queriers are closed if `DB.Querier()` fails.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Invert error handling in `DB.Querier()` and `DB.blockChunkQuerierForRange()` to make it clearer

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Ensure that queries that touch OOO data can't block OOO head garbage collection forever.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Address PR feedback: fix parameter name in comment

Co-authored-by: Jesus Vazquez <jesusvazquez@users.noreply.github.com>
Signed-off-by: Charles Korn <charleskorn@users.noreply.github.com>

* Address PR feedback: use `lastGarbageCollectedMmapRef`

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Address PR feedback: ensure pending reads are cleaned up if creating an OOO querier fails

Signed-off-by: Charles Korn <charles.korn@grafana.com>

---------

Signed-off-by: Charles Korn <charles.korn@grafana.com>
Signed-off-by: Charles Korn <charleskorn@users.noreply.github.com>
Co-authored-by: Jesus Vazquez <jesusvazquez@users.noreply.github.com>
2023-11-24 12:38:38 +01:00
Bryan Boreham f13bc1a5c9
Merge pull request #13040 from bboreham/smaller-stripeseries
TSDB: make the global hash lookup table smaller
2023-11-20 12:12:09 +00:00
Oleg Zaytsev f997c72f29
Make head block ULIDs descriptive (#13100)
* Make head block ULIDs descriptive

As far as I understand, these ULIDs aren't persisted anywhere, so it
should be safe to change them.

When debugging an issue, seeing an ULID like
`2ZBXFNYVVFDXFPGSB1CHFNYQTZ` or `33DXR7JA39CHDKMQ9C40H6YVVF` isn't very
helpful, so I propose to make them readable in their ULID string
version.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* Set a different ULID for RangeHead

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

---------

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2023-11-17 12:29:36 +01:00
Julien Pivotto 1b84c01b76
Merge pull request #13143 from mmorel-35/patch-3
tsdb/tsdbutil: use Go standard errors
2023-11-17 10:23:04 +01:00
Julien Pivotto 9cb96ad2ea
Merge pull request #13142 from mmorel-35/patch-2
tsdb/fileutil: use Go standard errors
2023-11-17 10:20:40 +01:00
Julien Pivotto 58eca19ac0
Merge pull request #13141 from mmorel-35/patch-1
tsdb/errors: fix errorlint linter
2023-11-17 10:20:00 +01:00
zenador 32ee1b15de
Fix error on ingesting out-of-order exemplars (#13021)
Fix and improve ingesting exemplars for native histograms.

See code comment for a detailed explanation of the algorithm.

Note that this changes the current behavior for all kind of samples slightly: We now allow exemplars with the same timestamp as during the last scrape if the value or the labels have changed.

Also note that we now do not ingest exemplars without timestamps for native histograms anymore.

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>

---------

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: zenador <zenador@users.noreply.github.com>
Co-authored-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
2023-11-16 15:07:37 +01:00
Matthieu MOREL d7c3bc4cb0
tsdb/tsdbutil: use Go standard errors
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-14 20:46:36 +01:00
Matthieu MOREL e60a508dd8 tsdb/errors: fix errorlint linter
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-14 19:16:12 +00:00
Matthieu MOREL e3041740e4 tsdb/fileutil: use Go standard errors
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-14 19:05:21 +00:00
Matthieu MOREL dd8871379a remplace errors.Errorf by fmt.Errorf
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-14 13:04:31 +00:00
Bryan Boreham 1bfb3ed062
Labels: reduce allocations when creating from TSDB WAL (#13044)
* Labels: reduce allocations when creating from TSDB

When reading the WAL, by passing references into the buffer we can avoid
copying strings under `-tags stringlabels`.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-14 11:36:35 +00:00
Julien Pivotto b718dca70b
Merge pull request #13108 from mmorel-35/patch-2
tsdb/chunkenc: use Go standard errors package
2023-11-14 00:54:35 +01:00
Julien Pivotto 3173feb5ef
Merge pull request #13107 from mmorel-35/patch-1
tsdb/agent: use Go standard errors package
2023-11-14 00:54:11 +01:00
Julien Pivotto 90ed7b08dc
Merge pull request #13124 from mmorel-35/patch-5
tsdb/index: use Go standard errors package
2023-11-14 00:53:49 +01:00
Julien Pivotto afc57eb306
Merge pull request #13109 from mmorel-35/patch-3
tsdb/chunks: use Go standard errors package
2023-11-14 00:52:59 +01:00
Julien Pivotto b6274ee747
Merge pull request #13114 from mmorel-35/patch-4
tsdb/wlog: use Go standard errors package
2023-11-14 00:52:21 +01:00
Julien Pivotto 7bcae56b6e
Merge pull request #13130 from mmorel-35/patch-6
tsdb/encoding: use Go standard errors package
2023-11-14 00:51:47 +01:00
Julien Pivotto f53af5c849
Merge pull request #13131 from mmorel-35/patch-7
tsdb/tombstones: use Go standard errors package
2023-11-14 00:51:31 +01:00
Julien Pivotto 17fe6bc296
Merge pull request #13133 from mmorel-35/patch-8
tsdb/record: use Go standard errors package
2023-11-14 00:51:16 +01:00
Bryan Boreham 65a443e6e3 TSDB: initialize conflicts map only when we need it.
Suggested by @songjiayang.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-13 16:20:31 +00:00
Bryan Boreham 37573d083b
Merge pull request #13056 from songjiayang/symbolCacheEntry-filed-alignment
TSDB block index writer: reduce memory used by symbol cache

Improve struct field alignment for symbolCacheEntry
2023-11-13 14:37:43 +00:00
George Krajcsovits acc114fe55
Fix panic during tsdb Commit (#13092)
* Fix panic during tsdb Commit

Fixes the following

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x19deb45]

goroutine 651118930 [running]:
github.com/prometheus/prometheus/tsdb.(*headAppender).Commit(0xc19100f7c0)
	/drone/src/vendor/github.com/prometheus/prometheus/tsdb/head_append.go:855 +0x245
github.com/prometheus/prometheus/tsdb.dbAppender.Commit({{0x35bd6f0?, 0xc19100f7c0?}, 0xc000fa4c00?})
	/drone/src/vendor/github.com/prometheus/prometheus/tsdb/db.go:1159 +0x2f

We theorize that the panic happened due the the series referenced by the
exemplar being removed between AppendExemplar and Commit due to being idle.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-11-12 14:51:37 +00:00
Matthieu MOREL 469e415d09
Update record.go
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-11 21:01:24 +01:00
Matthieu MOREL 69c07ec6ae
Update record_test.go
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-11 20:57:42 +01:00
Matthieu MOREL 63691d82a5
tsdb/record: use Go standard errors package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-11 20:52:49 +01:00
Matthieu MOREL c74b7ad4fb
Update tombstones.go
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-11 19:22:06 +01:00
Matthieu MOREL 118460a64f
tsdb/tombstones: use Go standard errors package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-11 19:19:50 +01:00
Matthieu MOREL 4d6d3c1715
tsdb/encoding: use Go standard errors package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-11 19:01:11 +01:00
George Krajcsovits 39a35d92bc
tsdb/head: wlog exemplars after samples (#13113)
When samples are committed in the head, they are also written to the WAL.
The order of WAL records should be sample then exemplar, but this was
not the case for native histogram samples. This PR fixes that.

The problem with the wrong order is that remote write reads the WAL and
sends the recorded timeseries in the WAL order, which means exemplars
arrived before histogram samples. If the receiving side is Prometheus
TSDB and the series has not existed before then the exemplar does not
currently create the series. Which means the exemplar is rejected and lost.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-11-11 17:30:16 +01:00
Matthieu MOREL 2972cc5e8f tsdb/index: use Go standard errors package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-09 21:37:41 +00:00
Bryan Boreham e6c0f69f98 TSDB: Only pay for hash collisions when they happen
Instead of a map of slices of `*memSeries`, ready for any of them to
hold series where hash values collide, split into a map of `*memSeries`
and a map of slices which is usually empty, since hash collisions are
a one-in-a-billion thing.

The `del` method gets more complicated, to maintain the invariant that
a series is only in one of the two maps.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-09 07:44:39 -06:00
Bryan Boreham ce4e757704 TSDB: refine variable naming in chunk gc
Slight further refactor.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-09 07:44:39 -06:00
Bryan Boreham 071d5732af TSDB: refactor cleanup of chunks and series
Extract the middle of the loop into a function, so it will be
easier to modify the `seriesHashmap` data structure.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-09 07:44:39 -06:00
machine424 a32fbc3658
head.go: Remove an unneeded snapshot trigger that was moved in https://github.com/prometheus/prometheus/pull/9328
and brougt back by mistake in 095f572d4a as part of https://github.com/prometheus/prometheus/pull/11447

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2023-11-09 11:46:46 +01:00
Matthieu MOREL fb48a351f0 tsdb/wlog: use Go standard errors package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-08 21:41:58 +00:00
songjiayang 443867f1aa symbolCacheEntry field type alignment, thus saving 8 bytes.
Signed-off-by: songjiayang <songjiayang1@gmail.com>
2023-11-09 00:43:27 +08:00
Arve Knudsen ae9221e152
tsdb/index.Symbols: Drop context argument from Lookup method (#13058)
Drop context argument from tsdb/index.Symbols.Lookup since lookup
should be fast and the context checking is a performance hit.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-11-08 13:08:33 +01:00
Matthieu MOREL ece8286305 tsdb/chunk: use Go standard errors package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-08 09:19:44 +00:00
Matthieu MOREL b60f9f801e tsdb/chunkenc: use Go standard errors package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-08 08:38:57 +00:00
Matthieu MOREL 724737006d
tsdb/agent: use Go standard errors package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>

Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-11-08 09:22:31 +01:00
Goutham Veeramachaneni b80617a248
Merge pull request #12881 from dimitarvdimitrov/dimitar/TestQuerierIndexQueriesRace-less-synchronisation
Improve sensitivity of TestQuerierIndexQueriesRace
2023-11-07 12:16:43 +01:00
Linas Medziunas ebed7d0612 Change Validate to be a method on histogram structs
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-03 16:47:59 +02:00
Linas Medziunas 1f8aea11d6 Move histogram validation code to model/histogram
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-03 16:17:24 +02:00
Linas Medziunas 1cd6c1cde5 ValidateHistogram: strict Count check in absence of NaNs
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-03 16:17:24 +02:00
beorn7 5dca994f64 Merge branch 'release-2.48' into beorn7/release 2023-11-02 19:58:33 +01:00
Jeanette Tan 52eb303031 Refactor assigning MinTime in histogram chunks
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-11-02 21:23:05 +08:00
Jeanette Tan 3ccaaa40ba Fix according to code review
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-11-02 13:37:07 +08:00
György Krajcsovits 4296ecbd14 tsdb/compact_test.go: test mixed typed series with PopulateBlock
Add testcase and update test so that it can test native histograms as well.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-11-02 13:33:42 +08:00
Jeanette Tan 27abf09e7f Fix missing MinTime in histogram chunks
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-11-02 13:33:39 +08:00
Jeanette Tan 2f7060bd5a Expand TestPopulateWithTombSeriesIterators to test earlier deletion intervals for histogram chunks as well as time-overlapping chunks
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-11-02 13:33:35 +08:00
Jeanette Tan 7a4a1127b7 Expand TestPopulateWithTombSeriesIterators to test min max times of chunks, including mixed chunks
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-11-02 13:33:33 +08:00
Jeanette Tan 04aabdd7cc Refactor TestPopulateWithDelSeriesIterator unit tests to reuse more code
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-11-02 13:33:30 +08:00
Jeanette Tan 46be85f2dc Make TestPopulateWithDelSeriesIterator tests cover histogram types and check MinTime
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-11-02 13:33:26 +08:00
Björn Rabenstein a43669e611
Merge pull request #12928 from alexandear/ci-enable-godot
ci(lint): enable godot; append dot at the end of comments
2023-11-01 17:15:41 +01:00
Julien Pivotto f568221610
Merge pull request #13057 from prometheus/release-2.48
Merge release-2.48 back into main
2023-10-31 15:24:39 -04:00
Oleksandr Redko fa90ca46e5 ci(lint): enable godot; append dot at the end of comments
Signed-off-by: Oleksandr Redko <Oleksandr_Redko@epam.com>
2023-10-31 19:53:38 +02:00
Oleksandr Redko 8e5f0387a2
ci(lint): enable nolintlint and remove redundant comments (#12926)
Signed-off-by: Oleksandr Redko <Oleksandr_Redko@epam.com>
2023-10-31 12:35:13 +01:00
zenador 80e977aae6
Remove `NewPossibleNonCounterInfo` and minimise creating empty annotations (#13012)
* Remove NewPossibleNonCounterInfo until it can be made more efficient, and avoid creating empty annotations as much as possible

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-10-24 17:36:07 +01:00
Márcio Carôso dff1c395f6
Expose --storage.tsdb.retention.time in metric prometheus_tsdb_retention_limit_seconds (#12986)
* Expose --storage.tsdb.retention.time in a metric

Signed-off-by: Marcio Caroso <msscaroso@gmail.com>

---------

Signed-off-by: Marcio Caroso <msscaroso@gmail.com>
2023-10-24 13:34:42 +02:00
Björn Rabenstein 059f7f0738
Merge pull request #12997 from prometheus/wal-samples-size
TSDB: Pre-size buffer to read samples from WAL
2023-10-24 13:26:06 +02:00
Bryan Boreham 90e98e0235 tsdb: create isolation transaction slice on demand
When Prometheus restarts it creates every series read in from the WAL,
but many of those series will be finished, and never receive any more
samples. By defering allocation of the txRing slice to when it is first
needed, we save 32 bytes per stale series.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-10-21 13:45:47 +00:00
Bryan Boreham 6fe8217ce4 tsdb: shrink txRing with smaller integers
4 billion active transactions ought to be enough for anyone.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-10-21 12:44:34 +00:00
Jeanette Tan 71a36d2396 Very minor refactor of the integer overflow fix
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-10-19 13:17:46 +08:00
Bryan Boreham 26fa2e8356 TSDB: Pre-size buffer to read samples from WAL
When reading the WAL this method is called with buffers from a pool, on
multiple goroutines. Pre-allocating sufficient size avoids slow growth
and many reallocations in `append`.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-10-17 17:31:26 +00:00
George Krajcsovits 7d7b9eacff
Fix int32 overflow issues (#12978)
On a 32 bit architecture the size of int is 32 bits. Thus converting from
int64, uint64 can overflow it and flip the sign.

Try for yourself in playground:
package main

import "fmt"

func main() {
	x := int64(0x1F0000001)
	y := int64(1)
	z := int32(x - y) // numerically this is 0x1F0000000
	fmt.Printf("%v\n", z)
}

Prints -268435456 as if x was smaller.

Followup to #12650

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-10-16 16:23:26 +02:00
Paschalis Tsilias 42b8f2f5fc tsdb/agent: allow ingestion of OOO samples (#12897)
Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com>
Signed-off-by: Levi Harrison <git@leviharrison.dev>
2023-10-15 13:47:42 -04:00
Ganesh Vernekar 4df2f2432b Additionally wrap WBL replay error (#12406)
* Additionally wrap WBL replay error

Although WBL replay is already wrapped with errLoadWbl,
there are other errors that can happen during a WBL replay.
We should not try to repair WAL in those cases.

This commit additionally wraps the final error in Head.Init again
with errLoadWbl so that WBL replay errors can be identified properly.

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com>
Co-authored-by: Jesus Vazquez <jesusvzpg@gmail.com>
Signed-off-by: Levi Harrison <git@leviharrison.dev>
2023-10-15 13:47:42 -04:00
Paschalis Tsilias afab845e65
tsdb/agent: allow ingestion of OOO samples (#12897)
Signed-off-by: Paschalis Tsilias <paschalis.tsilias@grafana.com>
2023-10-13 16:33:09 +02:00
Ganesh Vernekar f5913266a1
Additionally wrap WBL replay error (#12406)
* Additionally wrap WBL replay error

Although WBL replay is already wrapped with errLoadWbl,
there are other errors that can happen during a WBL replay.
We should not try to repair WAL in those cases.

This commit additionally wraps the final error in Head.Init again
with errLoadWbl so that WBL replay errors can be identified properly.

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com>
Co-authored-by: Jesus Vazquez <jesusvzpg@gmail.com>
2023-10-13 14:21:35 +02:00
Oleg Zaytsev fe90dcccff
Revert ListPostings change (#12955)
Reverts change from https://github.com/prometheus/prometheus/pull/12906

The benchmarks show that it's slower when intersecting, which is a
common usage for ListPostings (when intersecting matchers from Head)

(old is before #12906, new is #12906):

                           │     old     │                 new                 │
                           │   sec/op    │   sec/op     vs base                │
Intersect/LongPostings1-16   20.54µ ± 1%   21.11µ ± 1%   +2.76% (p=0.000 n=20)
Intersect/LongPostings2-16   51.03m ± 1%   52.40m ± 2%   +2.69% (p=0.000 n=20)
Intersect/ManyPostings-16    194.2m ± 3%   332.1m ± 1%  +71.00% (p=0.000 n=20)
geomean                      5.882m        7.161m       +21.74%

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2023-10-09 17:25:18 +02:00
Oleg Zaytsev 5bd8c8c561
Clarify Postings.At() contract (#12921)
It's implicit, but should be explicit. It is invalid to call At() after
a failed call to Next() or Seek().

Following up on https://github.com/prometheus/prometheus/pull/12906

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2023-10-09 16:15:06 +02:00
Oleg Zaytsev 1492031ef2
Optimize ListPostings Next() (#12906)
The Next() call of ListPostings() was updating two values, while we can
just update the position. This is up to 30% faster for high number of
Postings.

goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/tsdb/index
cpu: 11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz
                              │     old     │                 new                 │
                              │   sec/op    │   sec/op     vs base                │
ListPostings/count=100-16       819.2n ± 0%   732.6n ± 0%  -10.58% (p=0.000 n=20)
ListPostings/count=1000-16      2.685µ ± 1%   2.017µ ± 0%  -24.88% (p=0.000 n=20)
ListPostings/count=10000-16     21.43µ ± 1%   14.81µ ± 0%  -30.91% (p=0.000 n=20)
ListPostings/count=100000-16    209.4µ ± 1%   143.3µ ± 0%  -31.55% (p=0.000 n=20)
ListPostings/count=1000000-16   2.086m ± 1%   1.436m ± 1%  -31.18% (p=0.000 n=20)
geomean                         29.02µ        21.41µ       -26.22%

We're talking about microseconds here, but they just keep adding.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2023-10-02 16:24:25 +02:00
Arve Knudsen de7e057d3c
tsdb: Tighten up sub-benchmark scope in BenchmarkQuerier (#12718)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-10-02 12:16:37 +02:00
Björn Rabenstein 0de7f39e6a
Merge pull request #12894 from linasm/linasm/test-case-for-ValidateHistogram
Additional test case for ValidateHistogram
2023-09-27 14:16:57 +02:00
Linas Medziunas 1aad4004c3 Additional test case for ValidateHistogram
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-09-27 09:34:43 +03:00
Bryan Boreham 6dcbd653e9
tsdb: register metrics after Head is initialized (#12876)
This avoids situations where metrics are scraped before the data they
are trying to look at is initialized.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-09-25 21:57:08 +01:00
Goutham Veeramachaneni 86729d4d7b
Update exp package (#12650) 2023-09-21 22:53:51 +02:00
Björn Rabenstein f8dd8770ac
Merge pull request #12757 from bboreham/reuse-bufiter
TSDB: re-use iterator when moving between series
2023-09-21 14:08:53 +02:00
Dimitar Dimitrov 1155d736b6
Improve sensitivity of TestQuerierIndexQueriesRace
Currently, the two goroutines race against each other and it's possible that the main test goroutine finishes way earlier than appendSeries has had a chance to run at all.

I tested this change by breaking the code that X fixed and running the race test 100 times. Without the additional time.Sleep the test failed 11 times. With the sleep it failed 65 out of the 100 runs. Which is still not ideal, but it's a step forward.

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
2023-09-21 12:30:08 +02:00
Dimitar Dimitrov 6f1284ac93 Fix exit condition of TestQuerierIndexQueriesRace
The test was introduced in # but was changed during the code review and not reran with the faulty code since then.

Closes #

Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
2023-09-20 20:22:26 +01:00
Björn Rabenstein 864da019cd
Merge pull request #12874 from krajorama/outof-order-chunks
Fix duplicate sample detection at chunk size limit
2023-09-20 18:01:21 +02:00
Björn Rabenstein 9071913fd9
Merge pull request #12831 from aknuds1/arve/posting-context
Add context argument to `tsdb.PostingsForMatchers`
2023-09-20 17:15:15 +02:00
György Krajcsovits 9dbd100a5e Refactor solution to not repeat code
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-09-20 15:54:00 +02:00
György Krajcsovits 96d03b6f46 Fix duplicate sample detection at chunks size limit
Before cutting a new XOR chunk in case the chunk goes over the size
limit, check that the timestamp is in order and not equal or older
than the latest sample in the old chunk.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-09-20 14:49:56 +02:00
György Krajcsovits 56b3a015b6 Add regression test for duplicate detection at chunk size limit
TestHeadDetectsDuplcateSampleAtSizeLimit tests a regression where a
 duplicate sample,is appended to the head, right when the head chunk is
 at the size limit. The test adds all samples as duplicate, thus
 expecting that the result has exactly half of the samples.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-09-20 14:32:20 +02:00
Björn Rabenstein 83891135c6
Merge pull request #12838 from krajorama/fix-disappearing-span-panic
Fix counterResetInAnyBucket panic
2023-09-19 17:10:27 +02:00
George Krajcsovits 3512b2d678
storage: make histogram reset handling consistent in chainSampleIterator (#12779)
storage: make histogram reset handling consistent in chainSampleIterator

---------

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-09-19 17:06:46 +02:00
Alan Protasio 959c98441b Add context argument to tsdb.PostingsForMatchers
Signed-off-by: Alan Protasio <alanprot@gmail.com>
2023-09-16 18:13:32 +02:00
zenador 69edd8709b
Add warnings (and annotations) to PromQL query results (#12152)
Return annotations (warnings and infos) from PromQL queries

This generalizes the warnings we have already used before (but only for problems with remote read) as "annotations".

Annotations can be warnings or infos (the latter could be false positives). We do not treat them different in the API for now and return them all as "warnings". It would be easy to distinguish them and return infos separately, should that appear useful in the future.

The new annotations are then used to create a lot of warnings or infos during PromQL evaluations. Partially these are things we have wanted for a long time (e.g. inform the user that they have applied `rate` to a metric that doesn't look like a counter), but the new native histograms have created even more needs for those annotations (e.g. if a query tries to aggregate float numbers with histograms).

The annotations added here are not yet complete. A prominent example would be a warning about a range too short for a rate calculation. But such a warnings is more tricky to create with good fidelity and we will tackle it later.

Another TODO is to take annotations into account when evaluating recording rules.

---------

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-09-14 18:57:31 +02:00
Arve Knudsen 156222cc50
Add context argument to LabelQuerier.LabelValues (#12665)
Add context argument to LabelQuerier.LabelValues and
LabelQuerier.SortedLabelValues.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-09-14 16:02:04 +02:00
Arve Knudsen a964349e97
Add context argument to LabelQuerier.LabelNames (#12666)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-09-14 10:39:51 +02:00
Arve Knudsen 4451ba10b4
Add context argument to IndexReader.Postings (#12667)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-09-13 17:45:06 +02:00
Arve Knudsen 6ef9ed0bc3
Add context argument to DB.Delete (#12834)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-09-13 15:43:06 +02:00
György Krajcsovits b2fa4d910a Fix more counterResetInAnyBucket edgecases
Case a) empty span is at the beginning of the spans.
Case b) two consequtive empty spans with positive offsets.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-09-13 15:18:40 +02:00
Fiona Liao 4419399e4e
Do WBL mmap marker replay concurrently (#12801)
* Benchmark WBL

Extended WAL benchmark test with WBL parts too - added basic cases for
OOO handling - a percentage of series have a percentage of samples set
as OOO ones.

Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>
2023-09-12 21:31:10 +02:00
Shirley d3a1044354
WBL loading: don't send empty buffers over chan (#12808)
Signed-off-by: Shirley Leu <4163034+fridgepoet@users.noreply.github.com>
Co-authored-by: Fiona Liao <fiona.y.liao@gmail.com>
2023-09-12 16:26:02 +02:00
Arve Knudsen 6daee89e5f
Add context argument to Querier.Select (#12660)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-09-12 12:37:38 +02:00
Bryan Boreham f711d71aa8
Merge pull request #12798 from fionaliao/remove-duplicated-max-time
Remove duplicated ms.mmMaxTime check in processWALSamples
2023-09-06 09:17:42 +01:00
Fiona Liao f211fcd92d
Remove duplicated ms.mmMaxTime check in WAL
Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>
2023-09-05 15:23:03 +01:00
George Krajcsovits b6f903b5f9
Fix handling of explicit counter reset header in histograms. (#12772)
* Fix handling of explicit counter reset header in histograms.

Explicit counter reset were being ignored.
Also there was no unit test coverage.
Add test case for the first sample in a chunk.
Add test case for non first sample in chunk.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>

---------

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-09-01 23:39:15 +02:00
Dimitar Dimitrov b40865833d
PostingsForMatchers race with creating new series (#12558)
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
2023-08-29 11:03:27 +02:00
Bryan Boreham c5671c6d97
Merge pull request #12755 from bboreham/rangequery-benchmark-mmap
promql: force mmap of head chunks in BenchmarkRangeQuery
2023-08-26 15:56:52 +01:00
Bryan Boreham bdc7983956 TSDB: re-use iterator when moving between series
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-08-26 14:01:44 +00:00
Bryan Boreham 5d22d422ab
Merge pull request #12690 from michalbiesek/feat-go-bump
Update Go version to 1.21
2023-08-26 14:36:15 +01:00
Bryan Boreham 0d283effa8 promql: force mmap of head chunks in BenchmarkRangeQuery
Otherwise we have a highly unusual situation of over 100 chunks
in the headChunks list of each series, which heavily skews
performance.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-08-26 09:40:59 +00:00
Gregor Zeitlinger f01718262a
Unit tests for native histograms (#12668)
promql: Extend testing framework to support native histograms

This includes both the internal testing framework as well as the rules unit test feature of promtool.

This also adds a bunch of basic tests. Many of the code level tests can now be converted to tests within the framework, and more tests can be added easily.

---------

Signed-off-by: Harold Dost <h.dost@criteo.com>
Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
Signed-off-by: Stephen Lang <stephen.lang@grafana.com>
Co-authored-by: Harold Dost <h.dost@criteo.com>
Co-authored-by: Stephen Lang <stephen.lang@grafana.com>
Co-authored-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
2023-08-25 23:35:42 +02:00
Michal Biesek 04d7b4dbee
lint: Fix `SA1019` Using a deprecated function
`rand.Read` has been deprecated since Go 1.20
`crypto/rand.Read` is more appropriate

Ref: https://tip.golang.org/doc/go1.20

Signed-off-by: Michal Biesek <michalbiesek@gmail.com>
2023-08-25 17:47:41 +02:00
Justin Lei 8ef7dfdeeb
Add a chunk size limit in bytes (#12054)
Add a chunk size limit in bytes

This creates a hard cap for XOR chunks of 1024 bytes.

The limit for histogram chunk is also 1024 bytes, but it is a soft limit as a histogram has a dynamic size, and even a single one could be larger than 1024 bytes.

This also avoids cutting new histogram chunks if the existing chunk has fewer than 10 histograms yet. In that way, we are accepting "jumbo chunks" in order to have at least 10 histograms in a chunk, allowing compression to kick in.

Signed-off-by: Justin Lei <justin.lei@grafana.com>
2023-08-24 15:21:17 +02:00
beorn7 aa82fe198f tsdb: Fix histogram validation
So far, `ValidateHistogram` would not detect if the count did not
include the count in the zero bucket. This commit fixes the problem
and updates all the tests that have been undetected offenders so far.

Note that this problem would only ever create false negatives, so we
never falsely rejected to store a histogram because of it.

On the other hand, `ValidateFloatHistogram` has been to strict with
the count being at least as large as the sum of the counts in all the
buckets. Float precision issues could create false positives here, see
products of PromQL evaluations, it's actually quite hard to put an
upper limit no the floating point imprecision. Users could produce the
weirdest expressions, maxing out float precision problems. Therefore,
this commit simply removes that particular check from
`ValidateFloatHistogram`.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-08-22 23:04:01 +02:00
Mustafa Ateş Uzun e5e51bebef
fix: error message typo
Signed-off-by: Mustafa Ateş Uzun <mustafauzun0@gmail.com>
2023-08-17 16:34:45 +03:00
Julien Pivotto e3fabd5fdf
Merge pull request #12664 from prometheus/superq/cleanup_chunk_snapshots
Cleanup temporary chunk snapshot dirs
2023-08-08 13:02:39 +02:00
SuperQ 8d38d59fc5
Cleanup temporary chunk snapshot dirs
Simlar to cleanup of WAL files on startup, cleanup temporary
chunk_snapshot dirs. This prevents storage space leaks due to terminated
snapshots on shutdown.

Signed-off-by: SuperQ <superq@gmail.com>
2023-08-08 09:43:48 +02:00
Julien Pivotto c3311272d9
Merge pull request #12652 from colega/fix-typo-in-append-histogram-param-name
Fix typo in Appender.AppendHistogram() arg name
2023-08-04 16:37:40 +02:00
Oleg Zaytsev 6ea6def0d3
Use zeropool when replaying agent's DB WAL (#12651)
Same as https://github.com/prometheus/prometheus/pull/12189 but for
tsdb/agent/db.go

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2023-08-04 10:39:55 +02:00
Oleg Zaytsev c810e7cae3
Fix typo in Appender.AppendHistogram() arg name
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2023-08-04 10:21:16 +02:00
Oleg Zaytsev 61daa30bb1
Pass ref to SeriesLifecycleCallback.PostDeletion (#12626)
When a particular SeriesLifecycleCallback tries to optimize and run
closer to the Head, keeping track of the HeadSeriesRef instead of the
labelsets, it's impossible to handle the PostDeletion callback properly
as there's no way to know which series refs were deleted from the head.

This changes the callback to provide the series refs alongside the
labelsets, so the implementation can choose what to do.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2023-08-03 10:56:27 +02:00
Oleg Zaytsev cd7d0b69a2
Check nil err first when committing (#12625)
The most common case is to have a nil error when appending series, so
let's check that first instead of checking the 3 error types first.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2023-08-01 14:04:45 +02:00
cui fliter f26dfc95e6
fix struct name in comment (#12624)
Signed-off-by: cui fliter <imcusg@gmail.com>
2023-08-01 12:24:42 +02:00
Łukasz Mierzwa 3c80963e81
Use a linked list for memSeries.headChunk (#11818)
Currently memSeries holds a single head chunk in-memory and a slice of mmapped chunks.
When append() is called on memSeries it might decide that a new headChunk is needed to use for given append() call.
If that happens it will first mmap existing head chunk and only after that happens it will create a new empty headChunk and continue appending
our sample to it.

Since appending samples uses write lock on memSeries no other read or write can happen until any append is completed.
When we have an append() that must create a new head chunk the whole memSeries is blocked until mmapping of existing head chunk finishes.
Mmapping itself uses a lock as it needs to be serialised, which means that the more chunks to mmap we have the longer each chunk might wait
for it to be mmapped.
If there's enough chunks that require mmapping some memSeries will be locked for long enough that it will start affecting
queries and scrapes.
Queries might timeout, since by default they have a 2 minute timeout set.
Scrapes will be blocked inside append() call, which means there will be a gap between samples. This will first affect range queries
or calls using rate() and such, since the time range requested in the query might have too few samples to calculate anything.

To avoid this we need to remove mmapping from append path, since mmapping is blocking.
But this means that when we cut a new head chunk we need to keep the old one around, so we can mmap it later.
This change makes memSeries.headChunk a linked list, memSeries.headChunk still points to the 'open' head chunk that receives new samples,
while older, yet to be mmapped, chunks are linked to it.
Mmapping is done on a schedule by iterating all memSeries one by one. Thanks to this we control when mmapping is done, since we trigger
it manually, which reduces the risk that it will have to compete for mmap locks with other chunks.

Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>
2023-07-31 11:10:24 +02:00
Robert Fratto 886945cda7
tsdb/agent: ensure that new series get written to WAL on rollback (#12592)
If a new series is introduced in a storage.Appender instance, that
series should be written to the WAL once the storage.Appender is closed,
even on Rollback.

Previously, new series would only be written to the WAL when calling
Commit. However, because the series is stored in memory regardless,
subsequent calls to Commit may write samples to the WAL which reference
a series ID which that was never written.

Related to #11589. It's likely that this fix also resolves this issue,
but we need more testing from users to see if the problem persists after
this fix; there may be more cases where samples get written to the WAL
in Prometheus Agent mode without the corresponding series record.

Signed-off-by: Robert Fratto <robertfratto@gmail.com>
2023-07-27 09:28:26 -04:00
George Krajcsovits 6cd2d1621f
Hide histogram chunk append and reset header internals (#12352)
tsdb: Hide histogram chunk append and reset header internals

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>
2023-07-26 15:08:16 +02:00
Björn Rabenstein 0e12f11d61
Merge pull request #12583 from prometheus/release-2.46
Merge release-2.46 into main
2023-07-20 18:29:44 +02:00
György Krajcsovits d4e355243a tsdbutil/ChunkFromSamplesGeneric should not panic
Add error handling instead.
Prepares for #12352

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2023-07-20 17:01:34 +02:00
Julien Pivotto 7905594b52
Merge pull request #12557 from prometheus/beorn7/histogram
scrape: Enable ingestion of multiple exemplars per sample
2023-07-20 15:19:28 +02:00
Julien Pivotto 1f5934e7be
Merge pull request #10623 from songjiayang/update-index
make sure response error when TOC parse failed
2023-07-18 13:47:27 +02:00
cui fliter 096ceca44f
remove repetitive words (#12556)
Signed-off-by: cui fliter <imcusg@gmail.com>
2023-07-13 15:53:40 +02:00
beorn7 0e3f35324b scrape: Enable ingestion of multiple exemplars per sample
This has become a requirement for native histograms, as a single
histogram sample commonly has many buckets, so that providing many
exemplars makes sense.

Since OM text doesn't support native histograms yet, the test had to
be expanded to also support protobuf test cases.

Signed-off-by: beorn7 <beorn@grafana.com>
2023-07-13 14:16:10 +02:00
Julien Pivotto 89e213bc02
Merge pull request #12546 from roidelapluie/removeimport
TSDB: Remove usused import of sort
2023-07-11 15:06:48 +02:00
Justin Lei 32d87282ad
Add Zstandard compression option for wlog (#11666)
Snappy remains as the default compression but there is now a flag to switch 
the compression algorithm.

Signed-off-by: Justin Lei <justin.lei@grafana.com>
2023-07-11 14:57:57 +02:00
Julien Pivotto bf5bf1a4b3 TSDB: Remove usused import of sort
Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>
2023-07-11 14:29:31 +02:00
Julien Pivotto 8c8afec116
Merge pull request #12542 from merrickclay/tsdb-doc-comment
improve incorrect doc comment
2023-07-11 13:10:04 +02:00
Julien Pivotto 0f85e4f41d
Merge pull request #12539 from bboreham/slices-sorts
Replace sort.Slice with faster slices.SortFunc
2023-07-11 13:09:02 +02:00
Merrick Clay 70e41fc5ac improve incorrect doc comment
Signed-off-by: Merrick Clay <merrick.e.clay@gmail.com>
2023-07-10 16:52:00 -06:00
Bryan Boreham ce153e3fff Replace sort.Sort with faster slices.SortFunc
The generic version is more efficient.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-07-10 09:43:45 +00:00
Marc Tudurí 4851ced266
tsdb: Support native histograms in snapshot on shutdown (#12258)
Signed-off-by: Marc Tuduri <marctc@protonmail.com>
2023-07-05 11:44:13 +02:00
Julien Pivotto 9ff1f24efa
Merge pull request #12505 from pracucci/fix-infinite-loop-in-index-writer
Fix infinite loop in index Writer when a series contains duplicated label names
2023-07-04 13:08:36 +02:00
Patrick Oyarzun 68e5937474
Apply relevant label matchers in LabelValues before fetching extra postings (#12274)
* Apply matchers when fetching label values

Signed-off-by: Patrick Oyarzun <patrick.oyarzun@grafana.com>

* Avoid extra copying of label values

Signed-off-by: Patrick Oyarzun <patrick.oyarzun@grafana.com>

---------

Signed-off-by: Patrick Oyarzun <patrick.oyarzun@grafana.com>
2023-07-04 10:37:58 +01:00
Bryan Boreham 5255bf06ad Replace sort.Slice with faster slices.SortFunc
The generic version is more efficient.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-07-02 22:17:08 +00:00
Marco Pracucci 35069910f5
Fix infinite loop in index Writer when a series contains duplicated label names
Signed-off-by: Marco Pracucci <marco@pracucci.com>
2023-07-01 17:38:08 +02:00
Marco Pracucci 031d22df9e
Fix race condition in ChunkDiskMapper.Truncate() (#12500)
* Fix race condition in ChunkDiskMapper.Truncate()

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Added unit test

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Update tsdb/chunks/head_chunks.go

Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>

---------

Signed-off-by: Marco Pracucci <marco@pracucci.com>
Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>
2023-06-30 18:29:59 +05:30
Bartlomiej Plotka 4062f12573
Merge pull request #12396 from leizor/leizor/chunk-opts
Group args to append to memSeries in chunkOpts
2023-06-27 13:08:21 +02:00
Nidhey Nitin Indurkar a8772a4178
Feat: Get block by id directly on promtool analyze & get latest block if ID not provided (#12031)
* feat: analyze latest block or block by ID in CLI (promtool)

Signed-off-by: nidhey27 <nidhey.indurkar@infracloud.io>

* address remarks

Signed-off-by: nidhey60@gmail.com <nidhey.indurkar@infracloud.io>

* address latest review comments

Signed-off-by: nidhey60@gmail.com <nidhey.indurkar@infracloud.io>

---------

Signed-off-by: nidhey27 <nidhey.indurkar@infracloud.io>
Signed-off-by: nidhey60@gmail.com <nidhey.indurkar@infracloud.io>
2023-06-01 17:13:09 +05:30
Alan Protasio 73078bf738
Opmizing Group Regex (#12375)
Signed-off-by: Alan Protasio <alanprot@gmail.com>
2023-05-30 13:49:22 +02:00
Julien Pivotto 6f97641a51
Merge pull request #12380 from mmorel-35/patch-2
ci(lint): enable predeclared linter
2023-05-28 14:43:29 +02:00
Justin Lei e73d8b2084 Also pass chunkOpts into appendPreprocessor
Signed-off-by: Justin Lei <justin.lei@grafana.com>
2023-05-25 13:37:18 -07:00
Justin Lei 4c4454e4c9 Group args to append to memSeries in chunkOpts
Signed-off-by: Justin Lei <justin.lei@grafana.com>
2023-05-25 13:12:46 -07:00
Justin Lei 89af351730
Remove samplesPerChunk from memSeries (#12390)
Signed-off-by: Justin Lei <justin.lei@grafana.com>
2023-05-25 11:18:41 +02:00
zenador 37e5249e33
Use DefaultSamplesPerChunk in tsdb (#12387)
Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-05-24 13:00:21 +02:00
Baskar Shanmugam 905a0bd63a
Added 'limit' query parameter support to /api/v1/status/tsdb endpoint (#12336)
* Added 'topN' query parameter support to /api/v1/status/tsdb endpoint

Signed-off-by: Baskar Shanmugam <baskar.shanmugam.career@gmail.com>

* Updated query parameter for tsdb status to 'limit'

Signed-off-by: Baskar Shanmugam <baskar.shanmugam.career@gmail.com>

* Corrected Stats() parameter name from topN to limit

Signed-off-by: Baskar Shanmugam <baskar.shanmugam.career@gmail.com>

* Fixed p.Stats CI failure

Signed-off-by: Baskar Shanmugam <baskar.shanmugam.career@gmail.com>

---------

Signed-off-by: Baskar Shanmugam <baskar.shanmugam.career@gmail.com>
2023-05-22 14:37:07 +02:00
Alan Protasio 8c5d4b4add
Opmize MatchNotEqual (#12377)
Signed-off-by: Alan Protasio <alanprot@gmail.com>
2023-05-21 10:41:30 +02:00
Matthieu MOREL c8e7f95a3c ci(lint): enable predeclared linter
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2023-05-21 07:33:54 +00:00
George Krajcsovits 92d6980360
Fix populateWithDelChunkSeriesIterator and gauge histograms (#12330)
Use AppendableGauge to detect corrupt chunk with gauge histograms.
Detect if first sample is a gauge but the chunk is not set up to contain
gauge histograms.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>
2023-05-19 10:24:06 +02:00
Baskar Shanmugam f731a90a7f
Fix LabelValueStats in posting stats (#12342)
Problem:
LabelValueStats - This will provide a list of the label names and memory used in bytes.
It is calculated by adding the length of all values for a given label name.
But internally Prometheus stores the name and the value independently for each series.

Solution:
MemPostings struct maintains the values to seriesRef map which is used
to get the number of series which contains the label values.
Using that LabelValueStats is calculated as: seriesCnt * len(value
name)

Signed-off-by: Baskar Shanmugam <baskar.shanmugam.career@gmail.com>
2023-05-19 09:36:30 +02:00
Xiaochao Dong 80b7f73d26
Copy tombstone intervals to avoid race (#12245)
Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
2023-05-17 15:15:12 +02:00