Commit Graph

12677 Commits (8543f4827b19862c0de412b93e2f74804483a905)

Author SHA1 Message Date
Linas Medziunas 048886ae8a Histograms: optimize floatBucketIterator for common case
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-29 08:54:05 +02:00
Bartlomiej Plotka 911d0d0f7d
Update RELEASE.md
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2023-11-28 20:19:12 +00:00
Bartlomiej Plotka 34812fa759
Added Bartek as the next release shepherd.
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2023-11-28 20:18:19 +00:00
Björn Rabenstein 97f9ad9d31
Merge pull request #13191 from linasm/reduceResolution-inplace
Reuse receiver slices in [Float]Histogram.ReduceResolution
2023-11-28 17:48:52 +01:00
Björn Rabenstein 980e2895a2
Merge pull request #13129 from fatsheep9146/reduce-resolution-automatically
Native Histograms: automatically reduce resolution rather than fail scrape
2023-11-28 17:26:36 +01:00
Julien Pivotto 4293de96c7
Merge pull request #11299 from galexrt/sort_by_labels
promql: add sort_by_label and sort_by_label_desc functions
2023-11-28 15:31:48 +01:00
Julien Pivotto c1ec6ae851 sort_by_label: Switch to feature flag
Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>
2023-11-28 15:10:12 +01:00
Xiaochao Dong 28d8f1650c
tsdb: Make sure the cache for postings cardinality properly honors the label name (#12653)
Add a string remembering which label and limit the cache corresponds to.

Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
2023-11-28 13:54:37 +00:00
Linas Medziunas 49613823f8 Reuse slices in [Float]Histogram.ReduceResolution
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-28 15:45:31 +02:00
Arve Knudsen 1200c89d0c
Fix tsdb.stripeSeries.gc so it handles conflicts properly (#13195)
* Fix tsdb.stripeSeries.gc so it handles conflicts properly

tsdb.stripeSeries.gc needs to prune seriesHashmap.conflicts first,
otherwise seriesHashmap replaces the unique field with the first among
the conflicts. Also add regression test.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* TestStripeSeries_gc: Support stringlabels, don't use internals

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

---------

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-11-28 14:43:35 +01:00
Alexander Trost 5051a993ab promql: add sort_by_label and sort_by_label_desc functions
This adds functions to sort a vector by its label value.

Based on https://github.com/prometheus/prometheus/pull/1533

Signed-off-by: Alexander Trost <galexrt@googlemail.com>
2023-11-28 14:40:07 +01:00
Björn Rabenstein a6d4b8d97b
Merge pull request #13190 from linasm/fix-float-histogram-add-sub-mutating-argument
Fix FloatHistogram.Add/Sub mutating its argument
2023-11-28 14:04:50 +01:00
Björn Rabenstein 5f6c060284
Merge pull request #13189 from linasm/explicit-schema-check-in-histogram-reduce-resolution
Explicit schema check in [Float]Histogram.ReduceResolution
2023-11-28 14:00:34 +01:00
dependabot[bot] 8d058884a0
build(deps): bump sanitize-html from 2.10.0 to 2.11.0 in /web/ui (#12513)
Bumps [sanitize-html](https://github.com/apostrophecms/sanitize-html) from 2.10.0 to 2.11.0.
- [Changelog](https://github.com/apostrophecms/sanitize-html/blob/main/CHANGELOG.md)
- [Commits](https://github.com/apostrophecms/sanitize-html/compare/2.10.0...2.11.0)

---
updated-dependencies:
- dependency-name: sanitize-html
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-28 12:49:34 +01:00
David Peleg 969a9b7090
Fix docs: in labelname, convert to underscore (#12555)
Fixes https://github.com/prometheus/docs/issues/735

Signed-off-by: David Peleg <107049994+dapeleg-dn@users.noreply.github.com>
2023-11-28 12:30:05 +01:00
Bryan Boreham 1b92fb10de
Improvements to PR template (#13198)
I like a bit more of a hint what the PR is supposed to achieve.

Also, I took out a suggestion not to write tests.  We can decide
case-by-case if it's ok not to add a test; no need to encourage it.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-28 10:52:27 +00:00
Fiona Liao 5bee0cfce2
Change `ChunkReader.Chunk()` to `ChunkOrIterable()`
The ChunkReader interface's Chunk() has been changed to ChunkOrIterable(). 

This is a precursor to OOO native histogram support - with OOO native histograms, the chunks.Meta passed to Chunk() can result in multiple chunks being returned rather than just a single chunk (e.g. if oooMergedChunk has a counter reset in the middle). 

To support this, ChunkOrIterable() requires either a single chunk or an iterable to be returned. If an iterable is returned, the caller has the responsibility of converting the samples from the iterable into possibly multiple chunks. The OOOHeadChunkReader now returns an iterable rather than a chunk to prepare for the native histograms case. Also as a beneficial side effect, oooMergedChunk and boundedChunk has been simplified as they only need to implement the Iterable interface now, not the full Chunk interface.

---------

Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
2023-11-28 11:14:29 +01:00
Arve Knudsen ecc37588b0
tsdb: seriesHashmap.set by making receiver a pointer (#13193)
* Fix tsdb.seriesHashmap.set by making receiver a pointer

The method tsdb.seriesHashmap.set currently doesn't set the conflicts
field properly, due to the receiver being a non-pointer. Fix by turning
the receiver into a pointer, and add a corresponding regression test.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2023-11-27 15:40:30 +00:00
Ziqi Zhao 19ecc5dd94 add test case for bigGap
Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
2023-11-26 22:20:44 +08:00
Linas Medziunas f99ecc376e Fix FloatHistogram.Add/Sub mutating its argument
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-26 09:26:34 +02:00
Linas Medziunas ea1862aab4 Explicit schema check in [Float]Histogram.ReduceResolution
Signed-off-by: Linas Medziunas <linas.medziunas@gmail.com>
2023-11-25 11:38:15 +02:00
Julien Pivotto 965e603fa7
Merge pull request #13184 from bboreham/exemplar-sort
Scraping: use slices.sort for exemplars
2023-11-25 09:34:48 +01:00
zenador ccfe14d7e7
PromQL: ignore small errors for bucketQuantile (#13153)
promql: Improve histogram_quantile calculation for classic buckets

Tiny differences between classic buckets are most likely caused by floating point precision issues. With this commit, relative changes below a certain threshold are ignored. This makes the result of histogram_quantile more meaningful, and also avoids triggering the _input to histogram_quantile needed to be fixed for monotonicity_ annotations in unactionable cases.

This commit also adds explanation of the new adjustment and of the monotonicity annotation to the documentation of `histogram_quantile`.

---------

Signed-off-by: Jeanette Tan <jeanette.tan@grafana.com>
2023-11-25 00:05:38 +01:00
Yury Molodov 2e205ee95c
ui: heatmap visualization for histogram buckets (#13096)
ui: heatmap visualization for histogram buckets

Signed-off-by: Yury Moladau <yurymolodov@gmail.com>

---------

Signed-off-by: Yury Moladau <yurymolodov@gmail.com>
2023-11-24 22:44:48 +01:00
Julien Pivotto eda73dd3e5
Merge pull request #13187 from bboreham/refactor-newscrapeloop
Scraping tests: refactor scrapeLoop creation
2023-11-24 19:48:44 +01:00
Bryan Boreham 3e287e0170 Scraping tests: refactor scrapeLoop creation
Pull boilerplate code into a function. Where appropriate we set some
config on the returned object.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-24 17:28:09 +00:00
Bryan Boreham 784a2d2c74
Merge pull request #12992 from bboreham/single-scrape-buffer-pool
Scraping: share buffer pool across all scrapes
2023-11-24 16:26:19 +00:00
Bryan Boreham f0e1b592ab Scraping: use slices.sort for exemplars
The sort implementation using Go generics is used everywhere else
in Prometheus.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-24 14:42:26 +00:00
Charles Korn 59844498f7
Fix issue where queries can fail or omit OOO samples if OOO head compaction occurs between creating a querier and reading chunks (#13115)
* Add failing test.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Don't run OOO head garbage collection while reads are running.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Add further test cases for different order of operations.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Ensure all queriers are closed if `DB.blockChunkQuerierForRange()` fails.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Ensure all queriers are closed if `DB.Querier()` fails.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Invert error handling in `DB.Querier()` and `DB.blockChunkQuerierForRange()` to make it clearer

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Ensure that queries that touch OOO data can't block OOO head garbage collection forever.

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Address PR feedback: fix parameter name in comment

Co-authored-by: Jesus Vazquez <jesusvazquez@users.noreply.github.com>
Signed-off-by: Charles Korn <charleskorn@users.noreply.github.com>

* Address PR feedback: use `lastGarbageCollectedMmapRef`

Signed-off-by: Charles Korn <charles.korn@grafana.com>

* Address PR feedback: ensure pending reads are cleaned up if creating an OOO querier fails

Signed-off-by: Charles Korn <charles.korn@grafana.com>

---------

Signed-off-by: Charles Korn <charles.korn@grafana.com>
Signed-off-by: Charles Korn <charleskorn@users.noreply.github.com>
Co-authored-by: Jesus Vazquez <jesusvazquez@users.noreply.github.com>
2023-11-24 12:38:38 +01:00
Erik Sommer 0e585bf5c0 add cluster variable to Overview dashboard
Signed-off-by: Erik Sommer <ersotech@posteo.de>
2023-11-23 17:04:57 +01:00
Filip Petkovski 35a15e8f04
Add benchmark for native histograms (#13160)
* Add benchmark for native histograms

This commit adds a PromQL benchmark for queries on native histograms.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2023-11-23 14:09:17 +00:00
Paulin Todev 0102425af1
Use only one scrapeMetrics object per test. (#13051)
The scrape loop and scrape cache should use the same instance.
This brings the tests' behavior more in line with production.

Signed-off-by: Paulin Todev <paulin.todev@gmail.com>
2023-11-23 11:24:08 +00:00
Bryan Boreham 9051100aba Scraping: share buffer pool across all scrapes
Previously we had one per scrapePool, and one of those per configured
scraping job. Each pool holds a few unused buffers, so sharing one
across all scrapePools reduces total heap memory.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-23 10:23:34 +00:00
Bryan Boreham 34676a240e scrape: consistent function names for metadata
Too confusing to have `MetadataList` and `ListMetadata`, etc.
I standardised on the ones which are in an interface.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-23 09:08:02 +00:00
Charles Korn 2329fba0e5
Fix linting issues in comments (#13178)
Signed-off-by: Charles Korn <charles.korn@grafana.com>
2023-11-23 09:07:23 +00:00
Bryan Boreham a2d5c02298
Merge pull request #13084 from charleskorn/concatenatingchunkiterator
Fix issue where `concatenatingChunkIterator` can obscure errors
2023-11-22 08:16:39 +00:00
Etourneau Gwenn b37258c99b
Added Caching of network interface for Azure (#12622)
* Added Caching of network interface for Azure

Signed-off-by: Etourneau Gwenn <getourneau@yugabyte.com>

* Rename Counter for Azure cache

Signed-off-by: Etourneau Gwenn <getourneau@yugabyte.com>

* Format with goimports

Signed-off-by: Etourneau Gwenn <getourneau@yugabyte.com>

* Updated duration comparaison
Enabled cache by default with 5x the default refresh time

Signed-off-by: Etourneau Gwenn <getourneau@yugabyte.com>

* Change random function

Signed-off-by: Etourneau Gwenn <getourneau@yugabyte.com>

* Remove refresh interval

Signed-off-by: Etourneau Gwenn <getourneau@yugabyte.com>

* Remove from config as well

Signed-off-by: Etourneau Gwenn <getourneau@yugabyte.com>

* Reformat config_test
Removed uneeded error

Signed-off-by: Etourneau Gwenn <getourneau@yugabyte.com>

---------

Signed-off-by: Etourneau Gwenn <getourneau@yugabyte.com>
2023-11-21 11:59:17 +00:00
Julien Pivotto aeb8de0f27
Merge pull request #13159 from bboreham/labels-common
labels: extract common code between slice and stringlabels
2023-11-21 10:38:45 +01:00
Ziqi Zhao 8fe9250f7d optimize the logic of break the loop of reducing resolution
Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
2023-11-21 16:56:56 +08:00
Bryan Boreham f095c33da1 scrape: simplify TargetsActive function
Since everything was serialized on a single mutex, it's exactly the same
if we process targets in sequence without starting goroutines.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-20 19:28:08 +00:00
Julien Pivotto aaf5d643b6
Merge pull request #13164 from manas-rust/main
fix: The automatically generated file is inconsistent with the file i…
2023-11-20 16:17:14 +01:00
Julien Pivotto 684018dfc6
Merge pull request #13166 from prymitive/disable_scrape_compression
Add enable_compression scrape config option
2023-11-20 16:16:48 +01:00
Bryan Boreham f13bc1a5c9
Merge pull request #13040 from bboreham/smaller-stripeseries
TSDB: make the global hash lookup table smaller
2023-11-20 12:12:09 +00:00
Łukasz Mierzwa 870627fbed Add enable_compression scrape config option
Currently Prometheus will always request gzip compression from the target when sending scrape requests.
HTTP compression does reduce the amount of bytes sent over the wire and so is often desirable.
The downside of compression is that it requires extra resources - cpu & memory.

This also affects the resource usage on the target since it has to compress the response
before sending it to Prometheus.

This change adds a new option to the scrape job configuration block: enable_compression.
The default is true so it remains the same as current Prometheus behaviour.

Setting this option to false allows users to disable compression between Prometheus
and the scraped target, which will require more bandwidth but it lowers the resource
usage of both Prometheus and the target.

Fixes #12319.

Signed-off-by: Łukasz Mierzwa <l.mierzwa@gmail.com>
2023-11-20 12:02:55 +00:00
wangqing 7e2c6fc8f3 fix: The automatically generated file is inconsistent with the file in the code warehouse
reference: 3ef153b00c

Signed-off-by: wangqing <wangqing@uniontech.com>
2023-11-20 17:57:43 +08:00
Ben Kochie 9509ad082a
Merge pull request #12790 from prometheus/superq/bump-golangci-lint
Update golangci-lint.
2023-11-17 13:20:05 -06:00
Bryan Boreham a3e02f35d6 labels: extract common code between slice and stringlabels
This reduces bulk and should avoid issues if a fix is made in one file
and not the other.

A few methods now call `Range()` instead of `range`, but nothing
performance-sensitive.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2023-11-17 18:20:03 +00:00
Oleg Zaytsev f997c72f29
Make head block ULIDs descriptive (#13100)
* Make head block ULIDs descriptive

As far as I understand, these ULIDs aren't persisted anywhere, so it
should be safe to change them.

When debugging an issue, seeing an ULID like
`2ZBXFNYVVFDXFPGSB1CHFNYQTZ` or `33DXR7JA39CHDKMQ9C40H6YVVF` isn't very
helpful, so I propose to make them readable in their ULID string
version.

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* Set a different ULID for RangeHead

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

---------

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2023-11-17 12:29:36 +01:00
Julien Pivotto 1b84c01b76
Merge pull request #13143 from mmorel-35/patch-3
tsdb/tsdbutil: use Go standard errors
2023-11-17 10:23:04 +01:00
Julien Pivotto 9cb96ad2ea
Merge pull request #13142 from mmorel-35/patch-2
tsdb/fileutil: use Go standard errors
2023-11-17 10:20:40 +01:00