Commit Graph

1514 Commits (1bd16db101933f70308beb5b85e7fa3968a5228c)

Author SHA1 Message Date
Bryan Boreham 7996a13fdd
Merge pull request #15403 from bboreham/fix-rw-benchmark-startup
[TESTS] Remote-Write: Fix BenchmarkStartup
2024-11-25 17:31:24 +00:00
Arve Knudsen 89bbb885e5
Upgrade to golangci-lint v1.62.0 (#15424)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-11-20 17:22:20 +01:00
Arve Knudsen 9c02c26418
OTLP receiver: Convert also metric metadata (#15416)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-11-20 11:13:03 +01:00
Björn Rabenstein 384c5951ef
Merge pull request #14489 from harry671003/implement_metadata_limit
storage: Implement limit in mergeGenericQuerier
2024-11-19 17:32:16 +01:00
Đurica Yuri Nikolić 101b1c307f
[ENHANCEMEN] Remote-Write: optionally use a DNS resolver that picks a random IP (#15329)
When a remote-write is executed towards a host name that is resolved to multiple IP addresses, this PR introduces a possibility to force creation of new connections used for the remote-write request to a randomly chosen IP address from the ones corresponding to the host name. The default behavior remains unchanged, i.s., the IP address used for the connection creation remains the one chosen by Go.

This is an experimental feature, it is disabled by default.

Signed-off-by: Yuri Nikolic <durica.nikolic@grafana.com>
2024-11-15 15:41:49 +00:00
Bryan Boreham 0ef0b75a4f [TESTS] Remote-Write: Fix BenchmarkStartup
It was crashing due to uninitialized metrics, and not terminating due to
incorrectly reading segment names.

We need to export `SetMetrics` to avoid the first problem.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-11-15 11:22:07 +00:00
SuperQ bc1bc5c118
Update sigv4 library
Migrate use of prometheus/common/sigv4 to prometheus/sigv4.

Related to: https://github.com/prometheus/common/issues/709

Signed-off-by: SuperQ <superq@gmail.com>
2024-11-08 22:35:44 +01:00
Arthur Silva Sens 03995775a7
Allow UTF-8 characters in metric and label names as opt-in feature (#15258)
* Allow UTF-8 characters in metric and label names as opt-in feature

---------

Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
2024-11-08 10:02:48 -03:00
🌲 Harry 🌊 John 🏔 f9bc50b247 storage: Implement limit in mergeGenericQuerier
Signed-off-by: 🌲 Harry 🌊 John 🏔 <johrry@amazon.com>
2024-11-07 09:08:23 -08:00
Matthieu MOREL af1a19fc78 enable errorf rule from perfsprint linter
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2024-11-06 16:50:36 +01:00
Arve Knudsen 7c4f877881
otlptranslator: Harmonize non-UTF8 sanitization w/ naming rules. (#15314)
* otlptranslator: Harmonize non-UTF8 sanitization w/ naming rules.

Harmonize non-UTF8 sanitization w/ Prometheus naming rules.

---------

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-11-03 12:56:54 +00:00
Arthur Silva Sens 1a22b1d846
bugfix: Fix otlp translator switching colons to undescores (#15251)
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
2024-11-03 13:15:49 +01:00
Arthur Silva Sens 034d2b24bc
Fix typos in tests (#15312)
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
2024-11-02 15:38:51 +01:00
Bartlomiej Plotka dd4eb4590d
Merge pull request #15222 from jmichalek132/jm-add-tcp-events-to-tracing
tracing: add tcp events to remote store span
2024-11-01 14:26:15 +01:00
Arthur Silva Sens ea06f1a1d1
bugfix: Fix otlp translation of foreign characters
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
2024-10-30 19:29:31 -03:00
Arthur Silva Sens 8588289c24
otlp translator: Add test showing bugs
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
2024-10-30 19:28:53 -03:00
Juraj Michalek 76ff12b32a chore: only create span events
Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com>
2024-10-30 09:41:38 +01:00
Juraj Michalek 7ecdb55b02 chore: use otelhttptrace instead
Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com>
2024-10-30 09:41:38 +01:00
Juraj Michalek 3c1ffbb2fd chore: added idleTimeMs and fixed linting issues
Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com>
2024-10-30 09:41:38 +01:00
Juraj Michalek 2e7c739d44 chore: add tcp events to remote store span
Signed-off-by: Juraj Michalek <juraj.michalek132@gmail.com>
2024-10-30 09:41:38 +01:00
shenpengfeng a44db5f784 chore: fix function name in comment
Signed-off-by: shenpengfeng <xinhangzhou@icloud.com>
2024-10-29 17:58:44 +08:00
Pedro Tanaka bab587b9dc
Agent: allow for ingestion of CT samples (#15124)
* Remove unused option from HeadOptions

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Improve docs for appendable() method in head appender

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Ingest CT (float) samples in Agent DB

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* allow for ingestion of CT native histogram

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* adding some verification for ct ts

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Validating CT histogram before append and add newly created series to pending series

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* checking the wal for written samples

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Checking for samples in test

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* adding case for validations

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* fixing comparison when dedupelabels is enabled

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* unite tests, use table testing

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Implement CT related methods in timestampTracker for write storage

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* adding error case to test

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* removing unused fields

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Updating lastTs for series when adding CT to invalidate duplicates

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* making sure that updating the lastTS wont cause OOO later on in Commit();

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

---------

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
2024-10-27 01:06:34 +01:00
Ayoub Mrini 93db81dd3d
Merge pull request #14983 from machine424/dopp
fix(storage/mergeQuerier): fix a data race
2024-10-25 18:34:51 +02:00
Charlie Le d87f7440ca support int exemplar value type
When the exemplar type is an int, it incorrectly gets converted to a 0
when DoubleValue() is called on the exemplar. This adds a check to
ensure that the value is converted properly based on the type.

Signed-off-by: Charlie Le <charlie_le@apple.com>
2024-10-25 08:47:54 -07:00
Vanshika cccbe72514
TSDB: Fix some edge cases when OOO is enabled (#14710)
Fix some edge cases when OOO is enabled

Signed-off-by: Vanshikav123 <vanshikav928@gmail.com>
Signed-off-by: Vanshika <102902652+Vanshikav123@users.noreply.github.com>
Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com>
Co-authored-by: Jesus Vazquez <jesusvzpg@gmail.com>
2024-10-23 17:34:28 +02:00
machine424 cebcdce78a
fix(storage/mergeQuerier): copy the matcjers slice before passing it to queriers as
some of them may alter it.

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2024-10-22 14:08:47 +02:00
machine424 eb523a6b29
fix(storage/mergeQuerier): add a reproducer for data race that occurs when one of the queriers alters the passed matchers and propose a fix
Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2024-10-22 14:08:46 +02:00
Björn Rabenstein 5e987e7c7d
Merge pull request #15092 from jan--f/clarify-storage-contract
storage: require selectors to always return matching results
2024-10-09 13:07:59 +02:00
TJ Hoplock 6ebfbd2d54 chore!: adopt log/slog, remove go-kit/log
For: #14355

This commit updates Prometheus to adopt stdlib's log/slog package in
favor of go-kit/log. As part of converting to use slog, several other
related changes are required to get prometheus working, including:
- removed unused logging util func `RateLimit()`
- forward ported the util/logging/Deduper logging by implementing a small custom slog.Handler that does the deduping before chaining log calls to the underlying real slog.Logger
- move some of the json file logging functionality to use prom/common package functionality
- refactored some of the new json file logging for scraping
- changes to promql.QueryLogger interface to swap out logging methods for relevant slog sugar wrappers
- updated lots of tests that used/replicated custom logging functionality, attempting to keep the logical goal of the tests consistent after the transition
- added a healthy amount of `if logger == nil { $makeLogger }` type conditional checks amongst various functions where none were provided -- old code that used the go-kit/log.Logger interface had several places where there were nil references when trying to use functions like `With()` to add keyvals on the new *slog.Logger type

Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>
2024-10-07 15:58:50 -04:00
Matthieu MOREL ab64966e9d
fix: use "ErrorContains" or "EqualError" instead of "Contains(t, err.Error()" and "Equal(t, err.Error()" (#15094)
* fix: use "ErrorContains" or "EqualError" instead of "Contains(t, err.Error()" and "Equal(t, err.Error()"

---------

Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-10-06 16:35:29 +00:00
Jan Fajerski 563bfdd384 storage: require selectors to always return matching results
Signed-off-by: Jan Fajerski <jfajersk@redhat.com>
2024-10-04 12:10:16 +02:00
György Krajcsovits 44ebbb8458 Fix missing histogram copy in sampleRing
The specialized version of sample add to the ring:
func addH(s hSample, buf []hSample, r *sampleRing) []hSample
func addFH(s fhSample, buf []fhSample, r *sampleRing) []fhSample
already correctly copy histogram samples from the reused hReader, fhReader
buffers, but the generic version does not. This means that the
data is overwritten on the next read if the sample ring has seen histogram
and float samples at the same time and switched to generic mode.

The `genericAdd` function (which was commented anyway) is by now quite
different from the specialized functions so that this commit deletes
it.

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-10-02 13:57:28 +02:00
György Krajcsovits b5479831b8 Unit test for regression in rate vs float and histogram mixup
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-10-02 13:56:33 +02:00
Ayoub Mrini 105ab2e95a
fix(test): adjust defer invocations (#14996)
Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2024-09-27 17:13:51 +01:00
Arthur Silva Sens 6bd9b1a7cc
Histogram CT Zero ingestion
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
2024-09-26 11:29:22 -03:00
Callum Styan 005bd33fe2
support v2 proto for BenchmarkSampleSend (#14935)
Signed-off-by: Callum Styan <callumstyan@gmail.com>
2024-09-19 15:38:26 -07:00
Callum Styan c328d5fc88
fix rwv2 build write request benchmark, also change how the memory usage (#14925)
is reported for these benchmarks to more accurately reflect what's
actually allocated

Signed-off-by: Callum Styan <callumstyan@gmail.com>
2024-09-18 07:04:10 +01:00
Antoine Pultier d90d0976b5
fix(bstream/writeByte): ensure it appends only one byte (#14854)
fix(bstream/writeByte): ensure it appends only one byte

Signed-off-by: Antoine Pultier <antoine.pultier@sintef.no>
2024-09-17 16:28:33 +02:00
Carrie Edwards 14e3c05ce8
tsdb: Add support for ingestion of out-of-order native histogram samples (#14546)
Add support for ingesting OOO native histograms

* Add flag for enabling and disabling OOO native histogram ingestion
* Update OOO querying tests to include native histogram samples
* Add OOO head tests
* Add test for OOO native histogram counter reset headers

Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-authored by: Carrie Edwards <edwrdscarrie@gmail.com>
Co-authored by: Jeanette Tan <jeanette.tan@grafana.com>
Co-authored by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-authored by: Fiona Liao <fiona.liao@grafana.com>
2024-09-17 11:19:06 +02:00
Harry John 919dc0cbc6
storage: Update LabelQuerier interface to return sorted label values (#14849)
* Change LabelQuerier.LabelValues() to return sorted values

---------

Signed-off-by: 🌲 Harry 🌊 John 🏔 <johrry@amazon.com>
2024-09-17 08:55:02 +02:00
Nathan Baulch 50cd453c8f
chore: Fix typos (#14868)
* Fix typos

---------

Signed-off-by: Nathan Baulch <nathan.baulch@gmail.com>
2024-09-10 22:32:03 +02:00
Arve Knudsen 4fc562f9e7
OTLP: Support context cancellation/timeout during translation (#14612)
* OTLP: Support context cancellation/timeout during translation

---------

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-09-08 17:13:40 +02:00
György Krajcsovits a693dd19f2 Fix: chunkenc.MockSeriesIterator
Starts its index from 0 , but users call Next() before first sample
so it needs to start from -1

Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-08-30 16:44:36 +02:00
Arve Knudsen bc6c2c5d35
OTLP Receiver: Add tests (#14764)
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-08-30 11:30:57 +02:00
Bryan Boreham f90c7a11d1 [REFACTOR] OTLP translator: simplify time conversion
We don't need multiple levels of abstraction to convert nanoseconds to
milliseconds.
We do benefit from tests, however.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-08-29 12:00:01 +01:00
Owen Williams 9da75328ea
fix(utf8): ensure correct validation when legacy mode turned on (#14736)
fix(utf8): ensure correct validation when legacy mode turned on

This depends on the included update of the prometheus/common dependency.

---------

Signed-off-by: Owen Williams <owen.williams@grafana.com>
2024-08-28 17:15:42 +02:00
Justin Lei 3a82cd5a7e
Add streaming remote read to ReadClient (#11379)
* Add streaming remote read to ReadClient

Signed-off-by: Justin Lei <justin.lei@grafana.com>

* Apply suggestions from code review

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Justin Lei <justin.lei@grafana.com>

* Remote read instrumentation tweaks

Signed-off-by: Justin Lei <lei.justin@gmail.com>

* Minor cleanups

Signed-off-by: Justin Lei <lei.justin@gmail.com>

* In-line handleChunkedResponse

Signed-off-by: Justin Lei <lei.justin@gmail.com>

* Fix lints

Signed-off-by: Justin Lei <lei.justin@gmail.com>

* Explicitly call cancel() when needed

Signed-off-by: Justin Lei <lei.justin@gmail.com>

* Update chunkedSeries, chunkedSeriesIterator for new interfaces

Signed-off-by: Justin Lei <lei.justin@gmail.com>

* Adapt remote.chunkedSeries to use prompb.ChunkedSeries

Signed-off-by: Justin Lei <lei.justin@gmail.com>

* Fix lint

Signed-off-by: Justin Lei <lei.justin@gmail.com>

---------

Signed-off-by: Justin Lei <justin.lei@grafana.com>
Signed-off-by: Justin Lei <lei.justin@gmail.com>
Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
2024-08-28 07:23:54 +01:00
Devin Trejo d4994e5bc4
fix: Remote-write-reciever returns 4xx when request contains a time series with duplicate labels. (#14716)
Signed-off-by: Devin Trejo <dtrejo@palantir.com>
2024-08-23 15:15:27 -07:00
beorn7 0f760f63dd lint: Revamp our linting rules, mostly around doc comments
Several things done here:

- Set `max-issues-per-linter` to 0 so that we actually see all linter
  warnings and not just 50 per linter. (As we also set
  `max-same-issues` to 0, I assume this was the intention from the
  beginning.)

- Stop using the golangci-lint default excludes (by setting
  `exclude-use-default: false`. Those are too generous and don't match
  our style conventions. (I have re-added some of the excludes
  explicitly in this commit. See below.)

- Re-add the `errcheck` exclusion we have used so far via the
  defaults.

- Exclude the signature requirement `govet` has for `Seek` methods
  because we use non-standard `Seek` methods a lot. (But we keep other
  requirements, while the default excludes completely disabled the
  check for common method segnatures.)

- Exclude warnings about missing doc comments on exported symbols. (We
  used to be pretty adamant about doc comments, but stopped that at
  some point in the past. By now, we have about 500 missing doc
  comments. We may consider reintroducing this check, but that's
  outside of the scope of this commit. The default excludes of
  golangci-lint essentially ignore doc comments completely.)

- By stop using the default excludes, we now get warnings back on
  malformed doc comments. That's the most impactful change in this
  commit. It does not enforce doc comments (again), but _if_ there is
  a doc comment, it has to have the recommended form. (Most of the
  changes in this commit are fixing this form.)

- Improve wording/spelling of some comments in .golangci.yml, and
  remove an outdated comment.

- Leave `package-comments` inactive, but add a TODO asking if we
  should change that.

- Add a new sub-linter `comment-spacings` (and fix corresponding
  comments), which avoids missing spaces after the leading `//`.

Signed-off-by: beorn7 <beorn@grafana.com>
2024-08-22 17:36:11 +02:00
Bryan Boreham 95b3c49836
[TESTS] remote-write: fix slice init length (#14702)
fix: fix slice init length
2024-08-22 12:53:45 +01:00