Up to 32-byte values this saves garbage, runs faster.
For prefixes, only `toLower` the part we need for the map lookup.
Split toNormalisedLower into fast and slow paths, to avoid a penalty
for the `copy` call in the case where no allocations are done.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
In general aim for the happy case when the exposer lists the buckets
in ascending order.
Use Compact(2) to compact the result of nhcb convert.
This is more in line with how client_golang optimizes spans vs
buckets.
aef8aedb4b/prometheus/histogram.go (L1485)
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
The wrong source was used to return the created timestamp, leading to
index out of bound panic. One line fix.
Refactor the requirement test to be generic and be able to
test OpenMetrics and Prom parsers as well.
There are some differencies in what the parsers support, the Prom
parser doesn't have created timestamp.
The protobuf parser uses different formatting to identify the metric
for the scrape loop.
Each parser represents the sample timestamp differently.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Remove unused option from HeadOptions
Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
* Improve docs for appendable() method in head appender
Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
* Ingest CT (float) samples in Agent DB
Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
* allow for ingestion of CT native histogram
Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
* adding some verification for ct ts
Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
* Validating CT histogram before append and add newly created series to pending series
Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
* checking the wal for written samples
Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
* Checking for samples in test
Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
* adding case for validations
Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
* fixing comparison when dedupelabels is enabled
Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
* unite tests, use table testing
Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
* Implement CT related methods in timestampTracker for write storage
Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
* adding error case to test
Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
* removing unused fields
Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
* Updating lastTs for series when adding CT to invalidate duplicates
Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
* making sure that updating the lastTS wont cause OOO later on in Commit();
Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
---------
Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
Documented that WAL can still be written after memory-snapshot-on-shutdown - #10824
Co-authored-by: Björn Rabenstein <github@rabenste.in>
Signed-off-by: gopi <gopi.singaravelan.k@gmail.com>
---------
Signed-off-by: Gopi-eng2202 <gopi.singaravelan.k@gmail.com>
Signed-off-by: gopi <gopi.singaravelan.k@gmail.com>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
When the exemplar type is an int, it incorrectly gets converted to a 0
when DoubleValue() is called on the exemplar. This adds a check to
ensure that the value is converted properly based on the type.
Signed-off-by: Charlie Le <charlie_le@apple.com>
* docs: 2 to 3 migration guide
Signed-off-by: Jan Fajerski <jfajersk@redhat.com>
* docs/stability: add 3.0 section
Signed-off-by: Jan Fajerski <jfajersk@redhat.com>
* docs/migration: details on enabling legacy name validation
Signed-off-by: Owen Williams <owen.williams@grafana.com>\
* migration: add log format and `le` normalization
Signed-off-by: Jan Fajerski <jfajersk@redhat.com>
* migration: add new enable_http2 default for remote write
Signed-off-by: Jan Fajerski <jfajersk@redhat.com>
---------
Signed-off-by: Jan Fajerski <jfajersk@redhat.com>
Signed-off-by: Owen Williams <owen.williams@grafana.com>
Co-authored-by: Owen Williams <owen.williams@grafana.com>
I used these wrapper methods during initial development of the custom
handler that the deduper now implements. Since the deduper implements
slog.Handler and can be used directly as a logger, these wrapper methods
are no longer needed.
Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>
This change should have been included in the initial prometheus slog
conversion, but I must've lost track of it in all the rebases involved
in that PR.
This changes the dedupe logger so that the only method that needs to use
the lock is the `Handle()` method that actually interacts with the
deduplication map.
Ex:
```
==================
WARNING: DATA RACE
Write at 0x00c000518bc0 by goroutine 29481:
github.com/prometheus/prometheus/util/logging.(*Deduper).WithAttrs()
/home/tjhop/go/src/github.com/prometheus/prometheus/util/logging/dedupe.go:89 +0xef
log/slog.(*Logger).With()
/home/tjhop/.asdf/installs/golang/1.23.1/go/src/log/slog/logger.go:132 +0x106
github.com/prometheus/prometheus/storage/remote.NewQueueManager()
/home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/queue_manager.go:483 +0x7a9
github.com/prometheus/prometheus/storage/remote.(*WriteStorage).ApplyConfig()
/home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/write.go:201 +0x102c
github.com/prometheus/prometheus/storage/remote.(*Storage).ApplyConfig()
/home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/storage.go:92 +0xfd
github.com/prometheus/prometheus/storage/remote.TestWriteStorageApplyConfigsDuringCommit.func1()
/home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/storage_test.go:172 +0x3e4
github.com/prometheus/prometheus/storage/remote.TestWriteStorageApplyConfigsDuringCommit.gowrap1()
/home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/storage_test.go:174 +0x41
Previous read at 0x00c000518bc0 by goroutine 31261:
github.com/prometheus/prometheus/util/logging.(*Deduper).Handle()
/home/tjhop/go/src/github.com/prometheus/prometheus/util/logging/dedupe.go:82 +0x2b1
log/slog.(*Logger).log()
/home/tjhop/.asdf/installs/golang/1.23.1/go/src/log/slog/logger.go:257 +0x228
log/slog.(*Logger).Error()
/home/tjhop/.asdf/installs/golang/1.23.1/go/src/log/slog/logger.go:230 +0x3d4
github.com/prometheus/prometheus/tsdb/wlog.(*Watcher).loop()
/home/tjhop/go/src/github.com/prometheus/prometheus/tsdb/wlog/watcher.go:254 +0x2db
github.com/prometheus/prometheus/tsdb/wlog.(*Watcher).Start.gowrap1()
/home/tjhop/go/src/github.com/prometheus/prometheus/tsdb/wlog/watcher.go:227 +0x33
Goroutine 29481 (running) created at:
github.com/prometheus/prometheus/storage/remote.TestWriteStorageApplyConfigsDuringCommit()
/home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/storage_test.go:164 +0xe4
testing.tRunner()
/home/tjhop/.asdf/installs/golang/1.23.1/go/src/testing/testing.go:1690 +0x226
testing.(*T).Run.gowrap1()
/home/tjhop/.asdf/installs/golang/1.23.1/go/src/testing/testing.go:1743 +0x44
Goroutine 31261 (running) created at:
github.com/prometheus/prometheus/tsdb/wlog.(*Watcher).Start()
/home/tjhop/go/src/github.com/prometheus/prometheus/tsdb/wlog/watcher.go:227 +0x177
github.com/prometheus/prometheus/storage/remote.(*QueueManager).Start()
/home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/queue_manager.go:934 +0x304
github.com/prometheus/prometheus/storage/remote.(*WriteStorage).ApplyConfig()
/home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/write.go:232 +0x151b
github.com/prometheus/prometheus/storage/remote.(*Storage).ApplyConfig()
/home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/storage.go:92 +0xfd
github.com/prometheus/prometheus/storage/remote.TestWriteStorageApplyConfigsDuringCommit.func1()
/home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/storage_test.go:172 +0x3e4
github.com/prometheus/prometheus/storage/remote.TestWriteStorageApplyConfigsDuringCommit.gowrap1()
/home/tjhop/go/src/github.com/prometheus/prometheus/storage/remote/storage_test.go:174 +0x41
==================
--- FAIL: TestWriteStorageApplyConfigsDuringCommit (2.26s)
testing.go:1399: race detected during execution of test
FAIL
FAIL github.com/prometheus/prometheus/storage/remote 68.321s
```
Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>
Remote-write creates several shards to parallelise sending, each with
its own http connection. We do not want them all combined onto one
socket by http2.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Call through to the underlaying parser if we are not in a histogram
and the entry is a series or exponential native histogram. Otherwise store
and retrieve CT for NHCB.
* fix(omparser): losing exemplars when CT is parsed
Fixes: #15137
Ignore exemplars while peeking ahead during CT parsing.
Simplify state reset with defer().
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Fix some edge cases when OOO is enabled
Signed-off-by: Vanshikav123 <vanshikav928@gmail.com>
Signed-off-by: Vanshika <102902652+Vanshikav123@users.noreply.github.com>
Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com>
Co-authored-by: Jesus Vazquez <jesusvzpg@gmail.com>
* NHCB: scrape use state field and not booleans
From comment https://github.com/prometheus/prometheus/pull/14978#discussion_r1800898724
Also make compareLabels read only and move storeLabels to the first
processed classic histogram series.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* Speed up TestConvertClassicHistogramsToNHCB 3x
Reduce the startup time and timeouts
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* lint fix
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
---------
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* promtool: Add debug flag for rule tests
This makes it print out the tsdb state (both input_series and rules that
are run) at the end of a test, making reasoning about tests much easier.
Signed-off-by: David Leadbeater <dgl@dgl.cx>
* Reuse generated test name from junit testing
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
---------
Signed-off-by: David Leadbeater <dgl@dgl.cx>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-authored-by: David Leadbeater <dgl@dgl.cx>
Bryan Boreham