History

Jesus Vazquez 6cfe44d7fd WaitUntilIdle optimize idling time (#10878 ) Relates to @bboreham optimization in https://github.com/prometheus/prometheus/pull/10859 Bryan did reduce the sleep time improving the deltas on the benchmark by quite a lot. However I've been working on a similar implementation for out of order and I noticed that we actually get into this method thousands of times. @ywwg had the brilliant idea of not always sleeping before the select but actually make it a case in the select so we only sleep if we need to. The benchmark deltas are amazing ``` ❯ benchstat old_implementation.txt new_implementation_using_time_after.txt name old time/op new time/op delta LoadWAL/batches=10,seriesPerBatch=100,samplesPerSeries=7200,exemplarsPerSeries=0,mmappedChunkT=0-8 521ms ±25% 253ms ± 6% -51.47% (p=0.008 n=5+5) LoadWAL/batches=10,seriesPerBatch=100,samplesPerSeries=7200,exemplarsPerSeries=36,mmappedChunkT=0-8 773ms ± 3% 369ms ±31% -52.23% (p=0.008 n=5+5) LoadWAL/batches=10,seriesPerBatch=100,samplesPerSeries=7200,exemplarsPerSeries=72,mmappedChunkT=0-8 592ms ±28% 297ms ±28% -49.80% (p=0.008 n=5+5) LoadWAL/batches=10,seriesPerBatch=100,samplesPerSeries=7200,exemplarsPerSeries=360,mmappedChunkT=0-8 547ms ± 2% 999ms ±187% ~ (p=0.690 n=5+5) LoadWAL/batches=10,seriesPerBatch=10000,samplesPerSeries=50,exemplarsPerSeries=0,mmappedChunkT=0-8 11.3s ± 4% 1.3s ±44% -88.48% (p=0.008 n=5+5) LoadWAL/batches=10,seriesPerBatch=10000,samplesPerSeries=50,exemplarsPerSeries=2,mmappedChunkT=0-8 11.1s ± 1% 1.2s ±20% -89.08% (p=0.008 n=5+5) LoadWAL/batches=10,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=0,mmappedChunkT=0-8 1.24s ± 3% 0.18s ± 7% -85.76% (p=0.008 n=5+5) LoadWAL/batches=10,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=2,mmappedChunkT=0-8 1.24s ± 2% 0.18s ± 5% -85.24% (p=0.008 n=5+5) LoadWAL/batches=10,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=5,mmappedChunkT=0-8 1.23s ± 5% 0.27s ±33% -77.73% (p=0.008 n=5+5) LoadWAL/batches=10,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=24,mmappedChunkT=0-8 1.28s ± 1% 0.36s ± 7% -71.51% (p=0.008 n=5+5) LoadWAL/batches=100,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=0,mmappedChunkT=3800-8 12.1s ± 1% 3.1s ± 6% -74.33% (p=0.008 n=5+5) LoadWAL/batches=100,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=2,mmappedChunkT=3800-8 12.1s ± 1% 3.4s ± 4% -71.94% (p=0.008 n=5+5) LoadWAL/batches=100,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=5,mmappedChunkT=3800-8 12.1s ± 1% 3.8s ±17% -68.35% (p=0.008 n=5+5) LoadWAL/batches=100,seriesPerBatch=1000,samplesPerSeries=480,exemplarsPerSeries=24,mmappedChunkT=3800-8 12.4s ± 1% 4.0s ±18% -67.71% (p=0.008 n=5+5) ``` Benchmarked on Linux ``` goos: linux goarch: amd64 pkg: github.com/prometheus/prometheus/tsdb cpu: 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz ``` Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>		2 years ago
..
agent	…
chunkenc	…
chunks	Reduce chunk write queue memory usage 2 (#10874 )	2 years ago
docs	Update godoc links and remove note about TSDB versioning (#10754 )	3 years ago
encoding	…
errors	…
fileutil	Make necessary change to compile promql parser to wasm (#10683 )	3 years ago
goversion	…
index	Fix typo in symbol table size exceeded error message (#10746 )	3 years ago
record	…
test	…
testdata	…
tombstones	…
tsdbutil	…
wal	Remote Write: Rename confusing `walDir` parameter to `dir` (#10464 )	3 years ago
.gitignore	…
CHANGELOG.md	…
README.md	Update godoc links and remove note about TSDB versioning (#10754 )	3 years ago
block.go	fix typo (#10696 )	3 years ago
block_test.go	…
blockwriter.go	…
blockwriter_test.go	…
compact.go	…
compact_test.go	…
db.go	Don't increment prometheus_tsdb_compactions_failed_total when context is canceled (#10772 )	2 years ago
db_test.go	…
example_test.go	…
exemplar.go	…
exemplar_test.go	…
head.go	Fix panic if series is not found when deleting series	2 years ago
head_append.go	…
head_bench_test.go	…
head_read.go	…
head_test.go	…
head_wal.go	WaitUntilIdle optimize idling time (#10878 )	2 years ago
isolation.go	…
isolation_test.go	…
mocks_test.go	…
querier.go	…
querier_bench_test.go	…
querier_test.go	…
repair.go	…
repair_test.go	…
tsdbblockutil.go	tsdb: commit data periodically in CreateBlock (#10788 )	2 years ago
wal.go	…
wal_test.go	…

README.md

TSDB

This directory contains the Prometheus TSDB (Time Series DataBase) library, which handles storage and querying of all Prometheus v2 data.

Documentation

External resources

A writeup of the original design can be found here.
Video: Storing 16 Bytes at Scale from PromCon 2017.
Compression is based on the Gorilla TSDB white paper.

A series of blog posts explaining different components of TSDB: