The Prometheus monitoring system and time series database.

prometheus hacktoberfest metrics alerting graphing monitoring time-series

Ganesh Vernekar d4b9fe801f M-map full chunks of Head from disk (#6679 ) When appending to the head and a chunk is full it is flushed to the disk and m-mapped (memory mapped) to free up memory Prom startup now happens in these stages - Iterate the m-maped chunks from disk and keep a map of series reference to its slice of mmapped chunks. - Iterate the WAL as usual. Whenever we create a new series, look for it's mmapped chunks in the map created before and add it to that series. If a head chunk is corrupted the currpted one and all chunks after that are deleted and the data after the corruption is recovered from the existing WAL which means that a corruption in m-mapped files results in NO data loss. [Mmaped chunks format](https://github.com/prometheus/prometheus/blob/master/tsdb/docs/format/head_chunks.md) - main difference is that the chunk for mmaping now also includes series reference because there is no index for mapping series to chunks. [The block chunks](https://github.com/prometheus/prometheus/blob/master/tsdb/docs/format/chunks.md) are accessed from the index which includes the offsets for the chunks in the chunks file - example - chunks of series ID have offsets 200, 500 etc in the chunk files. In case of mmaped chunks, the offsets are stored in memory and accessed from that. During WAL replay, these offsets are restored by iterating all m-mapped chunks as stated above by matching the series id present in the chunk header and offset of that chunk in that file. Prombench results _WAL Replay_ 1h Wal reply time 30% less wal reply time - 4m31 vs 3m36 2h Wal reply time 20% less wal reply time - 8m16 vs 7m _Memory During WAL Replay_ High Churn: 10-15% less RAM - 32gb vs 28gb 20% less RAM after compaction 34gb vs 27gb No Churn: 20-30% less RAM - 23gb vs 18gb 40% less RAM after compaction 32.5gb vs 20gb Screenshots are in [this comment](https://github.com/prometheus/prometheus/pull/6679#issuecomment-621678932) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>		5 years ago
..
chunkenc	storage: Added Chunks{Queryable/Querier/SeriesSet/Series/Iteratable. Added generic Merge{SeriesSet/Querier} implementation. (#7005 )	5 years ago
chunks	M-map full chunks of Head from disk (#6679 )	5 years ago
cmd/tsdb	Use .UTC everywhere we use time.Unix (#7066 )	5 years ago
docs/format	M-map full chunks of Head from disk (#6679 )	5 years ago
encoding	Avoid WriteAt for Postings.	5 years ago
errors	Moving tsdb into its own subdirectory	5 years ago
fileutil	Replace fileutil.ReadDir with ioutil.ReadDir (#7029 ) (#7033 )	5 years ago
goversion	Fix some typo in comments (#6730 )	5 years ago
index	Goroutine: Fix ambiguous variable (#7175 )	5 years ago
record	Spelling (#6517 )	5 years ago
test	Port tsdb to use pkg/labels. (#6326 )	5 years ago
testdata	Handle V1 indexes, some of which have unsorted posting offset tables. (#6564 )	5 years ago
tombstones	tsdb: add tombstonesHeaderSize constant (#7028 )	5 years ago
tsdbutil	storage: Added Chunks{Queryable/Querier/SeriesSet/Series/Iteratable. Added generic Merge{SeriesSet/Querier} implementation. (#7005 )	5 years ago
wal	Capitalizing first letter of all log lines (#7043 )	5 years ago
.gitignore	Moving tsdb into its own subdirectory	5 years ago
CHANGELOG.md	Spelling (#6517 )	5 years ago
README.md	Edit TSDB README badges	5 years ago
block.go	Revert head posting optimization	5 years ago
block_test.go	M-map full chunks of Head from disk (#6679 )	5 years ago
compact.go	Capitalizing first letter of all log lines (#7043 )	5 years ago
compact_test.go	M-map full chunks of Head from disk (#6679 )	5 years ago
db.go	M-map full chunks of Head from disk (#6679 )	5 years ago
db_test.go	M-map full chunks of Head from disk (#6679 )	5 years ago
head.go	M-map full chunks of Head from disk (#6679 )	5 years ago
head_bench_test.go	M-map full chunks of Head from disk (#6679 )	5 years ago
head_test.go	M-map full chunks of Head from disk (#6679 )	5 years ago
isolation.go	Do not attempt isolation for appendID == 0	5 years ago
mocks_test.go	Revert head posting optimization	5 years ago
querier.go	Make sure deleted intervals are excluded from Seek (#6980 )	5 years ago
querier_bench_test.go	M-map full chunks of Head from disk (#6679 )	5 years ago
querier_test.go	M-map full chunks of Head from disk (#6679 )	5 years ago
repair.go	Capitalizing first letter of all log lines (#7043 )	5 years ago
repair_test.go	Port tsdb to use pkg/labels. (#6326 )	5 years ago
tsdbblockutil.go	M-map full chunks of Head from disk (#6679 )	5 years ago
wal.go	Capitalizing first letter of all log lines (#7043 )	5 years ago
wal_test.go	Replace fileutil.ReadDir with ioutil.ReadDir (#7029 ) (#7033 )	5 years ago

README.md

TSDB

This repository contains the Prometheus storage layer that is used in its 2.x releases.

A writeup of its design can be found here.

Based on the Gorilla TSDB white papers.

Video: Storing 16 Bytes at Scale from PromCon 2017.