127 Commits (e78fedf05475d182091472ff712f18c15a743259)

Author	SHA1	Message	Date
Ganesh Vernekar	c59ed492b2	Vertical query merging and compaction (#370) ... * Vertical series iterator Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Select overlapped blocks first in compactor Plan() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Added vertical compaction Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Code cleanup and comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix tests Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Add benchmark for compaction Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Perform vertical compaction only when blocks are overlapping. Actions for vertical compaction: * Sorting chunk metas * Calling chunks.MergeOverlappingChunks on the chunks Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Benchmark for vertical compaction * BenchmarkNormalCompaction => BenchmarkCompaction * Moved the benchmark from db_test.go to compact_test.go Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Benchmark for query iterator and seek for non overlapping blocks Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Vertical query merge only for overlapping blocks Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Simplify logging in Compact(...) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Updated CHANGELOG.md Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Calculate overlapping inside populateBlock Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * MinTime and MaxTime for BlockReader. Using this to find overlapping blocks in populateBlock() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Sort blocks w.r.t. MinTime in reload() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Log about overlapping in LeveledCompactor.write() instead of returning bool Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Log about overlapping inside LeveledCompactor.populateBlock() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Refactor createBlock to take optional []Series Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * review1 Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com> * Updated CHANGELOG and minor nits Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * nits Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Updated CHANGELOG Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Refactor iterator and seek benchmarks for Querier. Also has as overlapping blocks. Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Additional test case Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * genSeries takes optional labels. Updated BenchmarkQueryIterator and BenchmarkQuerySeek. Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Split genSeries into genSeries and populateSeries Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Check error in benchmark Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Warn about overlapping blocks in reload() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Krasi Georgiev	bf2239079d	refactor multi errors ... Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	d48606827c	simplify closers ... Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	315de4c782	fix windows tests ... Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	53c18e7a41	use a global indexFilename constant ... Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Ganesh Vernekar	1a9d08adc5	Don't write empty blocks (#374) ... * Dont write empty blocks when a compaction results in a block with no samples. Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
mknapphrt	ebf5d74325	Added storage size based retention method and new metrics (#343) ... Added methods needed to retain data based on a byte limitation rather than time. Limitation is only applied if the flag is set (defaults to 0). Both blocks that are older than the retention period and the blocks that make the size of the storage too large are removed. 2 new metrics for keeping track of the size of the local storage folder and the amount of times data has been deleted because the size restriction was exceeded. Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>	6 years ago
Brian Brazil	c93e261466	Reduce memory taken up by posting/symbol tables. ... Reuse the string already allocated for symbols in the posting tables. Use a slice for symbols in v2 format. Move symbol size logic into the index code. Avoid duplication of lookupSymbol logic. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	6 years ago
Krasi Georgiev	5a9ddeecef	fix lint errors (#439) ... unexported NewMemTombstones as this returns unexported memTombstones type which will not be shows in godoc. Added missing comments for exported methods. Removed unused RecordLogger,RecordReader interfaces. Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Ganesh Vernekar	3a08a71d86	LabelNames() method to get all unique label names (#369) ... * LabelNames() method to get all unique label names Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Ganesh Vernekar	6e712963e2	Fix updating of NumTombstones in block.Delete(..) (#385) ... Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Krasi Georgiev	d05611c027	removed some unused code and moved mockSeriesSet in querier_test (#394) ... Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Julius Volz	5ae6c60d39	Handle a bunch of unchecked errors (#365) ... As discovered by "gosec". Signed-off-by: Julius Volz <julius.volz@gmail.com>	6 years ago
Ganesh Vernekar	77d2c7cf3d	Added metric for symbol table size ... Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
codwu	667e539a7a	Merge branch 'master' of https://github.com/prometheus/tsdb into tsdb-delete	6 years ago
Benoît Knecht	1e1b2e163d	Make interval overlap comparisons more explicit ... Blocks are half-open intervals [a, b), while all other intervals (chunks, head, ...) are closed intervals [a, b]. Make that distinction explicit by defining `OverlapsClosedInterval()` methods for blocks and chunks, and using them in place of the more generic `intervalOverlap()` function. This change also fixes `db.Querier()` and `db.Delete()`, which could previously return one extraneous block at the end of the specified interval. Signed-off-by: Benoît Knecht <benoit.knecht@fsfe.org>	7 years ago
Fabian Reinartz	d907928e57	Clarify docs, error on unexpected meta read errors ... Signed-off-by: Fabian Reinartz <freinartz@google.com>	7 years ago
Fabian Reinartz	af9003dcef	Add resilience to crashes during deletion ... Signed-off-by: Fabian Reinartz <freinartz@google.com>	7 years ago
codwu	cd145c90d5	remove `put` function and use RLock in `Iter` function ... Signed-off-by: codwu <wuhan9087@163.com>	7 years ago
codwu	84a45cb79a	add rwmutex to prevent concurrent map read when delete series ... Signed-off-by: codwu <wuhan9087@163.com>	7 years ago
Ganesh Vernekar	528439aa93	Cleanup new blocks on 'CleanTombstones' faliure. ... Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	7 years ago
Goutham Veeramachaneni	f82a1fe4f2	Simplify stones counting. ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	7 years ago
Goutham Veeramachaneni	8a301b126a	Merge pull request #307 from mjtrangoni/fixes ... Fix some megacheck and unconvert issues	7 years ago
Mario Trangoni	c2182820ed	fix megacheck issue: should omit values from range	7 years ago
Simon Pasquier	79defa54df	Fix crash when a series has no block	7 years ago
Fabian Reinartz	d09283f60a	Add repair of bad index version	7 years ago
Goutham Veeramachaneni	2c3400ab4e	Add block ULID to index reader errors ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	7 years ago
Shubheksha Jalan	7b27dc2109	address review comments, fix index tests	7 years ago
Shubheksha Jalan	44052bc937	Make version for index format configurable	7 years ago
Shubheksha Jalan	129773b41a	merge BlockMeta and blockMeta struct, implement v1 and v2 index readers	7 years ago
Fabian Reinartz	1e55b7987f	Improve comments, handle allPostingsKey properly	7 years ago
Fabian Reinartz	67f0ca8f0e	Move index and chunk encoders to own packages	7 years ago
Goutham Veeramachaneni	3158b03e6c	Select series with label unset for != and !~ ... Fixes https://github.com/prometheus/prometheus/issues/3575 Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	7 years ago
Goutham Veeramachaneni	8a5ea9db74	Merge branch 'master' into tomb-clean ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	7 years ago
Goutham Veeramachaneni	24ff293dc9	Merge pull request #207 from Gouthamve/compact-fail ... Don't retry failed compactions.	7 years ago
Goutham Veeramachaneni	e7445d00b0	Add a function to cleanup tombstones. ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	7 years ago
Goutham Veeramachaneni	1fc94a02d1	Don't retry failed compactions. ... Fixes prometheus/prometheus#3487 Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	7 years ago
Fabian Reinartz	f1512a368a	Expose ChunkSeriesSet and lookups methods.	7 years ago
Fabian Reinartz	3ef4326114	Refactor tombstone reader types	7 years ago
Fabian Reinartz	e5ce2bef43	Add explicit error to Querier.Select ... This has been a frequent source of debugging pain since errors are potentially delayed to a much later point. They bubble up in an unrelated execution path.	7 years ago
Fabian Reinartz	d578b10d5c	chunk: make reader accept abstract ByteSlice	7 years ago
Fabian Reinartz	b7c3cfecbf	index: abstract ByteSlice and adjust indexReader ... This replaces the builtin byte slice with an interface for the index reader. This allows the complex decoding of the index file format to be used against more generalized implementations.	7 years ago
Sunny Klair	4fdf9b195c	Validate index TOC checksum on read	7 years ago
Fabian Reinartz	fb9da52b11	Add more verbose error handling for closing, reduce locking ... This commit introduces error returns in various places and is explicit about closing persisted blocks. {Index,Chunk,Tombstone}Readers are more consistent about their Close() method. Whenever a reader is retrieved, the corresponding close method must eventually be called. We use this to track pending readers against persisted blocks. Querier's against the DB no longer hold a read lock for their entire lifecycle. This avoids long running queriers to starve new ones when we have to acquire a write lock when reloading blocks.	7 years ago
Fabian Reinartz	78df406dac	Allocate and cache strings for persisted blocks ... This change loads the full symbol table when we open a persisted block and allocates a string for each. This ensures that strings retrieved through the index can be used after the block was closed. Before we backed the strings by the mmap'd byte regions which would segfault in this case. Also remove an inconsistency in the disk format and move both offset tables to the end (breaking change).	7 years ago
Goutham Veeramachaneni	8919baef03	Expose NewIndexReader() and cleanups ... Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	7 years ago
Fabian Reinartz	1ddedf2b30	Change series ID from uint32 to uint64	7 years ago
Fabian Reinartz	4cc37eecab	Refactor and add tests for compactor	7 years ago
Fabian Reinartz	5cf2662074	Refactor WAL into Head and misc improvements	7 years ago
Fabian Reinartz	3901b6e70b	Remove multiple heads ... This changes the structure to a single WAL backed by a single head block. Parts of the head block can be compacted. This relieves us from any head amangement and greatly simplifies any consistency and isolation concerns by just having a single head.	7 years ago