95 Commits (520b1d858f5b01a0b5cc6bbbaa2b5d43d7fbd7ca)

Author	SHA1	Message	Date
Vishnunarayan K I	7757fe6f21	Added ability to create db with NopWal (#519) ... Signed-off-by: Vishnunarayan K I <appukuttancr@gmail.com>	6 years ago
Krasi Georgiev	c3ffdf1a99	Test createBlock and check all os.RemoveAll in the tests for errors. (#549) ... Testing that createBlock creates blocks that can be opened. and checking the os.RemoveAll for errors will catch errors for un-closed files under windows. Many missing .Close() calls were added for fixing failing os.RemoveAll Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	ab8476dd06	Release 0.6.0 (#537)	6 years ago
Ganesh Vernekar	28c73f531f	Make vertical compaction and query merge optional ... Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Ganesh Vernekar	c59ed492b2	Vertical query merging and compaction (#370) ... * Vertical series iterator Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Select overlapped blocks first in compactor Plan() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Added vertical compaction Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Code cleanup and comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix tests Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Add benchmark for compaction Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Perform vertical compaction only when blocks are overlapping. Actions for vertical compaction: * Sorting chunk metas * Calling chunks.MergeOverlappingChunks on the chunks Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Benchmark for vertical compaction * BenchmarkNormalCompaction => BenchmarkCompaction * Moved the benchmark from db_test.go to compact_test.go Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Benchmark for query iterator and seek for non overlapping blocks Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Vertical query merge only for overlapping blocks Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Simplify logging in Compact(...) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Updated CHANGELOG.md Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Calculate overlapping inside populateBlock Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * MinTime and MaxTime for BlockReader. Using this to find overlapping blocks in populateBlock() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Sort blocks w.r.t. MinTime in reload() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Log about overlapping in LeveledCompactor.write() instead of returning bool Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Log about overlapping inside LeveledCompactor.populateBlock() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Refactor createBlock to take optional []Series Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * review1 Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com> * Updated CHANGELOG and minor nits Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * nits Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Updated CHANGELOG Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Refactor iterator and seek benchmarks for Querier. Also has as overlapping blocks. Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Additional test case Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * genSeries takes optional labels. Updated BenchmarkQueryIterator and BenchmarkQuerySeek. Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Split genSeries into genSeries and populateSeries Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Check error in benchmark Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Warn about overlapping blocks in reload() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Krasi Georgiev	07df4fd383	nits ... Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	e138c7ed7e	Merge remote-tracking branch 'upstream/master' into delete-compact-block-on-reload-error ... Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	0b72f9af4c	Merge pull request #270 from codesome/master ... Head: don't create stones, delete samples directly	6 years ago
Ganesh Vernekar	5481549324	Merge remote-tracking branch 'upstream/master' ... Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Simon Pasquier	95334f13c5	Merge branch 'master' into update-makefile-common ... Signed-off-by: Simon Pasquier <spasquie@redhat.com>	6 years ago
Krasi Georgiev	1603222bbc	small refactor of openTestDB to handle errors properly. ... Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	7245c6dc33	Merge remote-tracking branch 'upstream/master' into delete-compact-block-on-reload-error ... Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	3ec08eac50	use camelcase for rangeToTriggerCompaction ... Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
yeya24	6181d1f18f	fix two typos in db_test ... Signed-off-by: yeya24 <ben.ye@daocloud.io>	6 years ago
Krasi Georgiev	dac2b97dfd	make createBlock more generic so it can be used in other tests. (#489) ... Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Simon Pasquier	2e69508536	Merge branch 'master' into update-makefile-common ... Signed-off-by: Simon Pasquier <spasquie@redhat.com>	6 years ago
Ganesh Vernekar	1a9d08adc5	Don't write empty blocks (#374) ... * Dont write empty blocks when a compaction results in a block with no samples. Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
mknapphrt	ebf5d74325	Added storage size based retention method and new metrics (#343) ... Added methods needed to retain data based on a byte limitation rather than time. Limitation is only applied if the flag is set (defaults to 0). Both blocks that are older than the retention period and the blocks that make the size of the storage too large are removed. 2 new metrics for keeping track of the size of the local storage folder and the amount of times data has been deleted because the size restriction was exceeded. Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>	6 years ago
Ganesh Vernekar	d7e505db34	Dont store stones in head, delete samples directly ... Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Simon Pasquier	d5d7a097e1	Update Makefile.common ... This change also uses the latest staticcheck version which comes with new verifications, hence some clean up in the code. Signed-off-by: Simon Pasquier <spasquie@redhat.com>	6 years ago
Krasi Georgiev	48c439d26d	fix statick check errors (#475) ... fix the tests for `check_license` and `staticcheck` the static check also found some actual bugs. Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	eb6586f513	rename createPopulatedBlock to createBlock and use it instead ot createEmptyBlock (#488) ... Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	6d489a1004	fix TestWALSegmentSizeOption for windows. (#482) ... Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
glutamatt	22e3aeb107	Add WALSegmentSize as an option of tsdb creation (#450) ... Expose `WALSegmentSize` option to allow overriding the `DefaultOptions.WALSegmentSize`.	6 years ago
Krasi Georgiev	a2779cc901	fix flaky tests: TestDisableAutoCompactions,TestBlockRanges (#472) ... Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	bac9cbed2e	no overlapping on compaction when an existing block is not within default boundaries. (#461) ... closes https://github.com/prometheus/prometheus/issues/4643 Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Brian Brazil	d50b9a5619	Reload after reading the WAL. (#460) ... This causes the head to be GCed at startup, removing any series that were read from the WAL but have since been written to a block. In systems with low ingestion rates, this potentially could be many many hours of data. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	6 years ago
Ganesh Vernekar	7f30395115	LabelNames() for Querier (#455) ... * LabelNames() for Querier Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * nits Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Brian Brazil	41b54585d9	Use already open blocks while compacting. (#441) ... This roughly halves the RAM requirements of compaction. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	6 years ago
Krasi Georgiev	5a9ddeecef	fix lint errors (#439) ... unexported NewMemTombstones as this returns unexported memTombstones type which will not be shows in godoc. Added missing comments for exported methods. Removed unused RecordLogger,RecordReader interfaces. Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Ganesh Vernekar	3a08a71d86	LabelNames() method to get all unique label names (#369) ... * LabelNames() method to get all unique label names Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Krasi Georgiev	d804a27062	refactor util funcs to allow re-usage. (#419) ... * refactor util funcs to allow reusage. Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Ganesh Vernekar	61d0868966	Fix TestCorrectNumTombstones (#399) ... Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Ganesh Vernekar	6e712963e2	Fix updating of NumTombstones in block.Delete(..) (#385) ... Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Krasi Georgiev	d05611c027	removed some unused code and moved mockSeriesSet in querier_test (#394) ... Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	d38516b1c2	remove unused changes variable (#391) ... This was added in 55a9b5428a and later not used after some refactoring in following PRs. Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Chris Marchbanks	2db59a71a6	Fix assert order being backwards in a few places (#388) ... Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	6 years ago
Goutham Veeramachaneni	9c8ca47399	Fix filehandling for windows (#392) ... * Fix filehandling for windows Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Fix more windows filehandling issues Windows: Close files before deleting Checkpoints. Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> Windows: Close writers in case of errors so they can be deleted Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> Windows: Close block so that it can be deleted. Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> Windows: Close file to delete it Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> Windows: Close dir so that it can be deleted. Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> Windows: close files so that they can be deleted. Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> * Review feedback Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>	6 years ago
Fabian Reinartz	45071c657c	Properly initialize head time ... This fixes various issues when initializing the head time range under different starting conditions. Signed-off-by: Fabian Reinartz <freinartz@google.com>	6 years ago
Fabian Reinartz	7841d417b3	Ensure blocks are time-ordered in memory ... We assume in multiple places that the block list held by DB has blocks sequential by time. A regression caused us to hold them ordered by ULID, i.e. by creation time instead. Signed-off-by: Fabian Reinartz <freinartz@google.com>	6 years ago
codwu	667e539a7a	Merge branch 'master' of https://github.com/prometheus/tsdb into tsdb-delete	6 years ago
Benoît Knecht	24b223c161	db: add test for Querier returning too many blocks ... Due to the way blocks used to overlap by 1 millisecond (see #347), when requesting a 2-hour interval starting at `blocks[1].MinTime`, the `Querier` would consider three blocks: `blocks[0]`, `blocks[1]` and `blocks[2]`, because `blocks[0].MaxTime` and `blocks[2].MinTime` were in that interval. However, if the blocks don't overlap, only two blocks should be returned: `blocks[1]` and `blocks[2]`. This test ensures that it's indeed the case. Signed-off-by: Benoît Knecht <benoit.knecht@fsfe.org>	7 years ago
Benoît Knecht	4ed6b9ed72	db: add test for chunks that span beyond a block's boundaries ... Signed-off-by: Benoît Knecht <benoit.knecht@fsfe.org>	7 years ago
Fabian Reinartz	af9003dcef	Add resilience to crashes during deletion ... Signed-off-by: Fabian Reinartz <freinartz@google.com>	7 years ago
codwu	cd145c90d5	remove `put` function and use RLock in `Iter` function ... Signed-off-by: codwu <wuhan9087@163.com>	7 years ago
codwu	84a45cb79a	add rwmutex to prevent concurrent map read when delete series ... Signed-off-by: codwu <wuhan9087@163.com>	7 years ago
Krasi Georgiev	6094f35aa2	simplify if-else,test before the tombstone failure, more comments ... Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	7 years ago
Krasi Georgiev	f31a0d6457	add Test for Tombstone Cleaning after a failure ... Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	7 years ago
Fabian Reinartz	76c1b2cdb6	Ensure correct block order on reload ... Due to a regression blocks were no longer ordered by time before being stored in memory. This made data intermittently become unavailable for queries. Signed-off-by: Fabian Reinartz <freinartz@google.com>	7 years ago
Alexey Palazhchenko	64cce683cc	gofmt -s. ... Signed-off-by: Alexey Palazhchenko <alexey.palazhchenko@gmail.com>	7 years ago