prometheus

Commit Graph

Author	SHA1	Message	Date
Krasi Georgiev	8eeb70fee1	remove Fsync workaround for macos. (#574 ) since golang 1.12 no special handling is required for file.Sync() @pborzenkov thanks for the pointer. Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
zhulongcheng	aed16621c0	Add Head.compactable method (#542 ) * Add Head.compactable method Signed-off-by: zhulongcheng <zhulongcheng.me@gmail.com>	6 years ago
Vishnunarayan K I	7757fe6f21	Added ability to create db with NopWal (#519 ) Signed-off-by: Vishnunarayan K I <appukuttancr@gmail.com>	6 years ago
Krasi Georgiev	c3ffdf1a99	Test createBlock and check all os.RemoveAll in the tests for errors. (#549 ) Testing that createBlock creates blocks that can be opened. and checking the os.RemoveAll for errors will catch errors for un-closed files under windows. Many missing .Close() calls were added for fixing failing os.RemoveAll Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
naivewong	dd0d3c6f02	Several small fixes (#550 ) `if ds[i].meta.MinTime < t0 \|\| ds[i].meta.MaxTime > t0+tr `, ds[i].meta.MinTime is always larger or equal to t0, so no need for this check. `ulid.Parse` only checks if the length is 26. So changed to using `ulid.ParseStrict` to also check the validity of ulid.	6 years ago
zhulongcheng	8feb67e630	Update Appender.AddFast method comment (#551 ) Signed-off-by: zhulongcheng <zhulongcheng.me@gmail.com>	6 years ago
Krasi Georgiev	ab8476dd06	Release 0.6.0 (#537 )	6 years ago
Ganesh Vernekar	28c73f531f	Make vertical compaction and query merge optional Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Ganesh Vernekar	c59ed492b2	Vertical query merging and compaction (#370 ) * Vertical series iterator Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Select overlapped blocks first in compactor Plan() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Added vertical compaction Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Code cleanup and comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix tests Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Add benchmark for compaction Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Perform vertical compaction only when blocks are overlapping. Actions for vertical compaction: * Sorting chunk metas * Calling chunks.MergeOverlappingChunks on the chunks Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Benchmark for vertical compaction * BenchmarkNormalCompaction => BenchmarkCompaction * Moved the benchmark from db_test.go to compact_test.go Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Benchmark for query iterator and seek for non overlapping blocks Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Vertical query merge only for overlapping blocks Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Simplify logging in Compact(...) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Updated CHANGELOG.md Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Calculate overlapping inside populateBlock Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * MinTime and MaxTime for BlockReader. Using this to find overlapping blocks in populateBlock() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Sort blocks w.r.t. MinTime in reload() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Log about overlapping in LeveledCompactor.write() instead of returning bool Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Log about overlapping inside LeveledCompactor.populateBlock() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Refactor createBlock to take optional []Series Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * review1 Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com> * Updated CHANGELOG and minor nits Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * nits Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Updated CHANGELOG Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Refactor iterator and seek benchmarks for Querier. Also has as overlapping blocks. Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Additional test case Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * genSeries takes optional labels. Updated BenchmarkQueryIterator and BenchmarkQuerySeek. Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Split genSeries into genSeries and populateSeries Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Check error in benchmark Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Fix review comments Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * Warn about overlapping blocks in reload() Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Krasi Georgiev	beee5c58f3	Merge remote-tracking branch 'upstream/master' into shutdown-during-compaction Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	bf2239079d	refactor multi errors Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	c3a5c1d891	refactor error handling Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	45acaadd81	review changes Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	00d5e19baf	Merge remote-tracking branch 'upstream/master' into shutdown-during-compaction Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	6c34eb8b63	nits Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	315de4c782	fix windows tests Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	b521559c3b	do a proper cleanup for a failed reload after a compaction a failed reload immediately after a compaction should delete the resulting block to avoid creating blocks with the same time range. Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Brian Brazil	5db162568b	Remove _total from prometheus_tsdb_storage_blocks_bytes (#506 ) Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	6 years ago
Krasi Georgiev	9874377ead	nits Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	921f82cfc1	WIP Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	0b31b8af88	Merge remote-tracking branch 'upstream/master' into shutdown-during-compaction Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Ganesh Vernekar	1a9d08adc5	Don't write empty blocks (#374 ) * Dont write empty blocks when a compaction results in a block with no samples. Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
mknapphrt	ebf5d74325	Added storage size based retention method and new metrics (#343 ) Added methods needed to retain data based on a byte limitation rather than time. Limitation is only applied if the flag is set (defaults to 0). Both blocks that are older than the retention period and the blocks that make the size of the storage too large are removed. 2 new metrics for keeping track of the size of the local storage folder and the amount of times data has been deleted because the size restriction was exceeded. Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>	6 years ago
Krasi Georgiev	2e0571caba	remove unused WALFlushInterval option and NopWAL struct (#468 ) The WALFlushInterval is not used anywhere in the code base. The WAL is not an interface anymore to save some lookup time so can't use NopWAL in the tests. Instead can just pass nil as the code checks for that and it is essentially a noop. Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
glutamatt	22e3aeb107	Add WALSegmentSize as an option of tsdb creation (#450 ) Expose `WALSegmentSize` option to allow overriding the `DefaultOptions.WALSegmentSize`.	6 years ago
Krasi Georgiev	bd5ccee5c1	use context to cancel compactions Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	bac9cbed2e	no overlapping on compaction when an existing block is not within default boundaries. (#461 ) closes https://github.com/prometheus/prometheus/issues/4643 Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Krasi Georgiev	01e8296ee1	remove opaque metrics (#457 ) * more descriptive help text for the head metrics unit Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Brian Brazil	d50b9a5619	Reload after reading the WAL. (#460 ) This causes the head to be GCed at startup, removing any series that were read from the WAL but have since been written to a block. In systems with low ingestion rates, this potentially could be many many hours of data. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	6 years ago
Krasi Georgiev	7f00217d77	Allow manual compaction for tests when compaction is disabled globally. (#412 ) for tests we need to control when a compaction happens so with this change automated compaction can be disabled, but allow to run it manually it tests. fixes failing tests in : https://github.com/prometheus/tsdb/pull/374 Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
Ganesh Vernekar	7f30395115	LabelNames() for Querier (#455 ) * LabelNames() for Querier Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in> * nits Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Brian Brazil	41b54585d9	Use already open blocks while compacting. (#441 ) This roughly halves the RAM requirements of compaction. Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>	6 years ago
Alin Sinpalean	171fc4ab5d	Limit the returned db.Querier to the requested time range (#351 ) Limit the returned `db.Querier` to the requested time range. Preallocate the `baseChunkSeries.lset` and `baseChunkSeries.chks` slices to the previous series' slice sizes to avoid unnecessary grow slice reallocations.	6 years ago
Ganesh Vernekar	3a08a71d86	LabelNames() method to get all unique label names (#369 ) * LabelNames() method to get all unique label names Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Krasi Georgiev	d38516b1c2	remove unused changes variable (#391 ) This was added in `55a9b5428a` and later not used after some refactoring in following PRs. Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>	6 years ago
beorn7	3bc6c670fa	Revert "Remove `prometheus_` prefix from metrics" This reverts commit `98fe30438c`. After some discussion, it was concluded that we want the full `prometheus_tsdb_...` prefix hardcoded in the library. Signed-off-by: beorn7 <beorn@soundcloud.com>	6 years ago
Callum Styan	722f0ab920	break MigrateWAL into two functions, detection and migration (#371 ) Signed-off-by: Callum Styan <callumstyan@gmail.com>	6 years ago
beorn7	98fe30438c	Remove `prometheus_` prefix from metrics This can now be added by users of the library as needed with the new https://godoc.org/github.com/prometheus/client_golang/prometheus#WrapRegistererWithPrefix Signed-off-by: beorn7 <beorn@soundcloud.com>	6 years ago
Bob Shannon	cb7f320d42	Expose prometheus_tsdb_lowest_timestamp metric (#363 ) * Expose prometheus_tsdb_start_time_seconds metric Signed-off-by: Bob Shannon <bshannon@palantir.com> * Search for block with smallest minTime Signed-off-by: Bob Shannon <bshannon@palantir.com> * PR comments Signed-off-by: Bob Shannon <bshannon@palantir.com> * PR comment: Make metric name more accurate Signed-off-by: Bob Shannon <bshannon@palantir.com>	6 years ago
Ganesh Vernekar	2945db18ca	Changes in series names (and types) exposed (#376 ) Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Ganesh Vernekar	77d2c7cf3d	Added metric for symbol table size Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	6 years ago
Fabian Reinartz	1a5573b4ce	Migrate write ahead log On startup, rewrite the old write ahead log into the new format once. Signed-off-by: Fabian Reinartz <freinartz@google.com>	6 years ago
Fabian Reinartz	7841d417b3	Ensure blocks are time-ordered in memory We assume in multiple places that the block list held by DB has blocks sequential by time. A regression caused us to hold them ordered by ULID, i.e. by creation time instead. Signed-off-by: Fabian Reinartz <freinartz@google.com>	6 years ago
Fabian Reinartz	def912ce0e	Integrate new WAL and checkpoints Remove the old WAL and drop in the new one Signed-off-by: Fabian Reinartz <freinartz@google.com>	6 years ago
Benoît Knecht	1e1b2e163d	Make interval overlap comparisons more explicit Blocks are half-open intervals [a, b), while all other intervals (chunks, head, ...) are closed intervals [a, b]. Make that distinction explicit by defining `OverlapsClosedInterval()` methods for blocks and chunks, and using them in place of the more generic `intervalOverlap()` function. This change also fixes `db.Querier()` and `db.Delete()`, which could previously return one extraneous block at the end of the specified interval. Signed-off-by: Benoît Knecht <benoit.knecht@fsfe.org>	6 years ago
Benoît Knecht	0e4be5226a	db: block MaxTime should not be part of the block Block intervals are bound by `block.MinTime`, `block.MaxTime`, but they define a half-open interval: `[block.MinTime, block.MaxTime). However, when deciding if a chunk was part of a block or not, the `intervalOverlap()` function would consider both the chunk and the block intervals as being closed. Rather than modify the login in `intervalOverlap()`, we explicitly remove the last value from the interval when reading from head to persist blocks. Signed-off-by: Benoît Knecht <benoit.knecht@fsfe.org>	6 years ago
Fabian Reinartz	087c4c6d3a	Update doc comment Signed-off-by: Fabian Reinartz <freinartz@google.com>	7 years ago
Fabian Reinartz	d907928e57	Clarify docs, error on unexpected meta read errors Signed-off-by: Fabian Reinartz <freinartz@google.com>	7 years ago
Fabian Reinartz	af9003dcef	Add resilience to crashes during deletion Signed-off-by: Fabian Reinartz <freinartz@google.com>	7 years ago
Catalin Patulea	40766622ee	Update Appender.Add comment for uint64 refs. Follow-up to prometheus `0efecea6d4`. Signed-off-by: Catalin Patulea <catalinp@google.com>	7 years ago

1 2 3 4 5 ...

291 Commits (7a12d4588af017e495bcc6f5122a9e25e493dfb9)