* refactor block size calculation
The block size is kept in memory and not in the meta file anymore.
It now includes the size of the meta file itself for a more
correct block size.
It fixes a bug where the size didn't change when calling `block.Delete()`.
Adds a dedicated test to ensure correct block sizes.
This allows opening a db in a read only mode as it doesn't write to the meta file anymore.
Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>
In running Prometheus instances, compressing the records was shown to
reduce disk usage by half while incurring a negligible CPU cost.
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
* move the wal repair logic in db.Open
This is to allow opening a wal in a read oly mode without triggering a
repair.
Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>
since golang 1.12 no special handling is required for file.Sync()
@pborzenkov thanks for the pointer.
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
Testing that createBlock creates blocks that can be opened.
and checking the os.RemoveAll for errors will catch errors for un-closed files under windows.
Many missing .Close() calls were added for fixing failing os.RemoveAll
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
`if ds[i].meta.MinTime < t0 || ds[i].meta.MaxTime > t0+tr `, ds[i].meta.MinTime is always larger or equal to t0, so no need for this check.
`ulid.Parse` only checks if the length is 26. So changed to using `ulid.ParseStrict` to also check the validity of ulid.
a failed reload immediately after a compaction should delete the
resulting block to avoid creating blocks with the same time range.
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
Added methods needed to retain data based on a byte limitation rather than time. Limitation is only applied if the flag is set (defaults to 0). Both blocks that are older than the retention period and the blocks that make the size of the storage too large are removed.
2 new metrics for keeping track of the size of the local storage folder and the amount of times data has been deleted because the size restriction was exceeded.
Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>
The WALFlushInterval is not used anywhere in the code base.
The WAL is not an interface anymore to save some lookup time so can't use NopWAL in the tests. Instead can just pass nil as the code checks for that and it is essentially a noop.
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
This causes the head to be GCed at startup,
removing any series that were read from the WAL
but have since been written to a block. In
systems with low ingestion rates, this potentially
could be many many hours of data.
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
for tests we need to control when a compaction happens so with this
change automated compaction can be disabled, but allow to run it
manually it tests.
fixes failing tests in : https://github.com/prometheus/tsdb/pull/374
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
Limit the returned `db.Querier` to the requested time range. Preallocate the `baseChunkSeries.lset` and `baseChunkSeries.chks` slices to the previous series' slice sizes to avoid unnecessary grow slice reallocations.
This reverts commit 98fe30438c.
After some discussion, it was concluded that we want the full
`prometheus_tsdb_...` prefix hardcoded in the library.
Signed-off-by: beorn7 <beorn@soundcloud.com>
* Expose prometheus_tsdb_start_time_seconds metric
Signed-off-by: Bob Shannon <bshannon@palantir.com>
* Search for block with smallest minTime
Signed-off-by: Bob Shannon <bshannon@palantir.com>
* PR comments
Signed-off-by: Bob Shannon <bshannon@palantir.com>
* PR comment: Make metric name more accurate
Signed-off-by: Bob Shannon <bshannon@palantir.com>
We assume in multiple places that the block list held by DB
has blocks sequential by time.
A regression caused us to hold them ordered by ULID, i.e. by creation
time instead.
Signed-off-by: Fabian Reinartz <freinartz@google.com>
Blocks are half-open intervals [a, b), while all other intervals
(chunks, head, ...) are closed intervals [a, b].
Make that distinction explicit by defining `OverlapsClosedInterval()`
methods for blocks and chunks, and using them in place of the more
generic `intervalOverlap()` function.
This change also fixes `db.Querier()` and `db.Delete()`, which could
previously return one extraneous block at the end of the specified
interval.
Signed-off-by: Benoît Knecht <benoit.knecht@fsfe.org>
Block intervals are bound by `block.MinTime`, `block.MaxTime`, but they
define a half-open interval: `[block.MinTime, block.MaxTime).
However, when deciding if a chunk was part of a block or not, the
`intervalOverlap()` function would consider both the chunk and the block
intervals as being closed.
Rather than modify the login in `intervalOverlap()`, we explicitly
remove the last value from the interval when reading from head to
persist blocks.
Signed-off-by: Benoît Knecht <benoit.knecht@fsfe.org>
Yao Zengzeng