* Added CreateBlock and CreateHead functions to new file to make it reusable across packages.
Signed-off-by: Dipack P Panjabi <dipack.panjabi@gmail.com>
This PR gives the readonly DB the ability to create blocks from the WAL.
In order to implement this, we modify DBReadOnly.Blocks() to return an
empty slice and no error if no blocks are found.
xref: https://github.com/prometheus/tsdb/issues/346#issuecomment-520786524
Signed-off-by: Lucas Servén Marín <lserven@gmail.com>
* refactor block size calculation
The block size is kept in memory and not in the meta file anymore.
It now includes the size of the meta file itself for a more
correct block size.
It fixes a bug where the size didn't change when calling `block.Delete()`.
Adds a dedicated test to ensure correct block sizes.
This allows opening a db in a read only mode as it doesn't write to the meta file anymore.
Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>
In running Prometheus instances, compressing the records was shown to
reduce disk usage by half while incurring a negligible CPU cost.
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
* move the wal repair logic in db.Open
This is to allow opening a wal in a read oly mode without triggering a
repair.
Signed-off-by: Krasi Georgiev <8903888+krasi-georgiev@users.noreply.github.com>
since golang 1.12 no special handling is required for file.Sync()
@pborzenkov thanks for the pointer.
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
Testing that createBlock creates blocks that can be opened.
and checking the os.RemoveAll for errors will catch errors for un-closed files under windows.
Many missing .Close() calls were added for fixing failing os.RemoveAll
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
`if ds[i].meta.MinTime < t0 || ds[i].meta.MaxTime > t0+tr `, ds[i].meta.MinTime is always larger or equal to t0, so no need for this check.
`ulid.Parse` only checks if the length is 26. So changed to using `ulid.ParseStrict` to also check the validity of ulid.
a failed reload immediately after a compaction should delete the
resulting block to avoid creating blocks with the same time range.
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
Added methods needed to retain data based on a byte limitation rather than time. Limitation is only applied if the flag is set (defaults to 0). Both blocks that are older than the retention period and the blocks that make the size of the storage too large are removed.
2 new metrics for keeping track of the size of the local storage folder and the amount of times data has been deleted because the size restriction was exceeded.
Signed-off-by: Mark Knapp <mknapp@hudson-trading.com>
The WALFlushInterval is not used anywhere in the code base.
The WAL is not an interface anymore to save some lookup time so can't use NopWAL in the tests. Instead can just pass nil as the code checks for that and it is essentially a noop.
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
This causes the head to be GCed at startup,
removing any series that were read from the WAL
but have since been written to a block. In
systems with low ingestion rates, this potentially
could be many many hours of data.
Signed-off-by: Brian Brazil <brian.brazil@robustperception.io>
for tests we need to control when a compaction happens so with this
change automated compaction can be disabled, but allow to run it
manually it tests.
fixes failing tests in : https://github.com/prometheus/tsdb/pull/374
Signed-off-by: Krasi Georgiev <kgeorgie@redhat.com>
Limit the returned `db.Querier` to the requested time range. Preallocate the `baseChunkSeries.lset` and `baseChunkSeries.chks` slices to the previous series' slice sizes to avoid unnecessary grow slice reallocations.
This reverts commit 98fe30438c.
After some discussion, it was concluded that we want the full
`prometheus_tsdb_...` prefix hardcoded in the library.
Signed-off-by: beorn7 <beorn@soundcloud.com>
Dipack P Panjabi