prometheus

Commit Graph

Author	SHA1	Message	Date
Sunny Klair	4fdf9b195c	Validate index TOC checksum on read	7 years ago
Fabian Reinartz	fb9da52b11	Add more verbose error handling for closing, reduce locking This commit introduces error returns in various places and is explicit about closing persisted blocks. {Index,Chunk,Tombstone}Readers are more consistent about their Close() method. Whenever a reader is retrieved, the corresponding close method must eventually be called. We use this to track pending readers against persisted blocks. Querier's against the DB no longer hold a read lock for their entire lifecycle. This avoids long running queriers to starve new ones when we have to acquire a write lock when reloading blocks.	7 years ago
Fabian Reinartz	78df406dac	Allocate and cache strings for persisted blocks This change loads the full symbol table when we open a persisted block and allocates a string for each. This ensures that strings retrieved through the index can be used after the block was closed. Before we backed the strings by the mmap'd byte regions which would segfault in this case. Also remove an inconsistency in the disk format and move both offset tables to the end (breaking change).	7 years ago
Goutham Veeramachaneni	8919baef03	Expose NewIndexReader() and cleanups Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	7 years ago
Fabian Reinartz	1ddedf2b30	Change series ID from uint32 to uint64	7 years ago
Fabian Reinartz	4cc37eecab	Refactor and add tests for compactor	7 years ago
Fabian Reinartz	5cf2662074	Refactor WAL into Head and misc improvements	7 years ago
Fabian Reinartz	3901b6e70b	Remove multiple heads This changes the structure to a single WAL backed by a single head block. Parts of the head block can be compacted. This relieves us from any head amangement and greatly simplifies any consistency and isolation concerns by just having a single head.	7 years ago
Goutham Veeramachaneni	7438ed7035	Expose Intervals type for use by TombstoneReader. TombstoneReader is exposed but Intervals is not. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	7 years ago
Fabian Reinartz	905af27cf9	Refactor compactor	7 years ago
Fabian Reinartz	66ff7b12e9	Pool Chunk objects during compaction	7 years ago
Fabian Reinartz	2644c8665c	Don't allocate ChunkMetas, reuse postings slices	7 years ago
Fabian Reinartz	96d7f540d4	Persist series without allocating the full set Change index persistence for series to not be accumulated in memory before being written as one large batch. `Labels` and `ChunkMeta` objects are reused. This cuts down memory spikes during compaction of multiple blocks significantly. As part of the the Index{Reader,Writer} now have an explicit notion of symbols and series must be inserted in order.	7 years ago
Fabian Reinartz	3410559c1b	Compact head block early Let older head blocks be compacted once the newest once has samples at 50% of its total range. This allows the memory of the compacted blocks to be released and garbage collected before a new head block gets created. Thereby the number of head blocks is 1 or 2 instead of 2 or 3 and memory spikes are reduced.	8 years ago
Fabian Reinartz	f006e2d1ab	Merge pull request #92 from prometheus/sources Add list of source head blocks to meta.json	8 years ago
Fabian Reinartz	683b4c8c2a	Add list of source head blocks to meta.json	8 years ago
Goutham Veeramachaneni	261cd9f393	Incorporate feedback. Move back to {Enable, Disable}Compactions. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	a110a64abd	Add full Snapshot support Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	a1c8425357	Initial implementation of HeadBlock Snapshots Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	29c73f05f2	Make sure that mint and maxt are not modified. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	44e9ae38b5	Incorporate PR feedback. * Expose Stone as it is used in an exported method. * Move from tombstoneReader to []Stone for the same reason as above. * Make WAL reading a little cleaner. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	6febabeb28	Final delete fixes. * Make sure no reads happen on the block when delete is in progress. * Fix bugs in compaction. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	f29fb62fba	Make TombstoneReader a Getter. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	31cf939448	Add NumTombstones to BlockMeta. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	3eb4119ab1	Make HeadBlock use WAL. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	244b73fce1	Rename for clarity and consistency. Misc. changes for code cleanliness. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	8434019ad9	Merge branch 'master' into deletes-1 Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	662d8173fe	Make Appends after Delete visible. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	0b70333ef6	Add tests for tombstones. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	3dbb400bef	Initial implementation of Delete on DB. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	45d3db4e9e	Use a *mapTombstoneReader instead of map We need to recalculate the sorted ref list everytime we make a Tombstones() call. This avoids that. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Fabian Reinartz	39df7e2bba	Switch blocks to ULID directories, drop sequenc numbers	8 years ago
Goutham Veeramachaneni	34a86af3c6	Move tombstones to their own thing. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	cea3c88f17	Add Tombstones() method to Block. Also add Seek() to TombstoneReader Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	3de55171d3	Add tests for tombstones and deletedIterator Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	4f1d857590	Implement Delete on HeadBlock Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	5579efbd5b	Initial implentation of Deletes on persistedBlock Very much a WIP Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Fabian Reinartz	5534e6c53c	Make HeadBlock impl public, make interface private	8 years ago
Fabian Reinartz	5b3cb11f3d	Merge pull request #65 from prometheus/comment-fixes Minor comment fixes and additions.	8 years ago
Fabian Reinartz	2c46815adb	Merge pull request #66 from prometheus/fix-meta-error-handling Fix closing of meta file if meta encoding failed	8 years ago
Julius Volz	e374db22bc	Remove unused skiplist types	8 years ago
Julius Volz	94f1ac599e	Fix closing of meta file if meta encoding failed	8 years ago
Julius Volz	8d1fb4fa01	Minor comment fixes and additions.	8 years ago
Fabian Reinartz	778103b450	Add liecence file and headers	8 years ago
Fabian Reinartz	3635569257	Trigger reload correctly on interrupted compaction	8 years ago
Fabian Reinartz	2c999836fb	Add Queryable interface to Block This adds the Queryable interface to the Block interface. Head and persisted blocks now implement their own Querier() method and thus isolate customization (e.g. remapPostings) more cleanly.	8 years ago
Fabian Reinartz	11be2cc585	Add composed Block interfaces, remove head generation This adds more lower-leve interfaces which are used to compose to different Block interfaces. The DB only uses interfaces instead of explicit persistedBlock and headBlock. The headBlock generation property is dropped as the use-case can be implemented using block sequence numbers.	8 years ago
Fabian Reinartz	2c3b56350a	Write meta.json file atomically	8 years ago
Fabian Reinartz	306831f151	Add per-block state ULID	8 years ago
Fabian Reinartz	9a5dfadb09	Tweak IndexWriter interface, serialize directly into dir	8 years ago

1 2

85 Commits (c354d6bd590f5857b563ec23ce986fcaa7cabadb)