230 Commits (cc306ef0d521c1d7e957aebd7418b903d55ebc6b)

Author	SHA1	Message	Date
Fabian Reinartz	70909ca8ad	Ensure GC runs after each compactor call ... GC is triggered rarely, which may cause unnecessarily high memory spikes when running several compaction cycles in a row. Explicitly run GC so we don't have idle bytes marked as used from the previous cycle.	8 years ago
Fabian Reinartz	55ee4b5b3b	Merge branch 'master' of github.com:fabxc/tsdb	8 years ago
Fabian Reinartz	c18e055d7c	Fix races and add comments on remaining ones	8 years ago
Fabian Reinartz	e837034360	Merge pull request #14 from Gouthamve/log-update ... Update kit/log To New API	8 years ago
Fabian Reinartz	9c93f8f2aa	Fix various races ... This fixes different race condition encoutnered when running Prometheus. It reduces the overall performance in the synthetic benchmark a fair bit but has no indiciations of impacting a real-world setup notably.	8 years ago
Fabian Reinartz	3635569257	Trigger reload correctly on interrupted compaction	8 years ago
Fabian Reinartz	2c999836fb	Add Queryable interface to Block ... This adds the Queryable interface to the Block interface. Head and persisted blocks now implement their own Querier() method and thus isolate customization (e.g. remapPostings) more cleanly.	8 years ago
Fabian Reinartz	11be2cc585	Add composed Block interfaces, remove head generation ... This adds more lower-leve interfaces which are used to compose to different Block interfaces. The DB only uses interfaces instead of explicit persistedBlock and headBlock. The headBlock generation property is dropped as the use-case can be implemented using block sequence numbers.	8 years ago
Goutham Veeramachaneni	df7db4ac07	Update kit/log To New API ... NewContext has been removed couple of weeks back. Ref: https://github.com/go-kit/kit/releases/tag/v0.4.0	8 years ago
Fabian Reinartz	303a4ec3bc	Sync directory on retention delation	8 years ago
Fabian Reinartz	863d38dfee	Fix unreturned lock	8 years ago
Fabian Reinartz	d0770302ed	Add retention deletion	8 years ago
Fabian Reinartz	4962175218	Fix deadlock between heads and headmtx ... With hundreds of concurrent appenders the locking order between the headBlocks on instantiating appenders and write locking the headmtx is hard to impossible to get consistent. Just never instantiate appenders while holding the headmtx lock in any way.	8 years ago
Fabian Reinartz	65b846ae5b	Remove unreturned locks, detect writes on closed heads	8 years ago
Fabian Reinartz	32c32013a6	Fix loading of persisted blocks	8 years ago
Fabian Reinartz	cb4dde7659	Fix WAL log recovery bug ... This fixes a bug where the last WAL file was closed after consuming it instead of being left open for further writes. Reloading of blocks on startup considers loading head blocks now.	8 years ago
Fabian Reinartz	3910b913bd	Revert interval overlap comparison changes	8 years ago
Fabian Reinartz	87805fb83f	Remove Partitioned* code	8 years ago
Fabian Reinartz	eedbebe1d7	Misc cleanups	8 years ago
Fabian Reinartz	50791a412e	Remove leaky Appender abstraction for hashedAppend	8 years ago
Fabian Reinartz	87d270fe11	Fix interval check, make DB shutdown concurrent	8 years ago
Fabian Reinartz	04783831e2	Correctly close compacted blocks.	8 years ago
Fabian Reinartz	55a9b5428a	Add separate head mutex ... Introduce a seperate mutex for the head blocks to avoid a race where a post-compaction reload may run between switching the DB's base mutex to create a new head block in an appender.	8 years ago
Fabian Reinartz	3e569bc964	Optimize memory allocations in writer	8 years ago
Fabian Reinartz	ffb24a98f4	Add missing unlock, run debug endpoint in benchmark	8 years ago
Fabian Reinartz	92120448c2	Properly cleanup compacted dirs, fixes, docs	8 years ago
Fabian Reinartz	2c3e778d90	Compactor interface	8 years ago
Fabian Reinartz	2c3b56350a	Write meta.json file atomically	8 years ago
Fabian Reinartz	6c4217276f	Periodically trigger compaction ... This addresses an issue where the compaction triggered on cutting a new block doesn't find anything as the writers are still active on the block that should be ready for compaction.	8 years ago
Fabian Reinartz	db5c88ea9a	Misc compaction fixes	8 years ago
Fabian Reinartz	b281e4e39b	Accept prometheus.Registerer in constructor	8 years ago
Fabian Reinartz	306831f151	Add per-block state ULID	8 years ago
Fabian Reinartz	78780cd2ba	Segment chunk file ... This adds write path support for segmented chunk data files. Files of 512MB are pre-allocated and written to. If the file size is exceeded, the next file is started. On completion, files are truncated to their final size.	8 years ago
Fabian Reinartz	a3b47c4929	Create default logger for DB	8 years ago
Fabian Reinartz	f734773214	Replace per-file locking with single PID lock file ... File locks have a multitude of problems that make them hard to use correctly. As they are just advisory, they are only meaningful to prevent accidents like running the same process twice. A simple PID file lock works reliably in those cases and is simpler.	8 years ago
Fabian Reinartz	79944a5912	Break out WAL into segment files	8 years ago
Fabian Reinartz	987a90d149	Don't delete files if retention duration is 0	8 years ago
Fabian Reinartz	9c76624df2	Add initial retention cutoff	8 years ago
Fabian Reinartz	f1435f2e2c	Track appended samples properly in metric	8 years ago
Fabian Reinartz	012cf4ef25	Count writer references on head blocks	8 years ago
Fabian Reinartz	5a1c8eaa0e	Fix missing appends after reference lookups	8 years ago
Fabian Reinartz	f556036037	Close blocks before overwriting with compacted one ... This prevents the meta.json file of a compacted block to be overwritten with data of the old block.	8 years ago
Fabian Reinartz	ac5229e1b4	Correctly write empty blocks and extend appenders by new blocks	8 years ago
Fabian Reinartz	42fa342229	Improve multi-head handling ... This improves handling of multiple head blocks. Configuration is simplified to specify the number of concurrently appendable blocks.	8 years ago
Fabian Reinartz	30efe4a58c	Support writing to multiple head blocks ... This is an initial (and hacky) first pass on allowing appending to multiple blocks simultaniously to avoid dropping samples right after cutting a new head block. It's also required for cases like the PGW, where a scrape may contain varying timestamps.	8 years ago
Fabian Reinartz	a61a31a5d7	compaction: add fast-path for compacting mem blocks	8 years ago
Fabian Reinartz	c20cc44b06	Add docs, write sequence number to meta.json	8 years ago
Fabian Reinartz	30d8866c13	Limit compaction range, make cut and compact concurrent	8 years ago
Fabian Reinartz	67d185ceb9	Compact based on compaction generation	8 years ago
Fabian Reinartz	d4779b374c	Properly track and write meta file	8 years ago
Fabian Reinartz	9ddbd64d00	Move stats into meta.json file, cleanup, docs	8 years ago
Fabian Reinartz	e006bc6dc6	Improve error messages, create regular dir for block	8 years ago
Fabian Reinartz	d2322f6095	Improve compaction processing	8 years ago
Fabian Reinartz	343dd9d94c	Fix wrong byte size in WAL base ref	8 years ago
Fabian Reinartz	dd0b69fe1b	Export ErrNotFound	8 years ago
Fabian Reinartz	1c80c33e72	Fix bug of unsorted postings lists being created ... The former approach created unordered postings list by either map iteration of new series being unsorted (fixable) or concurrent writers creating new series interleaved. We switch back to generating ephemeral references for a single batch. Newly created series have to be re-set upon the next insert.	8 years ago
Fabian Reinartz	c7f5590a71	Ensure order of postings when adding new series	8 years ago
Fabian Reinartz	d970f0256a	Add Rollback() and docs to Appender interface	8 years ago
Fabian Reinartz	22db9c3413	Remove old appendBatch methods	8 years ago
Fabian Reinartz	a317f252b9	Expose series references to clients ... This exposes a reference number of a series represented by a label set to clients. Subsequent samples can be directly added via the reference rather than repeatedly passing in the full labels. This drasitcally speeds up the append process. The appender chain uses different sections of the reference number for assignment to child appenders and invalidating reference numbers as necessary. Clients can either pass out reference numbers themselves or have their own optimized lookup, i.e. by directly associating unparsed metric descriptors strings with reference numbers.	8 years ago
Fabian Reinartz	c32a94d409	Unexport HeadBlock, export Block interface	8 years ago
Fabian Reinartz	29883a18fc	Add own Appender() method for DB	8 years ago
Fabian Reinartz	142c89b8b0	Fix/update metrics	8 years ago
Fabian Reinartz	8c31c6e934	Make concurrent head chunk reads safe, fix misc races ... This adds a 4 sample buffer to every head chunk. The XOR compression scheme may edit bytes in place. The minimum size of a sample is 2 bits. So keeping the last 4 samples in an in-memory buffer makes it safe to query the preceeding ones while samples are added	8 years ago
Fabian Reinartz	1943f8d1bb	Fix head block stats races	8 years ago
Fabian Reinartz	300f4e2abf	Use separate lock for series creation ... This uses the head block's own lock to only lock if new series were encountered. In the general append case we just need to hold a	8 years ago
Fabian Reinartz	63e12807da	Don't update head postings mapper on every append	8 years ago
Fabian Reinartz	71efd2e08d	Periodically fsync WAL, make head cut async	8 years ago
Fabian Reinartz	2eb544c98e	Change file names and maker parsing safer	8 years ago
Fabian Reinartz	96c2bd249f	Handle compaction trigger and reinitializing in DB	8 years ago
Fabian Reinartz	3ed2c2a14b	Rename Partition to regular DB, DB to PartitionedDB	8 years ago
Fabian Reinartz	937cdb579c	Switch to sequential block names ... This changes block directory names from the int64 timestamp to sequential numbering.	8 years ago
Fabian Reinartz	4590b61343	Rename shard to partition	8 years ago
Fabian Reinartz	5aa7f7cce8	Compact head block into persisted block	8 years ago
Fabian Reinartz	3f72d5d027	Fix last timestamp initialization ... This initializes the chunkDesc's last timestamp to the minimum value so initial samples with a timestamp of 0 (e.g. in tests) are not accidentally dropped.	8 years ago
Fabian Reinartz	ec99f99d3d	Fix and test bug in shardSeriesSet	8 years ago
Fabian Reinartz	ac49f8c15e	Consolidate persistence and compaction	8 years ago
Fabian Reinartz	91b65b55e7	Run persistence in separate goroutine	8 years ago
Fabian Reinartz	a648ef5252	Convert persister into function	8 years ago
Fabian Reinartz	dbd2b21d2e	Make persistence atomic	8 years ago
Fabian Reinartz	62b8ded0a5	Rename iterator value getters to At()	8 years ago
Fabian Reinartz	5d75a3dc7b	Add basic compaction ... This adds a basic compactor that will merge two persisted blocks into one. It simply fully rewrites the index and concatenates the chunk lists. It just writes into the current working dir and doesn't properly handle which blocks to compact for now.	8 years ago
Fabian Reinartz	c00d17e691	Modify IndexReader API to accomodate compaction ... This changes the IndexReader API to expose plain labels and chunk meta information instead of a Series interface. Dropping of irrelevant chunks is moved into the querier. A LabelIndices method is added to query for existing label value indices.	8 years ago
Fabian Reinartz	1e1a37b15b	Remove double-reference in chunk hashmap	8 years ago
Fabian Reinartz	675f0886f0	Append to chunks cannot error	8 years ago
Fabian Reinartz	7280533c42	Add basic shard metrics	8 years ago
Fabian Reinartz	a009247ab7	Add matching of empty label	8 years ago
Fabian Reinartz	eb4f366988	Add return error to Appender.Add	8 years ago
Fabian Reinartz	3a5468f251	Lock shards during querying and shutdown	8 years ago
Fabian Reinartz	ab7fbc05ad	Zero timestamp as base, use binary search list postings	8 years ago
Fabian Reinartz	f8111cef0e	Fix chunk series iterator seeking	8 years ago
Fabian Reinartz	787199a88e	Fix erroneous value assignments	8 years ago
Fabian Reinartz	8aba95048a	Rename to OpenHeadBlock	8 years ago
Fabian Reinartz	9c6a72aadd	Load head with WALs correctly	8 years ago
Fabian Reinartz	1dde3b6d31	Add WAL decoder+loading and benchmarks	8 years ago
Fabian Reinartz	0b8c77361e	Add initial WAL writing	8 years ago
Fabian Reinartz	2a825f6c28	Consolidate mem index into HeadBlock	8 years ago
Fabian Reinartz	da2beb3e6d	Fix zero division, add buffer series iterator	8 years ago
Fabian Reinartz	ede733ab6c	Extract labels package	8 years ago
Fabian Reinartz	ee217adc7e	Redfine append interface, remove old Prometheus storage from bench	8 years ago
Fabian Reinartz	d9ca4b47f5	Fix offset errors, fix persisted postings order	8 years ago
Fabian Reinartz	1b23d62e3f	Properly close files before reopening	8 years ago
Fabian Reinartz	00a503129b	Use contextualized and traced errors in reader	8 years ago
Fabian Reinartz	bad93d8d57	Extract head serialization into Head method	8 years ago
Fabian Reinartz	b08f82fa4e	Pre-select relevant chunks on series access. ... This adds interval metadata to indexed chunks. The queried interval is used to filter chunks when queried from the index to save unnecessary accesses of the chunks file. This is especially relevant for series that come and go often and larger files.	8 years ago
Fabian Reinartz	bd77103a49	Add stats serialization, load querier of all blocks	8 years ago
Fabian Reinartz	b2f1db5666	Add unsafe string and slice conversions	8 years ago
Fabian Reinartz	5424a0cf75	Rename SeriesShard to Shard	8 years ago
Fabian Reinartz	9873e18b75	Add loading of persisted blocks	8 years ago
Fabian Reinartz	ca89080128	Misc fixes for initial Prometheus integration	8 years ago
Fabian Reinartz	fc992fafc2	Change querier interface, initial implementations	8 years ago
Fabian Reinartz	6eeb0ef01c	Add queriers and partial mocks	8 years ago
Fabian Reinartz	5e02e28f9c	Add proper mmap calls	8 years ago
Fabian Reinartz	14dbc59f2b	cleanup and switching removal of unsafe calls.	8 years ago
Fabian Reinartz	3a528c3078	Write plain postings list index	8 years ago
Fabian Reinartz	8cbc95c316	Write label value indices	8 years ago
Fabian Reinartz	1e0edf367b	Write index with symbol table	8 years ago
Fabian Reinartz	40a451694f	Refactor persistence into interfaces	8 years ago
Fabian Reinartz	62f9dc311c	misc	8 years ago
Fabian Reinartz	74f8dfd95d	Persist blocks periodically	8 years ago
Fabian Reinartz	0cf8bb9e53	Move sub-indexes into single index structure	8 years ago
Fabian Reinartz	8aa99a3ebd	misc	8 years ago
Fabian Reinartz	2c34a15fe6	Add initial seriailization of block data	8 years ago
Fabian Reinartz	3ef7da33c8	Restructure files	8 years ago
Fabian Reinartz	b845f8d3a1	Reduce test data allocations	8 years ago
Fabian Reinartz	52276c6966	Bucket samples before appending. ... This pre-sorts samples into buckets before appending them to reduce locking of shards.	8 years ago
Fabian Reinartz	9b459458d0	Docs and interface definitions	8 years ago
Fabian Reinartz	83574b1565	Add new interfaces and skeleton	8 years ago
Fabian Reinartz	6f93a699e6	wipe	8 years ago
Fabian Reinartz	f9f11c4a08	Add main db implementation	8 years ago