prometheus

Commit Graph

Author	SHA1	Message	Date
Goutham Veeramachaneni	73cc5bae51	Colocate defer statements near relevant functions Signed-off-by: Goutham Veeramachaneni <goutham@boomerangcommerce.com>	8 years ago
Goutham Veeramachaneni	b51a05044e	Fix race condition for 2 appenders having same ts Race: Suppose we have 100 existing series inside a HeadBlock. Now we open two appenders in two routines A1, A2 and append 30 new series and 60 new series respectively with some common series. Both try to commit at the same time and the following happens in the given order: A2 executes createSeries() A1 executes createSeries() (with its common series referencing the ids from A2) A1 persists its newlabels, samples A2 persists its newlabels, samples Now when reading it back, we read A1's samples which reference A2's id and thereby fail. Ref: prometheus/promtheus#2795 Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Fabian Reinartz	05e411a8eb	Improve heuristic to spread chunks across block	8 years ago
Goutham Veeramachaneni	a110a64abd	Add full Snapshot support Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	a1c8425357	Initial implementation of HeadBlock Snapshots Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	29c73f05f2	Make sure that mint and maxt are not modified. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	44e9ae38b5	Incorporate PR feedback. * Expose Stone as it is used in an exported method. * Move from tombstoneReader to []Stone for the same reason as above. * Make WAL reading a little cleaner. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	6febabeb28	Final delete fixes. * Make sure no reads happen on the block when delete is in progress. * Fix bugs in compaction. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	c211ec4f49	Fix concurrent map access. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	f29fb62fba	Make TombstoneReader a Getter. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	9bf7aa9af1	Misc. fixes incorporating feedback. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	31cf939448	Add NumTombstones to BlockMeta. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	3eb4119ab1	Make HeadBlock use WAL. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	244b73fce1	Rename for clarity and consistency. Misc. changes for code cleanliness. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	8434019ad9	Merge branch 'master' into deletes-1 Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	662d8173fe	Make Appends after Delete visible. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	d6bd64357b	Fix Delete on HeadBlock Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	45d3db4e9e	Use a *mapTombstoneReader instead of map We need to recalculate the sorted ref list everytime we make a Tombstones() call. This avoids that. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Fabian Reinartz	39df7e2bba	Switch blocks to ULID directories, drop sequenc numbers	8 years ago
Fabian Reinartz	285bc07030	Switch append refs to string	8 years ago
Goutham Veeramachaneni	22c1b5b492	Make SeriesSets use tombstones. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	34a86af3c6	Move tombstones to their own thing. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	cea3c88f17	Add Tombstones() method to Block. Also add Seek() to TombstoneReader Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	4f1d857590	Implement Delete on HeadBlock Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	5579efbd5b	Initial implentation of Deletes on persistedBlock Very much a WIP Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Fabian Reinartz	8b51b7e2be	Make WAL for HeadBlock composeable.	8 years ago
Fabian Reinartz	4862b261d0	Abstract WAL into interface	8 years ago
Fabian Reinartz	535532ca02	Export refdSample The type was part of a exported method signatures and should therefore be exported as well.	8 years ago
Fabian Reinartz	5534e6c53c	Make HeadBlock impl public, make interface private	8 years ago
Goutham Veeramachaneni	2fa647f50b	Fix missing postings in Merge and Intersect (#77 ) * Test for a previous implematation of Intersect Before we were moving the postings list everytime we create a new chained `intersectPostings`. That was causing some postings to be skipped. This test fails on the older version. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Advance on Seek only when valid. Issue: Before in mergedPostings and others we advance everytime we `Seek`, which causes issues with `Intersect`. Take the case, where we have a mergedPostings = m merging, a: {10, 20, 30} and b: {15, 25, 35}. Everytime we `Seek`, we do a.Seek and b.Seek. Now if we Intersect m with {21, 22, 23, 30}, we would do Seek({21,22,23}) which would advance a and b beyond 30. Fix: Now we advance only when the seeking value is greater than the current value, as the definition specifies. Also, posting 0 will not be a valid posting and will be used to signal finished or un-initialized PostingsList. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Add test for Merge+Intersect edgecase. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in> * Add comments to trivial tests. Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Fabian Reinartz	291137781b	Merge branch 'panic-fix2' of https://github.com/Gouthamve/tsdb into Gouthamve-panic-fix2	8 years ago
Fabian Reinartz	09cd2021de	Merge pull request #75 from Gouthamve/head-gen E2E test for headBlock	8 years ago
Goutham Veeramachaneni	8096d11e4e	Add bounds check to headBlockAppender Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Goutham Veeramachaneni	adaf4d2099	Handle duplicate & out of order values in same txn Add docs about not erroring out on exact dupes. Moved tests to require.* Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Brian Brazil	72443bddfc	Remove dead code.	8 years ago
Brian Brazil	bceb5c1b16	When checking for amended points, do it in terms of bits. NaN != NaN, so the previous code would incorrectly report it as changed. There's also plans to take advantage of the NaN payload, so look at the entire value.	8 years ago
Fabian Reinartz	778103b450	Add liecence file and headers	8 years ago
Fabian Reinartz	c73a397da2	Adjust maximum samples per chunk.	8 years ago
Goutham Veeramachaneni	a51b2666d7	Fix Panic When Accessing Uncut memorySeries When calling AddFast, we check the details of the head chunk of the referred memorySeries. But it could happen that there are no chunks in the series at all. Currently, we are deferring chunk creation to when we actually append samples, but we can be sure that there will be samples if the series is created. We will be consuming no extra memory by cutting a chunk when we create the series. Ref: #28 comment 2 Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>	8 years ago
Fabian Reinartz	10c7c9acbe	Adjust import names to new repository organisation	8 years ago
Fabian Reinartz	87d48bf9de	Merge branch 'master' of github.com:fabxc/tsdb	8 years ago
Fabian Reinartz	a52980e0a8	Add workaround for deadlocks This adds a workaround to avoid deadlocks for inconsistent write lock order across headBlocks. Things keep working if transactions only append data for the same timestamp, which is generally the case for Prometheus. Full behavior should be restored in a subsequent change.	8 years ago
Goutham Veeramachaneni	61f866bb94	Add Sample Back The compilation and tests are broken as head.go requires sample which has been moved to another package while moving BufferedSeriesIterator. Duplication seemed better compared to exposing sample from tsdbutil.	8 years ago
Fabian Reinartz	3be4ef94ce	Move BufferedSeriesIterator in own package This functionality is useful for a lot of clients but not relevant to the TSDB's core features.	8 years ago
Fabian Reinartz	e478d0e3bc	Actually close olds blocks in reloadBlocks This fixes a bug leaking memory because blocks were not actually closed as the closing call references the initial, empty slice	8 years ago
Fabian Reinartz	789e8224ff	Fix wrong comparison in head block resorting	8 years ago
Fabian Reinartz	55ee4b5b3b	Merge branch 'master' of github.com:fabxc/tsdb	8 years ago
Fabian Reinartz	c18e055d7c	Fix races and add comments on remaining ones	8 years ago
Fabian Reinartz	e837034360	Merge pull request #14 from Gouthamve/log-update Update kit/log To New API	8 years ago
Fabian Reinartz	9c93f8f2aa	Fix various races This fixes different race condition encoutnered when running Prometheus. It reduces the overall performance in the synthetic benchmark a fair bit but has no indiciations of impacting a real-world setup notably.	8 years ago
Fabian Reinartz	3635569257	Trigger reload correctly on interrupted compaction	8 years ago
Fabian Reinartz	2c999836fb	Add Queryable interface to Block This adds the Queryable interface to the Block interface. Head and persisted blocks now implement their own Querier() method and thus isolate customization (e.g. remapPostings) more cleanly.	8 years ago
Fabian Reinartz	11be2cc585	Add composed Block interfaces, remove head generation This adds more lower-leve interfaces which are used to compose to different Block interfaces. The DB only uses interfaces instead of explicit persistedBlock and headBlock. The headBlock generation property is dropped as the use-case can be implemented using block sequence numbers.	8 years ago
Goutham Veeramachaneni	df7db4ac07	Update kit/log To New API NewContext has been removed couple of weeks back. Ref: https://github.com/go-kit/kit/releases/tag/v0.4.0	8 years ago
Fabian Reinartz	65b846ae5b	Remove unreturned locks, detect writes on closed heads	8 years ago
Fabian Reinartz	e0b33a7a28	Remove leftover signaling channels from headBlock	8 years ago
Fabian Reinartz	34efe4e2c8	Drop position mapper from head block The position mapper was intended to pre-computed "expensive" ordering of label sets. It was expensive to update and caused a lot of trouble. Skipping this optimization entirely actually revelead it was pointless and even harmful from the e2e perspective.	8 years ago
Fabian Reinartz	ad5812d83a	Handle WAL corruption by truncating This adds handling for various corruption scenarios of the WAL. If corruption is encountered, we truncate the WAL after the last valid entry transparently and continue appending after the offset.	8 years ago
Fabian Reinartz	a8e8903350	Use ChunkMeta references for clarity This has been a common source of hard to debug issues. Its a premature and unbenchmarked optimization and semantically, we want ChunkMetas to be references in all changed cases.	8 years ago
Fabian Reinartz	e825a0b40c	Make mapper updates asynchronous	8 years ago
Fabian Reinartz	cb4dde7659	Fix WAL log recovery bug This fixes a bug where the last WAL file was closed after consuming it instead of being left open for further writes. Reloading of blocks on startup considers loading head blocks now.	8 years ago
Fabian Reinartz	87805fb83f	Remove Partitioned* code	8 years ago
Fabian Reinartz	50791a412e	Remove leaky Appender abstraction for hashedAppend	8 years ago
Fabian Reinartz	55a9b5428a	Add separate head mutex Introduce a seperate mutex for the head blocks to avoid a race where a post-compaction reload may run between switching the DB's base mutex to create a new head block in an appender.	8 years ago
Fabian Reinartz	92120448c2	Properly cleanup compacted dirs, fixes, docs	8 years ago
Fabian Reinartz	2c3e778d90	Compactor interface	8 years ago
Fabian Reinartz	306831f151	Add per-block state ULID	8 years ago
Fabian Reinartz	78780cd2ba	Segment chunk file This adds write path support for segmented chunk data files. Files of 512MB are pre-allocated and written to. If the file size is exceeded, the next file is started. On completion, files are truncated to their final size.	8 years ago
Fabian Reinartz	a3d042b54e	Support multiple chunk files in read path	8 years ago
Fabian Reinartz	9c7a88223e	Add full encode/decode WAL cycle test	8 years ago
Fabian Reinartz	f1435f2e2c	Track appended samples properly in metric	8 years ago
Fabian Reinartz	012cf4ef25	Count writer references on head blocks	8 years ago
Fabian Reinartz	5a1c8eaa0e	Fix missing appends after reference lookups	8 years ago
Fabian Reinartz	ac5229e1b4	Correctly write empty blocks and extend appenders by new blocks	8 years ago
Fabian Reinartz	30efe4a58c	Support writing to multiple head blocks This is an initial (and hacky) first pass on allowing appending to multiple blocks simultaniously to avoid dropping samples right after cutting a new head block. It's also required for cases like the PGW, where a scrape may contain varying timestamps.	8 years ago
Fabian Reinartz	c20cc44b06	Add docs, write sequence number to meta.json	8 years ago
Fabian Reinartz	472c618c39	Drop out-of-bound samples	8 years ago
Fabian Reinartz	d4779b374c	Properly track and write meta file	8 years ago
Fabian Reinartz	9ddbd64d00	Move stats into meta.json file, cleanup, docs	8 years ago
Fabian Reinartz	5ceca3c810	Write to WAL before appending to memory storage	8 years ago
Fabian Reinartz	343dd9d94c	Fix wrong byte size in WAL base ref	8 years ago
Fabian Reinartz	5fb01d41aa	Use new Prometheus text format parser	8 years ago
Fabian Reinartz	dd0b69fe1b	Export ErrNotFound	8 years ago
Fabian Reinartz	1c80c33e72	Fix bug of unsorted postings lists being created The former approach created unordered postings list by either map iteration of new series being unsorted (fixable) or concurrent writers creating new series interleaved. We switch back to generating ephemeral references for a single batch. Newly created series have to be re-set upon the next insert.	8 years ago
Fabian Reinartz	c7f5590a71	Ensure order of postings when adding new series	8 years ago
Fabian Reinartz	d970f0256a	Add Rollback() and docs to Appender interface	8 years ago
Fabian Reinartz	22db9c3413	Remove old appendBatch methods	8 years ago
Fabian Reinartz	fde69dab49	Use buffer pool for head appenders	8 years ago
Fabian Reinartz	a317f252b9	Expose series references to clients This exposes a reference number of a series represented by a label set to clients. Subsequent samples can be directly added via the reference rather than repeatedly passing in the full labels. This drasitcally speeds up the append process. The appender chain uses different sections of the reference number for assignment to child appenders and invalidating reference numbers as necessary. Clients can either pass out reference numbers themselves or have their own optimized lookup, i.e. by directly associating unparsed metric descriptors strings with reference numbers.	8 years ago
Fabian Reinartz	5e028710d5	Add fast past to validation after lock switch	8 years ago
Fabian Reinartz	1b39887baa	Revalidate series existance after lock switch	8 years ago
Fabian Reinartz	ca5791efbc	Simplify creation of new series	8 years ago
Fabian Reinartz	0ca755b4ae	Replace single head chunk per series with memSeries This adds a memory series holding several chunk to replace the single head chunk per series so far. This is necessary for uniform maximum chunk sizes in cases where some series have higher frequency samples than others.	8 years ago
Fabian Reinartz	c32a94d409	Unexport HeadBlock, export Block interface	8 years ago
Fabian Reinartz	d86e8a63c7	Report correct number of appended samples	8 years ago
Fabian Reinartz	29883a18fc	Add own Appender() method for DB	8 years ago
Fabian Reinartz	4c4e0c614e	Simplify position mapper updating	8 years ago
Fabian Reinartz	0dffd52238	Use page writer in compaction	8 years ago
Fabian Reinartz	89d8467f5c	Add missing lock	8 years ago
Fabian Reinartz	8c31c6e934	Make concurrent head chunk reads safe, fix misc races This adds a 4 sample buffer to every head chunk. The XOR compression scheme may edit bytes in place. The minimum size of a sample is 2 bits. So keeping the last 4 samples in an in-memory buffer makes it safe to query the preceeding ones while samples are added	8 years ago
Fabian Reinartz	1943f8d1bb	Fix head block stats races	8 years ago
Fabian Reinartz	6aa922c5a6	Fix races	8 years ago
Fabian Reinartz	300f4e2abf	Use separate lock for series creation This uses the head block's own lock to only lock if new series were encountered. In the general append case we just need to hold a	8 years ago
Fabian Reinartz	63e12807da	Don't update head postings mapper on every append	8 years ago
Fabian Reinartz	71efd2e08d	Periodically fsync WAL, make head cut async	8 years ago
Fabian Reinartz	96c2bd249f	Handle compaction trigger and reinitializing in DB	8 years ago
Fabian Reinartz	937cdb579c	Switch to sequential block names This changes block directory names from the int64 timestamp to sequential numbering.	8 years ago
Fabian Reinartz	9790aa98ac	Add postings wrapper that emits head postings in label set order This adds a position mapper that takes series from a head block in the order they were appended and creates a mapping representing them in order of their label sets. Write-repair of the postings list would cause very expensive writing. Hence, we keep them as they are and only apply the postition mapping at the very end, after a postings list has been sufficienctly reduced through intersections etc.	8 years ago
Fabian Reinartz	5aa7f7cce8	Compact head block into persisted block	8 years ago
Fabian Reinartz	3f72d5d027	Fix last timestamp initialization This initializes the chunkDesc's last timestamp to the minimum value so initial samples with a timestamp of 0 (e.g. in tests) are not accidentally dropped.	8 years ago
Fabian Reinartz	ac49f8c15e	Consolidate persistence and compaction	8 years ago
Fabian Reinartz	e7f04d14d5	Lock mmapped files	8 years ago
Fabian Reinartz	91b65b55e7	Run persistence in separate goroutine	8 years ago
Fabian Reinartz	a648ef5252	Convert persister into function	8 years ago
Fabian Reinartz	dbd2b21d2e	Make persistence atomic	8 years ago
Fabian Reinartz	beb842a856	Change block interface This changes the block interface to directly expose index and series readers rather than the direct querier	8 years ago
Fabian Reinartz	5d7ec06e04	Use ChunkMeta in SeriesWriter	8 years ago
Fabian Reinartz	c00d17e691	Modify IndexReader API to accomodate compaction This changes the IndexReader API to expose plain labels and chunk meta information instead of a Series interface. Dropping of irrelevant chunks is moved into the querier. A LabelIndices method is added to query for existing label value indices.	8 years ago
Fabian Reinartz	1e1a37b15b	Remove double-reference in chunk hashmap	8 years ago
Fabian Reinartz	675f0886f0	Append to chunks cannot error	8 years ago
Fabian Reinartz	7280533c42	Add basic shard metrics	8 years ago
Fabian Reinartz	201d7687b6	Fix uniqueness of new series	8 years ago
Fabian Reinartz	787199a88e	Fix erroneous value assignments	8 years ago
Fabian Reinartz	8aba95048a	Rename to OpenHeadBlock	8 years ago
Fabian Reinartz	1dde3b6d31	Add WAL decoder+loading and benchmarks	8 years ago
Fabian Reinartz	0b8c77361e	Add initial WAL writing	8 years ago
Fabian Reinartz	2a825f6c28	Consolidate mem index into HeadBlock	8 years ago
Fabian Reinartz	ede733ab6c	Extract labels package	8 years ago
Fabian Reinartz	1b23d62e3f	Properly close files before reopening	8 years ago
Fabian Reinartz	282d9ae6e2	Implement label value queries in all layers.	8 years ago
Fabian Reinartz	bad93d8d57	Extract head serialization into Head method	8 years ago
Fabian Reinartz	b08f82fa4e	Pre-select relevant chunks on series access. This adds interval metadata to indexed chunks. The queried interval is used to filter chunks when queried from the index to save unnecessary accesses of the chunks file. This is especially relevant for series that come and go often and larger files.	8 years ago
Fabian Reinartz	bd77103a49	Add stats serialization, load querier of all blocks	8 years ago
Fabian Reinartz	d56b281006	Rename Iterator to Postings	8 years ago
Fabian Reinartz	ca89080128	Misc fixes for initial Prometheus integration	8 years ago
Fabian Reinartz	14dbc59f2b	cleanup and switching removal of unsafe calls.	8 years ago
Fabian Reinartz	40a451694f	Refactor persistence into interfaces	8 years ago
Fabian Reinartz	74f8dfd95d	Persist blocks periodically	8 years ago
Fabian Reinartz	0cf8bb9e53	Move sub-indexes into single index structure	8 years ago
Fabian Reinartz	8aa99a3ebd	misc	8 years ago
Fabian Reinartz	2c34a15fe6	Add initial seriailization of block data	8 years ago
Fabian Reinartz	3ef7da33c8	Restructure files	8 years ago

1 2 3 4 5

242 Commits (6d489a1004dc3f7fa1ebffb66f6cebc2a0278fe7)