prometheus

Commit Graph

Author	SHA1	Message	Date
beorn7	13fcf1ddbc	Implement double-delta encoded chunks.	10 years ago
Julius Volz	795704f0df	Merge pull request #565 from fabxc/fabxc/labelmatcher_test Tests for retrieving fingerprints for label matchers added.	10 years ago
Fabian Reinartz	4bff5d29bf	Add tests for retrieving fingerprints for label matchers. This checks for the basic behaviour of GetFingerprintsForLabelMatchers, that is, whether the different matcher types filter the correct fingerprints and intersections are correct.	10 years ago
beorn7	edd716e63c	Fix the embarrassing bug introduced in commit `0851945`. In that commit, the 'maintainSeries' call was accidentally removed. This commit refactors things a bit so that there is now a clean 'maintainMemorySeries' and a 'maintainArchivedSeries' call. Straighten the nomenclature a bit (consistently use 'drop' for chunks and 'purge' for series/metrics). Remove the annoying 'Completed maintenance sweep through archived fingerprints' message if there were no archived fingerprints to do maintenance on.	10 years ago
beorn7	af91fb8e31	Improve persisting chunks to disk. This is done by bucketing chunks by fingerprint. If the persisting to disk falls behind, more and more chunks are in the queue. As soon as there are "double hits", we will now persist both chunks in one go, doubling the disk throughput (assuming it is limited by disk seeks). Should even more pile up so that we end wit "triple hits", we will persist those first, and so on. Even if we have millions of time series, this will still help, assuming not all of them are growing with the same speed. Series that get many samples and/or are not very compressable will accumulate chunks faster, and they will soon get double- or triple-writes. To improve the chance of double writes, -storage.local.persistence-queue-capacity could be set to a higher value. However, that will slow down shutdown a lot (as the queue has to be worked through). So we leave it to the user to set it to a really high value. A more fundamental solution would be to checkpoint not only head chunks, but also chunks still in the persist queue. That would be quite complicated for a rather limited use-case (running many time series with high ingestion rate on slow spinning disks).	10 years ago
beorn7	5d3cd65a5d	Improve performance of ingestion. - Parallelize AppendSamples as much as possible without breaking the contract about temporal order. - Allocate more fingerprint locker slots. - Do not run early checkpoints if we are behind on chunk persistence. - Increase fpMinWaitDuration to give the disk more time for more important things. Also, switch math.MaxInt64 and math.MinInt64 to the new constants.	10 years ago
Bjoern Rabenstein	5859b74f1b	Clean up license issues. - Move CONTRIBUTORS.md to the more common AUTHORS. - Added the required NOTICE file. - Changed "Prometheus Team" to "The Prometheus Authors". - Reverted the erroneous changes to the Apache License.	10 years ago
Bjoern Rabenstein	622e8350cd	Fix a bug handling freshly unarchived series. Usually, if you unarchive a series, it is to add something to it, which will create a new head chunk. However, if a series in unarchived, and before anything is added to it, it is handled by the maintenance loop, it will be archived again. In that case, we have to load the chunkDescs to know the lastTime of the series to be archived. Usually, this case will happen only rarely (as a race, has never happened so far, possibly because the locking around unarchiving and the subsequent sample append is smart enough). However, during crash recovery, we sometimes treat series as "freshly unarchived" without directly appending a sample. We might add more cases of that type later, so better deal with archiving properly and load chunkDescs if required.	10 years ago
Bjoern Rabenstein	bb42cc2e2d	Evict based on memory pressure. Evict recently used chunks last. Change-Id: Ie6168f0cdb3917bdc63b6fe15585dd70c1e42afe	10 years ago
Bjoern Rabenstein	e23ee0f7cc	Fix race in test. Change-Id: I53e1a4c5a6b5f846acd76043166b6cb7bf7d5dc7	10 years ago
Bjoern Rabenstein	3f61d304ce	Reorganize maintenance loop. Change-Id: Iac10f988ba3e93ffb188f49c30f92e0b6adce5a3	10 years ago
Bjoern Rabenstein	7af42eda65	Optimize purging. Now only purge if there is something to purge. Also, set savedFirstTime and archived time range appropriately. (Which is needed for the optimization.) Change-Id: Idcd33319a84def3ce0318d886f10c6800369e7f9	10 years ago
Bjoern Rabenstein	7a9efc9c59	Fix typo in test. Change-Id: I3c2fd76bc5f50446c58f8ef693d9c6595197feaa	10 years ago
Bjoern Rabenstein	4efc60174b	Tweak and verify a few parameters. Remove TODOs accordingly. Change-Id: Ic062e13b6ae89a9135d3f14011114fe1cca1cef8	10 years ago
Bjoern Rabenstein	5f8e9617ef	Add more tests. Add an end-to-end fuzz and race test. Fix a race exposed by the above. Change-Id: Ifaa39a90cefbde8d4c29bda197cc92592ded21bb	10 years ago
Bjoern Rabenstein	d215e013b7	Fix the weird chunkDesc shuffling bug. The root cause was that after chunkDesc eviction, the offset between memory representation of chunk layout (via chunkDescs in memory) was shiftet against chunks as layed out on disk. Keeping the offset up to date is by no means trivial, so this commit is pretty involved. Also, found a race that for some reason didn't bite us so far: Persisting chunks was completel unlocked, so if chunks were purged on disk at the same time, disaster would strike. However, locking the persisting of chunk revealed interesting dead locks. Basically, never queue under the fp lock. Change-Id: I1ea9e4e71024cabbc1f9601b28e74db0c5c55db8	10 years ago
Julius Volz	c3fcea45e3	Support finer time resolutions than 1 second. Change-Id: I4c5f1d6d2361e841999b23283d1961b1bd0c2859	10 years ago
Bjoern Rabenstein	8fba3302bc	Bold changes to concurrency. (WIP. Probably doesn't work yet.) Change-Id: Id1537dfcca53831a1d428078a5863ece7bdf4875	10 years ago
Bjoern Rabenstein	71206dbc06	More code cleanups. Add license text everywhere. And others.... Change-Id: I11ccde267a2ef7eb366c4788ba7aeae14ba7545c	10 years ago
Bjoern Rabenstein	f5f9f3514a	Major code cleanup. - Make it go-vet and golint clean. - Add comments, TODOs, etc. Change-Id: If1392d96f3d5b4cdde597b10c8dff1769fcfabe2	10 years ago
Bjoern Rabenstein	1ca7f24137	Remove float diff tolerance altogether. Change-Id: I9ea9683a4665d5800fca75560bb4b8a8b4406d55	10 years ago
Bjoern Rabenstein	d742edfe0d	Fix precision loss. Large delta values often imply a difference between a large base value and the large delta value, potentially resulting in small numbers with a huge precision error. Since large delta values need 8 bytes anyway, we are not even saving memory. As a solution, always save the absoluto value rather than a delta once 8 bytes would be needed for the delta. Timestamps are then saved as 8 byte integers, while values are always saved as float64 in that case. Change-Id: I01100d600515e16df58ce508b50982ffd762cc49	10 years ago
Bjoern Rabenstein	dc2e463a97	Improvements after review. Change-Id: I484359282d4c7113518bbbb131f4f18383c08fdb	10 years ago
Bjoern Rabenstein	52c9dc43a3	Improve testing. In particular, create a fuzz test for time series. Change-Id: I523a17912405a0b6b46bd395c781d201dfe55036	10 years ago
Julius Volz	e7ed39c9a6	Initial experimental snapshot of next-gen storage. Change-Id: Ifb8709960dbedd1d9f5efd88cdd359ee9fa9d26d	10 years ago

25 Commits (a8d4f8af9a3b65ddabe0498f78b2f8c59701a775)