12 Commits (9c953064c312fbcf008745927b543df37eb81e4b)

Author	SHA1	Message	Date
beorn7	7199a9d9d4	storage: Guard against appending to evicted chunk ... Fixes #2480. For certain definition of "fixes". This is something that should never happen. Sadly, it does happen, albeit extremely rarely. This could be some weird cornercase we haven't covered yet. Or it happens as a consequesnce of data corruption or a crash recovery gone bad. This is not a "real" fix as we don't know the root cause of the incident reported in #2480. However, this makes sure the server does not crash, but deals gracefully with the problem: The series in question is quarantined, which even makes it available for forensics.	8 years ago
beorn7	434ab2a6a3	storage: Evict chunks and calculate persistence pressure based on target heap size ... This is a fairly easy attempt to dynamically evict chunks based on the heap size. A target heap size has to be set as a command line flage, so that users can essentially say "utilize 4GiB of RAM, and please don't OOM". The -storage.local.max-chunks-to-persist and -storage.local.memory-chunks flags are deprecated by this change. Backwards compatibility is provided by ignoring -storage.local.max-chunks-to-persist and use -storage.local.memory-chunks to set the new -storage.local.target-heap-size to a reasonable (and conservative) value (both with a warning). This also makes the metrics intstrumentation more consistent (in naming and implementation) and cleans up a few quirks in the tests. Answers to anticipated comments: There is a chance that Go 1.9 will allow programs better control over the Go memory management. I don't expect those changes to be in contradiction with the approach here, but I do expect them to complement them and allow them to be more precise and controlled. In any case, once those Go changes are available, this code has to be revisted. One might be tempted to let the user specify an estimated value for the RSS usage, and then internall set a target heap size of a certain fraction of that. (In my experience, 2/3 is a fairly safe bet.) However, investigations have shown that RSS size and its relation to the heap size is really really complicated. It depends on so many factors that I wouldn't even start listing them in a commit description. It depends on many circumstances and not at least on the risk trade-off of each individual user between RAM utilization and probability of OOMing during a RAM usage peak. To not add even more to the confusion, we need to stick to the well-defined number we also use in the targeting here, the sum of the sizes of heap objects.	8 years ago
beorn7	4719482f5f	storage: Make tests go-vet and golint clean	8 years ago
beorn7	485ac8dff7	storage: Verify validity of byte length when unmarshalling (double)delta chunks ... This makes sure a division-by-zero crash cannot happen in the Len() method. Fixes #2773	8 years ago
Tom Wilkie	585878cdb2	Add call to estimate number of samples in a chunk to the API	8 years ago
Dominik Schulz	182e17958a	Trivial spelling corrections and a small comment.	8 years ago
beorn7	719508752b	Re-add counting of evict chunk ops and decrementing NumMemChunks ... Also, modify test to expose the regression.	8 years ago
Julius Volz	cb02f017ee	Clean up some doc comments	8 years ago
Julius Volz	c212ef0326	Add Chunk.Utilization() methods ... When using the chunking code in other projects (both Weave Prism and ChronixDB ingester), you sometimes want to know how well you are utilizing your chunks when closing/storing them.	8 years ago
Julius Volz	c25f0de5ae	Remove local.ZeroSample{,Pair}, use model definitions	8 years ago
Julius Volz	044ebce779	Review fixups.	8 years ago
Julius Volz	ab80ced756	storage: separate chunk package, publish more names ... This is a followup to https://github.com/prometheus/prometheus/pull/2011. This publishes more of the methods and other names of the chunk code and moves the chunk code to its own package. There's some unavoidable ugliness: the chunk and chunkDesc metrics are used by both packages, so I had to move them to the chunk package. That isn't great, but I don't see how to do it better without a larger redesign of everything. Same for the evict requests and some other types.	8 years ago