prometheus

Commit Graph

Author	SHA1	Message	Date
Björn Rabenstein	c44e7cd105	Merge pull request #706 from prometheus/beorn7/persistence2 Improve iterator performance.	10 years ago
Fabian Reinartz	112a778922	Align int64s for atomic operations	10 years ago
beorn7	3b9c421a69	Weed out all the [Gg]et* method names. The only exception is getNumChunksToPersist to avoid naming the struct member numChunksToPersist in a weird way.	10 years ago
Julius Volz	267fd34156	Switch Prometheus to use github.com/prometheus/log. This change is conceptually very simple, although the diff is large. It switches logging from "github.com/golang/glog" to "github.com/prometheus/log", while not actually changing any log messages. V(1)-style logging has been changed to be log.Debug*().	10 years ago
beorn7	81b190bf45	Remove locking from series iterator. Cache chunk iterators.	10 years ago
beorn7	cd5574bf8a	Make chunk and series iterators more efficient.	10 years ago
beorn7	f79c694be5	Add benchmarks for series iterator methods.	10 years ago
Fabian Reinartz	f59a449a24	Fix storage test	10 years ago
Fabian Reinartz	d8440d75f1	Do not start storage processing before Start() is called.	10 years ago
beorn7	d1a93655a1	Fix typo.	10 years ago
beorn7	7c6466d476	Reserve only ~1M FPs for the mapping. That reduces the chance of having a fingerprint in the reserved area.	10 years ago
beorn7	ac75dc2812	Avoid archive lookup for known mapped FPs.	10 years ago
beorn7	ed810b45bf	Improvements after review.	10 years ago
beorn7	c36e0e05f1	Add crash recovery of fingerprint mappings.	10 years ago
beorn7	2235cec175	Handle fingerprint collisions.	10 years ago
beorn7	9820e5fe99	Use FastFingerprint where appropriate.	10 years ago
Scott Worley	e5f92d35fe	Fix storage/local tests for 32-bit systems	10 years ago
beorn7	a052d32609	Comment improvement.	10 years ago
beorn7	66fc61f9b7	Make bufPool a member of the persistence struct.	10 years ago
beorn7	b02d900e61	Improve chunk and chunkDesc loading. Also, clean up some things in the code (especially introduction of the chunkLenWithHeader constant to avoid the same expression all over the place). Benchmark results: BEFORE BenchmarkLoadChunksSequentially 5000 283580 ns/op 152143 B/op 312 allocs/op BenchmarkLoadChunksRandomly 20000 82936 ns/op 39310 B/op 99 allocs/op BenchmarkLoadChunkDescs 10000 110833 ns/op 15092 B/op 345 allocs/op AFTER BenchmarkLoadChunksSequentially 10000 146785 ns/op 152285 B/op 315 allocs/op BenchmarkLoadChunksRandomly 20000 67598 ns/op 39438 B/op 103 allocs/op BenchmarkLoadChunkDescs 20000 99631 ns/op 12636 B/op 192 allocs/op Note that everything is obviously loaded from the page cache (as the benchmark runs thousands of times with very small series files). In a real-world scenario, I expect a larger impact, as the disk operations will more often actually hit the disk. To load ~50 sequential chunks, this reduces the iops from 100 seeks and 100 reads to 1 seek and 1 read.	10 years ago
beorn7	c563398c68	Remove obsolete debug message.	10 years ago
beorn7	c5fa0b90c3	Fix the case where a series in memory has 0 chunks, but chunks on disk. This is actually completely normal for a freshly unarchived series. Test added to expose.	10 years ago
beorn7	3035b8bfdd	Adaptively reduce the wait time for memory series maintenance. This will make in-memory series maintenance the faster the more chunks are waiting for persistence.	10 years ago
beorn7	fbc44d8f95	Add benchmark for loading chunks and chunk descs.	10 years ago
beorn7	6a21f73898	Fixes after review.	10 years ago
beorn7	51d35f4481	Instrument series maintenance durations.	10 years ago
beorn7	12ae6e9203	Increase resilience of the storage against data corruption - step 4. Step 4: Add a configurable sync'ing of series files after modification.	10 years ago
beorn7	11bd9ce1bd	Increase resilience of the storage against data corruption - step 3. Step 3: Remember the mtime of series files and make use of it to detect series files that are not the one the checkpoint thinks they are.	10 years ago
beorn7	e25cca823c	Increase resilience of the storage against data corruption - step 2. Step 2: Add a flag -storage.local.pedantic-checks to check every series file. Also, remove countPersistedHeadChunks channel, which is unused.	10 years ago
beorn7	3d8d8928be	Increase resilience of the storage against data corruption - step 1. Step 1: Admit the problem by turning the various "panic"s into logged errors, followed by marking the persistence as dirty.	10 years ago
beorn7	da7c0461c6	Rename persist queue len/cap to num/max chunks to persist. Remove deprecated flag storage.incoming-samples-queue-capacity.	10 years ago
beorn7	a075900f9a	Merge branch 'beorn7/persistence' into beorn7/ingestion-tweaks	10 years ago
beorn7	1d8fc7d56f	Change minor things after code review.	10 years ago
beorn7	be11cb2b07	Remove the sample ingestion channel. The one central sample ingestion channel has caused a variety of trouble. This commit removes it. Targets and rule evaluation call an Append method directly now. To incorporate multiple storage backends (like OpenTSDB), storage.Tee forks the Append into two different appenders. Note that the tsdb queue manager had its own queue anyway. It was a queue after a queue... Much queue, so overhead... Targets have their own little buffer (implemented as a channel) to avoid stalling during an http scrape. But a new scrape will only be started once the old one is fully ingested. The contraption of three pipelined ingesters was removed. A Target is an ingester itself now. Despite more logic in Target, things should be less confusing now. Also, remove lint and vet warnings in ast.go.	10 years ago
beorn7	0056eaeb4f	Redesign series maintenance and chunk persistence.	10 years ago
beorn7	5bea942d8e	Improve various things around chunk encoding. A number of mostly minor things: - Rename chunk type -> chunk encoding. - After all, do not carry around the chunk encoding to all parts of the system, but just have one place where the encoding for new chunks is set based on the flag. The new approach has caveats as well, but the polution of so many method signatures is worse. - Use the default chunk encoding for new chunks of existing series. (Previously, only new _series_ would get chunks with the default encoding.) - Use an enum for chunk encoding. (But keep the version number for the flag, for reasons discussed previously.) - Add encoding() to the chunk interface (so that a chunk knows its own encoding - no need to have that in a different top-level function). - Got rid of newFollowUpChunk (which would keep the existing encoding for all chunks of a time series). Now only use newChunk(), which will create a chunk encoding according to the flag. - Simplified transcodeAndAdd. - Reordered methods of deltaEncodedChunk and doubleDeltaEncoded chunk to match the order in the chunk interface. - Only transcode if the chunk is not yet half full. If more than half full, add a new chunk instead.	10 years ago
beorn7	9ecf93526d	Sync the checkpoints. Because that's what should be done with checkpoints.	10 years ago
beorn7	853f971540	Actually use double-delta encoding for transcoding. :-o	10 years ago
beorn7	23ba8a5516	Make floats exact again. This should do the right thing for the old delta chunks, too.	10 years ago
beorn7	a8d4f8af9a	Improve minor things after review. The problem of float precision will be addressed in the next commit.	10 years ago
beorn7	13fcf1ddbc	Implement double-delta encoded chunks.	10 years ago
beorn7	5ed8f6c205	Update persistQueueLength after chunks were persisted.	10 years ago
beorn7	0167083da6	Improvements after review.	10 years ago
beorn7	ebac14eff3	Add version guard to persistence.	10 years ago
Julius Volz	795704f0df	Merge pull request #565 from fabxc/fabxc/labelmatcher_test Tests for retrieving fingerprints for label matchers added.	10 years ago
Fabian Reinartz	4bff5d29bf	Add tests for retrieving fingerprints for label matchers. This checks for the basic behaviour of GetFingerprintsForLabelMatchers, that is, whether the different matcher types filter the correct fingerprints and intersections are correct.	10 years ago
beorn7	92991026bb	Fix chunkDescsTotal count in case of errors. Only increment the counter if we actually add the memory series to the fingerprintToSeries map.	10 years ago
beorn7	1db7589081	Reduce the capacity of countPersistedHeadChunks. The capacity is basically how many persisted head chunks we will count at most while doing other things, in particular checkpointing. To limit the amount of already counted head chunks, keep this number low, otherwise we will easily checkpoint too often if checkpoints take long anyway.	10 years ago
beorn7	9406afad72	Do not double-count non-persisted head chunks on loading.	10 years ago
beorn7	dbc22b972c	Check last time in head chunk for head chunk timeout, not first.	10 years ago

1 2 3 4

157 Commits (c44e7cd10560ffea7024e3f890261c123968b469)