134 Commits (26dbd0776e7b7b190c2d19f1c1ab86f4c11b1f57)

Author	SHA1	Message	Date
Matt T. Proud	b3e34c6658	Implement batch database sample curator. ... This commit introduces to Prometheus a batch database sample curator, which corroborates the high watermarks for sample series against the curation watermark table to see whether a curator of a given type needs to be run. The curator is an abstract executor, which runs various curation strategies across the database. It remarks the progress for each type of curation processor that runs for a given sample series. A curation procesor is responsible for effectuating the underlying batch changes that are request. In this commit, we introduce the CompactionProcessor, which takes several bits of runtime metadata and combine sparse sample entries in the database together to form larger groups. For instance, for a given series it would be possible to have the curator effectuate the following grouping: - Samples Older than Two Weeks: Grouped into Bunches of 10000 - Samples Older than One Week: Grouped into Bunches of 1000 - Samples Older than One Day: Grouped into Bunches of 100 - Samples Older than One Hour: Grouped into Bunches of 10 The benefits hereof of such a compaction are 1. a smaller search space in the database keyspace, 2. better employment of compression for repetious values, and 3. reduced seek times.	12 years ago
Julius Volz	2202cd71c9	Track alerts over time and write out alert timeseries.	12 years ago
Johannes 'fish' Ziemke	1ad41d4c00	Call closer.Close() earlier.	12 years ago
Johannes 'fish' Ziemke	22da76e8ab	Close of reportTicker to exit goroutine.	12 years ago
Johannes 'fish' Ziemke	5043c6fce7	Have goroutine exit on signal via defer block.	12 years ago
juliusv	af7ddc36e2	Merge pull request #176 from prometheus/optimization/view-materialization/slice-chunking ... Truncate irrelevant chunk values.	12 years ago
Julius Volz	9b8c671ec9	Fixes/cleanups to renderView() samples truncation.	12 years ago
Matt T. Proud	05504d3642	WIP - Truncate irrelevant chunk values. ... This does not work with the view tests.	12 years ago
Matt T. Proud	a32602140e	Convert the TestInstant value into UTC. ... For the forthcoming Curator, we don't record timezone information in the samples, nor do we in the curation remarks. All times are recorded UTC. That said, for the test environment to better match production, the special instant should be in UTC.	12 years ago
Matt T. Proud	b1a8e51b07	Extract dto.SampleValueSeries into model.Values.	12 years ago
Matt T. Proud	422003da8e	Convert trailing float64s.	12 years ago
Matt T. Proud	db4ffbb262	Wrap dto.SampleKey with business logic type. ... The curator work can be done easier if dto.SampleKey is no longer directly accessed but rather has a higher level type around it that captures a certain modicum of business logic. This doesn't look terribly interesting today, but it will get more so.	12 years ago
Matt T. Proud	f9e99bd08a	Refresh SampleValue to 64-bit floating point. ... We always knew that this needed to be fixed.	12 years ago
Matt T. Proud	092c7bd88e	Stochastic test support plural SampleValueSeries. ... After SampleValue was refactored into SampleValueSeries, which involves plural values under a common super key, the stochastic test was never refreshed to reflect this reality. We had other tests that validated the functionality, but this one was insufficently forward-ported.	12 years ago
Julius Volz	99dcbe0f94	Integrate memory and disk layers in view rendering.	12 years ago
Julius Volz	63625bd244	Make view use memory persistence, remove obsolete code. ... This makes the memory persistence the backing store for views and adjusts the MetricPersistence interface accordingly. It also removes unused Get* method implementations from the LevelDB persistence so they don't need to be adapted to the new interface. In the future, we should rethink these interfaces. All staleness and interpolation handling is now removed from the storage layer and will be handled only by the query layer in the future.	12 years ago
Matt T. Proud	d468271e2f	Fix append queue telemetry and parameterize sizes. ... The original append queue telemetry never worked, because it was updated only upon the exit of the select statement, which would usually liberate the queues of contents. This has been fixed to be reported arbitrarily. The queue sizes are now parameterizable via flags.	12 years ago
Julius Volz	95b081f9bc	Stop serving tiered storage after draining it.	12 years ago
Matt T. Proud	a55602df4a	Validate diskFrontier domain for series candidate. ... It is the case with the benchmark tool that we thought that we generated multiple series and saved them to the disk as such, when in reality, we overwrote the fields of the outgoing metrics via Go map reference behavior. This was accidental. In the course of diagnosing this, a few errors were found: 1. ``newSeriesFrontier`` should check to see if the candidate fingerprint is within the given domain of the ``diskFrontier``. If not, as the contract in the docstring stipulates, a ``nil`` ``seriesFrontier`` should be emitted. 2. In the interests of aiding debugging, the raw LevelDB ``levigoIterator`` type now includes a helpful forensics ``String()`` method. This work produced additional cleanups: 1. ``Close() error`` with the storage stack is technically incorrect, since nowhere in the bowels of it does an error actually occur. The interface has been simplified to remove this for now.	12 years ago
Matt T. Proud	d79c932a8e	Merge pull request #120 from prometheus/feature/storage/compaction ... Spin up curator run in the tests.	12 years ago
Matt T. Proud	c3e3460ca6	Spin up curator run in the tests. ... After this commit, we'll need to add validations that it does the desired work, which we presently know that it doesn't. Given the changes I made with a plethora of renamings, I want to commit this now before it gets even larger.	12 years ago
Matt T. Proud	461da0b3a8	Merge pull request #117 from prometheus/feature/storage/compaction ... Spin up storage layers for made fixtures.	12 years ago
Matt T. Proud	d0ad6cbeaa	Spin up storage layers for made fixtures.	12 years ago
Julius Volz	c59f3fc538	Fix formatting in tiered_test.go.	12 years ago
juliusv	39826d7335	Merge pull request #107 from prometheus/julius-fix-get-fingerprints ... Fix bug in GetFingerprintsForLabelSet().	12 years ago
Julius Volz	2668700e54	Fix bug in GetFingerprintsForLabelSet().	12 years ago
Matt T. Proud	c53a72a894	Test data for the curator.	12 years ago
Julius Volz	55ca65aa6e	More userfriendly output when we fail to create the tiered storage.	12 years ago
Matt T. Proud	c4e971d7d9	Merge pull request #101 from prometheus/refactor/test/directory-extraction ... Create temporary directory handler.	12 years ago
Matt T. Proud	b86b0ea41a	Create temporary directory handler.	12 years ago
Julius Volz	8cf2af3923	Abort view job processing on timeout.	12 years ago
Julius Volz	2b8f0b2cc7	Constantize metric name label name.	12 years ago
Julius Volz	e096896932	PR comment fixups.	12 years ago
Julius Volz	dd67ab115b	Change GetAllMetricNames() to GetAllValuesForLabel().	12 years ago
Julius Volz	42bdf921d1	Fetch integrated memory/disk data for simple Get* functions.	12 years ago
Julius Volz	11bb94a7e5	Implement GetAllMetricNames() for memory storage.	12 years ago
Julius Volz	991dc68d78	Rename misnamed oldestSampleTimestamp variable.	12 years ago
Matt T. Proud	3e97a3630d	Include nascent curator scaffolding. ... The curator doesn't do anything yet; rather, this is the type definition including the anciliary testing scaffold. Improve Makefile and Git developer experience. The top-level Makefile was a bit overloaded in terms of generation of assets and their management. This has been offloaded into separate Makefiles. The Git developer experience sucked due to lack of .gitignore policies. Also: Fix faulty skiplist naming from old merge.	12 years ago
Matt T. Proud	b2e4c88b80	Wrap LevelDB iterator operations behind interface. ... The LevelDB storage types return an interface type now that wraps around the underlying iterator. This both enhances testability but improves upon, in my opinion, the interface design for the LevelDB iterator. Secondarily, the resource reaping behaviors for the LevelDB iterators have been improved by dropping the externalized io.Closer object. Finally, the iterator provisioning methods provide the option for indicating whether one wants a snapshotted iterator or not.	12 years ago
Matt T. Proud	70448711ec	Merge pull request #95 from prometheus/feature/persistence/batching ... Several interface cleanups.	12 years ago
Matt T. Proud	8f6b55be71	Several interface cleanups. ... - Kill Close in Persistent and document interface. - Extract batching behavior into interface. - Kill IteratorManager, which was used for unknown reasons.	12 years ago
Julius Volz	a33d2726bc	Mark range op as consumed if it receives no data points in range.	12 years ago
Julius Volz	3c9d6cb66c	Add several needed persistence proxy methods to tiered storage.	12 years ago
Julius Volz	081d250929	Fix view's GetRangeValues() reverse iteration behavior.	12 years ago
Julius Volz	0be0aa59c2	Wait until storage is drained before closing the underlying leveldb.	12 years ago
Julius Volz	becc278eb6	Fix two bugs in range op time advancement.	12 years ago
Matt T. Proud	ceb6611957	Fix regression in subsequent range op. compactions. ... We have an anomaly whereby subsequent range operations fail to be compacted into one single range operation. This fixes such behavior.	12 years ago
Matt T. Proud	669abdfefe	``make format`` invocation.	12 years ago
Julius Volz	bdb067b47f	Implement remaining View Get* methods.	12 years ago
Julius Volz	1f42364733	Fix typo in comment.	12 years ago