prometheus

Commit Graph

Author	SHA1	Message	Date
Julius Volz	1eee448bc1	Store samples in custom binary encoding. This has been shown to provide immense decoding speed benefits. See also: https://groups.google.com/forum/#!topic/prometheus-developers/FeGl_qzGrYs Change-Id: I7d45b4650e44ddecaa91dad9d7fdb3cd0b9f15fe	11 years ago
Bjoern Rabenstein	6bc083f38b	Major code cleanup in storage. - Mostly docstring fixed/additions. (Please review these carefully, since most of them were missing, I had to guess them from an outsider's perspective. (Which on the other hand proves how desperately required many of these docstrings are.)) - Removed all uses of new(...) to meet our own style guide (draft). - Fixed all other 'go vet' and 'golint' issues (except those that are not fixable (i.e. caused by bugs in or by design of 'go vet' and 'golint')). - Some trivial refactorings, like reorder functions, minor renames, ... - Some slightly less trivial refactoring, mostly to reduce code duplication by embedding types instead of writing many explicit forwarders. - Cleaned up the interface structure a bit. (Most significant probably the removal of the View-like methods from MetricPersistenc. Now they are only in View and not duplicated anymore.) - Removed dead code. (Probably not all of it, but it's a first step...) - Fixed a leftover in storage/metric/end_to_end_test.go (that made some parts of the code never execute (incidentally, those parts were broken (and I fixed them, too))). Change-Id: Ibcac069940d118a88f783314f5b4595dce6641d5	11 years ago
Julius Volz	718ad2224b	Fix LevelDB closing order. The storage itself should be closed before any of the objects passed into it are closed (otherwise closing the storage can randomly freeze). Defers are executed in reverse order, so closing the storage should be the last of the defer statements. Change-Id: Id920318b876f5b94767ed48c81221b3456770620	11 years ago
Matt T. Proud	4a87c002e8	Update low-level i'faces to reflect wireformats. This commit fixes a critique of the old storage API design, whereby the input parameters were always as raw bytes and never Protocol Buffer messages that encapsulated the data, meaning every place a read or mutation was conducted needed to manually perform said translations on its own. This is taxing. Change-Id: I4786938d0d207cefb7782bd2bd96a517eead186f	11 years ago
Julius Volz	aa5d251f8d	Use github.com/golang/glog for all logging.	11 years ago
Matt T. Proud	a5141e4d0a	Depointerize storage conf. and chain ingester. The storage builders need to work with the assumption that they have a copy of the underlying configuration data if any mutations are made.	11 years ago
Matt T. Proud	cc989c68e1	Replace direct curation table access with wrapper.	11 years ago
Matt T. Proud	07ac921aec	Code Review: First pass.	11 years ago
Matt T. Proud	d8792cfd86	Extract HighWatermarking. Clean up the rest.	11 years ago
Matt T. Proud	f4669a812c	Extract index storage into separate types.	11 years ago
Matt T. Proud	772d3d6b11	Consolidate LevelDB storage construction. There are too many parameters to constructing a LevelDB storage instance for a construction method, so I've opted to take an idiomatic approach of embedding them in a struct for easier mediation and versioning.	11 years ago
Matt T. Proud	30b1cf80b5	WIP - Snapshot of Moving to Client Model.	12 years ago
Matt T. Proud	b811ccc161	Disable paranoid checks and expose max FDs option. We shouldn't need paranoid checks now. We also shouldn't need too many FDs being open due to rule evaluator hitting in-memory values stream.	12 years ago
Matt T. Proud	a73f061d3c	Persist solely Protocol Buffers. An design question was open for me in the beginning was whether to serialize other types to disk, but Protocol Buffers quickly won out, which allows us to drop support for other types. This is a good start to cleaning up a lot of cruft in the storage stack and can let us eventually decouple the various moving parts into separate subsystems for easier reasoning. This commit is not strictly required, but it is a start to making the rest a lot more enjoyable to interact with.	12 years ago
Matt T. Proud	8f4c7ece92	Destroy naked returns in half of corpus. The use of naked return values is frowned upon. This is the first of two bulk updates to remove them.	12 years ago
Matt T. Proud	4e0c932a4f	Simplify Encoder's encoding signature. The reality is that if we ever try to encode a Protocol Buffer and it fails, it's likely that such an error is ultimately not a runtime error and should be fixed forthwith. Thusly, we should rename ``Encoder.Encode`` to ``Encoder.MustEncode`` and drop the error return value.	12 years ago
Matt T. Proud	b224251981	Simplify compaction and expose database sizes. This commit simplifies the way that compactions across a database's keyspace occur due to reading the LevelDB internals. Secondarily it introduces the database size estimation mechanisms. Include database health and help interfaces. Add database statistics; remove status goroutines. This commit kills the use of Go routines to expose status throughout the web components of Prometheus. It also dumps raw LevelDB status on a separate /databases endpoint.	12 years ago
Matt T. Proud	1f7f89b4e3	Simplify compaction and expose database sizes. This commit simplifies the way that compactions across a database's keyspace occur due to reading the LevelDB internals. Secondarily it introduces the database size estimation mechanisms.	12 years ago
Matt T. Proud	fa6a1f97d0	Expose interfaces for pruner and make pruner tool. In order to run database cleanups and diagnostics, we should have a means for pruning a database---even if LevelDB does this for us.	12 years ago
Matt T. Proud	b3e34c6658	Implement batch database sample curator. This commit introduces to Prometheus a batch database sample curator, which corroborates the high watermarks for sample series against the curation watermark table to see whether a curator of a given type needs to be run. The curator is an abstract executor, which runs various curation strategies across the database. It remarks the progress for each type of curation processor that runs for a given sample series. A curation procesor is responsible for effectuating the underlying batch changes that are request. In this commit, we introduce the CompactionProcessor, which takes several bits of runtime metadata and combine sparse sample entries in the database together to form larger groups. For instance, for a given series it would be possible to have the curator effectuate the following grouping: - Samples Older than Two Weeks: Grouped into Bunches of 10000 - Samples Older than One Week: Grouped into Bunches of 1000 - Samples Older than One Day: Grouped into Bunches of 100 - Samples Older than One Hour: Grouped into Bunches of 10 The benefits hereof of such a compaction are 1. a smaller search space in the database keyspace, 2. better employment of compression for repetious values, and 3. reduced seek times.	12 years ago
Matt T. Proud	a55602df4a	Validate diskFrontier domain for series candidate. It is the case with the benchmark tool that we thought that we generated multiple series and saved them to the disk as such, when in reality, we overwrote the fields of the outgoing metrics via Go map reference behavior. This was accidental. In the course of diagnosing this, a few errors were found: 1. ``newSeriesFrontier`` should check to see if the candidate fingerprint is within the given domain of the ``diskFrontier``. If not, as the contract in the docstring stipulates, a ``nil`` ``seriesFrontier`` should be emitted. 2. In the interests of aiding debugging, the raw LevelDB ``levigoIterator`` type now includes a helpful forensics ``String()`` method. This work produced additional cleanups: 1. ``Close() error`` with the storage stack is technically incorrect, since nowhere in the bowels of it does an error actually occur. The interface has been simplified to remove this for now.	12 years ago
Matt T. Proud	c3e3460ca6	Spin up curator run in the tests. After this commit, we'll need to add validations that it does the desired work, which we presently know that it doesn't. Given the changes I made with a plethora of renamings, I want to commit this now before it gets even larger.	12 years ago
Matt T. Proud	c53a72a894	Test data for the curator.	12 years ago
Matt T. Proud	6dcaa28806	Include LevelDB fixture generators for curator. This will help reduce common boilerplate for our test process with respect to LevelDB-related things.	12 years ago
Matt T. Proud	b2e4c88b80	Wrap LevelDB iterator operations behind interface. The LevelDB storage types return an interface type now that wraps around the underlying iterator. This both enhances testability but improves upon, in my opinion, the interface design for the LevelDB iterator. Secondarily, the resource reaping behaviors for the LevelDB iterators have been improved by dropping the externalized io.Closer object. Finally, the iterator provisioning methods provide the option for indicating whether one wants a snapshotted iterator or not.	12 years ago
Matt T. Proud	f2a30cf20c	Several important cleanups and deprecations. EachFunc is deprecated. Remove deprecated ``Pair`` and ``GetAll``. These were originally used for forensic and the old gorest impl. Nothing today in the user-facing path nor the tests uses them, especially since the advent of the ForEach protocol in the interface.	12 years ago
Matt T. Proud	8f6b55be71	Several interface cleanups. - Kill Close in Persistent and document interface. - Extract batching behavior into interface. - Kill IteratorManager, which was used for unknown reasons.	12 years ago
Julius Volz	f238b23b04	Set -leveldbFlushOnMutate to false by default.	12 years ago
Matt T. Proud	d5380897c3	Cleanups and adds performance regression.	12 years ago
Matt T. Proud	41068c2e84	Checkpoint.	12 years ago
Julius Volz	d67e4b9131	Address outstanding comments from PR/47 and other cleanups.	12 years ago
Julius Volz	e23ac0adbd	Implement bulk iteration/operation storage interface.	12 years ago
Matt T. Proud	ec3e98b365	Include Snappy in Runtime. Snappy should have been explicitly included in the runtime, for I erroneously thought that LevelDB bundled this into its runtime as-is. It turns out that this assumption is wrong, and I thought we had Snappy compression support all-along.	12 years ago
Matt T. Proud	79ba248bbe	Make LevelDB operating modes invocation-time conf. Presently our use of LevelDB and its operating modes are hardcoded into the storage stack. This pull request decouples this and re-exposes this through flags. We can now perform benchmarking and remedial tuning.	12 years ago
Matt T. Proud	ea54751431	Update import paths to new location. This repository moved from matttproud/prometheus to prometheus/prometheus, and all import paths need to be updated.	12 years ago
Matt T. Proud	3ac5d48b1a	Impl' storage i'faces and fix non-idiomatic warts. This change includes implementation of most major storage layer features, albeit some imperfect. It also includes nascent telemetry bindings, too.	12 years ago
Matt T. Proud	577acf4fe7	Exploding the storage infrastructure by contexts.	12 years ago
Matt T. Proud	15a6681651	Various cleanups. Kill LevelDB watermarks due to redundancy. General interface documentation has begun. Creating custom types for the model to prevent errors down the road. Renaming of components for easier comprehension. Exposition of interface in LevelDB. Slew of simple refactorings.	12 years ago
Matt T. Proud	c1f0d8aefd	Levigo -> LevelDB in terminology and references.	12 years ago
Matt T. Proud	6ed67b02e2	Incorporate new interface tests.	12 years ago
Matt T. Proud	44f8802ae7	Add Apache License 2.0 boilerplate.	12 years ago
Matt T. Proud	2bbdaa5790	Initial directory re-arrangement for storage.	12 years ago

42 Commits (b3a78d2202e2b80c73f0d11bcf202ad5c606d818)