The LevelDB index retrievals could be repeated in a given operation
batch if multiple queued mutations affect the same (Label Name) singles
and (Label Name, Label Value) doubles. This is wasteful and
inefficient, as a single retrieval suffices. Thusly this commit
retrieves the canonical index mappings if the said mapping has not
been looked up in a given batch.
Snappy should have been explicitly included in the runtime, for I
erroneously thought that LevelDB bundled this into its runtime as-is.
It turns out that this assumption is wrong, and I thought we had
Snappy compression support all-along.
Presently our use of LevelDB and its operating modes are hardcoded
into the storage stack. This pull request decouples this and
re-exposes this through flags. We can now perform benchmarking
and remedial tuning.
client_golang was updated to support full label-oriented telemetry,
which introduced interface incompatibilities with the previous
version of Prometheus. To alleviate this, a general fetching and
processing dispatching system has been created, which discriminates
and processes according to the version of input.
This implementation needs to be cleaned up considerably, but it
should hopefully provide some breathing room. Ultimately this will
be extracted out of the storage layer---specifically the LevelDB
hierarchy---to interface with a generic set of abstractions for a
storage system, but this will be where it remains for now.
The LevelDB stochastic tests could take a long time.
Original:
```
Matt:prometheus mtp$ make test
make -C model
protoc --go_out=generated/ data.proto
go build ./...
go test ./...
warning: building out-of-date packages:
github.com/matttproud/golang_instrumentation/maths
github.com/matttproud/golang_instrumentation/utility
github.com/matttproud/golang_instrumentation/metrics
github.com/matttproud/golang_instrumentation
installing these packages with 'go test -i ./...' will speed future tests.
? github.com/matttproud/prometheus [no test files]
? github.com/matttproud/prometheus/coding [no test files]
ok github.com/matttproud/prometheus/coding/indexable 0.012s
? github.com/matttproud/prometheus/model [no test files]
? github.com/matttproud/prometheus/model/generated [no test files]
? github.com/matttproud/prometheus/retrieval [no test files]
? github.com/matttproud/prometheus/storage/metric [no test files]
ok github.com/matttproud/prometheus/storage/metric/leveldb 70.800s
? github.com/matttproud/prometheus/storage/raw [no test files]
? github.com/matttproud/prometheus/storage/raw/index [no test files]
ok github.com/matttproud/prometheus/storage/raw/index/leveldb 0.012s
ok github.com/matttproud/prometheus/storage/raw/leveldb 0.016s
ok github.com/matttproud/prometheus/utility 0.012s
? github.com/matttproud/prometheus/utility/test [no test files]
```
to
```
Matt:prometheus mtp$ make test
make -C model
protoc --go_out=generated/ data.proto
go build ./...
go test ./...
warning: building out-of-date packages:
github.com/matttproud/golang_instrumentation/maths
github.com/matttproud/golang_instrumentation/utility
github.com/matttproud/golang_instrumentation/metrics
github.com/matttproud/golang_instrumentation
installing these packages with 'go test -i ./...' will speed future tests.
? github.com/matttproud/prometheus [no test files]
? github.com/matttproud/prometheus/coding [no test files]
ok github.com/matttproud/prometheus/coding/indexable 0.011s
? github.com/matttproud/prometheus/model [no test files]
? github.com/matttproud/prometheus/model/generated [no test files]
? github.com/matttproud/prometheus/retrieval [no test files]
? github.com/matttproud/prometheus/storage/metric [no test files]
ok github.com/matttproud/prometheus/storage/metric/leveldb 2.158s
? github.com/matttproud/prometheus/storage/raw [no test files]
? github.com/matttproud/prometheus/storage/raw/index [no test files]
ok github.com/matttproud/prometheus/storage/raw/index/leveldb 0.013s
ok github.com/matttproud/prometheus/storage/raw/leveldb 0.013s
ok github.com/matttproud/prometheus/utility 0.013s
? github.com/matttproud/prometheus/utility/test [no test files]
```
Kill LevelDB watermarks due to redundancy.
General interface documentation has begun.
Creating custom types for the model to prevent errors down the
road.
Renaming of components for easier comprehension.
Exposition of interface in LevelDB.
Slew of simple refactorings.
Matt T. Proud