prometheus

Commit Graph

Author	SHA1	Message	Date
Julius Volz	e7ed39c9a6	Initial experimental snapshot of next-gen storage. Change-Id: Ifb8709960dbedd1d9f5efd88cdd359ee9fa9d26d	10 years ago
Julius Volz	b3901827ee	Fix persistence references in tools subdir. These had escaped me because the tools aren't rebuilt if there are changes outside of the respective tool itself. Change-Id: I3e69631babdd95b18e698eb79098dfa59f60f597	11 years ago
Julius Volz	01f652cb4c	Separate storage implementation from interfaces. This was initially motivated by wanting to distribute the rule checker tool under `tools/rule_checker`. However, this was not possible without also distributing the LevelDB dynamic libraries because the tool transitively depended on Levigo: rule checker -> query layer -> tiered storage layer -> leveldb This change separates external storage interfaces from the implementation (tiered storage, leveldb storage, memory storage) by putting them into separate packages: - storage/metric: public, implementation-agnostic interfaces - storage/metric/tiered: tiered storage implementation, including memory and LevelDB storage. I initially also considered splitting up the implementation into separate packages for tiered storage, memory storage, and LevelDB storage, but these are currently so intertwined that it would be another major project in itself. The query layers and most other parts of Prometheus now have notion of the storage implementation anymore and just use whatever implementation they get passed in via interfaces. The rule_checker is now a static binary :) Change-Id: I793bbf631a8648ca31790e7e772ecf9c2b92f7a0	11 years ago
Bjoern Rabenstein	c342ad33a0	Fix OperatorError. This used to work with Go 1.1, but only because of a compiler bug. The bug is fixed in Go 1.2, so we have to fix our code now. Change-Id: I5a9f3a15878afd750e848be33e90b05f3aa055e1	11 years ago
Julius Volz	740d448983	Use custom timestamp type for sample timestamps and related code. So far we've been using Go's native time.Time for anything related to sample timestamps. Since the range of time.Time is much bigger than what we need, this has created two problems: - there could be time.Time values which were out of the range/precision of the time type that we persist to disk, therefore causing incorrectly ordered keys. One bug caused by this was: https://github.com/prometheus/prometheus/issues/367 It would be good to use a timestamp type that's more closely aligned with what the underlying storage supports. - sizeof(time.Time) is 192, while Prometheus should be ok with a single 64-bit Unix timestamp (possibly even a 32-bit one). Since we store samples in large numbers, this seriously affects memory usage. Furthermore, copying/working with the data will be faster if it's smaller. MEMORY USAGE RESULTS Initial memory usage comparisons for a running Prometheus with 1 timeseries and 100,000 samples show roughly a 13% decrease in total (VIRT) memory usage. In my tests, this advantage for some reason decreased a bit the more samples the timeseries had (to 5-7% for millions of samples). This I can't fully explain, but perhaps garbage collection issues were involved. WHEN TO USE THE NEW TIMESTAMP TYPE The new clientmodel.Timestamp type should be used whenever time calculations are either directly or indirectly related to sample timestamps. For example: - the timestamp of a sample itself - all kinds of watermarks - anything that may become or is compared to a sample timestamp (like the timestamp passed into Target.Scrape()). When to still use time.Time: - for measuring durations/times not related to sample timestamps, like duration telemetry exporting, timers that indicate how frequently to execute some action, etc. NOTE ON OPERATOR OPTIMIZATION TESTS We don't use operator optimization code anymore, but it still lives in the code as dead code. It still has tests, but I couldn't get all of them to pass with the new timestamp format. I commented out the failing cases for now, but we should probably remove the dead code soon. I just didn't want to do that in the same change as this. Change-Id: I821787414b0debe85c9fffaeb57abd453727af0f	11 years ago
Julius Volz	eb461a707d	Add chunk sanity checking to dumper tool. Also, move codecs/filters to common location so they can be used in subsequent test. Change-Id: I3ffeb09188b8f4552e42683cbc9279645f45b32e	11 years ago
Julius Volz	0003027dce	Add needed trailing spaces in logs.	11 years ago
Julius Volz	aa5d251f8d	Use github.com/golang/glog for all logging.	11 years ago
Julius Volz	b9c90925dd	Add simple rule file checker tool. This tool does the simple job of checking a Prometheus rule file for validity and printing the parsed rules if the loading succeeds. This came out of an immediate need to quickly verify changes to existing rule files.	11 years ago
Matt T. Proud	cc989c68e1	Replace direct curation table access with wrapper.	11 years ago
Matt T. Proud	07ac921aec	Code Review: First pass.	11 years ago
Matt T. Proud	30b1cf80b5	WIP - Snapshot of Moving to Client Model.	12 years ago
Johannes 'fish' Ziemke	56249320e3	Remove gvm on travis.	12 years ago
Johannes 'fish' Ziemke	4524a5fe37	Remove gvm and install go by hand.	12 years ago
Matt T. Proud	a73f061d3c	Persist solely Protocol Buffers. An design question was open for me in the beginning was whether to serialize other types to disk, but Protocol Buffers quickly won out, which allows us to drop support for other types. This is a good start to cleaning up a lot of cruft in the storage stack and can let us eventually decouple the various moving parts into separate subsystems for easier reasoning. This commit is not strictly required, but it is a start to making the rest a lot more enjoyable to interact with.	12 years ago
Julius Volz	6d68626125	Simple LevelDB sample dumper tool. This very simple tool dumps all samples in a LevelDB metrics storage together with their chunk information. This is very crude for now - I just whipped it up because I needed to debug something. We should add printf-style formatting support for it in the future to indicate what information to output for each sample in the database. We could also support inspecting the index databases.	12 years ago
Matt T. Proud	1f7f89b4e3	Simplify compaction and expose database sizes. This commit simplifies the way that compactions across a database's keyspace occur due to reading the LevelDB internals. Secondarily it introduces the database size estimation mechanisms.	12 years ago
Matt T. Proud	fa6a1f97d0	Expose interfaces for pruner and make pruner tool. In order to run database cleanups and diagnostics, we should have a means for pruning a database---even if LevelDB does this for us.	12 years ago

18 Commits (af77d5ef0bad5248904675e7d689449ff74705bb)