prometheus

Commit Graph

Author	SHA1	Message	Date
Julius Volz	d5ef0c64dc	Merge "Add optional sample replication to OpenTSDB."	11 years ago
Julius Volz	61d26e8445	Add optional sample replication to OpenTSDB. Prometheus needs long-term storage. Since we don't have enough resources to build our own timeseries storage from scratch ontop of Riak, Cassandra or a similar distributed datastore at the moment, we're planning on using OpenTSDB as long-term storage for Prometheus. It's data model is roughly compatible with that of Prometheus, with some caveats. As a first step, this adds write-only replication from Prometheus to OpenTSDB, with the following things worth noting: 1) I tried to keep the integration lightweight, meaning that anything related to OpenTSDB is isolated to its own package and only main knows about it (essentially it tees all samples to both the existing storage and TSDB). It's not touching the existing TieredStorage at all to avoid more complexity in that area. This might change in the future, especially if we decide to implement a read path for OpenTSDB through Prometheus as well. 2) Backpressure while sending to OpenTSDB is handled by simply dropping samples on the floor when the in-memory queue of samples destined for OpenTSDB runs full. Prometheus also only attempts to send samples once, rather than implementing a complex retry algorithm. Thus, replication to OpenTSDB is best-effort for now. If needed, this may be extended in the future. 3) Samples are sent in batches of limited size to OpenTSDB. The optimal batch size, timeout parameters, etc. may need to be adjusted in the future. 4) OpenTSDB has different rules for legal characters in tag (label) values. While Prometheus allows any characters in label values, OpenTSDB limits them to a to z, A to Z, 0 to 9, -, _, . and /. Currently any illegal characters in Prometheus label values are simply replaced by an underscore. Especially when integrating OpenTSDB with the read path in Prometheus, we'll need to reconsider this: either we'll need to introduce the same limitations for Prometheus labels or escape/encode illegal characters in OpenTSDB in such a way that they are fully decodable again when reading through Prometheus, so that corresponding timeseries in both systems match in their labelsets. Change-Id: I8394c9c55dbac3946a0fa497f566d5e6e2d600b5	11 years ago
Stuart Nelson	0c58e388f6	rename curation metrics to prometheus_curation Change-Id: I6a0bf277e88ea8eb737670b7e865ae20f2cbfb91	11 years ago
Stuart Nelson	28f59edf16	Added telemetry for counting stored samples Change-Id: I0f36f7c2738d070ca2f107fcb315f98e46803af3	11 years ago
Tobias Schmidt	6947ee9bc9	Try to create metrics root directory if missing This change tries to be nice and create the metrics directoy first before erroring out. Change-Id: I72691cdc32469708cd671c6ef1fb7db55fe60430	11 years ago
Julius Volz	740d448983	Use custom timestamp type for sample timestamps and related code. So far we've been using Go's native time.Time for anything related to sample timestamps. Since the range of time.Time is much bigger than what we need, this has created two problems: - there could be time.Time values which were out of the range/precision of the time type that we persist to disk, therefore causing incorrectly ordered keys. One bug caused by this was: https://github.com/prometheus/prometheus/issues/367 It would be good to use a timestamp type that's more closely aligned with what the underlying storage supports. - sizeof(time.Time) is 192, while Prometheus should be ok with a single 64-bit Unix timestamp (possibly even a 32-bit one). Since we store samples in large numbers, this seriously affects memory usage. Furthermore, copying/working with the data will be faster if it's smaller. MEMORY USAGE RESULTS Initial memory usage comparisons for a running Prometheus with 1 timeseries and 100,000 samples show roughly a 13% decrease in total (VIRT) memory usage. In my tests, this advantage for some reason decreased a bit the more samples the timeseries had (to 5-7% for millions of samples). This I can't fully explain, but perhaps garbage collection issues were involved. WHEN TO USE THE NEW TIMESTAMP TYPE The new clientmodel.Timestamp type should be used whenever time calculations are either directly or indirectly related to sample timestamps. For example: - the timestamp of a sample itself - all kinds of watermarks - anything that may become or is compared to a sample timestamp (like the timestamp passed into Target.Scrape()). When to still use time.Time: - for measuring durations/times not related to sample timestamps, like duration telemetry exporting, timers that indicate how frequently to execute some action, etc. NOTE ON OPERATOR OPTIMIZATION TESTS We don't use operator optimization code anymore, but it still lives in the code as dead code. It still has tests, but I couldn't get all of them to pass with the new timestamp format. I commented out the failing cases for now, but we should probably remove the dead code soon. I just didn't want to do that in the same change as this. Change-Id: I821787414b0debe85c9fffaeb57abd453727af0f	11 years ago
Julius Volz	6b7de31a3c	Upgrade to LevelDB 1.14.0 to fix LevelDB bugs. This tentatively fixes https://github.com/prometheus/prometheus/issues/368 due to an upstream bugfix in snapshotted LevelDB iterator handling, which got fixed in LevelDB 1.14.0: https://code.google.com/p/leveldb/issues/detail?id=200 Change-Id: Ib0cc67b7d3dc33913a1c16736eff32ef702c63bf	11 years ago
Julius Volz	db015de65b	Comment and "go fmt" fixups in compaction tests. Change-Id: Iaa0eda6a22a5caa0590bae87ff579f9ace21e80a	11 years ago
Julius Volz	51408bdfe8	Merge changes I3ffeb091,Idffefea4 * changes: Add chunk sanity checking to dumper tool. Add compaction regression tests.	11 years ago
Julius Volz	2162e57784	Merge "Fix watermarker default time / LevelDB key ordering bug."	11 years ago
Julius Volz	5e18255920	Merge "Fix chunk corruption compaction bug."	11 years ago
Julius Volz	eb461a707d	Add chunk sanity checking to dumper tool. Also, move codecs/filters to common location so they can be used in subsequent test. Change-Id: I3ffeb09188b8f4552e42683cbc9279645f45b32e	11 years ago
Julius Volz	6ea22f2bf9	Add compaction regression tests. This adds regression tests that catch the two error cases reported in https://github.com/prometheus/prometheus/issues/367 It also adds a commented-out test case for the crash in https://github.com/prometheus/prometheus/issues/368 but there's no fix for the latter crash yet. Change-Id: Idffefea4ed7cc281caae660bcad2e3c13ec3bd17	11 years ago
Conor Hennessy	9a48010cec	Add a check for metrics directory existence. Previously on startup the program would just quit without stating explicitly why. Change-Id: I833b85eb74d2dd27cdc3f0f2e65d7bb1c42caa39	11 years ago
Julius Volz	b5f6e3c90c	Fix watermarker default time / LevelDB key ordering bug. This fixes part 2) of https://github.com/prometheus/prometheus/issues/367 (uninitialized time.Time mapping to a higher LevelDB key than "normal" timestamps). Change-Id: Ib079974110a7b7c4757948f81fc47d3d29ae43c9	11 years ago
Julius Volz	a1a97ed064	Fix chunk corruption compaction bug. This fixes part 1) of https://github.com/prometheus/prometheus/issues/367 (the storing of samples with the wrong fingerprint into a compacted chunk, thus corrupting it). Change-Id: I4c36d0d2e508e37a0aba90b8ca2ecc78ee03e3f1	11 years ago
Matt T. Proud	86fcbe5bde	Retain DTO on each cycle. Change-Id: Ifc6f68f98eacb01097771d0dbf043c98bba1d518	11 years ago
Matt T. Proud	4a87c002e8	Update low-level i'faces to reflect wireformats. This commit fixes a critique of the old storage API design, whereby the input parameters were always as raw bytes and never Protocol Buffer messages that encapsulated the data, meaning every place a read or mutation was conducted needed to manually perform said translations on its own. This is taxing. Change-Id: I4786938d0d207cefb7782bd2bd96a517eead186f	11 years ago
Matt T. Proud	7910f6e863	Prevent total storage locking during memory flush. While a hack, this change should allow us to serve queries expeditiously during a flush operation. Change-Id: I9a483fd1dd2b0638ab24ace960df08773c4a5079	11 years ago
Matt T. Proud	12d5e6ca5a	Curation should not starve user-interactive ops. The background curation should be staggered to ensure that disk I/O yields to user-interactive operations in a timely manner. The lack of routine prioritization necessitates this. Change-Id: I9b498a74ccd933ffb856e06fedc167430e521d86	11 years ago
Matt T. Proud	2b42fd0068	Snapshot of no more frontier. Change-Id: Icd52da3f52bfe4529829ea70b4865ed7c9f6c446	11 years ago
Matt T. Proud	7db518d3a0	Abstract high watermark cache into standard LRU. Conflicts: storage/metric/memory.go storage/metric/tiered.go storage/metric/watermark.go Change-Id: Iab2aedbd8f83dc4ce633421bd4a55990fa026b85	11 years ago
Matt T. Proud	d74c2c54d4	Interfacification of stream. Move the stream to an interface, for a number of additional changes around it are underway. Conflicts: storage/metric/memory.go Change-Id: I4a5fc176f4a5274a64ebdb1cad52600954c463c3	11 years ago
Matt T. Proud	c262907fec	Kill interface cruft. These pieces were never used and should be thusly removed. Change-Id: I8dd151ec4c40b6d3ccffad1bb9b8b75a92e9ee37	11 years ago
Matt T. Proud	b23acccea8	Kill AppendSample interface definition. AppendSample will be repcated with AppendSamples, which will take advantage of bulks appends. This is a necessary step for indexing pipeline decoupling. Change-Id: Ia83811a87bcc89973d3b64d64b85a28710253ebc	11 years ago
Matt T. Proud	aaaf3367d6	Include forgotten imports. This fixes the build. Change-Id: Id132f4342adb9ed20116191086f157ca7f7cf515	11 years ago
Matt T. Proud	acf91f38bd	Build layered indexers. The indexers will be extracted in a short while and wrapped accordingly with these types. Change-Id: I4d1abda4e46117210babad5aa0d42f9ca1f6594f	11 years ago
Matt T. Proud	972e856d9b	Kill the curation state channel. The use of the channels for curation state were always unidiomatic. Change-Id: I1cb1d7175ebfb4faf28dff84201066278d6a0d92	11 years ago
Matt T. Proud	1ceb25b701	Publication of LevelDBMetricPersistence Fields. This will enable us to break down the onerous construction method. Change-Id: Ia89337ba39d6745af6757180af2485ec8a990a3b	11 years ago
Julius Volz	0003027dce	Add needed trailing spaces in logs.	11 years ago
Julius Volz	aa5d251f8d	Use github.com/golang/glog for all logging.	11 years ago
Matt T. Proud	a5141e4d0a	Depointerize storage conf. and chain ingester. The storage builders need to work with the assumption that they have a copy of the underlying configuration data if any mutations are made.	11 years ago
Matt T. Proud	820e551988	Code Review: Nits.	11 years ago
Matt T. Proud	a3bf2efdd5	Replace index writes with wrapped interface. This commit is the first of several and should not be regarded as the desired end state for these cleanups. What this one does it, however, is wrap the query index writing behind an interface type that can be injected into the storage stack and have its lifecycle managed separately as needed. It also would mean we can swap out underlying implementations to support remote indexing, buffering, no-op indexing very easily. In the future, most of the individual index interface members in the tiered storage will go away in favor of agents that can query and resolve what they need from the datastore without the user knowing how and why they work.	11 years ago
Matt T. Proud	52664f701a	Hot Fix: Use extracted time.	11 years ago
Matt T. Proud	38dac35b3e	Code Review: Short name consistency.	11 years ago
Matt T. Proud	a00f18d78b	Code Review: Manual re-alignment.	11 years ago
Matt T. Proud	cc989c68e1	Replace direct curation table access with wrapper.	11 years ago
Matt T. Proud	07ac921aec	Code Review: First pass.	11 years ago
Matt T. Proud	d8792cfd86	Extract HighWatermarking. Clean up the rest.	11 years ago
Matt T. Proud	f4669a812c	Extract index storage into separate types.	11 years ago
Matt T. Proud	772d3d6b11	Consolidate LevelDB storage construction. There are too many parameters to constructing a LevelDB storage instance for a construction method, so I've opted to take an idiomatic approach of embedding them in a struct for easier mediation and versioning.	11 years ago
Julius Volz	e3415e953f	Add notifications telemetry.	11 years ago
juliusv	927435d68e	Merge pull request #333 from prometheus/round-time Round time to nearest second in memory storage.	12 years ago
Julius Volz	5d88e8cc45	Round time to nearest second in memory storage. When samples get flushed to disk, they lose sub-second precision anyways. By already dropping sub-second precision, data fetched from memory vs. disk will behave the same. Later, we should consider also storing a more compact representation than time.Time in memory if we're not going to use its full precision.	12 years ago
Matt T. Proud	f7704af4f8	Code Review: Formatting comments.	12 years ago
Julius Volz	a76a797f3f	Always treat series without watermarks as too old. Current series always get watermarks written out upon append now. This drops support for old series without any watermarks by always reporting them as too old (stale) during queries.	12 years ago
Julius Volz	d2da21121c	Implement getValueRangeAtIntervalOp for faster range queries. This also short-circuits optimize() for now, since it is complex to implement for the new operator, and ops generated by the query layer already fulfill the needed invariants. We should still investigate later whether to completely delete operator optimization code or extend it to support getValueRangeAtIntervalOp operators.	12 years ago
Julius Volz	e7f049c85b	Fix expunging of empty memory series (loop var pointerization bug)	12 years ago
Julius Volz	baa5b07829	Fix condition for dropping empty memory series.	12 years ago

1 2 3 4 5 ...

278 Commits (d5ef0c64dc1befc88ee4ec0560aa7d5cefd5fc8d)