acbc2b8cb6
storage: Fix float->uint conversions on some compilers.
...
See https://github.com/prometheus/prometheus/issues/887 , which will at
least be partially fixed by this.
From the spec https://golang.org/ref/spec#Conversions :
"In all non-constant conversions involving floating-point or complex
values, if the result type cannot represent the value the conversion
succeeds but the result value is implementation-dependent."
This ended up setting the converted values to 0 on Debian's Go 1.4.2
compiler, at least on 32-bit Debians.
2015-07-13 11:19:11 +02:00
8c196c1028
Minor doc fixes.
2015-06-23 17:07:18 +02:00
6bfb4549a6
storage: add LastSamplePairForFingerprint method
2015-06-23 13:45:15 +02:00
dc7d27ab9a
retrieval: add honor label handling and parametrized querying.
...
This commit adds the honor_labels and params arguments to the scrape
config. This allows to specify query parameters used by the scrapers
and handling scraped labels with precedence.
2015-06-23 13:45:14 +02:00
9016917d1c
Increment dirty counter only if setDirty(true) is called.
...
Currently, we increment the counter even if setDirty(false) is called,
which sets the storage clean.
2015-06-22 18:12:55 +02:00
1eff186555
Merge pull request #810 from prometheus/fabxc/lmatch
...
Match empty labels.
2015-06-22 15:45:50 +02:00
5b91ea9b36
storage: improve label matching and allow unset matching.
...
Matching of empty labels now also matches metrics where the label
was not explicitly set to the empty string.
2015-06-22 15:33:44 +02:00
46df1fd5ea
storage/local: add benchmark for label matching.
2015-06-22 15:33:44 +02:00
b105e26f4d
storage: remove global flags
2015-06-15 19:01:06 +02:00
5c6c0e2faa
Add storage method to delete time series
2015-06-01 21:23:32 +02:00
0de6edbdfc
Move pkg/ to util/
2015-06-01 21:12:32 +02:00
2317b001d0
Move flock package to pkg/flock
2015-06-01 21:12:31 +02:00
3c8fbf1e15
Move test package to pkg/testutil
2015-06-01 21:12:31 +02:00
aff01e29c3
Limit retrievable samples to retention window.
...
The storage does not delete data immediately after the retention period.
We don't want to retrieve this data as it causes artifacts.
2015-05-27 13:13:59 +02:00
a92134a947
Merge pull request #724 from prometheus/fabxc/storage-startup
...
Read from indexing queue during crash recovery.
2015-05-23 16:50:47 +02:00
6e319532cf
Read from indexing queue during crash recovery.
...
Change #704 introduced a regression that started reading the queue only
after potential crash recovery. When more than the queue capacity was
indexed, Prometheus deadlocked.
2015-05-23 15:32:35 +02:00
dbcb3d9333
Use an RW lock to checkpoint fingerprint mappings.
...
This has to be backported to 0.13.x.
2015-05-23 14:05:05 +02:00
3b9ab546e6
Add metrics to count inconsistencies and fp collisions.
2015-05-21 18:46:20 +02:00
c44e7cd105
Merge pull request #706 from prometheus/beorn7/persistence2
...
Improve iterator performance.
2015-05-21 13:48:52 +02:00
112a778922
Align int64s for atomic operations
2015-05-21 01:38:50 +02:00
3b9c421a69
Weed out all the [Gg]et* method names.
...
The only exception is getNumChunksToPersist to avoid naming the struct
member numChunksToPersist in a weird way.
2015-05-20 19:13:06 +02:00
267fd34156
Switch Prometheus to use github.com/prometheus/log.
...
This change is conceptually very simple, although the diff is large. It
switches logging from "github.com/golang/glog" to
"github.com/prometheus/log", while not actually changing any log
messages. V(1)-style logging has been changed to be log.Debug*().
2015-05-20 18:19:32 +02:00
81b190bf45
Remove locking from series iterator. Cache chunk iterators.
2015-05-20 16:19:34 +02:00
cd5574bf8a
Make chunk and series iterators more efficient.
2015-05-20 16:19:34 +02:00
f79c694be5
Add benchmarks for series iterator methods.
2015-05-20 16:19:34 +02:00
f59a449a24
Fix storage test
2015-05-20 16:12:07 +02:00
d8440d75f1
Do not start storage processing before Start() is called.
2015-05-19 13:51:45 +02:00
d1a93655a1
Fix typo.
2015-05-11 17:15:30 +02:00
7c6466d476
Reserve only ~1M FPs for the mapping.
...
That reduces the chance of having a fingerprint in the reserved area.
2015-05-08 18:10:56 +02:00
ac75dc2812
Avoid archive lookup for known mapped FPs.
2015-05-08 16:39:26 +02:00
ed810b45bf
Improvements after review.
2015-05-08 13:35:39 +02:00
c36e0e05f1
Add crash recovery of fingerprint mappings.
2015-05-07 18:58:14 +02:00
2235cec175
Handle fingerprint collisions.
2015-05-07 18:17:59 +02:00
9820e5fe99
Use FastFingerprint where appropriate.
2015-05-06 12:00:58 +02:00
e5f92d35fe
Fix storage/local tests for 32-bit systems
2015-04-30 14:19:48 -07:00
a052d32609
Comment improvement.
2015-04-14 10:49:43 +02:00
66fc61f9b7
Make bufPool a member of the persistence struct.
2015-04-14 10:43:09 +02:00
b02d900e61
Improve chunk and chunkDesc loading.
...
Also, clean up some things in the code (especially introduction of the
chunkLenWithHeader constant to avoid the same expression all over the place).
Benchmark results:
BEFORE
BenchmarkLoadChunksSequentially 5000 283580 ns/op 152143 B/op 312 allocs/op
BenchmarkLoadChunksRandomly 20000 82936 ns/op 39310 B/op 99 allocs/op
BenchmarkLoadChunkDescs 10000 110833 ns/op 15092 B/op 345 allocs/op
AFTER
BenchmarkLoadChunksSequentially 10000 146785 ns/op 152285 B/op 315 allocs/op
BenchmarkLoadChunksRandomly 20000 67598 ns/op 39438 B/op 103 allocs/op
BenchmarkLoadChunkDescs 20000 99631 ns/op 12636 B/op 192 allocs/op
Note that everything is obviously loaded from the page cache (as the
benchmark runs thousands of times with very small series files). In a
real-world scenario, I expect a larger impact, as the disk operations
will more often actually hit the disk. To load ~50 sequential chunks,
this reduces the iops from 100 seeks and 100 reads to 1 seek and 1
read.
2015-04-13 21:06:04 +02:00
c563398c68
Remove obsolete debug message.
2015-04-13 16:59:52 +02:00
c5fa0b90c3
Fix the case where a series in memory has 0 chunks, but chunks on disk.
...
This is actually completely normal for a freshly unarchived series.
Test added to expose.
2015-04-09 15:57:11 +02:00
3035b8bfdd
Adaptively reduce the wait time for memory series maintenance.
...
This will make in-memory series maintenance the faster the more chunks
are waiting for persistence.
2015-04-01 17:52:03 +02:00
fbc44d8f95
Add benchmark for loading chunks and chunk descs.
2015-03-19 19:28:21 +01:00
6a21f73898
Fixes after review.
2015-03-19 17:54:59 +01:00
51d35f4481
Instrument series maintenance durations.
2015-03-19 17:06:16 +01:00
12ae6e9203
Increase resilience of the storage against data corruption - step 4.
...
Step 4: Add a configurable sync'ing of series files after modification.
2015-03-19 15:58:02 +01:00
11bd9ce1bd
Increase resilience of the storage against data corruption - step 3.
...
Step 3: Remember the mtime of series files and make use of it to
detect series files that are not the one the checkpoint thinks they
are.
2015-03-19 15:44:11 +01:00
e25cca823c
Increase resilience of the storage against data corruption - step 2.
...
Step 2: Add a flag -storage.local.pedantic-checks to check every
series file.
Also, remove countPersistedHeadChunks channel, which is unused.
2015-03-19 12:06:15 +01:00
3d8d8928be
Increase resilience of the storage against data corruption - step 1.
...
Step 1: Admit the problem by turning the various "panic"s into logged
errors, followed by marking the persistence as dirty.
2015-03-19 11:49:18 +01:00
da7c0461c6
Rename persist queue len/cap to num/max chunks to persist.
...
Remove deprecated flag storage.incoming-samples-queue-capacity.
2015-03-18 19:36:41 +01:00
a075900f9a
Merge branch 'beorn7/persistence' into beorn7/ingestion-tweaks
2015-03-18 19:09:31 +01:00
Julius Volz