Commit Graph

1000 Commits (1a38075f83831cb26162dd521b22b2da05bbe6e4)

Author SHA1 Message Date
Callum Styan 5f1be2cf45 Refactor calculateDesiredShards + don't reshard if we're having issues sending samples. (#6111)
* Refactor calculateDesiredShards + don't reshard if we're having issues
sending samples.
* Track lastSendTimestamp via an int64 with atomic add/load, add a test
for reshard calculation.
* Simplify conditional for skipping resharding, add samplesIn/Out to shard
testcase struct.

Signed-off-by: Callum Styan <callumstyan@gmail.com>
2019-10-21 15:54:25 -06:00
Krasi Georgiev 81d284f806
Merge the 2.13 release branch to master (#6117) 2019-10-09 17:41:46 +02:00
Callum Styan 84ff928606 Make sure the remote write storage uses a dedupe logger. (#6113)
Signed-off-by: Callum Styan <callumstyan@gmail.com>
2019-10-08 11:42:00 -06:00
Chris Marchbanks 8df4bca470
Garbage collect asynchronously in the WAL Watcher
The WAL Watcher replays a checkpoint after it is created in order to
garbage collect series that no longer exist in the WAL. Currently the
garbage collection process is done serially with reading from the tip of
the WAL which can cause large delays in writing samples to remote
storage just after compaction occurs.

This also fixes a memory leak where dropped series are not cleaned up as
part of the SeriesReset process.

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2019-10-07 14:36:10 -06:00
George Felix 895abbb7d0 Replaced test validations with testutils on storage/remote/codec_test.go (#6097)
* Replaced test validations with testutils on storage/remote/codec_test.go

Signed-off-by: George Felix <george.felix@ubeeqo.com>

* gofmt

Signed-off-by: George Felix <george.felix@ubeeqo.com>

* Removed shouldPass assertion

Signed-off-by: George Felix <gfelixc@gmail.com>

* Fixes to improve readability

Signed-off-by: George Felix <george.felix@ubeeqo.com>

* Fixes based on code review comments

Signed-off-by: George Felix <george.felix@ubeeqo.com>
2019-10-07 11:35:53 -06:00
Joe Elliott 95dc59ec7e Replaced t.Fatalf() with testutil.Assert() in buffer_test.go (#6084)
* Added Fatal method and used it in buffer_test

Signed-off-by: Joe Elliott <number101010@gmail.com>

* Added period to meet contributing guidelines

Signed-off-by: Joe Elliott <number101010@gmail.com>

* Removed fatal testutil method.  Refactored test cases to use testutil.Assert

Signed-off-by: Joe Elliott <number101010@gmail.com>

* Added if found condition for clarity

Signed-off-by: Joe Elliott <number101010@gmail.com>
2019-10-02 06:28:08 +01:00
陈谭军 103f26d188 fix the wrong word (#6069)
Signed-off-by: chentanjun <2799194073@qq.com>
2019-09-30 09:54:55 -06:00
Callum Styan 3344bb5c33 Move WAL watcher code to tsdb/wal package. (#5999)
* Move WAL watcher code to tsdb/wal package.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Fix tests after moving WAL watcher code.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Lint fixes.

Signed-off-by: Callum Styan <callumstyan@gmail.com>
2019-09-19 14:45:41 +05:30
Björn Rabenstein 3b3eaf3496
Merge pull request #5787 from cstyan/reshard-max-logging
Add metrics for max/min/desired shards to queue manager.
2019-09-09 22:32:54 +02:00
Chris Marchbanks b4317768b9
Merge pull request #5849 from csmarchbanks/rw-use-labels
Cache labels.Labels to Identify Series in Remote Write
2019-09-04 14:35:52 -06:00
Yao Zengzeng f65b7c296d fix TODO: only stop & recreate remote write queues which have changes (#5540)
Signed-off-by: YaoZengzeng <yaozengzeng@zju.edu.cn>
2019-09-04 11:21:53 -06:00
Chris Marchbanks 791a2409a2
Pre-allocate pendingSamples to reduce allocations
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2019-09-03 15:41:47 -06:00
Chris Marchbanks 160186da18
Store labels.Labels instead of []prompb.Label
This will use half the steady state memory as required by prompb.Label.

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2019-09-03 15:41:46 -06:00
Stanislav Putrya 6141a8bd7c Show warnings in UI if query have returned some warnings (#5964)
* Show warnings in UI if query have returned some warnings
+ improve warning (error) text if query to remote was finished with error
* Add prefixes for remote_read errors

Signed-off-by: Stan Putrya <root.vagner@gmail.com>
2019-08-28 14:25:28 +01:00
Bartek Płotka 48b2c9c8ea
remote-read: streamed chunked server side; Extended protobuf; Added chunked, checksumed reader (#5703)
Part of: https://github.com/prometheus/prometheus/issues/4517 and https://github.com/improbable-eng/thanos/issues/488

Changes:
* Extended protobuf for chunked remote read and negotation.
* Added checksumed, chunked Writer/Reader.
* Added Server side implementation for chunked streamed remote-read.


Signed-off-by: Bartek Plotka <bwplotka@gmail.com>
2019-08-19 21:16:10 +01:00
Julius Volz b5c833ca21
Update go.mod dependencies before release (#5883)
* Update go.mod dependencies before release

Signed-off-by: Julius Volz <julius.volz@gmail.com>

* Add issue for showing query warnings in promtool

Signed-off-by: Julius Volz <julius.volz@gmail.com>

* Revert json-iterator back to 1.1.6

It produced errors when marshaling Point values with special float
values.

Signed-off-by: Julius Volz <julius.volz@gmail.com>

* Fix expected step values in promtool tests after client_golang update

Signed-off-by: Julius Volz <julius.volz@gmail.com>

* Update generated protobuf code after proto dep updates

Signed-off-by: Julius Volz <julius.volz@gmail.com>
2019-08-14 11:00:39 +02:00
Bartek Płotka 32be514845
Merge pull request #5805 from codesome/merge-tsdb
Merge tsdb into prometheus
2019-08-13 11:39:41 +01:00
Chris Marchbanks a6a55c433c Improve desired shards calculation (#5763)
The desired shards calculation now properly keeps track of the rate of
pending samples, and uses the previously unused integralAccumulator to
adjust for missing information in the desired shards calculation.

Also, configure more capacity for each shard.  The default 10 capacity
causes shards to block on each other while
sending remote requests. Default to a 500 sample capacity and explain in
the documentation that having more capacity will help throughput.

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2019-08-13 10:10:21 +01:00
Ganesh Vernekar 5ecef3542d
Cleanup after merging tsdb into prometheus
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2019-08-13 14:04:14 +05:30
ethan 38ccf0157e cleanup: correct func name in log message (#5852)
Signed-off-by: Guangming Wang <guangming.wang@daocloud.io>
2019-08-10 16:24:58 +01:00
Chris Marchbanks 529ccff07b
Remove all usages of stretchr/testify
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2019-08-08 19:49:27 -06:00
Chris Marchbanks 0685eb5395
Refactor testutil.NewStorage into a new package
This avoids a circular dependency between the testutil and storage
packages.

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2019-08-08 19:43:04 -06:00
Vadym Martsynovskyy 8318aa2d5d Check for duplicate label names in remote read (#5829)
* Check for duplicate label names in remote read

Also add test to confirm that #5731 is fixed

* Use subtests in TestValidateLabelsAndMetricName

* Really check that expectedErr matches err

Signed-off-by: Vadym Martsynovskyy <vmartsynovskyy@gmail.com>
2019-08-07 16:13:10 +01:00
Callum Styan c40a83f386 Add metrics for max shards, min shards, and desired shards.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
2019-08-04 20:04:19 -07:00
AllenZMC 758c71b980 fix word `encourter` to `encounter`
Signed-off-by: czm <zhongming.chang@daocloud.io>
2019-07-29 22:16:23 +08:00
Devin Trejo d77f2aa29c Only check last directory when discovering checkpoint number (#5756)
* Only check last directory when discovering checkpoint number

Signed-off-by: Devin Trejo <dtrejo@palantir.com>

* Comments for checkpointNum

Signed-off-by: Devin Trejo <dtrejo@palantir.com>
2019-07-15 17:53:58 +01:00
Yao Zengzeng 3cde8a9941 pass error up if WALWathcer.segments() return err (#5741)
Signed-off-by: YaoZengzeng <yaozengzeng@zju.edu.cn>
2019-07-15 17:52:03 +01:00
Xigang Wang 445bcd1251 Update the runShard method and change len(pendingSamples) to n=len(pendingSamples) (#5708)
Signed-off-by: xigang <wangxigang2014@gmail.com>
2019-07-09 19:09:11 +01:00
Chris Marchbanks 06f1ba73eb
Provide flag to compress the tsdb WAL
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2019-07-03 08:03:29 -06:00
Chris Marchbanks 475ca2ecd0
Update to tsdb 0.9.1
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2019-07-03 07:51:11 -06:00
Chris Marchbanks 06bdaf076f Remote Write Allocs Improvements (#5614)
* Add benchmark for sample delivery
* Simplify StoreSeries to have only one loop
* Reduce allocations for pending samples in runShard
* Only allocate one send slice per segment
* Cache a buffer in each shard for snappy to use
* Remove queue manager seriesMtx

  It is not possible for any of the places protected by the seriesMtx to
  be called concurrently so it is safe to remove. By removing the mutex we
  can simplify the Append code to one loop.

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2019-06-27 19:48:21 +01:00
Chris Marchbanks a38a54fa11 Split remote write storage into its own type
This allows other processes to reuse just the remote write code without
having to use the remote read code as well. This will be used to create
a sidecar capable of sending remote write payloads.

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2019-06-27 10:11:02 +01:00
Thomas Jackson 91d7175eaa Add storage.Warnings to LabelValues and LabelNames (#5673)
Fixes #5661

Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>
2019-06-17 08:31:17 +01:00
Dmitry Shmulevich 0c0638b080 resolve race condition in maxGauge (#5647)
* resolve race condition in maxGauge

Signed-off-by: Dmitry Shmulevich <dmitry.shmulevich@sysdig.com>
2019-06-13 20:55:08 +01:00
Chris Marchbanks 840872a6f8 Fix remote storage config not updating correctly (#5555)
* Update remote write and remote read separately
* Add external labels to the remote write conf hash
* Add unit tests for remote storage lifecycle

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2019-05-17 10:29:49 +01:00
Simon Pasquier 45506841e6
*: enable all default linters (#5504)
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2019-05-03 15:11:28 +02:00
Callum Styan 3639d51eb6 Remote Storage: string interner should not panic in release (#5487)
* Don't panic if we try to release a string that is not in the interner.

* Move seriesMtx locking in QueueManager's StoreSeries function.

This stops us from calling release for strings that aren't interned if
there's a race between reading a checkpoint and storing new series
labels, which could happen during checkpointing or reloading config.

Signed-off-by: Callum Styan <callumstyan@gmail.com>
2019-04-24 10:46:31 +01:00
Callum Styan e87449b59d Remote Write: Queue Manager specific metrics shouldn't exist if the queue no longer exists (#5445)
* Unregister remote write queue manager specific metrics when stopping the
queue manager.

* Use DeleteLabelValues instead of Unregister to remove queue and watcher
related metrics when we stop them. Create those metrics in the structs
start functions rather than in their constructors because of the
ordering of creation, start, and stop in remote storage ApplyConfig.

* Add setMetrics function to WAL watcher so we can set
the watchers metrics in it's Start function, but not
have to call Start in some tests (causes data race).

Signed-off-by: Callum Styan <callumstyan@gmail.com>
2019-04-23 09:49:17 +01:00
Callum Styan b7538e7b49 Don't stop, recreate, and start remote storage QueueManagers if the (#5485)
remote write config hasn't changed at all.

Signed-off-by: Callum Styan <callumstyan@gmail.com>
2019-04-23 09:47:18 +01:00
Romain Baugue 95193fa027 Exhaust every request body before closing it (#5166) (#5479)
From the documentation:
> The default HTTP client's Transport may not
> reuse HTTP/1.x "keep-alive" TCP connections if the Body is
> not read to completion and closed.

This effectively enable keep-alive for the fixed requests.

Signed-off-by: Romain Baugue <romain.baugue@elwinar.com>
2019-04-18 09:50:37 +01:00
Vasily Sliouniaev 5be9a1426f Prevent reshard concurrent with calling stop (#5460)
* Prevent reshard concurrent with calling stop

Signed-off-by: Vasily <v.sliouniaev@gmail.com>
2019-04-16 11:25:19 +01:00
Callum Styan c2b88992a3 Remote Write: fix checkpoint reading (#5429)
* Fix ReadCheckpoint to ensure that it actually reads all the contents of
each segment in a checkpoint dir, or returns an error.

Signed-off-by: Callum Styan <callumstyan@gmail.com>
2019-04-09 10:52:44 +01:00
Tariq Ibrahim 8fdfa8abea refine error handling in prometheus (#5388)
i) Uses the more idiomatic Wrap and Wrapf methods for creating nested errors.
ii) Fixes some incorrect usages of fmt.Errorf where the error messages don't have any formatting directives.
iii) Does away with the use of fmt package for errors in favour of pkg/errors

Signed-off-by: tariqibrahim <tariq181290@gmail.com>
2019-03-26 00:01:12 +01:00
Tom Wilkie 807fd33ecc Review feedback.
- Update read path to use labels.Labels.
- Fix the tests.
- Remove pack.
- Remove unused function.
- Fix race in tests.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
2019-03-18 20:31:12 +00:00
Callum Styan 1a7923dde3 Add ref counting to string interning so we can remove
a string when there are no longer any refs. Add tests for interning.

Co-authored-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Callum Styan <callumstyan@gmail.com>
2019-03-18 20:31:12 +00:00
Tom Wilkie cbf5f13285 Naive string iterning for labes & values in the remote_write path.
Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
2019-03-18 20:31:12 +00:00
Tom Wilkie c7b3535997 Use pkg/relabelling in remote write.
- Unmarshall external_labels config as labels.Labels, add tests.
- Convert some more uses of model.LabelSet to labels.Labels.
- Remove old relabel pkg (fixes #3647).
- Validate external label names.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
2019-03-18 20:31:12 +00:00
Tom Wilkie 2fa93595d6
More WAL remote_write tweaks. (#5300)
* Consistently pre-lookup the metrics for a given queue in queue manager.
* Don't open the WAL (for writing) in the remote_write code.
* Add some more logging.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
2019-03-05 12:21:11 +00:00
Krasi Georgiev 1684dc750a
updated tsdb to 0.6.0 (#5292)
* updated tsdb to 0.6.0

as part of the update also added the new storage.tsdb.allow-overlapping-blocks flag and mark it as experimental.
2019-03-04 21:42:45 +02:00
Tariq Ibrahim 1adb91738d fix typo in recordType method of wal_watcher.go (#5297)
Signed-off-by: tariqibrahim <tariq181290@gmail.com>
2019-03-04 17:33:35 +01:00