prometheus

Commit Graph

Author	SHA1	Message	Date
Callum Styan	5f1be2cf45	Refactor calculateDesiredShards + don't reshard if we're having issues sending samples. (#6111 ) * Refactor calculateDesiredShards + don't reshard if we're having issues sending samples. * Track lastSendTimestamp via an int64 with atomic add/load, add a test for reshard calculation. * Simplify conditional for skipping resharding, add samplesIn/Out to shard testcase struct. Signed-off-by: Callum Styan <callumstyan@gmail.com>	5 years ago
Krasi Georgiev	81d284f806	Merge the 2.13 release branch to master (#6117 )	5 years ago
Callum Styan	84ff928606	Make sure the remote write storage uses a dedupe logger. (#6113 ) Signed-off-by: Callum Styan <callumstyan@gmail.com>	5 years ago
Chris Marchbanks	8df4bca470	Garbage collect asynchronously in the WAL Watcher The WAL Watcher replays a checkpoint after it is created in order to garbage collect series that no longer exist in the WAL. Currently the garbage collection process is done serially with reading from the tip of the WAL which can cause large delays in writing samples to remote storage just after compaction occurs. This also fixes a memory leak where dropped series are not cleaned up as part of the SeriesReset process. Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	5 years ago
George Felix	895abbb7d0	Replaced test validations with testutils on storage/remote/codec_test.go (#6097 ) * Replaced test validations with testutils on storage/remote/codec_test.go Signed-off-by: George Felix <george.felix@ubeeqo.com> * gofmt Signed-off-by: George Felix <george.felix@ubeeqo.com> * Removed shouldPass assertion Signed-off-by: George Felix <gfelixc@gmail.com> * Fixes to improve readability Signed-off-by: George Felix <george.felix@ubeeqo.com> * Fixes based on code review comments Signed-off-by: George Felix <george.felix@ubeeqo.com>	5 years ago
Joe Elliott	95dc59ec7e	Replaced t.Fatalf() with testutil.Assert() in buffer_test.go (#6084 ) * Added Fatal method and used it in buffer_test Signed-off-by: Joe Elliott <number101010@gmail.com> * Added period to meet contributing guidelines Signed-off-by: Joe Elliott <number101010@gmail.com> * Removed fatal testutil method. Refactored test cases to use testutil.Assert Signed-off-by: Joe Elliott <number101010@gmail.com> * Added if found condition for clarity Signed-off-by: Joe Elliott <number101010@gmail.com>	5 years ago
陈谭军	103f26d188	fix the wrong word (#6069 ) Signed-off-by: chentanjun <2799194073@qq.com>	5 years ago
Callum Styan	3344bb5c33	Move WAL watcher code to tsdb/wal package. (#5999 ) * Move WAL watcher code to tsdb/wal package. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Fix tests after moving WAL watcher code. Signed-off-by: Callum Styan <callumstyan@gmail.com> * Lint fixes. Signed-off-by: Callum Styan <callumstyan@gmail.com>	5 years ago
Björn Rabenstein	3b3eaf3496	Merge pull request #5787 from cstyan/reshard-max-logging Add metrics for max/min/desired shards to queue manager.	5 years ago
Chris Marchbanks	b4317768b9	Merge pull request #5849 from csmarchbanks/rw-use-labels Cache labels.Labels to Identify Series in Remote Write	5 years ago
Yao Zengzeng	f65b7c296d	fix TODO: only stop & recreate remote write queues which have changes (#5540 ) Signed-off-by: YaoZengzeng <yaozengzeng@zju.edu.cn>	5 years ago
Chris Marchbanks	791a2409a2	Pre-allocate pendingSamples to reduce allocations Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	5 years ago
Chris Marchbanks	160186da18	Store labels.Labels instead of []prompb.Label This will use half the steady state memory as required by prompb.Label. Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	5 years ago
Stanislav Putrya	6141a8bd7c	Show warnings in UI if query have returned some warnings (#5964 ) * Show warnings in UI if query have returned some warnings + improve warning (error) text if query to remote was finished with error * Add prefixes for remote_read errors Signed-off-by: Stan Putrya <root.vagner@gmail.com>	5 years ago
Bartek Płotka	48b2c9c8ea	remote-read: streamed chunked server side; Extended protobuf; Added chunked, checksumed reader (#5703 ) Part of: https://github.com/prometheus/prometheus/issues/4517 and https://github.com/improbable-eng/thanos/issues/488 Changes: * Extended protobuf for chunked remote read and negotation. * Added checksumed, chunked Writer/Reader. * Added Server side implementation for chunked streamed remote-read. Signed-off-by: Bartek Plotka <bwplotka@gmail.com>	5 years ago
Julius Volz	b5c833ca21	Update go.mod dependencies before release (#5883 ) * Update go.mod dependencies before release Signed-off-by: Julius Volz <julius.volz@gmail.com> * Add issue for showing query warnings in promtool Signed-off-by: Julius Volz <julius.volz@gmail.com> * Revert json-iterator back to 1.1.6 It produced errors when marshaling Point values with special float values. Signed-off-by: Julius Volz <julius.volz@gmail.com> * Fix expected step values in promtool tests after client_golang update Signed-off-by: Julius Volz <julius.volz@gmail.com> * Update generated protobuf code after proto dep updates Signed-off-by: Julius Volz <julius.volz@gmail.com>	5 years ago
Bartek Płotka	32be514845	Merge pull request #5805 from codesome/merge-tsdb Merge tsdb into prometheus	5 years ago
Chris Marchbanks	a6a55c433c	Improve desired shards calculation (#5763 ) The desired shards calculation now properly keeps track of the rate of pending samples, and uses the previously unused integralAccumulator to adjust for missing information in the desired shards calculation. Also, configure more capacity for each shard. The default 10 capacity causes shards to block on each other while sending remote requests. Default to a 500 sample capacity and explain in the documentation that having more capacity will help throughput. Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	5 years ago
Ganesh Vernekar	5ecef3542d	Cleanup after merging tsdb into prometheus Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>	5 years ago
ethan	38ccf0157e	cleanup: correct func name in log message (#5852 ) Signed-off-by: Guangming Wang <guangming.wang@daocloud.io>	5 years ago
Chris Marchbanks	529ccff07b	Remove all usages of stretchr/testify Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	5 years ago
Chris Marchbanks	0685eb5395	Refactor testutil.NewStorage into a new package This avoids a circular dependency between the testutil and storage packages. Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	5 years ago
Vadym Martsynovskyy	8318aa2d5d	Check for duplicate label names in remote read (#5829 ) * Check for duplicate label names in remote read Also add test to confirm that #5731 is fixed * Use subtests in TestValidateLabelsAndMetricName * Really check that expectedErr matches err Signed-off-by: Vadym Martsynovskyy <vmartsynovskyy@gmail.com>	5 years ago
Callum Styan	c40a83f386	Add metrics for max shards, min shards, and desired shards. Signed-off-by: Callum Styan <callumstyan@gmail.com>	5 years ago
AllenZMC	758c71b980	fix word `encourter` to `encounter` Signed-off-by: czm <zhongming.chang@daocloud.io>	5 years ago
Devin Trejo	d77f2aa29c	Only check last directory when discovering checkpoint number (#5756 ) * Only check last directory when discovering checkpoint number Signed-off-by: Devin Trejo <dtrejo@palantir.com> * Comments for checkpointNum Signed-off-by: Devin Trejo <dtrejo@palantir.com>	5 years ago
Yao Zengzeng	3cde8a9941	pass error up if WALWathcer.segments() return err (#5741 ) Signed-off-by: YaoZengzeng <yaozengzeng@zju.edu.cn>	5 years ago
Xigang Wang	445bcd1251	Update the runShard method and change len(pendingSamples) to n=len(pendingSamples) (#5708 ) Signed-off-by: xigang <wangxigang2014@gmail.com>	5 years ago
Chris Marchbanks	06f1ba73eb	Provide flag to compress the tsdb WAL Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	5 years ago
Chris Marchbanks	475ca2ecd0	Update to tsdb 0.9.1 Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	5 years ago
Chris Marchbanks	06bdaf076f	Remote Write Allocs Improvements (#5614 ) * Add benchmark for sample delivery * Simplify StoreSeries to have only one loop * Reduce allocations for pending samples in runShard * Only allocate one send slice per segment * Cache a buffer in each shard for snappy to use * Remove queue manager seriesMtx It is not possible for any of the places protected by the seriesMtx to be called concurrently so it is safe to remove. By removing the mutex we can simplify the Append code to one loop. Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	5 years ago
Chris Marchbanks	a38a54fa11	Split remote write storage into its own type This allows other processes to reuse just the remote write code without having to use the remote read code as well. This will be used to create a sidecar capable of sending remote write payloads. Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	5 years ago
Thomas Jackson	91d7175eaa	Add storage.Warnings to LabelValues and LabelNames (#5673 ) Fixes #5661 Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com>	6 years ago
Dmitry Shmulevich	0c0638b080	resolve race condition in maxGauge (#5647 ) * resolve race condition in maxGauge Signed-off-by: Dmitry Shmulevich <dmitry.shmulevich@sysdig.com>	6 years ago
Chris Marchbanks	840872a6f8	Fix remote storage config not updating correctly (#5555 ) * Update remote write and remote read separately * Add external labels to the remote write conf hash * Add unit tests for remote storage lifecycle Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>	6 years ago
Simon Pasquier	45506841e6	*: enable all default linters (#5504 ) Signed-off-by: Simon Pasquier <spasquie@redhat.com>	6 years ago
Callum Styan	3639d51eb6	Remote Storage: string interner should not panic in release (#5487 ) * Don't panic if we try to release a string that is not in the interner. * Move seriesMtx locking in QueueManager's StoreSeries function. This stops us from calling release for strings that aren't interned if there's a race between reading a checkpoint and storing new series labels, which could happen during checkpointing or reloading config. Signed-off-by: Callum Styan <callumstyan@gmail.com>	6 years ago
Callum Styan	e87449b59d	Remote Write: Queue Manager specific metrics shouldn't exist if the queue no longer exists (#5445 ) * Unregister remote write queue manager specific metrics when stopping the queue manager. * Use DeleteLabelValues instead of Unregister to remove queue and watcher related metrics when we stop them. Create those metrics in the structs start functions rather than in their constructors because of the ordering of creation, start, and stop in remote storage ApplyConfig. * Add setMetrics function to WAL watcher so we can set the watchers metrics in it's Start function, but not have to call Start in some tests (causes data race). Signed-off-by: Callum Styan <callumstyan@gmail.com>	6 years ago
Callum Styan	b7538e7b49	Don't stop, recreate, and start remote storage QueueManagers if the (#5485 ) remote write config hasn't changed at all. Signed-off-by: Callum Styan <callumstyan@gmail.com>	6 years ago
Romain Baugue	95193fa027	Exhaust every request body before closing it (#5166 ) (#5479 ) From the documentation: > The default HTTP client's Transport may not > reuse HTTP/1.x "keep-alive" TCP connections if the Body is > not read to completion and closed. This effectively enable keep-alive for the fixed requests. Signed-off-by: Romain Baugue <romain.baugue@elwinar.com>	6 years ago
Vasily Sliouniaev	5be9a1426f	Prevent reshard concurrent with calling stop (#5460 ) * Prevent reshard concurrent with calling stop Signed-off-by: Vasily <v.sliouniaev@gmail.com>	6 years ago
Callum Styan	c2b88992a3	Remote Write: fix checkpoint reading (#5429 ) * Fix ReadCheckpoint to ensure that it actually reads all the contents of each segment in a checkpoint dir, or returns an error. Signed-off-by: Callum Styan <callumstyan@gmail.com>	6 years ago
Tariq Ibrahim	8fdfa8abea	refine error handling in prometheus (#5388 ) i) Uses the more idiomatic Wrap and Wrapf methods for creating nested errors. ii) Fixes some incorrect usages of fmt.Errorf where the error messages don't have any formatting directives. iii) Does away with the use of fmt package for errors in favour of pkg/errors Signed-off-by: tariqibrahim <tariq181290@gmail.com>	6 years ago
Tom Wilkie	807fd33ecc	Review feedback. - Update read path to use labels.Labels. - Fix the tests. - Remove pack. - Remove unused function. - Fix race in tests. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Callum Styan	1a7923dde3	Add ref counting to string interning so we can remove a string when there are no longer any refs. Add tests for interning. Co-authored-by: Tom Wilkie <tom.wilkie@gmail.com> Signed-off-by: Callum Styan <callumstyan@gmail.com>	6 years ago
Tom Wilkie	cbf5f13285	Naive string iterning for labes & values in the remote_write path. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Tom Wilkie	c7b3535997	Use pkg/relabelling in remote write. - Unmarshall external_labels config as labels.Labels, add tests. - Convert some more uses of model.LabelSet to labels.Labels. - Remove old relabel pkg (fixes #3647). - Validate external label names. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Tom Wilkie	2fa93595d6	More WAL remote_write tweaks. (#5300 ) * Consistently pre-lookup the metrics for a given queue in queue manager. * Don't open the WAL (for writing) in the remote_write code. * Add some more logging. Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>	6 years ago
Krasi Georgiev	1684dc750a	updated tsdb to 0.6.0 (#5292 ) * updated tsdb to 0.6.0 as part of the update also added the new storage.tsdb.allow-overlapping-blocks flag and mark it as experimental.	6 years ago
Tariq Ibrahim	1adb91738d	fix typo in recordType method of wal_watcher.go (#5297 ) Signed-off-by: tariqibrahim <tariq181290@gmail.com>	6 years ago

1 2 3 4 5 ...

1000 Commits (8cf054849f6adc3718c8081ad0ffd91d1cfac023)