Commit Graph

1126 Commits (f3b2d2a99889257022de5923a070e770b8d41b02)

Author SHA1 Message Date
Bartlomiej Plotka fb79f515fc Fixed second bug.
5 years ago
Bartlomiej Plotka 2cf637fbf5 Addressed comments.
5 years ago
Bartlomiej Plotka 34426766d8 Unify Iterator interfaces. All point to storage now.
5 years ago
李国忠 3cd6a5b050
Storage concurrently tests and bug fix (#6808)
5 years ago
李国忠 40dd13b074
Storage concurrently (#6770)
5 years ago
helenxu1221 7df4fe3faa reset counter after collecting metric (#6798)
5 years ago
Robert Fratto a53e00f9fd
pass registerer from storage to queue manager for its metrics (#6728)
5 years ago
Peter Štibraný 08c5549055
Document that NewMergeSeriesSet expects individual sets to be sorted. (#6718)
5 years ago
Brian Brazil 38d32e0686 Don't sort postings if we only have one block.
5 years ago
Anand Singh Kunwar aa61e392b2 Make remote client `Store` use passed context (#6673)
5 years ago
Julien Pivotto cf42888e4d Fix order of testutil.Equals (#6695)
5 years ago
Julien Pivotto aad8f89ecb Remote storage: propagate json marshal errors (#6622)
5 years ago
Chris Marchbanks 7f3aca62c4 Only reduce the number of shards when caught up.
5 years ago
Chris Marchbanks 9e24e1f9e8 Use samplesPending rather than integral
5 years ago
Chris Marchbanks 847c66a843 Add sharding test
5 years ago
Josh Soref 91d76c8023 Spelling (#6517)
5 years ago
Julien Pivotto 31700a05df Improve testutil.ErrorEqual (#6471)
5 years ago
Callum Styan 67838643ee
Add config option for remote job name (#6043)
5 years ago
Garrett 5a9c4acfbf Pushdown aggregator group by through read hint (#6401)
5 years ago
Chris Marchbanks 5000c05378
Merge pull request #6378 from prometheus/accurate-desired-shards-metric
5 years ago
Callum Styan 5830e03691
Merge pull request #6337 from cstyan/rw-log-replay
5 years ago
Callum Styan 6a24eee340 Simplify duration check for watcher WAL replay.
5 years ago
Chris Marchbanks 6f34e35b3e
Record the exact value of desired shards in metric
5 years ago
Chris Marchbanks 0e684ca205
Fix unknown type in sharding up log
5 years ago
Callum Styan c2cb1e4103 Add a metric to track total bytes sent per remote write queue. (#6344)
5 years ago
Tom Wilkie de0a772b8e Port tsdb to use pkg/labels. (#6326)
5 years ago
Callum Styan 5f1be2cf45 Refactor calculateDesiredShards + don't reshard if we're having issues sending samples. (#6111)
5 years ago
Krasi Georgiev 81d284f806
Merge the 2.13 release branch to master (#6117)
5 years ago
Callum Styan 84ff928606 Make sure the remote write storage uses a dedupe logger. (#6113)
5 years ago
Chris Marchbanks 8df4bca470
Garbage collect asynchronously in the WAL Watcher
5 years ago
George Felix 895abbb7d0 Replaced test validations with testutils on storage/remote/codec_test.go (#6097)
5 years ago
Joe Elliott 95dc59ec7e Replaced t.Fatalf() with testutil.Assert() in buffer_test.go (#6084)
5 years ago
陈谭军 103f26d188 fix the wrong word (#6069)
5 years ago
Callum Styan 3344bb5c33 Move WAL watcher code to tsdb/wal package. (#5999)
5 years ago
Björn Rabenstein 3b3eaf3496
Merge pull request #5787 from cstyan/reshard-max-logging
5 years ago
Chris Marchbanks b4317768b9
Merge pull request #5849 from csmarchbanks/rw-use-labels
5 years ago
Yao Zengzeng f65b7c296d fix TODO: only stop & recreate remote write queues which have changes (#5540)
5 years ago
Chris Marchbanks 791a2409a2
Pre-allocate pendingSamples to reduce allocations
5 years ago
Chris Marchbanks 160186da18
Store labels.Labels instead of []prompb.Label
5 years ago
Stanislav Putrya 6141a8bd7c Show warnings in UI if query have returned some warnings (#5964)
5 years ago
Bartek Płotka 48b2c9c8ea
remote-read: streamed chunked server side; Extended protobuf; Added chunked, checksumed reader (#5703)
5 years ago
Julius Volz b5c833ca21
Update go.mod dependencies before release (#5883)
5 years ago
Bartek Płotka 32be514845
Merge pull request #5805 from codesome/merge-tsdb
5 years ago
Chris Marchbanks a6a55c433c Improve desired shards calculation (#5763)
5 years ago
Ganesh Vernekar 5ecef3542d
Cleanup after merging tsdb into prometheus
5 years ago
ethan 38ccf0157e cleanup: correct func name in log message (#5852)
5 years ago
Chris Marchbanks 529ccff07b
Remove all usages of stretchr/testify
5 years ago
Chris Marchbanks 0685eb5395
Refactor testutil.NewStorage into a new package
5 years ago
Vadym Martsynovskyy 8318aa2d5d Check for duplicate label names in remote read (#5829)
5 years ago
Callum Styan c40a83f386 Add metrics for max shards, min shards, and desired shards.
5 years ago
AllenZMC 758c71b980 fix word `encourter` to `encounter`
5 years ago
Devin Trejo d77f2aa29c Only check last directory when discovering checkpoint number (#5756)
5 years ago
Yao Zengzeng 3cde8a9941 pass error up if WALWathcer.segments() return err (#5741)
5 years ago
Xigang Wang 445bcd1251 Update the runShard method and change len(pendingSamples) to n=len(pendingSamples) (#5708)
5 years ago
Chris Marchbanks 06f1ba73eb
Provide flag to compress the tsdb WAL
5 years ago
Chris Marchbanks 475ca2ecd0
Update to tsdb 0.9.1
5 years ago
Chris Marchbanks 06bdaf076f Remote Write Allocs Improvements (#5614)
5 years ago
Chris Marchbanks a38a54fa11 Split remote write storage into its own type
5 years ago
Thomas Jackson 91d7175eaa Add storage.Warnings to LabelValues and LabelNames (#5673)
6 years ago
Dmitry Shmulevich 0c0638b080 resolve race condition in maxGauge (#5647)
6 years ago
Chris Marchbanks 840872a6f8 Fix remote storage config not updating correctly (#5555)
6 years ago
Simon Pasquier 45506841e6
*: enable all default linters (#5504)
6 years ago
Callum Styan 3639d51eb6 Remote Storage: string interner should not panic in release (#5487)
6 years ago
Callum Styan e87449b59d Remote Write: Queue Manager specific metrics shouldn't exist if the queue no longer exists (#5445)
6 years ago
Callum Styan b7538e7b49 Don't stop, recreate, and start remote storage QueueManagers if the (#5485)
6 years ago
Romain Baugue 95193fa027 Exhaust every request body before closing it (#5166) (#5479)
6 years ago
Vasily Sliouniaev 5be9a1426f Prevent reshard concurrent with calling stop (#5460)
6 years ago
Callum Styan c2b88992a3 Remote Write: fix checkpoint reading (#5429)
6 years ago
Tariq Ibrahim 8fdfa8abea refine error handling in prometheus (#5388)
6 years ago
Tom Wilkie 807fd33ecc Review feedback.
6 years ago
Callum Styan 1a7923dde3 Add ref counting to string interning so we can remove
6 years ago
Tom Wilkie cbf5f13285 Naive string iterning for labes & values in the remote_write path.
6 years ago
Tom Wilkie c7b3535997 Use pkg/relabelling in remote write.
6 years ago
Tom Wilkie 2fa93595d6
More WAL remote_write tweaks. (#5300)
6 years ago
Krasi Georgiev 1684dc750a
updated tsdb to 0.6.0 (#5292)
6 years ago
Tariq Ibrahim 1adb91738d fix typo in recordType method of wal_watcher.go (#5297)
6 years ago
Tariq Ibrahim ab8e9b7423 fix typo in queue_manager.go comment (#5294)
6 years ago
Tom Wilkie 67da8e7b46
Refactor and fix queue resharding (#5286)
6 years ago
Callum Styan b8106dd459 Review feedback:
6 years ago
Callum Styan 512f549064 Refactor: inline decodeRecord in readSegment and don't bother decoding samples records if we're not tailing the segment, add a benchmark test and fix some other tests
6 years ago
Tom Wilkie f795942572 Decrement pending sample when queue exits.
6 years ago
Tom Wilkie ee7efa93fe Fix some tests.
6 years ago
Callum Styan b69bdfb4d1 Store the checkpoint we read last, so that we don't keep reading the same checkpoint on each tick.
6 years ago
Tom Wilkie efbd9559f4 Deal with corruptions in the WAL:
6 years ago
Tom Wilkie adf5307470 Update wal LiveReader to ensure EOF is correctly propagated.
6 years ago
Callum Styan d6258aea8f Fix up remote write tests:
6 years ago
Tom Wilkie 184f06a981 Combine the record decoding metrics into one; break out garbage collection into a separate function.
6 years ago
Tom Wilkie 859cda27ff Remove some 'global' state, moving segment numbers to parameters.
6 years ago
Tom Wilkie bdc6b764b0 If reading the WAL fails, try again. Also, read from the segment containing the index for the last checkpoint, not the first segment.
6 years ago
Tom Wilkie d6f911b511 Factor out logging ratelimit & dedupe middleware.
6 years ago
Tom Wilkie a5c20642b3 Refactor WAL watcher to remove some duplication.
6 years ago
Tom Wilkie 37ad4db485 Export timestamps in seconds since epoch.
6 years ago
JoeWrightss 362873f72b Fix .Log() error message (#5257)
6 years ago
Simon Pasquier b41d6d54f2
storage/remote: increase timeouts for Travis CI (#5224)
6 years ago
Callum Styan 37e35f9e0c Various improvements to WAL based remote write.
6 years ago
Tom Wilkie b93bafeee1 Various fixes to locking & shutdown for WAL-based remote write.
6 years ago
Callum Styan 6f69e31398 Tail the TSDB WAL for remote_write
6 years ago
Goutham Veeramachaneni 384cba1211
Add flag for size based retention (#5109)
6 years ago
Krasi Georgiev 3bd41cc92c Udpate tsdb to 0.4 (#5110)
6 years ago
Matt Layher 302148fd69 *: apply gofmt -s
6 years ago
Callum Styan 5358f76c5c update remote write path proto so that Labels/Timeseries can't be nil (#4957)
6 years ago
Simon Pasquier f678e27eb6
*: use latest release of staticcheck (#5057)
6 years ago
glutamatt 5ddde1965b tune the "Wal segment size" with a flag (#5029)
6 years ago
Tom Wilkie 6e08029b56
Move err to be the last return value from storage.Select. (#5054)
6 years ago
AixesHunter 806632790e update inconsistent comment (#5046)
6 years ago
Bartek Płotka 62c8337e77 Moved configuration into `relabel` package. (#4955)
6 years ago
Alin Sinpalean 44bec482fb Minor optimization for BufferedSeriesIterator: actually drop the samples falling outside of the new delta from the underlying sampleRing, when ReduceDelta is called. (#4849)
6 years ago
Alin Sinpalean d6adfe2ae2 Use a fake SeriesIterator (that generates samples on the fly instead of using a slice) for BufferedSeriesIterator, to reduce the variance of benchmark results due to memory pressure. (#4847)
6 years ago
Ryota Arai 135d580ab2 Introduce min_shards for remote write to set minimum number of shards. (#4924)
6 years ago
mknapphrt f0e9196dca Return warnings on a remote read fail (#4832)
6 years ago
Ben Kochie c6399296dc
Fix spelling/typos (#4921)
6 years ago
Daniele Sluijters f25a6baedb remote: Set User-Agent header in requests (#4891)
6 years ago
Krasi Georgiev bd100182b2
added tsdb/head mint maxt metrics (#4888)
6 years ago
Simon Pasquier ed19373a78
*: remove use of golang.org/x/net/context (#4869)
6 years ago
Ganesh Vernekar ca93fd544b /api/v1/labels endpoint for getting all label names (#4835)
6 years ago
fengyuceNv 94fff219ab improve remote storage enqueue performance (#4772)
6 years ago
Tariq Ibrahim 3f7ed7de49 Adding new metric type to track in-flight remote read queries. (#4677)
6 years ago
Tom Wilkie d3a1ff1abf
Reduce memory usage of remote read by reducing pointer usage. (#4655)
6 years ago
yzpeninsula 4ae3bce260 Fix typo (#4497)
6 years ago
Tom Wilkie 457e4bb58e
Limit the number of samples remote read can return. (#4532)
6 years ago
Daisy T 7d01ead689 change time.duration to model.duration for standardization (#4479)
6 years ago
Julius Volz 8fbe1b5133
Handle a bunch of unchecked errors (#4461)
6 years ago
Henri DF ffb7836c14 Send "Accept-Encoding" header in read request (#4421)
6 years ago
Henri DF 3abb2cc349 Fix typo (#4423)
6 years ago
Alin Sinpalean 372e7652b7 Reuse (copy) overlapping matrix samples between range evaluation steps (#4315)
6 years ago
Goutham Veeramachaneni c28cc5076c Saner defaults and metrics for remote-write (#4279)
6 years ago
Alin Sinpalean e3b775b78b Simplify BufferedSeriesIterator usage (#4294)
6 years ago
Thomas Jackson 92c6f0c92e Add offset to selectParams (#4226)
6 years ago
Brian Brazil fb695fb435
Merge pull request #4285 from prometheus/release-2.3
7 years ago
Tom Wilkie b8217720ac Review feedback.
7 years ago
Corentin Chary db9dbeeaec federation: nil pointer deference when using remove read
7 years ago
Brian Brazil 78efdc6d6b
Avoid infinite loop on duplicate NaN values. (#4275)
7 years ago
Tom Wilkie 0b189b2da9 Review feedback.
7 years ago
Corentin Chary 530107f8ef federation: nil pointer deference when using remove read
7 years ago
Andreas Auernhammer 37d1bcf495 limit size of POST requests against remote read endpoint (#4239)
7 years ago
Fabian Reinartz fe80dddbc4
Merge pull request #4210 from bboreham/log-remote-name
7 years ago
Brian Brazil dd6781add2 Optimise PromQL (#3966)
7 years ago
Bryan Boreham 3277aeefaa Add queue name to logger for remote writes
7 years ago
Tom Wilkie b58199bf12 Review feedback.
7 years ago
Tom Wilkie 3353bbd018 Add proper unclean shutdown handling with a cancellable context.
7 years ago
Tom Wilkie e51d6c4b6c Make remote flush deadline a command line param.
7 years ago
Tom Wilkie a6c353613a Make the flush deadline configurable.
7 years ago
Tom Wilkie aa17263edd Remove WaitGroup and extra goroutine.
7 years ago
Tom Wilkie f3c61f8bb2 Only give remote queues 1 minute to flush samples on shutdown.
7 years ago
Tom Wilkie ba418780be Dedupe samples in the mergeIterator.
7 years ago
Henri DF 2952387ed1 Pass query hints down into remote read query proto (#4122)
7 years ago
Adam Shannon 809881d7f5 support reading basic_auth password_file for HTTP basic auth (#4077)
7 years ago
Mario Trangoni 464e747f1e fix some comments typos (#4059)
7 years ago
ferhat elmas ec8e4d8a7c all: remove unnecessary type conversions (#3992)
7 years ago
Tom Wilkie 02a154ced6
Merge pull request #3941 from prometheus/3809-correctly-stop-timer
7 years ago