Commit Graph

59 Commits (16f4adbd65127e548d2d365c57f3af7d112045ca)

Author SHA1 Message Date
Bryan Boreham 6ca88a9b59
Merge pull request #14637 from prometheus/superq/update_storage_docs
Reword storage docs on corruption issues
2024-12-03 11:44:43 +00:00
Nathan Baulch 50cd453c8f
chore: Fix typos (#14868)
* Fix typos

---------

Signed-off-by: Nathan Baulch <nathan.baulch@gmail.com>
2024-09-10 22:32:03 +02:00
Jan Fajerski fa318711f4 Merge branch 'main' into 3.0-main-sync-24-09-09
Conflicts:
	cmd/prometheus/main.go
	docs/command-line/prometheus.md
	docs/feature_flags.md
	web/ui/build_ui.sh
	web/web.go
    Resolved by dropping the UTF-8 feature flag and adding the
    `auto-reload-config` feature flag.
    For the new web ui pick all changes from `main`.
2024-09-09 15:44:22 +02:00
Bartlomiej Plotka edc83ed164
Update storage.md with the latest around Remote Write 2.0 (#14848)
* Update storage.md with the latest around Remote Write 2.0

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Update storage.md

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

---------

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2024-09-06 13:50:10 +01:00
Julien Pivotto 7711cd5ab5 Remove deprecated storage.tsdb.retention flag
Signed-off-by: Julien Pivotto <roidelapluie@o11y.eu>
Signed-off-by: Julien <roidelapluie@o11y.eu>
2024-08-16 13:53:09 +02:00
SuperQ b7dd209663
Reword storage docs on corruption issues
Reword the section on what to do if major corruption happens. Users are
being confused by the existing wording and cherry-picking the meaning
from the single sentance about durability.

Signed-off-by: SuperQ <superq@gmail.com>
2024-08-09 10:48:55 +02:00
Sean Killeen f031fe1fa3
Update storage.md to provide right-sizing advice on retention
Signed-off-by: Sean Killeen <SeanKilleen@gmail.com>
2024-07-15 12:30:16 -04:00
anarcat 545d31f184
docs: clarify backup requirements for storage (#14297)
* clarify backup requirements for storage

After reading this (again) recently, I was under the impression that our backup strategy ("just throw Bacula at it") was just not good enough and that our backups were inconsistent. I filed [an issue internally][41627] about this because of that concern.

But reading a conversation with @SuperQ on IRC, I came under the impression that only the WAL files would be lost. This is an attempt at documenting this more clearly.

[41627]: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41627
---------

Signed-off-by: anarcat <anarcat@users.noreply.github.com>
Co-authored-by: Ben Kochie <superq@gmail.com>
2024-06-19 07:46:13 +02:00
beorn7 3127a4029e doc: Clarify the limits of dumping/backfilling via OpenMetrics
This is about native histograms (not yet supported) and staleness
markers (for which OpenMetrics support isn't even planned).

Signed-off-by: beorn7 <beorn@grafana.com>
2024-05-21 14:50:06 +02:00
Nishant Singh c8b23980c9
Update docs/storage.md
Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com>
Signed-off-by: Nishant Singh <nishant@heim.id>
2024-04-27 13:50:50 +05:30
Nishant Singh 801314901c
Update docs/storage.md
Co-authored-by: Ayoub Mrini <ayoubmrini424@gmail.com>
Signed-off-by: Nishant Singh <nishant@heim.id>
2024-04-27 13:50:41 +05:30
tesla59 5e638b7f44 docs: storage.md: clarify storage.tsdb.retention.time description
Signed-off-by: tesla59 <nishant@heim.id>
2024-04-24 02:58:25 +05:30
Michael 67294b0925 Tweak line wrappings in docs/storage.md
Signed-off-by: Michael <cloudyonspring@126.com>
2022-11-15 21:06:08 +08:00
Ganesh Vernekar f34aeefe6e
Allow overlapping blocks by default (#11331)
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2022-09-28 19:17:54 +05:30
Rodrigo Queiro 70c1446a64
Clarify units of --storage.tsdb.retention.size (#10154)
The flag uses Base2Bytes: 129ed4ec8b/cmd/prometheus/main.go (L1476)

Signed-off-by: Rodrigo Queiro <rodrigoq@google.com>
2022-01-13 00:55:57 +01:00
beorn7 8fdfa52976 API: Promote remote-write-receiver to stable
Since `/api/v1/write` is a mutating endpoint, we should still activate
the remote-write-receiver explicitly. But we should do it in the same
way as the other mutating endpoints, i.e. via a flag
`--web.enable-remote-write-receiver`.

This commit marks the feature flag as deprecated, i.e. it still works
but logs a warning on startup. This enables users to seamlessly
migrate. With the next minor release, we can start ignoring the
feature flag (but still warn a user that is trying to use it).

Signed-off-by: beorn7 <beorn@grafana.com>
2022-01-05 15:36:07 +01:00
Dean Meehan 5751bfb61f Removed Duplication Typo (to to)
Signed-off-by: Dean Meehan <dean@dean.technology>
2021-09-13 19:34:07 +01:00
Ganesh Vernekar 3d4c5f890d
Clarify in docs about the min disk space requirement when using size retention (#9245)
Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
2021-08-24 17:26:10 +05:30
Ankit Goel d437cee73a
Move storage.tsdb.retention.size out of experimental #8728 (#9004)
* Move storage.tsdb.retention.size out of experimental #8728

Signed-off-by: Ankit Goel <ankit.goel@deliveryhero.com>
2021-06-30 01:30:11 +02:00
Steve Kuznetsov fd6c852567
promtool: backfill: allow configuring block duration (#8919)
* promtool: backfill: allow configuring block duration

When backfilling large amounts of data across long periods of time, it
may in certain circumstances be useful to use a longer block duration to
increase the efficiency and speed of the backfilling process. This patch
adds a flag --block-duration-power to allow a user to choose the power N
where the block duration is 2^(N+1)h.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

* promtool: use sub-tests in backfill testing

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

* backfill: add messages to tests for clarity

When someone new breaks a test, seeing "expected: false, got: true" is
really not useful. A nice message helps here.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>

* backfill: test long block durations

A test that uses a long block duration to write bigger blocks is added.
The check to make sure all blocks are the default duration is removed.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
2021-06-29 14:53:38 +05:30
Julien Pivotto 2c9875d0e7 Storage: Create a code block for example command.
The current doc is broken:
https://prometheus.io/docs/prometheus/2.27/storage/

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-06-17 12:21:45 +02:00
Ikko Ashimine 446e5cc160 Fix typo in storage.md
mulitple -> multiple

Signed-off-by: Ikko Ashimine <eltociear@gmail.com>
2021-05-20 20:32:03 +09:00
nberkley f9e2dd0697
Add support for smaller block chunk segment allocations (#8478)
* Add support for --storage.tsdb.max-chunk-size to suport small chunks for space limited prometheus instances.

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

* Update tsdb/compact.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

* Update tsdb/db.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

* Update cmd/prometheus/main.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

* Change naming scheme to

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

* Add a lower bound to --storage.tsdb.max-block-chunk-segment-size

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

* Update storage.md to explain what a chunk segment is

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

* Apply suggestions from code review

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

* Force tests

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

* Fix code style

Signed-off-by: Nathan Berkley <nberkley@tripadvisor.com>

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2021-04-15 14:25:01 +05:30
Jess G 731545ad34
Add documentation for recording rule backfiller (#8674)
* add docs for rule backfiller

Signed-off-by: jessicagreben <jessicagrebens@gmail.com>
2021-04-01 22:38:00 +02:00
Julien Pivotto 432d5ebc6c Rename default branch to main
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2021-02-22 20:28:02 +01:00
Tom Wilkie d479151f1f Various enhancements and refactorings for remote write receiver:
- Remove unrelated changes
- Refactor code out of the API module - that is already getting pretty crowded.
- Don't track reference for AddFast in remote write.  This has the potential to consume unlimited server-side memory if a malicious client pushes a different label set for every series.  For now, its easier and safer to always use the 'slow' path.
- Return 400 on out of order samples.
- Use remote.DecodeWriteRequest in the remote write adapters.
- Put this behing the 'remote-write-server' feature flag
- Add some (very) basic docs.
- Used named return & add test for commit error propagation

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>
2021-02-08 20:41:23 +00:00
Atibhi Agrawal c08009139a
Add docs for backfill (#8230)
* Add docs for backfill

Signed-off-by: aSquare14 <atibhi.a@gmail.com>
2020-12-10 16:29:44 +01:00
Mike DeSimone 759150878c
fix trivial typos (#8238)
Signed-off-by: Mike DeSimone <michael.2.desimone@gmail.com>
2020-11-29 01:23:10 +00:00
Anthony D'Atri 2cbc0f9bfe
Various doc tweaks (#8111)
Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>
2020-10-27 09:50:37 +00:00
venkatbvc 6f28c46f4c
Updating storage.md for retention.size (#7942)
Signed-off-by: venkatbvc <chalapathirao@gmail.com>
2020-09-16 10:49:02 +05:30
showuon 791b84830d
Update the storage.md to fix typo, and some grammar/format issue (#7871)
* Update the storage.md to fix typo, and some grammar/format issue

Signed-off-by: Luke Chen <showuon@gmail.com>
2020-08-31 09:50:42 +01:00
Julien Pivotto 32c2d897ec
Storage: add chunks_head and fix links (#7693)
The links fix makes the links point to the current version of the files,
e.g. the relase branch.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-07-31 19:17:16 +02:00
Julien Pivotto 01e3bfcd1a
Add warnings about NFS (#7691)
* Add warnings about NFS

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-07-30 11:22:44 +02:00
Ben Kochie 8d3c2f6829
Enable WAL compression by default (#7410)
Enable the `--storage.tsdb.wal-compression` flag by defualt.

Signed-off-by: Ben Kochie <superq@gmail.com>
2020-06-18 17:59:40 +01:00
Arthur Silva Sens 7727b9012e
Correction of misleading help text(#5142) (#7231)
* Correction of misleading help text(#5142)

Signed-off-by: arthursens <arthursens2005@gmail.com>
2020-05-11 12:15:01 +01:00
Nguyen Hai Truong 26574eb04d
docs: Fix typo in docs/storage.md (#6799)
`availiablity` -> `availability`

Signed-off-by: Nguyen Hai Truong <truongnh@vn.fujitsu.com>
2020-02-10 18:32:58 +00:00
Stuart Spence 5ade524a1b grammar typo (#6617)
Signed-off-by: Stuart Spence <stuart.spence@canada.ca>
2020-01-13 22:14:52 +00:00
Kien Nguyen 34e2f15e18 Correct the default max block duratioh value
The default value is 31 days [1]

[1] https://github.com/prometheus/prometheus/blob/master/cmd/prometheus/main.go#L312

Signed-off-by: Kien Nguyen <kiennt2609@gmail.com>
2019-11-25 11:12:26 +07:00
yuxiaobo 7850f1b35c new world spelling mistake
Signed-off-by: yuxiaobo <yuxiaobogo@163.com>
2019-10-17 19:09:54 +08:00
Ben Kochie 16370e6880
Update storage considerations documentation (#6134)
* Improve clarity of internal storage durability.
* Add notes about remote storage.

Signed-off-by: Ben Kochie <superq@gmail.com>
2019-10-16 21:45:01 +02:00
abejidefemi1@gmail.com 5264b9628b fix punctuation mark
fix typo
Signed-off-by: abejidefemi1@gmail.com <abejidefemi1@gmail.com>
2019-10-01 09:02:11 +01:00
yuxiaobo96 5f6a56a0bb Third correction (#6012)
Signed-off-by: yuxiaobo <yuxiaobogo@163.com>
2019-09-20 10:48:56 +01:00
Ganesh Vernekar 5ecef3542d
Cleanup after merging tsdb into prometheus
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2019-08-13 14:04:14 +05:30
Ben Kochie ff40de7ca6 Add more details about retention to storage docs (#5842)
* Make compaction docs a little more clear, easy to find.
* Expand compaction docs slightly.
* Add notes about block cleanup to operational section.

Signed-off-by: Ben Kochie <superq@gmail.com>
2019-08-07 17:04:48 +01:00
Chris Marchbanks 08f6a43619
Add documentation for the WAL compression flag
Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>
2019-07-09 10:33:59 -06:00
Hrishikesh Barman 27404e6226 better readability for data directory tree in storage docs (#5128)
* better readabilty for data dir tree in storage docs

Signed-off-by: geekodour <hrishikeshbman@gmail.com>
2019-04-16 11:40:13 +01:00
Frederic Branczyk 32079f351f
docs: Specifically call out NFS and POSIX
Signed-off-by: Frederic Branczyk <fbranczyk@gmail.com>
2019-01-31 12:57:48 +01:00
Frederic Branczyk 3de734d8de
docs: Add filesystem POSIX requirement
Signed-off-by: Frederic Branczyk <fbranczyk@gmail.com>
2019-01-29 13:51:16 +01:00
Goutham Veeramachaneni 384cba1211
Add flag for size based retention (#5109)
* Add flag for size based retention

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Deprecate the old retention flag for a new one.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Add ability to take a suffix for size flag

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Address feedback

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
2019-01-18 19:18:36 +05:30
Daniel Lo Nigro edaa4835a2 Update WAL directory structure, and mention that the last three are always kept (#5096)
* Small tweaks to storage doc page

Signed-off-by: Daniel Lo Nigro <d@d.sb>
2019-01-15 10:32:29 +00:00