Commit Graph

42 Commits (b90c7a55da04231643895b192d15d499adbc480f)

Author SHA1 Message Date
Julien Pivotto e208afcc95
web: Remove APIv2 (#7935)
* web: Remove APIv2

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-09-15 09:30:55 +02:00
Julien Pivotto 6573bf42f2
web: Close and exhaust all response bodies in tests + cleanup snapshot directory (#7850)
* web: Close and exhaust all response bodies in tests

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* Remove empty snapshot directory

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

* Fix lint

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-08-29 11:22:00 +01:00
Li Zhijian 369245e606
cleanup one more tempfile (#7839)
Introduced by a recent PR(#7810)

Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
2020-08-25 18:13:48 +02:00
Li Zhijian 6f6ef1451e
Cleanup tempfiles (#7696)
* cleanup tempfiles for web_test

Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>

* cleanup tempfiles for api_test

Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>

* cleanup tempfiles for file_test

Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
2020-08-24 13:03:22 +05:30
Julien Pivotto e96d786fb8
Stop CMUX and GRPC servers even with stale connections (#7810)
I have received a recent report for a user which confirms that sometime
the GRPC server does not stop propery.

It appears that there are 2 issues:

1. The cmux server can refuse to stop if there are stale connections.
For that we set the ReadTimeout.
2. The GRPC server graceful stop can never finish.

What this PR avoids is:

```
goroutine 227 [semacquire, 2 minutes]:
sync.runtime_Semacquire(0xc00059a75c)
    /usr/local/go/src/runtime/sema.go:56 +0x42
sync.(*WaitGroup).Wait(0xc00059a75c)
    /usr/local/go/src/sync/waitgroup.go:130 +0x64
google.golang.org/grpc.(*Server).GracefulStop(0xc00059a600)
```

This PR stops the GRPC server after 15s. Related to the go routing dumps
in #6747.

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
2020-08-17 09:50:32 +01:00
Ganesh Vernekar d4b9fe801f
M-map full chunks of Head from disk (#6679)
When appending to the head and a chunk is full it is flushed to the disk and m-mapped (memory mapped) to free up memory

Prom startup now happens in these stages
 - Iterate the m-maped chunks from disk and keep a map of series reference to its slice of mmapped chunks.
- Iterate the WAL as usual. Whenever we create a new series, look for it's mmapped chunks in the map created before and add it to that series.

If a head chunk is corrupted the currpted one and all chunks after that are deleted and the data after the corruption is recovered from the existing WAL which means that a corruption in m-mapped files results in NO data loss.

[Mmaped chunks format](https://github.com/prometheus/prometheus/blob/master/tsdb/docs/format/head_chunks.md)  - main difference is that the chunk for mmaping now also includes series reference because there is no index for mapping series to chunks.
[The block chunks](https://github.com/prometheus/prometheus/blob/master/tsdb/docs/format/chunks.md) are accessed from the index which includes the offsets for the chunks in the chunks file - example - chunks of series ID have offsets 200, 500 etc in the chunk files.
In case of mmaped chunks, the offsets are stored in memory and accessed from that. During WAL replay, these offsets are restored by iterating all m-mapped chunks as stated above by matching the series id present in the chunk header and offset of that chunk in that file.

**Prombench results**

_WAL Replay_

1h Wal reply time
30% less wal reply time - 4m31 vs 3m36
2h Wal reply time
20% less wal reply time - 8m16 vs 7m

_Memory During WAL Replay_

High Churn:
10-15% less RAM -  32gb vs 28gb
20% less RAM after compaction 34gb vs 27gb
No Churn:
20-30% less RAM -  23gb vs 18gb
40% less RAM after compaction 32.5gb vs 20gb

Screenshots are in [this comment](https://github.com/prometheus/prometheus/pull/6679#issuecomment-621678932)


Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2020-05-06 21:00:00 +05:30
Ben Ye 1e4e37144d
Fixed wrongly handled not ready TSDB on web and API. (#7182)
* fix federate endpoint panic

Signed-off-by: yeya24 <yb532204897@gmail.com>

* Fixed all cases of not ready TSDB being wrongly handled.

* Fixed issue for federation.
* Ensured this will never happen again thanks to interfaces
* Fixes same issue for stats.
* Added tests for readiness.
* Fixed bug in stats. It was:
   status.MaxTime = db.Head().MaxTime()
   status.MinTime = db.Head().MaxTime()


Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed Brian's comments.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Addressed Brian's comments.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-04-29 17:16:14 +01:00
Ben Ye ecda6013ed
Use only local tsdb for federation (#7096)
Signed-off-by: yeya24 <yb532204897@gmail.com>
2020-04-07 16:42:42 +01:00
Bartlomiej Plotka a20bebf7eb Moved readyStorage to main.
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-02-17 18:03:57 +00:00
Bartlomiej Plotka 34426766d8 Unify Iterator interfaces. All point to storage now.
This is part of https://github.com/prometheus/prometheus/pull/5882 that can be done to simplify things.
All todos I added will be fixed in follow up PRs.

* querier.Querier, querier.Appender, querier.SeriesSet, and querier.Series interfaces merged
with storage interface.go. All imports that.
* querier.SeriesIterator replaced by chunkenc.Iterator
* Added chunkenc.Iterator.Seek method and tests for xor implementation (?)
* Since we properly handle SelectParams for Select methods I adjusted min max
based on that. This should help in terms of performance for queries with functions like offset.
* added Seek to deletedIterator and test.
* storage/tsdb was removed as it was only a unnecessary glue with incompatible structs.

No logic was changed, only different source of abstractions, so no need for benchmarks.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
2020-02-17 18:03:54 +00:00
Harkishen Singh 489a9aa7b9
Adds normalization of localhost urls in targets page react (#6794)
* support for globalurls in targets page react

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* fixed tests

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* removed fmts

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* implemented suggestions

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* formatted

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* implemented suggestions. fixed tests.

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* formated go code

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>

* implemented suggestions

Signed-off-by: Harkishen Singh <harkishensingh@hotmail.com>
2020-02-17 18:19:15 +01:00
Ganesh Vernekar 5ecef3542d
Cleanup after merging tsdb into prometheus
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
2019-08-13 14:04:14 +05:30
Simon Pasquier be67b8d460
web: fix flaky TestHTTPMetrics() (#5695)
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2019-06-24 15:48:15 +02:00
Simon Pasquier d7f38dfdde
web: add 'code' label to HTTP metrics (#5640)
* web: add prometheus_http_requests_total metrics

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

* Add unit test for requestCounter metric

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2019-06-19 17:56:04 +02:00
Minh-Long Do b26b5c9e96 Add rendering test of template based web endpoints (#5188)
Signed-off-by: Minh-Long  Do <minhlong.langos@gmail.com>
2019-02-08 10:17:47 +00:00
Simon Pasquier 6fa8de132b
web/v1/api: add tests for admin actions (#4767)
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2018-11-15 14:22:16 +01:00
Simon Pasquier 2d7f562ed6 web: fix asset paths for Windows platforms (#4616)
* web: fix asset paths for Windows platforms

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

* web: add tests

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2018-09-19 12:50:53 +05:30
Fabian Reinartz e7cfc7dae5 web: restore old path prefix behavior
Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-06-15 09:08:14 +01:00
Fabian Reinartz 8a4bda8d57 web: remove security headers
Signed-off-by: Fabian Reinartz <freinartz@google.com>
2018-06-12 08:59:13 -04:00
Eric V c207920500 Added HTTP security headers to web.go (#3583)
Signed-off-by: Eric Valenzuela <evalenzuela@thousandeyes.com>
2018-05-03 07:27:34 +01:00
beorn7 94ff07b81d Merge branch 'release-2.2'
Signed-off-by: beorn7 <beorn@soundcloud.com>
2018-04-10 16:50:35 +02:00
Ganesh Vernekar b44ce11d1b Added test to check pathPrefix 2018-03-30 11:55:54 +05:30
Anton Tereshchenkov 4cb8f6c260 web: remove unused MetricsPath option (#3964) 2018-03-21 09:29:40 +00:00
Krasi Georgiev 4be9a40b89 fatalf is not thread safe so using panic instead (#3680) 2018-01-13 10:37:38 +00:00
Krasi Georgiev c94fa731aa bypass the proxy for the tests 2017-12-20 18:21:10 +00:00
Krasi Georgiev 0e58cda1fe use Fatalf instead of panic 2017-12-19 12:55:43 +00:00
Krasi Georgiev 61d793cb1f handle web_test webhandler errors. 2017-12-19 12:55:43 +00:00
Krasi Georgiev aca8f85699 fixed the tests 2017-12-18 17:22:50 +00:00
Fabian Reinartz 53875da7fb Merge pull request #3286 from prometheus/dev-2.0
Dev 2.0
2017-10-12 16:19:10 +02:00
Brian Brazil 8740fcc657 Use panic in goroutine in test, not t.Fatalf 2017-10-09 19:44:03 +01:00
Julien Levesy 26a067b23d use testutil assertion helpers in web package testsuite 2017-10-05 22:40:10 +02:00
Fabian Reinartz 7a3261aa99 Merge pull request #3240 from Gouthamve/fix-admin-api
web/api: Fix typo which broke the 2.0 admin APIs.
2017-10-05 14:33:12 +02:00
Fabian Reinartz 2d0b8e8b94 Merge branch 'master' into dev-2.0 2017-10-05 13:09:18 +02:00
Goutham Veeramachaneni 35aaa2002d
web/api: Add tests for v2 admin API
Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>
2017-10-05 15:25:09 +05:30
Fabian Reinartz 7b02bfee0a web: start web handler while TSDB is starting up 2017-09-20 15:03:19 +02:00
Krasi Georgiev b4b0999e7f add prefix to the api when prometheus started with custom web.external-url
Signed-off-by: Krasi Georgiev <krasi.root@gmail.com>
2017-09-18 17:59:27 +03:00
Tom Wilkie bbc9671d50 Get profile handlers working again after #3054 and #3146.
Ensures the pprod endpoints deal with path-prefixes correctly; adds a test so we don't break it again.
2017-09-18 13:27:09 +01:00
Fabian Reinartz d21f149745 *: migrate to go-kit/log 2017-09-08 22:01:51 +05:30
Fabian Reinartz 25f3e1c424 Merge branch 'master' into mergemaster 2017-08-10 17:04:25 +02:00
emluque ff54c5c11a 2831 Add Healthy and Ready endpoints 2017-08-07 17:34:04 -03:00
Jonathan Boulle 38098f8c95 Add missing license headers
Prometheus is Apache 2 licensed, and most source files have the
appropriate copyright license header, but some were missing it without
apparent reason. Correct that by adding it.
2016-04-13 16:08:22 +02:00
Julius Volz 0f57b403a9 Add tests for "globalURL" template function. 2015-07-03 14:47:52 +02:00