2607 Commits (fc7de5374a2c3f4f07a24ba56ae3e3d3f6fa7c7d)

 

All Branches   

Search

Author	SHA1	Message	Date
beorn7	fc7de5374a	Quarantine series upon problem writing to the series file ... This fixes https://github.com/prometheus/prometheus/issues/1059 , but not in the obvious way (simply not updating the persist watermark, because that's actually not that simple - we don't really know what has gone wrong exactly). As any errors relevant here are most likely caused by severe and unrecoverable problems with the series file, Using the now quarantine feature is the right step. We don't really have to be worried about any inconsistent state of the series because it will be removed for good ASAP. Another plus is that we don't have to declare the whole storage dirty anymore.	9 years ago
beorn7	0ea5801e47	Handle errors caused by data corruption more gracefully ... This requires all the panic calls upon unexpected data to be converted into errors returned. This pollute the function signatures quite lot. Well, this is Go... The ideas behind this are the following: - panic only if it's a programming error. Data corruptions happen, and they are not programming errors. - If we detect a data corruption, we "quarantine" the series, essentially removing it from the database and putting its data into a separate directory for forensics. - Failure during writing to a series file is not considered corruption automatically. It will call setDirty, though, so that a crashrecovery upon the next restart will commence and check for that. - Series quarantining and setDirty calls are logged and counted in metrics, but are hidden from the user of the interfaces in interface.go, whith the notable exception of Append(). The reasoning is that we treat corruption by removing the corrupted series, i.e. a query for it will return no results on its next call anyway, so return no results right now. In the case of Append(), we want to tell the user that no data has been appended, though. Minor side effects: - Now consistently using filepath.* instead of path.*. - Introduced structured logging where I touched it. This makes things less consistent, but a complete change to structured logging would be out of scope for this PR.	9 years ago
beorn7	8766f99085	Merge branch 'beorn7/storage2' into beorn7/storage3	9 years ago
beorn7	162f6fa6f6	Merge branch 'beorn7/storage' into beorn7/storage2	9 years ago
beorn7	79a2ae2d2e	Add missing test file	9 years ago
beorn7	b6840997a7	Merge branch 'beorn7/storage2' into beorn7/storage3	9 years ago
beorn7	ce58fd357b	Merge branch 'beorn7/storage' into beorn7/storage2 ... Conflicts: storage/local/chunk.go storage/local/interface.go	9 years ago
beorn7	2581648f70	Separate iterators by offset ... Add test that exposes the problem.	9 years ago
beorn7	536e0bc86b	Merge branch 'beorn7/fix-deadlock' into beorn7/storage3	9 years ago
beorn7	33a50e69f7	Fix a deadlock ... Double acquisition of the RLock usually doesn't blow up, but if the write lock is called for between the two RLock's, we are deadlocked. This deadlock does not exist in release-0.17, BTW.	9 years ago
beorn7	c740789ce3	Improve predict_linear ... Fixes https://github.com/prometheus/prometheus/issues/1401 This remove the last (and in fact bogus) use of BoundaryValues. Thus, a whole lot of unused (and arguably sub-optimal / ugly) code can be removed here, too.	9 years ago
Julius Volz	73399f826a	Merge pull request #1427 from prometheus/fix-scrape-timeout ... Remove invalid scrape timeout from example config.	9 years ago
Julius Volz	657d65d6d6	Remove invalid scrape timeout from example config. ... It can't be greater than the scrape interval. Let's just remove it.	9 years ago
beorn7	4b503ed9a5	Merge branch 'master' into beorn7/storage2	9 years ago
beorn7	059295332f	Merge remote-tracking branch 'origin/master' into beorn7/storage	9 years ago
beorn7	53005c3085	Merge branch 'beorn7/storage' into beorn7/storage2	9 years ago
beorn7	28e9bbc15f	Populate chunkDesc.chunkLastTime during checkpoint loading, too	9 years ago
Björn Rabenstein	a8c79f0a0c	Merge pull request #1422 from prometheus/release-0.17 ... Merge more commits from 0.17.	9 years ago
Björn Rabenstein	5eff37ccbe	Merge pull request #1421 from prometheus/beorn7/fix ... Fix a very special case of handling the checkpoint timer	9 years ago
beorn7	8fa1560e48	Fix a very special case of handling the checkpoint timer	9 years ago
Björn Rabenstein	17bfe798eb	Merge pull request #1419 from prometheus/release-note-fixes ... Improve 0.17.0 changelog	9 years ago
Tobias Schmidt	b7e6651e06	Improve 0.17.0 changelog ... * remove wrong release date until 0.17.0 gets actually released * fix wrong alertmanager version number * add example for regex anchor change	9 years ago
Brian Brazil	e4e00b6f24	Merge pull request #1418 from igncp/patch-1 ... Fix minor typo	9 years ago
Ignacio Carbajo	0c537d6af6	Fix minor typo	9 years ago
beorn7	41e44f6ab9	Merge branch 'master' into beorn7/storage2	9 years ago
Björn Rabenstein	888c77cb06	Merge pull request #1416 from prometheus/beorn7/fix-test ... Fix a targetmanager test	9 years ago
beorn7	fd5108b038	Fix a targetmanager test	9 years ago
Björn Rabenstein	d9eb624322	Merge pull request #1415 from prometheus/release-0.17 ... Forward-merge release-0.17 into master	9 years ago
Björn Rabenstein	51aad630b6	Merge pull request #1414 from prometheus/beorn7/rushed-race ... Fix a race condition in calculatePersistenceUrgencyScore	9 years ago
beorn7	4d1f7b49b6	Fix a race condition in calculatePersistenceUrgencyScore	9 years ago
Brian Brazil	04946afd0a	Merge pull request #1412 from prometheus/fingerprintfix ... Remove fullLabels method and fix target updating	9 years ago
Fabian Reinartz	6df1f49c13	Remove fullLabels method and fix target updating ... With recent changes to a Target's internal data representation updating by fullLabels() assigns the additional default instance label. This breaks target identity comparison and causes identical targets from service discovery to be constantly swapped.	9 years ago
beorn7	454ecf3f52	Rework the way ranges and instants are handled ... In a way, our instants were also ranges, just with the staleness delta as range length. They are no treated equally, just that in one case, the range length is set as range, in the other the staleness delta. However, there are "real" instants where start and and time of a query is the same. In those cases, we only want to return a single value (the one closest before or at the equal start and end time). If that value is the last sample in the series, odds are we have it already in the series object. In that case, there is no need to pin or load any chunks. A special singleSampleSeriesIterator is created for that. This should greatly speed up instant queries as they happen frequently for rule evaluations.	9 years ago
Fabian Reinartz	209c4ad64f	Merge pull request #1410 from bluecmd/patch-1 ... Allow custom ldflags for go build	9 years ago
Christian Svensson	69ebf45649	Allow custom ldflags for go build ... This allows users to use CGO and external linker when building Prometheus.	9 years ago
beorn7	b876f8e6a5	Move lastSamplePair method up to memorySeries ... This implies a slight change of behavior as only samples added to the respective instance of a memorySeries are returned. However, this is most likely anyway what we want. Following cases: - Server has been restarted: Given the time it takes to cleanly shutdown and start up a server, the series are now stale anyway. An improved staleness handling (still to be implemented) will be based on tracking if a given target is continuing to expose samples for a given time series. In that case, we need a full scrape cycle to decide about staleness. So again, it makes sense to consider everything stale directly after a server restart. - Series unarchived due to a read request: The series is definitely stale so we don't want to return anything anyway. - Freshly created time series or series unarchived because of a sample append: That happens because appending a sample is imminent. Before the fingerprint lock is released, the series will have received a sample, and lastSamplePair will always returned the expected value.	9 years ago
beorn7	1e13f89039	Return SamplePair istead of *SamplePair consistently ... Formalize ZeroSamplePair as return value for non-existing samples. Change LastSamplePairForFingerprint to return a SamplePair (and not a pointer to it), which saves allocations in a potentially extremely frequent call.	9 years ago
beorn7	d290340367	Fix and improve chunkDesc locking	9 years ago
beorn7	0e202dacb4	Streamline series iterator creation ... This will fix issue #1035 and will also help to make issue #1264 less bad. The fundamental problem in the current code: In the preload phase, we quite accurately determine which chunks will be used for the query being executed. However, in the subsequent step of creating series iterators, the created iterators are referencing _all_ in-memory chunks in their series, even the un-pinned ones. In iterator creation, we copy a pointer to each in-memory chunk of a series into the iterator. While this creates a certain amount of allocation churn, the worst thing about it is that copying the chunk pointer out of the chunkDesc requires a mutex acquisition. (Remember that the iterator will also reference un-pinned chunks, so we need to acquire the mutex to protect against concurrent eviction.) The worst case happens if a series doesn't even contain any relevant samples for the query time range. We notice that during preloading but then we will still create a series iterator for it. But even for series that do contain relevant samples, the overhead is quite bad for instant queries that retrieve a single sample from each series, but still go through all the effort of series iterator creation. All of that is particularly bad if a series has many in-memory chunks. This commit addresses the problem from two sides: First, it merges preloading and iterator creation into one step, i.e. the preload call returns an iterator for exactly the preloaded chunks. Second, the required mutex acquisition in chunkDesc has been greatly reduced. That was enabled by a side effect of the first step, which is that the iterator is only referencing pinned chunks, so there is no risk of concurrent eviction anymore, and chunks can be accessed without mutex acquisition. To simplify the code changes for the above, the long-planned change of ValueAtTime to ValueAtOrBefore time was performed at the same time. (It should have been done first, but it kind of accidentally happened while I was in the middle of writing the series iterator changes. Sorry for that.) So far, we actively filtered the up to two values that were returned by ValueAtTime, i.e. we invested work to retrieve up to two values, and then we invested more work to throw one of them away. The SeriesIterator.BoundaryValues method can be removed once #1401 is fixed. But I really didn't want to load even more changes into this PR. Benchmarks: The BenchmarkFuzz.* benchmarks run 83% faster (i.e. about six times faster) and allocate 95% fewer bytes. The reason for that is that the benchmark reads one sample after another from the time series and creates a new series iterator for each sample read. To find out how much these improvements matter in practice, I have mirrored a beefy Prometheus server at SoundCloud that suffers from both issues #1035 and #1264. To reach steady state that would be comparable, the server needs to run for 15d. So far, it has run for 1d. The test server currently has only half as many memory time series and 60% of the memory chunks the main server has. The 90th percentile rule evaluation cycle time is ~11s on the main server and only ~3s on the test server. However, these numbers might get much closer over time. In addition to performance improvements, this commit removes about 150 LOC.	9 years ago
Fabian Reinartz	fce17b41c5	Merge pull request #1408 from prometheus/hostname ... Log argument parse errors	9 years ago
Fabian Reinartz	e62677d7ba	Log argument parse errors ... Fixes #1407	9 years ago
Brian Brazil	cd85352fe1	Merge pull request #1403 from igncp/master ... Fix minor typo	9 years ago
Ignacio Carbajo	6a323b1e6d	Fix minor typo	9 years ago
Brian Brazil	b447002309	Merge pull request #1402 from prometheus/fabxc/target-identity ... Use fingerprint for target identity comparison	9 years ago
Fabian Reinartz	825831e98f	Use fingerprint for target identity comparison ... So far we were using the InstanceIdentifier to compare equality of targets. This is not always accurate, for example for the blackbox exporter where the actual target is in the parameter.	9 years ago
Fabian Reinartz	c24c5e6fb3	Merge pull request #1400 from prometheus/beorn7/instrumentation ... Fix the instrumentation fixes	9 years ago
beorn7	663a1550d0	Fix the instrumentation fixes	9 years ago
Fabian Reinartz	73e38c534a	Merge pull request #1398 from prometheus/scraperef2 ... Handle scrape timeout on request.	9 years ago
Fabian Reinartz	66767121ab	Handle scrape timeout on request. ... For historic reasons we were enforcing a timeout directly via the TCP dialer. This is no longer necessary for quite a while now. Switching to context.Context will allow us to properly terminate requests on shutdown as well.	9 years ago
Fabian Reinartz	1f70345d0c	Merge pull request #1397 from prometheus/remove-old-scrapetime-setting ... Remove old superfluous calls to setLastScrape().	9 years ago