Commit Graph

42 Commits (b4b5bfdb4665c7fd0e94303c2999921b873d930a)

Author SHA1 Message Date
Eric Paris df590da6ab Return early from eviction debug helpers if !glog.V(3)
Should keep us from running a bunch of loops needlessly.
2017-03-08 20:19:52 -05:00
Kubernetes Submit Queue f9ccee7714 Merge pull request #42435 from dashpole/timestamps_for_fsstats
Automatic merge from submit-queue (batch tested with PRs 42369, 42375, 42397, 42435, 42455)

[Bug Fix]: Avoid evicting more pods than necessary by adding Timestamps for fsstats and ignoring stale stats

Continuation of #33121.  Credit for most of this goes to @sjenning.  I added volume fs timestamps.

**why is this a bug** 
This PR attempts to fix part of https://github.com/kubernetes/kubernetes/issues/31362 which results in multiple pods getting evicted unnecessarily whenever the node runs into resource pressure. This PR reduces the chances of such disruptions by avoiding reacting to old/stale metrics.
Without this PR, kubernetes nodes under resource pressure will cause unnecessary disruptions to user workloads. 
This PR will also help deflake a node e2e test suite.

The eviction manager currently avoids evicting pods if metrics are old.  However, timestamp data is not available for filesystem data, and this causes lots of extra evictions.
See the [inode eviction test flakes](https://k8s-testgrid.appspot.com/google-node#kubelet-flaky-gce-e2e) for examples.
This should probably be treated as a bugfix, as it should help mitigate extra evictions.

cc: @kubernetes/sig-storage-pr-reviews  @kubernetes/sig-node-pr-reviews @vishh @derekwaynecarr @sjenning
2017-03-03 23:21:48 -08:00
Seth Jennings c5faf1c156 kubelet: eviction: add timestamp to FsStats 2017-03-02 11:20:24 -08:00
David Ashpole ac612eab8e eviction manager changes for allocatable 2017-03-02 07:36:24 -08:00
Vishnu kannan 9b4a8f7464 fix eviction helper function description
Signed-off-by: Vishnu kannan <vishnuk@google.com>
2017-02-27 21:24:45 -08:00
Vishnu Kannan cc5f5474d5 add support for node allocatable phase 2 to kubelet
Signed-off-by: Vishnu Kannan <vishnuk@google.com>
2017-02-27 21:24:44 -08:00
Derek Carr 0171121486 Add debug logging to eviction manager 2017-02-08 15:01:12 -05:00
Dr. Stefan Schimanski bc6fdd925d pkg/api/resource: move to apimachinery 2017-01-29 21:41:44 +01:00
deads2k 6a4d5cd7cc start the apimachinery repo 2017-01-11 09:09:48 -05:00
Seth Jennings 4c30459e49 switch from local qos types to api types 2017-01-10 10:54:30 -06:00
Derek Carr 459a7a05f1 Ability to quota storage by storage class 2016-12-09 13:26:59 -05:00
Chao Xu 5e1adf91df cmd/kubelet 2016-11-23 15:53:09 -08:00
Seth Jennings 2583116f1a kubelet: eviction: add memcg threshold notifier to improve eviction responsiveness 2016-11-06 20:47:49 -06:00
David Ashpole d68572e581 eviction message now includes resource that was exhausted 2016-11-04 09:30:02 -07:00
David Ashpole 9aca40dee6 revert #33218. dont need #36180. We only use diskpressure 2016-11-04 08:29:27 -07:00
David Ashpole b8fc546d60 eviction manager ecivts pod using the most inodes. 2016-10-31 11:32:49 -07:00
Kubernetes Submit Queue 6a9d56b35a Merge pull request #32724 from sjenning/eviction-timestamp
Automatic merge from submit-queue

kubelet: eviction: avoid duplicate action on stale stats

Currently, the eviction code can be overly aggressive when synchronize() is called two (or more) times before a particular stat has been recollected by cadvisor.  The eviction manager will take additional  action based on information for which it has already taken actions.

This PR provides a method for the eviction manager to track the timestamp of the last obversation and not take action if the stat has not been updated since the last time synchronize() was run.

@derekwaynecarr @vishh @kubernetes/rh-cluster-infra
2016-10-06 11:05:34 -07:00
Seth Jennings 98e97a475a kubelet: eviction: avoid duplicate action on stale stats 2016-10-06 11:39:05 -05:00
Kubernetes Submit Queue 07eba4c6ef Merge pull request #33392 from sjenning/min-reclaim-percent
Automatic merge from submit-queue

kubelet: eviction: allow minimum reclaim as percentage

Fixes #33354 

xref #32537

**Release note**:
```release-note
The kubelet --eviction-minimum-reclaim option can now take precentages as well as absolute values for resources quantities
```
@derekwaynecarr @vishh @mtaufen
2016-10-06 09:25:46 -07:00
David Ashpole fed3f37eef Split NodeDiskPressure into NodeInodePressure and NodeDiskPressure 2016-10-03 11:42:56 -07:00
Seth Jennings f52dce9319 kubelet: eviction: allow minreclaim as percentage 2016-09-30 11:12:36 -05:00
Angus Salkeld f785f3d3ef Clean up IPTables caps i.e.: sed -i "s/Iptables/IPTables/g" 2016-08-29 10:34:42 +10:00
Kubernetes Submit Queue cdbc3b56eb Merge pull request #31523 from derekwaynecarr/imagefs-observations
Automatic merge from submit-queue

Set imagefs rank and reclaim functions when nodefs+imagefs share comm…

Fixes #31192 

I decided that the behavior should match the current output of the kubelet summary API.  With no dedicated imagefs, the ranking and reclaim functions will match the nodefs ranking and reclaim functions.

/cc @ronnielai @vishh
2016-08-27 02:58:42 -07:00
derekwaynecarr 6a7eb7b420 Set imagefs rank and reclaim functions when nodefs+imagefs share common device 2016-08-26 11:08:43 -04:00
Yu-Ju Hong a072bda6fd Print out resource name when evicting pods 2016-08-25 14:11:21 -07:00
derekwaynecarr a65a24a850 kubelet eviction on inode exhaustion 2016-08-17 16:57:44 -04:00
bindata-mockuser d2dd03aecc Support percentage representation for eviction thresholds 2016-08-11 14:10:25 -07:00
derekwaynecarr 68bc47ecc6 Add support to invoke image gc in response to disk eviction thresholds 2016-08-04 17:13:08 -04:00
derekwaynecarr 611c127f0d kubelet eviction manager support for min-reclaim 2016-08-04 11:17:44 -04:00
Ron Lai 8bc4444f16 Delete containers when pod is deleted 2016-08-03 15:56:04 -07:00
derekwaynecarr 0de1e62b30 modify fsStats to fsStatsType to avoid confusion with cadvisor types 2016-07-28 16:01:38 -04:00
derekwaynecarr c3324b88a0 Eviction manager observes and acts on disk pressure 2016-07-28 16:01:38 -04:00
Avesh Agarwal cb7766de19 Fix kubelet to not accept negative eviction (hard, soft) thresholds
and add unit tests
2016-07-27 10:56:31 -04:00
derekwaynecarr 2f4a4ea4cf Kubelet parses and validates eviction-minimum-reclaim flag 2016-07-21 16:46:12 -04:00
David McMahon ef0c9f0c5b Remove "All rights reserved" from all the headers. 2016-06-29 17:47:36 -07:00
Buddha Prakash 4acb64f8bd Make Qos naming consistent across the codebase 2016-06-26 16:19:47 -07:00
Buddha Prakash c3551ae6cd Refactor qos package
Signed-off-by: Buddha Prakash <buddhap@google.com>
2016-06-24 15:36:04 -07:00
derekwaynecarr 2a1d3faf08 kubelet support for max grace period during soft eviction 2016-05-20 11:30:08 -04:00
Clayton Coleman e3c8c4c22b
Handle resource.Quantity changes in eviction thresholds 2016-05-19 08:41:43 -04:00
Clayton Coleman 5e4308f91d
Update use of Quantity in other classes 2016-05-19 08:41:43 -04:00
derekwaynecarr edc76f6d4f out of resource killing (memory) 2016-05-14 11:34:45 -04:00
derekwaynecarr 725af223aa Add parsers for eviction thresholds 2016-05-06 12:06:03 -04:00