Automatic merge from submit-queue (batch tested with PRs 53228, 53232, 53353). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fixes a regression introduced by PR 52290 that extended resource
capacity may temporarily drop to zero after kubelet restarts and PODs restarted during
that time window could fail to be scheduled.
**What this PR does / why we need it**:
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
https://github.com/kubernetes/kubernetes/issues/53342
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue (batch tested with PRs 53403, 53233). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Remove containers from deleted pods once containers have exited
Issue #51899
Since container deletion is currently done through periodic garbage collection every 30 seconds, it takes a long time for pods to be deleted, and causes the kubelet to send all delete pod requests at the same time, which has performance issues. This PR makes the kubelet actively remove containers of deleted pods rather than wait for them to be removed in periodic garbage collection.
/release-note-none
Automatic merge from submit-queue (batch tested with PRs 50555, 51152). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Implement CRI stats in Docker Shim
**What this PR does / why we need it**:
This PR implements CRI Stats in the Docker Shim. It is needed to enable CRI stats for Docker and ongoing /stats/summary API changes in moving to use CRI.
Related issues:
#46984 (CRI: instruct kubelet to (optionally) consume container stats from CRI)
#45614 (CRI: add methods for container stats)
This PR is also a followup to my original PR (https://github.com/kubernetes/kubernetes/pull/50396) to implement Windows Container Stats. The plan is that Windows Stats will use a hybrid model: pod and container level stats will come from CRI (via dockershim) and that node level stats will come from a "winstats" package that exports cadvisor like datastructures using windows specific perf counters from the node. I will update that PR to only export node level stats.
@yujuhong @yguo0905 @dchen1107 @jdumars @anhowe @michmike
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
enable to specific unconfined AppArmor profile
**What this PR does / why we need it**:
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#52370
**Special notes for your reviewer**:
/assign @tallclair @liggitt
**Release note**:
```release-note
enable to specific unconfined AppArmor profile
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
create separate transports for liveness and readiness probes
There is currently an issue with the http2 connection pools in golang such that two GETs to the same host:port using the same Transport can collide and one gets rejected with `http2: no cached connection was available`. This happens with readiness and liveness probes if the intervals line up such that worker goroutines invoke the two probes at the exact same time.
The result is a transient probe error that appears in the events. If the failureThreshold is 1, which is kinda crazy, it would cause a pod restart.
The PR creates a separate `httprobe` instance for readiness and liveness probes so that they don't share a Transport and connection pool.
Fixes https://github.com/kubernetes/kubernetes/issues/49740
@smarterclayton @jhorwit2
Implement CRI stats for dockershim using docker stats. This enables use
of the summary api to get container metrics on Windows where CRI stats
are enabled.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Eliminate extra CRI call during processing cpu set
**What this PR does / why we need it**:
Encountered this during `kubernetes/frakti` node e2e test.
When cpuset is not set, there's still plenty of `runtime.UpdateContainerResources` been called, which seems unnecessary.
cc @ConnorDoyle Make sense? Fixes: #53304
**Special notes for your reviewer**:
**Release note**:
```release-note
Only do UpdateContainerResources when cpuset is set
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix broken links in kubelet after moving proposals to subdirs
**What this PR does / why we need it**:
fix incorrect links in kubelet after kubernetes/community#1010
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes kubernetes/community#918
**Special notes for your reviewer**:
CC @bgrant0607
**Release note**:
```
NONE
```
Automatic merge from submit-queue (batch tested with PRs 51021, 53225, 53094, 53219). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Change ImageGCManage to consume ImageFS stats from StatsProvider
Fixes#53083.
**Release note**:
```
Change ImageGCManage to consume ImageFS stats from StatsProvider
```
/assign @Random-Liu
Automatic merge from submit-queue (batch tested with PRs 44596, 52708, 53163, 53167, 52692). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix the bad code comment and make the format unify
**What this PR does / why we need it**:
Fix the bad code comment and make the format unify
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 44596, 52708, 53163, 53167, 52692). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Do not GC exited containers in running pods
This fixes a regression introduced by #45896, and was identified by #52462.
This bug causes the kubelet to garbage collect exited containers in a running pod.
This manifests in strange and confusing state when viewing the cluster. For example, it can show running pods as having no init container (see #52462), if that container has exited and been removed.
This PR solves this problem by only removing containers and sandboxes from terminated pods.
The important line change is:
` if cgc.podDeletionProvider.IsPodDeleted(podUID) || evictNonDeletedPods {` --->
`if cgc.podStateProvider.IsPodDeleted(podUID) || (cgc.podStateProvider.IsPodTerminated(podUID) && evictTerminatedPods) {`
cc @MrHohn @yujuhong @kubernetes/sig-node-bugs
```release-note
BugFix: Exited containers are not Garbage Collected by the kubelet while the pod is running
```
Automatic merge from submit-queue (batch tested with PRs 44596, 52708, 53163, 53167, 52692). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix --kube-reserved storage key name and add UTs for node allocatable reservation
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: part of #52463
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
/assign @jingxu97
Automatic merge from submit-queue (batch tested with PRs 50280, 52529, 53093, 53108, 53168). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
remove certificate manager unused code
**What this PR does / why we need it**:
remove unused const
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 53157, 52628). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Revert "Make kubelet touch iptables lock file during initialization"
**What this PR does / why we need it**: Revert #47212. #36485 is fixed so this is no longer needed.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #NONE
**Special notes for your reviewer**:
/assign @yujuhong @dchen1107
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 52634, 53121, 53161). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Normalize RepoTags before checking for match
**What this PR does / why we need it**:
on projectatomic-based docker, we get "docker.io/library/busybox:latest"
when someone uses an unqualified name like "busybox". Though when we
inspect, the RepoTag will still say "docker.io/busybox:latest", So
we have reparse the tag, normalize it and try again. Please see the
additional test case.
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
Fixes#52110
**Special notes for your reviewer**:
**Release note**:
```release-note
Fixes an issue pulling pod specs referencing unqualified images from docker.io on centos/fedora/rhel
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Improve codes which checks whether sandbox contains containers
Currently evictSandboxes() checks whether sandbox contains
containers, it traverses all the containers for every sandbox,
but when cluster has many containres, it wastes a lot of time.
It is better to use sets in this case.
**Release note**:
```release-note
NONE
```
on projectatomic-based docker, we get "docker.io/library/busybox:latest"
when someone uses an unqualified name like "busybox". Though when we
inspect, the RepoTag will still say "docker.io/busybox:latest", So
we have reparse the tag, normalize it and try again. Please see the
additional test case.
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
Automatic merge from submit-queue (batch tested with PRs 50988, 50509, 52660, 52663, 52250). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Create container log symlink for all containers
**What this PR does / why we need it**:
dockershim only makes log symlink for running containers now, we should also create the log symlink for failed containers.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#50499
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix typo in docs of remote package
**What this PR does / why we need it**:
Fix typo in docs of kubelet/remote package
**Which issue this PR fixes**: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 52990, 53064, 52686, 52221, 53069). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Align imagefs eviction defaults with image gc defaults
**What this PR does / why we need it**:
If a node is configured to use an imagefs for container storage, we should align the default imagefs eviction threshold with the default image-gc threshold. This PR updates the default imagesfs.available threshold to trigger when below 15% available space, which is same as default image-gc high threshold for 85%.
Fixes https://github.com/kubernetes/kubernetes/issues/53074
**Special notes for your reviewer**:
none, this only impacts nodes running an imagefs otherwise its ignored.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 52990, 53064, 52686, 52221, 53069). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
CRI: Allow configuring stdout/stderr streams for Exec/Attach requests
Add stdout/stderr to exec and attach requests. Also check the request to
ensure it meets the requirements.
**Which issue this PR fixes**: fixes#44448
```release-note
CRI: Add stdout/stderr fields to Exec and Attach requests.
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add an OWNERS file for deviceplugin package. Update OWNERS file for gpu package.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 52721, 53057, 52493, 52998, 52896). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix a potential file leak
Previously, if a write or sync error occurred, we would not have called
Close(). This commit refactors ReplaceFile() so that we are sure to call
Close(), and also attempts to delete the temporary file if errors occur.
See: https://github.com/kubernetes/kubernetes/pull/52119#discussion_r137916659Fixes: #53060
```release-note
NONE
```
@yujuhong @ash2k
When using the legacy docker container runtime and when a container has
terminationMessagePolicy=FallbackToLogsOnError and when docker is
configured with a log driver other than json-log (such as journald),
the kubelet should not try to get the container's log from the
json log file (since it's not there) but should instead ask docker for
the logs.
Previously, if a write or sync error occurred, we would not have called
Close(). This commit refactors ReplaceFile() so that we are sure to call
Close(), and also attempts to delete the temporary file if errors occur.
In the current test, there is a race that the new device plugin endpoint
may not be added to the device plugin manager endpoints at the time when
we call manager.Devices(). Added the checking and waiting for endpoint
updates before calling manager.Devices() in the test.
Tested:
go test -race -count 500 k8s.io/kubernetes/pkg/kubelet/deviceplugin -run
TestDevicePluginReRegistration -timeout 5h
Automatic merge from submit-queue (batch tested with PRs 51067, 52319, 52803, 52961, 51972). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Move prometheus metrics for docker operations into dockershim
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
kubelet: remove the --docker-exec-handler flag
Stop supporting the "nsenter" exec handler. Only the Docker native exec
handler is supported.
The flag was deprecated in Kubernetes 1.6 and is safe to remove
in Kubernetes 1.9 according to the deprecation policy.
**What this PR does / why we need it**:
**Which issue this PR fixes** : fixes#40229
**Special notes for your reviewer**:
N/A
**Release note**:
```release-note
Remove the --docker-exec-handler flag. Only native exec handler is supported.
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
dockershim: fine-tune network-ready handling on sandbox teardown and removal
If sandbox teardown results in an error, GC will periodically attempt
to again remove the sandbox. Until the sandbox is removed, pod
sandbox status calls will attempt to enter the pod's namespace and
retrieve the pod IP, but the first teardown attempt may have already
removed the network namespace, resulting in a pointless log error
message that the network namespace doesn't exist, or that nsenter
can't find eth0.
The network-ready mechanism originally attempted to suppress those
messages by ensuring that pod sandbox status skipped network checks
when networking was already torn down, but unfortunately the ready
value was cleared too early.
Also, don't tear down the pod network multiple times if the first
time we tore it down, it succeeded.
**What this PR does / why we need it**:
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
bazel: build/test almost everything
**What this PR does / why we need it**: Miscellaneous cleanups and bug fixes. The main motivating idea here was to make `bazel build //...` and `bazel test //...` mostly work. (There's a few reasons these still don't work, but we're a lot closer.)
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
/assign @BenTheElder @mikedanese @spxtr
Automatic merge from submit-queue (batch tested with PRs 50890, 52484, 52542, 52567, 50672). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Do not set message when terminationMessagePath not found
If terminationMessagePath is set to a file that does not exist, we should not log an error message and instead try falling back to logs (based on the user's request).
This also slightly simplifies the terminationMessagePath processing.
Seen in #50499
```release-note
If a container does not create a file at the `terminationMessagePath`, no message should be output about being unable to find the file.
```
Automatic merge from submit-queue (batch tested with PRs 52168, 48939, 51889, 52051, 50396). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Add Windows Server Containers Stats and Metrics to Kubelet
**What this PR does / why we need it**:
This PR implements stats for Windows Server Containers. This adds the ability to monitor Windows Server containers via the existing stats/summary endpoint inside the kubelet. Windows metrics can now be ingested into heapster and monitored using existing tools (like Grafana).
Previously, the /stats/summary api would consistently crash the kubelet on Windows server containers. This PR implements a new package "winstats" which reads windows server metrics from a combination of windows specific perf counters as well as docker stats. The "winstats" package exports functions that return CAdvisor data structures, which the existing summary api can read.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#49398
This PR addresses my plan to implement windows server container stats https://github.com/kubernetes/kubernetes/issues/49398 .
**Release note**:
```release-note
Add monitoring of Windows Server containers metrics in the kubelet via the stats/summary endpoint.
```
Automatic merge from submit-queue (batch tested with PRs 52168, 48939, 51889, 52051, 50396). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Fix typo in kubelet kuberuntime container test
Changes "Expetected" to "Expected"
**What this PR does / why we need it**: Fixes a typo in a test
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*:
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 50294, 50422, 51757, 52379, 52014). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Fix AnnotationProvidedIPAddr annotation for externalCloudProvider
**What this PR does / why we need it**:
In #44258, it introduced `AnnotationProvidedIPAddr`. When kubelet has 'node-ip' parameter set, and cloud provider not set, this annotation would be populated, and then will be validated by cloud-controller-manager:
https://github.com/kubernetes/kubernetes/pull/44258/files#diff-6b0808bd1afb15f9f77986f4459601c2R465
Later with #47152, externalCloudProvider is checked and func returns before that annotation got set. In this case, that annotation will not get populated.
This fix is to bring that annotation assignment to a proper location.
Please correct me if I have any misunderstanding.
@wlan0 @ublubu
**Which issue this PR fixes**
**Special notes for your reviewer**:
**Release note**:
Automatic merge from submit-queue (batch tested with PRs 52109, 52235, 51809, 52161, 50080). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Remove backward compatibility of hostportChainName
**What this PR does / why we need it**:
fix TODO.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
/assign @freehan
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 52240, 48145, 52220, 51698, 51777). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
dockershim: remove support for legacy containers
The code was first introduced in 1.6 to help pre-CRI-kubelet upgrade to
using the CRI implementation. They can safely be removed now.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
kubelet: fix inconsistent display of terminated pod IPs
PLEG and kubelet race when reading and sending pod status to the apiserver. PLEG
inserts status into a cache, and then signals kubelet. Kubelet then eventually
reads the status out of that cache, but in the mean time the status could have
been changed by PLEG.
When a pod exits, pod status will no longer include the pod's IP address because
the network plugin/runtime will report "" for terminated pod IPs. If this status
gets inserted into the PLEG cache before kubelet gets the status out of the cache,
kubelet will see a blank pod IP address. This happens in about 1/5 of cases when
pods are short-lived, and somewhat less frequently for longer running pods.
To ensure consistency for properties of dead pods, copy an old status update's
IP address over to the new status update if (a) the new status update's IP is
missing and (b) all sandboxes of the pod are dead/not-ready (eg, no possibility
for a valid IP from the sandbox).
Fixes: https://github.com/kubernetes/kubernetes/issues/47265
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1449373
@eparis @freehan @kubernetes/rh-networking @kubernetes/sig-network-misc
Stop supporting the "nsenter" exec handler. Only the Docker native exec
handler is supported.
The flag was deprecated in Kubernetes 1.6 and is safe to remove
in Kubernetes 1.9 according to the deprecation policy.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Fix volume remount on reboot
**What this PR does / why we need it**:
Check the mount is actually attached & mounted before marking actual state of world of Kubelet reconciler.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#51982
**Special notes for your reviewer**:
Added explicit check to make sure volumes are attached and are mounted before marking the state in actual state of world.
**Release note**:
NONE
If sandbox teardown results in an error, GC will periodically attempt
to again remove the sandbox. Until the sandbox is removed, pod
sandbox status calls will attempt to enter the pod's namespace and
retrieve the pod IP, but the first teardown attempt may have already
removed the network namespace, resulting in a pointless log error
message that the network namespace doesn't exist, or that nsenter
can't find eth0.
The network-ready mechanism originally attempted to suppress those
messages by ensuring that pod sandbox status skipped network checks
when networking was already torn down, but unfortunately the ready
value was cleared too early.
Also, don't tear down the pod network multiple times if the first
time we tore it down, it succeeded.
Automatic merge from submit-queue (batch tested with PRs 51337, 47080, 52646, 52635, 52666). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Fix CRI container/imagefs stats.
`ContainerStats`, `ListContainerStats` and `ImageFsInfo` are returning `not implemented` error now.
This PR fixes it.
@yujuhong @feiskyer @yguo0905
Automatic merge from submit-queue (batch tested with PRs 51337, 47080, 52646, 52635, 52666). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Map a resource to multiple signals in eviction manager
It is possible to have multiple signals that point to the same type of
resource, e.g., both SignalNodeFsAvailable and
SignalAllocatableNodeFsAvailable refer to the same resource NodeFs.
Change the map from map[v1.ResourceName]evictionapi.Signal to
map[v1.ResourceName][]evictionapi.Signal
**What this PR does / why we need it**:
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#52661
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Fixes a race in deviceplugin/manager_test.go and a race in deviceplug…
…in/manager.go.
**What this PR does / why we need it**:
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
https://github.com/kubernetes/kubernetes/issues/52560
**Special notes for your reviewer**:
Tested with go test -count 50 -race k8s.io/kubernetes/pkg/kubelet/deviceplugin and all runs passed.
**Release note**:
```release-note
```
Automatic merge from submit-queue (batch tested with PRs 48970, 52497, 51367, 52549, 52541). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Recreate pod sandbox when the sandbox does not have an IP address.
**What this PR does / why we need it**:
Attempts to fix a bug where Pods do not receive networking when the kubelet restarts during pod creation.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*:
fixes # https://github.com/kubernetes/kubernetes/issues/48510
**Release note**:
```release-note
NONE
```
Currently when evictSandboxes() checks whether sandbox contains
containers, it traverses all the containers for every sandbox,
but when cluster has many containres, it wastes a lot of time.
It is better to use sets in this case.
Automatic merge from submit-queue (batch tested with PRs 52176, 43152). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Eliminate hangs/throttling of node heartbeat
Fixes https://github.com/kubernetes/kubernetes/issues/48638Fixes#50304
Stops kubelet from wedging when updating node status if unable to establish tcp connection.
Notes that this only affects the node status loop. The pod sync loop would still hang until the dead TCP connections timed out, so more work is needed to keep the sync loop responsive in the face of network issues, but this change lets existing pods coast without the node controller trying to evict them
```release-note
kubelet to master communication when doing node status updates now has a timeout to prevent indefinite hangs
```
If terminationMessagePath is set to a file that does not exist, we
should not log an error message and instead try falling back to logs
(based on the user's request).
Automatic merge from submit-queue (batch tested with PRs 52452, 52115, 52260, 52290)
Fixes device plugin re-registration handling logic to make sure:
- If a device plugin exits, its exported resource will be removed.
- No capacity change if a new device plugin instance comes up to replace the old instance.
**What this PR does / why we need it**:
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes https://github.com/kubernetes/kubernetes/issues/52510
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue (batch tested with PRs 52442, 52247, 46542, 52363, 51781)
Make CPU manager release CPUs when Pod enters completed phase.
**What this PR does / why we need it**: When CPU manager is enabled, this PR releases allocated CPUs when container is not running and is non-restartable.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#52351
**Special notes for your reviewer**:
This bug is only reproduced for pods with `restartPolicy` = `Never` or `OnFailure`. The following output is from a 4 CPU node. This bug can be reproduced as long >= half the cores are requested.
pod1.yaml:
```
apiVersion: v1
kind: Pod
metadata:
name: test-pod1
spec:
containers:
- image: ubuntu
command: ["/bin/bash"]
args: ["-c", "sleep 5"]
name: test-container1
resources:
requests:
cpu: 2
memory: 100Mi
limits:
cpu: 2
memory: 100Mi
restartPolicy: "Never"
```
pod2.yaml:
```
apiVersion: v1
kind: Pod
metadata:
name: test-pod2
spec:
containers:
- image: ubuntu
command: ["/bin/bash"]
args: ["-c", "sleep 5"]
name: test-container1
resources:
requests:
cpu: 2
memory: 100Mi
limits:
cpu: 2
memory: 100Mi
restartPolicy: "Never"
```
Run a local Kubernetes cluster with CPU manager enabled.
```sh
KUBELET_FLAGS='--feature-gates=CPUManager=true --cpu-manager-policy=static --cpu-manager-reconcile-period=1s --kube-reserved=cpu=500m' ./hack/local-up-cluster.sh
```
_Before:_
Create `test-pod1` using pod1.yaml.
```
./cluster/kubectl.sh create -f pod1.yaml
```
Wait for the pod to complete and wait another 90 seconds (give enough time for GC to kick-in).
Create `test-pod2` using pod2.yaml.
```
./cluster/kubectl.sh create -f pod2.yaml
```
Get all pods in the cluster.
```
./cluster/kubectl.sh get pods -a
NAME READY STATUS RESTARTS AGE
test-pod1 0/1 Completed 0 1m
test-pod2 0/1 not enough cpus available to satisfy request 0 9s
```
_After:_
Create `test-pod1` using pod1.yaml.
```
./cluster/kubectl.sh create -f pod1.yaml
```
Wait for the pod to complete and wait another 90 seconds (give enough time for GC to kick-in).
Create `test-pod2` using pod2.yaml.
```
./cluster/kubectl.sh create -f pod2.yaml
```
Get all pods in the cluster.
```
./cluster/kubectl.sh get pods -a
NAME READY STATUS RESTARTS AGE
test-pod1 0/1 Completed 0 1m
test-pod2 0/1 Completed 0 9s
```
Automatic merge from submit-queue (batch tested with PRs 52442, 52247, 46542, 52363, 51781)
Ignore pods for quota marked for deletion whose node is unreachable
**What this PR does / why we need it**:
Traditionally, we charge to quota all pods that are in a non-terminal phase. We have a user report that noted the behavior change in kube 1.5 for the node controller to no longer force delete pods whose nodes have been lost. Instead, the pod is marked for deletion, and the reason is updated to state that the node is unreachable. The user expected the quota to be released. If the user was at their quota limit, their application may not be able to create a new replica given the current behavior. As a result, this PR ignores pods marked for deletion that have exceeded their grace period.
**Which issue this PR fixes**
xref https://bugzilla.redhat.com/show_bug.cgi?id=1455743
fixes https://github.com/kubernetes/kubernetes/issues/52436
**Release note**:
```release-note
Ignore pods marked for deletion that exceed their grace period in ResourceQuota
```
- If a device plugin exits, its exported resource will be removed.
- No capacity change if a new device plugin instance comes up to replace the old instance.
This implements stats for windows nodes in a new package, winstats.
WinStats exports methods to get cadvisor like datastructures, however
with windows specific metrics. WinStats only gets node level metrics and
information, container stats will go via the CRI. This enables the
use of the summary api to get metrics for windows nodes.
Automatic merge from submit-queue (batch tested with PRs 52339, 52343, 52125, 52360, 52301)
dockershim: check if f.Sync() returns an error and surface it
```release-note
dockershim: check the error when syncing the checkpoint.
```
Automatic merge from submit-queue (batch tested with PRs 48226, 52046, 52231, 52344, 52352)
Log at higher verbosity levels some common SyncPod errors
This log message was 90% of all glog.Errorf level statements reported on a production cluster, hiding other more impactful errors. We already log it in start container, but for extra caution we continue to log it at v(3) here (the downside of not logging a start container error is worse than some log spam at higher levels).
HandleError() is intended only for unknown and unexpected errors.
```release-note
NONE
```
@derekwaynecarr @sjenning
Automatic merge from submit-queue (batch tested with PRs 48226, 52046, 52231, 52344, 52352)
[BugFix] Soft Eviction timer works correctly
fixes#51516
thresholdsMet should not exclude previously met thresholds when we do not have new stats for a threshold.
/assign @vishh @derekwaynecarr
cc @kubernetes/sig-node-bugs
Automatic merge from submit-queue (batch tested with PRs 51041, 52297, 52296, 52335, 52338)
Use cAdvisor constant for crio imagefs
**What this PR does / why we need it**:
code hygiene to use a constant from cAdvisor
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue
fsync config checkpoint files after writing
@yujuhong brought up that it's possible for a hard reboot to result in empty checkpoint files, if they haven't been synced to disk yet. This PR ensures that Kubelet configuration checkpoints are synced after writing to avoid this issue.
fixes#52222
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 52264, 51870)
Use credentials from providers for docker sandbox image
**What this PR does / why we need it**:
Sandbox image lookup uses creds from docker config only; other credential providers are ignored. This is a regression introduced in dockershim.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#51293
**Special notes for your reviewer**:
Should also cherry-pick this to release-1.6 and release-1.7.
**Release note**:
```release-note
Fix credentials providers for docker sandbox image.
```
Automatic merge from submit-queue (batch tested with PRs 52047, 52063, 51528)
Improve dynamic kubelet config e2e node test and fix bugs
Rather than just changing the config once to see if dynamic kubelet
config at-least-sort-of-works, this extends the test to check that the
Kubelet reports the expected Node condition and the expected configuration
values after several possible state transitions.
Additionally, this adds a stress test that changes the configuration 100
times. It is possible for resource leaks across Kubelet restarts to
eventually prevent the Kubelet from restarting. For example, this test
revealed that cAdvisor's leaking journalctl processes (see:
https://github.com/google/cadvisor/issues/1725) could break dynamic
kubelet config. This test will help reveal these problems earlier.
This commit also makes better use of const strings and fixes a few bugs
that the new testing turned up.
Related issue: #50217
I had been sitting on this until the cAdvisor fix merged in #51751, as these tests fail without that fix.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue
Added large topology tests for static policy in CPU Manager.
**What this PR does / why we need it**: This PR adds a very large topology test case for the CPU Manager feature.
Related to #51180.
CC @ConnorDoyle
Rather than just changing the config once to see if dynamic kubelet
config at-least-sort-of-works, this extends the test to check that the
Kubelet reports the expected Node condition and the expected configuration
values after several possible state transitions.
Additionally, this adds a stress test that changes the configuration 100
times. It is possible for resource leaks across Kubelet restarts to
eventually prevent the Kubelet from restarting. For example, this test
revealed that cAdvisor's leaking journalctl processes (see:
https://github.com/google/cadvisor/issues/1725) could break dynamic
kubelet config. This test will help reveal these problems earlier.
This commit also makes better use of const strings and fixes a few bugs
that the new testing turned up.
Related issue: #50217
Automatic merge from submit-queue (batch tested with PRs 51239, 51644, 52076)
do not update init containers status if terminated
fixes#29972#41580
This fixes an issue where, if a completed init container is removed while the pod or subsequent init containers are still running, the status for that init container will be reset to `Waiting` with `PodInitializing`.
This can manifest in a number of ways.
If the init container is removed why the main pod containers are running, the status will be reset with no functional problem but the status will be reported incorrectly in `kubectl get pod` for example
If the init container is removed why a subsequent init container is running, the init container will be **re-executed** leading to all manner of badness.
@derekwaynecarr @bparees
Automatic merge from submit-queue (batch tested with PRs 51728, 49202)
Fix setNodeAddress when a node IP and a cloud provider are set
**What this PR does / why we need it**:
When a node IP is set and a cloud provider returns the same address with
several types, only the first address was accepted. With the changes made
in PR #45201, the vSphere cloud provider returned the ExternalIP first,
which led to a node without any InternalIP.
The behaviour is modified to return all the address types for the
specified node IP.
**Which issue this PR fixes**: fixes#48760
**Special notes for your reviewer**:
* I'm not a golang expert, is it possible to mock `kubelet.validateNodeIP()` to avoid the need of real host interface addresses in the test ?
* It would be great to have it backported for a next 1.6.8 release.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 51728, 49202)
Enable CRI-O stats from cAdvisor
**What this PR does / why we need it**:
cAdvisor may support multiple container runtimes (docker, rkt, cri-o, systemd, etc.)
As long as the kubelet continues to run cAdvisor, runtimes with native cAdvisor support may not want to run multiple monitoring agents to avoid performance regression in production. Pending kubelet running a more light-weight monitoring solution, this PR allows remote runtimes to have their stats pulled from cAdvisor when cAdvisor is registered stats provider by introspection of the runtime endpoint.
See issue https://github.com/kubernetes/kubernetes/issues/51798
**Special notes for your reviewer**:
cAdvisor will be bumped to pick up https://github.com/google/cadvisor/pull/1741
At that time, CRI-O will support fetching stats from cAdvisor.
**Release note**:
```release-note
NONE
```
* Expected value comes before actual value in assert.Equal()
* Use assert.Equal() instead of assert.True() when possible
* Add a unit test that verifies no-op pod updates to the
secret_manager and the configmap_manager
* Add a clarifying comment about why it's good to seemingly
delete a secret on updates.
* Fix (for now, non-buggy) variable shadowing issue
Automatic merge from submit-queue
Make *fakeMountInterface in container_manager_unsupported_test.go implement mount.Interface again.
This was broken in #45724
**Release note**:
```release-note
NONE
```
/sig storage
/sig node
/cc @jsafrane, @vishh
Automatic merge from submit-queue (batch tested with PRs 51186, 50350, 51751, 51645, 51837)
Set up DNS server in containerized mounter path
During NFS/GlusterFS mount, it requires to have DNS server to be able to
resolve service name. This PR gets the DNS server ip from kubelet and
add it to the containerized mounter path. So if containerized mounter is
used, service name could be resolved during mount
**Release note**:
```release-note
Allow DNS resolution of service name for COS using containerized mounter. It fixed the issue with DNS resolution of NFS and Gluster services.
```
Automatic merge from submit-queue (batch tested with PRs 51186, 50350, 51751, 51645, 51837)
Update Cadvisor Dependency
Fixes: https://github.com/kubernetes/kubernetes/issues/51832
This is the worst dependency update ever...
The root of the problem is the [name change of Sirupsen -> sirupsen](https://github.com/sirupsen/logrus/issues/570#issuecomment-313933276). This means that in order to update cadvisor, which venders the lowercase, we need to update all dependencies to use the lower-cased version. With that being said, this PR updates the following packages:
`github.com/docker/docker`
- `github.com/docker/distribution`
- `github.com/opencontainers/go-digest`
- `github.com/opencontainers/image-spec`
- `github.com/opencontainers/runtime-spec`
- `github.com/opencontainers/selinux`
- `github.com/opencontainers/runc`
- `github.com/mrunalp/fileutils`
- `golang.org/x/crypto`
- `golang.org/x/sys`
- `github.com/docker/go-connections`
- `github.com/docker/go-units`
- `github.com/docker/libnetwork`
- `github.com/docker/libtrust`
- `github.com/sirupsen/logrus`
- `github.com/vishvananda/netlink`
`github.com/google/cadvisor`
- `github.com/euank/go-kmsg-parser`
`github.com/json-iterator/go`
Fixed https://github.com/kubernetes/kubernetes/issues/51832
```release-note
Fix journalctl leak on kubelet restart
Fix container memory rss
Add hugepages monitoring support
Fix incorrect CPU usage metrics with 4.7 kernel
Add tmpfs monitoring support
```
Automatic merge from submit-queue (batch tested with PRs 51186, 50350, 51751, 51645, 51837)
Wait for container cleanup before deletion
We should wait to delete pod API objects until the pod's containers have been cleaned up. See issue: #50268 for background.
This changes the kubelet container gc, which deletes containers belonging to pods considered "deleted".
It adds two conditions under which a pod is considered "deleted", allowing containers to be deleted:
Pods where deletionTimestamp is set, and containers are not running
Pods that are evicted
This PR also changes the function PodResourcesAreReclaimed by making it return false if containers still exist.
The eviction manager will wait for containers of previous evicted pod to be deleted before evicting another pod.
The status manager will wait for containers to be deleted before removing the pod API object.
/assign @vishh
Automatic merge from submit-queue (batch tested with PRs 50072, 51744)
Deviceplugin checkpoint
**What this PR does / why we need it**:
Extends on top of PR 51209 to checkpoint device to pod allocation information on Kubelet to recover from Kubelet restarts.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
```
During NFS/GlusterFS mount, it requires to have DNS server to be able to
resolve service name. This PR gets the DNS server ip from kubelet and
add it to the containerized mounter path. So if containerized mounter is
used, service name could be resolved during mount
Modifies the VolumeZonePredicate to handle a PV that belongs to more
then one zone or region. This is indicated by the zone or region label
value containing a comma separated list.
Introduce feature gate for expanding PVs
Add a field to SC
Add new Conditions and feature tag pvc update
Add tests for size update via feature gate
register the resize admission plugin
Update golint failures
Automatic merge from submit-queue
Remove deprecated init-container in annotations
fixes#50655fixes#51816closes#41004fixes#51816
Builds on #50654 and drops the initContainer annotations on conversion to prevent bypassing API server validation/security and targeting version-skewed kubelets that still honor the annotations
```release-note
The deprecated alpha and beta initContainer annotations are no longer supported. Init containers must be specified using the initContainers field in the pod spec.
```
Automatic merge from submit-queue (batch tested with PRs 51805, 51725, 50925, 51474, 51638)
Flexvolume dynamic plugin discovery: Prober unit tests and basic e2e test.
**What this PR does / why we need it**: Tests for changes introduced in PR #50031 .
As part of the prober unit test, I mocked filesystem, filesystem watch, and Flexvolume plugin initialization.
Moved the filesystem event goroutine to watcher implementation.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#51147
**Special notes for your reviewer**:
First commit contains added functionality of the mock filesystem.
Second commit is the refactor for moving mock filesystem into a common util directory.
Third commit is the unit and e2e tests.
**Release note**:
```release-note
NONE
```
/release-note-none
/sig storage
/assign @saad-ali @liggitt
/cc @mtaufen @chakri-nelluri @wongma7
Automatic merge from submit-queue (batch tested with PRs 51666, 49829, 51058, 51004, 50938)
Fix threshold notifier build tags
**What this PR does / why we need it**:
Cross building from darwin is currently broken on the following error:
```
# k8s.io/kubernetes/pkg/kubelet/eviction
pkg/kubelet/eviction/threshold_notifier_unsupported.go:25: NewMemCGThresholdNotifier redeclared in this block
previous declaration at pkg/kubelet/eviction/threshold_notifier_linux.go:38
```
It looks like #49300 broke the build tags introduced in #38630 and #37384. This fixes the build tag on `threshold_notifier_unsupported.go` as the cgo requirement was removed from `threshold_notifier_linux.go`.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#50935
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 45724, 48051, 46444, 51056, 51605)
Mount propagation in kubelet
Together with #45724 it implements mount propagation as proposed in https://github.com/kubernetes/community/pull/659
There is:
- New alpha annotation that allows user to explicitly set propagation mode for each `VolumeMount` in pod containers (to be replaced with real `VolumeMount.Propagation` field during beta) + validation + tests. "Private" is the default one (= no change to existing pods).
I know about proposal for real API fields for alpha feature in https://docs.google.com/document/d/1wuoSqHkeT51mQQ7dIFhUKrdi3-1wbKrNWeIL4cKb9zU/edit, but it seems it's not implemented yet. It would save me quite lot of code and ugly annotation.
- Updated CRI API to transport chosen propagation to Docker.
- New `kubelet --experimental-mount-propagation` option to enable the previous bullet without modifying types.go (worked around with changing `KubeletDeps`... not nice, but it's better than adding a parameter to `NewMainKubelet` and removing it in the next release...)
```release-note
kubelet has alpha support for mount propagation. It is disabled by default and it is there for testing only. This feature may be redesigned or even removed in a future release.
```
@derekwaynecarr @dchen1107 @kubernetes/sig-node-pr-reviews
Automatic merge from submit-queue
Make /var/lib/kubelet as shared during startup
This is part of ~~https://github.com/kubernetes/community/pull/589~~https://github.com/kubernetes/community/pull/659
We'd like kubelet to be able to consume mounts from containers in the future, therefore kubelet should make sure that `/var/lib/kubelet` has shared mount propagation to be able to see these mounts.
On most distros, root directory is already mounted with shared mount propagation and this code will not do anything. On older distros such as Debian Wheezy, this code detects that `/var/lib/kubelet` is a directory on `/` which has private mount propagation and kubelet bind-mounts `/var/lib/kubelet` as rshared.
Both "regular" linux mounter and `NsenterMounter` are updated here.
@kubernetes/sig-storage-pr-reviews @kubernetes/sig-node-pr-reviews
@vishh
Release note:
```release-note
Kubelet re-binds /var/lib/kubelet directory with rshared mount propagation during startup if it is not shared yet.
```
Automatic merge from submit-queue (batch tested with PRs 51590, 48217, 51209, 51575, 48627)
Skip system container cgroup stats if undefined
**What this PR does / why we need it**:
the kubelet /stats/summary endpoint tried to look up cgroup stats for containers that are not required. this polluted logs with messages about not finding stats for "" container. this pr skips cgroup stats if the cgroup name is not specified (they are optional anyway)
**Special notes for your reviewer**:
i think this was a regression from recent refactor.
**Release note**:
```release-note
NONE
```
Kubernetes Submit Queue