Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
should not fmt.printf in the e2e test
**What this PR does / why we need it**:
should not fmt.printf in the e2e test
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
none
```
Automatic merge from submit-queue (batch tested with PRs 67071, 66906, 66722, 67276, 67039). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
#50102 Task 1: Move apimachinery/pkg/watch.Until into client-go/tools/watch.UntilWithoutRetry
**What this PR does / why we need it**:
This is a split off from https://github.com/kubernetes/kubernetes/pull/50102 to go in smaller pieces.
Moves `apimachinery/pkg/watch.Until` into `client-go/tools/watch.UntilWithoutRetry` and adds context so it is cancelable.
**Release note**:
```release-note
NONE
```
**Dev release note**:
```dev-release-note
`apimachinery/pkg/watch.Until` has been moved to `client-go/tools/watch.UntilWithoutRetry`.
While switching please consider using the new `client-go/tools/watch.UntilWithSync` or `client-go/tools/watch.Until`.
```
/cc @smarterclayton @kubernetes/sig-api-machinery-pr-reviews
/milestone v1.12
/priority important-soon
/kind bug
(bug after the main PR which is this split from)
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Remove ARCH specific image consideration from e2e tests
Change-Id: I309fec49b030a4d457890f25d2f69e7c641c03fd
**What this PR does / why we need it**:
All e2e test images are now using multi-arch manifests so we should stop
looking up and using images that are specific to runtime.GOARCH
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 67058, 67083, 67220, 67222, 67209). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
use const v1.ProtocolTCP replace of string TCP
**What this PR does / why we need it**:
use const v1.ProtocolTCP replace of string TCP
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
All e2e test images are now using multi-arch manifests so we should stop
looking up and using images that are specific to runtime.GOARCH
Change-Id: I5f3fd6e9a42b9fb88891c19e28a2dfcf7a14be82
Automatic merge from submit-queue (batch tested with PRs 67061, 66589, 67121, 67149). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Get load balancer name per provider
**What this PR does / why we need it**:
GetLoadBalancerName() should be implemented per cloud provider as opposed to one neutral implementation.
This PR will address this by moving `cloudprovider.GetLoadBalancerName()` to the `LoadBalancer interface` and then provide an implementation for each cloud provider, while maintaining previously expected functionality.
**Which issue(s) this PR fixes**:
Fixes [#43173](https://github.com/kubernetes/kubernetes/issues/43173)
**Special notes for your reviewer**:
This is a work in progress. Looking for feedback as I work on this, from any interested parties.
**Release note**:
```release-note
NONE
```
Use the same pattern everywhere in the e2e test
harness, use busybox (from dockerhub) instead
of using the one from k8s.gcr.io registry.
Change-Id: I57c3b867408c1f9478a8909c26744ea0368ff003
Automatic merge from submit-queue (batch tested with PRs 62444, 66358, 66724, 66726). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
GetMasterHost should not return port
**What this PR does / why we need it**:
masterUrl.Host may include port, e.g. 10.114.51.201:8443
GetMasterHost() should not return port
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
NONE
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 64695, 65982, 65908). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Collecting etcd metrics
**What this PR does / why we need it**:
Adding etcd metrics to performance test log.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
ref #64030
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 64593, 65117, 65629, 65827, 65686). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add e2e tests for volumeMode of persistent volume
This set of e2e tests is to confirm that persistent volume works well for all volumeModes.
Coverage of the tests are shown in the figure of [Test cases], below.
Once implementation policy is confirmed to be good, we can add plugins and test cases to this.
[Test cases]
```
# plugin volumeMode Test case Expectation
--- ---------- -------------- ------------------------------------------------------ ------------
1 iSCSI Block (a) Create Pod with PV and confirm Read/Write to PV Success
2 iSCSI Block (b) Create Pod and restart kubelet and confirm PV Success
3 iSCSI Block (c) Create Pod and restart kubelet while deleting PV Success
4 iSCSI FileSystem (a) Create Pod with PV and confirm Read/Write to PV Success
5 iSCSI FileSystem (b) Create Pod and restart kubelet and confirm PV Success
6 iSCSI FileSystem (c) Create Pod and restart kubelet while deleting PV Success
7 RBD Block (a) Create Pod with PV and confirm Read/Write to PV Success
8 RBD Block (b) Create Pod and restart kubelet and confirm PV Success
9 RBD Block (c) Create Pod and restart kubelet while deleting PV Success
10 RBD FileSystem (a) Create Pod with PV and confirm Read/Write to PV Success
11 RBD FileSystem (b) Create Pod and restart kubelet and confirm PV Success
12 RBD FileSystem (c) Create Pod and restart kubelet while deleting PV Success
13 CephFS Block (a) Create Pod with PV and confirm Read/Write to PV Fail
14 CephFS FileSystem (a) Create Pod with PV and confirm Read/Write to PV Success
15 CephFS FileSystem (b) Create Pod and restart kubelet and confirm PV Success
16 CephFS FileSystem (c) Create Pod and restart kubelet while deleting PV Success
17 NFS Block (a) Create Pod with PV and confirm Read/Write to PV Fail
18 NFS FileSystem (a) Create Pod with PV and confirm Read/Write to PV Success
19 NFS FileSystem (b) Create Pod and restart kubelet and confirm PV Success
20 NFS FileSystem (c) Create Pod and restart kubelet while deleting PV Success
```
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#56803
**Special notes for your reviewer**:
/sig storage
@gnufied @rootfs @jsafrane @jeffvance
This PR is intend to reflect the comments for the below PR.
https://github.com/kubernetes/kubernetes/pull/56804
**Release note**:
```release-note
NONE
```
This set of e2e tests is to confirm that persistent volume works well for all volumeModes.
Coverage of the tests are shown in the figure of [Test cases], below.
Once implementation policy is confirmed to be good, we can add plugins and test cases to this.
[Test cases]
# plugin volumeMode Test case Expectation
--- ---------- -------------- ------------------------------------------------------ ------------
1 iSCSI Block (a) Create Pod with PV and confirm Read/Write to PV Success
2 iSCSI FileSystem (a) Create Pod with PV and confirm Read/Write to PV Success
3 RBD Block (a) Create Pod with PV and confirm Read/Write to PV Success
4 RBD FileSystem (a) Create Pod with PV and confirm Read/Write to PV Success
5 CephFS Block (a) Create Pod with PV and confirm Read/Write to PV Fail
6 CephFS FileSystem (a) Create Pod with PV and confirm Read/Write to PV Success
7 NFS Block (a) Create Pod with PV and confirm Read/Write to PV Fail
8 NFS FileSystem (a) Create Pod with PV and confirm Read/Write to PV Success
fixes: #56803
Automatic merge from submit-queue (batch tested with PRs 65339, 65343, 65324, 65335, 65367). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Introduce scheduler CPU/Memory profile-gathering in density test
This should help us get more reliable/realistic data for scheduler (from our real-cluster scalability tests).
/cc @wojtek-t
fyi - @davidopp @bsalamat @misterikkit
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 65301, 65291, 65307, 63845, 65313). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix scheduler reset metrics bug in testinfra
/cc @krzysied
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Split scheduler latency metric to fine-grained steps
This splits the summary metric we recently added into finer steps. It should be very useful for performance experiments.
/cc @wojtek-t
fyi - @bsalamat @misterikkit
Strictly speaking this is a breaking change, but since this metric was added only ~week ago I think it should fine (we should port this change to 1.11).
```release-note
Split 'scheduling_latency_seconds' metric into finer steps (predicate, priority, premption)
```
Automatic merge from submit-queue (batch tested with PRs 64140, 64898, 65022, 65037, 65027). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add e2e regression tests for the kubelet being secure
**What this PR does / why we need it**:
This PR does,
1. The kubelet cAdvisor port (4194) can't be reached, neither via the API server proxy nor directly on the public IP address
2. The kubelet read-only port (10255) can't be reached, neither via the API server proxy nor directly on the public IP address
3. The kubelet can delegate ServiceAccount tokens to the API server
4. The kubelet's main port (10250) has both authentication (should fail with no credentials) and authorization (should fail with insufficient permissions) set-up
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixeskubernetes/kubeadm#838
**Special notes for your reviewer**:
/cc luxas tallclair
**Release note**:
```release-note
Add e2e regression tests for the kubelet being secure
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Better log line in e2e
**What this PR does / why we need it**:
Very minor improvement to logs in e2e tests when creating a namespace.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 64252, 64307, 64163, 64378, 64179). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add reliable wait for volume server startup.
Remove sleep(20) and check for readiness of volume servers by checking logs.
**Special notes for your reviewer**:
I prefer checking logs over readiness probe, as it's not easy to probe for Ceph/iSCSI/NFS/Gluster server readiness.
```release-note
NONE
```
@jeffvance @copejon @rootfs @msau42, PTAL
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Adding scale error retries
**What this PR does / why we need it**:
ScaleWithRetries will retry all retryable errors, not only conflict error.
ref #63030
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 60699, 63780). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
e2e/storage: parameterize container images
**What this PR does / why we need it**:
The CSI integration test for hostpath was hard-coded to use the latest
stable release of the sidecar and hostpath container images. This
makes sense for regression testing of changes made in Kubernetes
itself, but the same test is also useful for testing the "canary"
images on quay.io before tagging them as a new release or for testing
locally produced images. Both is now possible via command line
parameters.
**Which issue(s) this PR fixes**:
Related-to: kubernetes-csi/docs#23
**Special notes for your reviewer**:
The commit message has usage instructions.
```release-note
NONE
```
/sig storage
Automatic merge from submit-queue (batch tested with PRs 63348, 63839, 63143, 64447, 64567). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Move pkg/scheduler/schedulercache -> pkg/scheduler/cache
**What this PR does / why we need it**:
Move pkg/scheduler/schedulercache -> pkg/scheduler/cache
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#63813
**Special notes for your reviewer**:
In order to prevent name conflicts still rename the `cache` to `schedulercache`.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 62460, 64480, 63774, 64540, 64337). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Modified regional PD test to fetch template name from GCE
**What this PR does / why we need it**: Previously, the regional PD failover e2e test assumes a specific relationship between the names of an instance group and its corresponding template. It turns out to not always hold true for different types of clusters. Instead, the test should fetch the correct template name by calling out to GCE.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Part of #59988
Need to cherry pick this back to 1.10 along with #64223
**Release note**:
```release-note
NONE
```
/assign @saad-ali @wojtek-t
/sig storage
Automatic merge from submit-queue (batch tested with PRs 63328, 64316, 64444, 64449, 64453). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fixing scheduling latency metrics
**What this PR does / why we need it**:
Allows to measure and to display scheduling latency metrics during tests. Provides new functionality of resetting scheduler latency metrics.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#63493
**Special notes for your reviewer**:
E2eSchedulingLatency, SchedulingAlgorithmLatency, BindingLatency are now available
as subtypes of OperationLatency.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Move Ceph server secret creation to common code.
The secret should be created only on one place.
**Release note**:
```release-note
NONE
```
@jeffvance @copejon @rootfs @msau42 PTAL
Automatic merge from submit-queue (batch tested with PRs 64399, 64324, 64404, 64406, 64396). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add daemonset when to getReplicasFromRuntimeObject when cleaning objects in e2e
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#64359
**Special notes for your reviewer**:
/assign @dims
**Release note**:
```release-note
NONE
```
Putting the command line argument handling into the central test
context seems like the better solution, in particular considering that
argument handling might get changed in the future to use Viper.
Automatic merge from submit-queue (batch tested with PRs 63859, 63979). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Drop reapers
/assign @deads2k @juanvallejo
**Release note**:
```release-note
kubectl delete does not use reapers for removing objects anymore, but relies on server-side GC entirely
```
Automatic merge from submit-queue (batch tested with PRs 63232, 64257, 64183, 64266, 64134). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Measure scheduler throughput in density test
This is a step towards exposing scheduler-related metrics on [perf-dash](http://perf-dash.k8s.io/).
This particular PR adds scheduler throughput computation and makes the results available in our test artifacts.
So if you do some experiments, you'll have some historical baseline data to compare against.
xref https://github.com/kubernetes/kubernetes/issues/63493
fyi - @wojtek-t @davidopp @bsalamat @misterikkit
cc @kubernetes/sig-scheduling-misc @kubernetes/sig-scalability-misc
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 64013, 63896, 64139, 57527, 62102). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Refactor test utils that deal with Kubelet metrics for clarity
I found these functions hard to understand, because the names did not
accurately reflect their behavior. For example, GetKubeletMetrics
assumed that all of the metrics passed in were measuring latency.
The caller of GetKubeletMetrics was implicitly making this assumption,
but it was not obvious at the call site.
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 62756, 63862, 61419, 64015, 64063). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Handle TERM signal to reduce pod terminating time.
**What this PR does / why we need it**:
Register signal handler for `TERM`.
For container, process is run as PID 1. PID 1 is special in linux, it ignore any signal with the default action. So process will not terminate on `SIGINT` or `SIGTERM` unless it is coded to do so.
By default, docker use `TERM` signal to termiante pods, it will wait 10 seconds before sending `KILL` signal.
```
$ docker run -d --rm --name test busybox sh -c 'while true; do sleep 1; done'
aa827df5d7bfffc5ca4fae2429d0b761a5a142c57ba3e1faa59b305c92f1c875
$ time docker stop test
test
real 0m10.408s
user 0m0.048s
sys 0m0.020s
```
It's better to register a exit handler for `TERM`, it reduces a lot of time in waiting pods to termiante.
```
$ docker run -d --rm --name test busybox sh -c 'trap exit TERM; while true; do sleep 1; done'
e331bff454dba8e45df6065c3bd2a928e1c41303aafdf88ede38def3e4e5781f
$ time docker stop test
test
real 0m0.690s
user 0m0.048s
sys 0m0.020s
```
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 62756, 63862, 61419, 64015, 64063). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Use apps/v1 Deployment/ReplicaSet in controller and kubectl
This updates the Deployment controller and integration/e2e tests to use apps/v1, as part of #55714.
This also requires updating any other components that use the `deployment/util` package, most notably `kubectl`. That means client versions 1.11 and above will only work with server versions 1.9 and above. This is well within our client-server version skew policy of +/-1 minor version.
However, this PR *only* updates the parts of `kubectl` that used `deployment/util`. So although kubectl now requires apps/v1, it still also depends on extensions/v1beta1. Migrating other parts of kubectl to apps/v1 is beyond the scope of this PR, which was just to change the Deployment controller and fix all the fallout.
```release-note
kubectl: This client version requires the `apps/v1` APIs, so it will not work against a cluster version older than v1.9.0. Note that kubectl only guarantees compatibility with clusters that are +/-1 minor version away.
```
I found these functions hard to understand, because the names did not
accurately reflect their behavior. For example, GetKubeletMetrics
assumed that all of the metrics passed in were measuring latency.
The caller of GetKubeletMetrics was implicitly making this assumption,
but it was not obvious at the call site.
Automatic merge from submit-queue (batch tested with PRs 63920, 63716, 63928, 60553, 63946). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Wait for pod deletion instead of termination in reconstruction test
**What this PR does / why we need it**:
Change volume test to wait for pod deletion instead of pod termination
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Addresses https://github.com/kubernetes/kubernetes/issues/63923
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Deprecate InfluxDB cluster monitoring
InfluxDB cluster monitoring addon will no longer be supported and will be removed in k8s 1.12.
Default monitoring solution will be changed to `standalone`.
Heapster will still be deployed for backward compatibility of `kubectl top`
```release-note
Stop using InfluxDB as default cluster monitoring
InfluxDB cluster monitoring is deprecated and will be removed in v1.12
```
cc @piosz
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Auto-calculate allowed-not-ready-nodes in test framework
Actually we (sig-scalability) are pretty much the only users of this flag.
This reduces the overhead of having to provide its value based on num-nodes each time we run our tests.
/cc @wojtek-t
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 63367, 63718, 63446, 63723, 63720). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
finish new dynamic client and deprecate old dynamic client
Builds on a couple other pulls. This completes the transition to the new dynamic client.
@kubernetes/sig-api-machinery-pr-reviews
@caesarxuchao @sttts
```release-note
The old dynamic client has been replaced by a new one. The previous dynamic client will exist for one release in `client-go/deprecated-dynamic`. Switch as soon as possible.
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
e2e: add a tooling argument to differentiate tooling
We expect lots of tools to be able to install on provider=gce - the cluster
API, kops, kube-up etc.
We introduce a new optional flag to e2e ('tooling') to enable switching on
the tooling, not just the cloud.
This will prove useful for upgrade tests, for example, where the mechanism
will likely vary by tooling, but is currently tightly bound to the provider
(i.e. cloud)
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 63246, 63185). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add scale-up test from 0 for GPU node-pool
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
should use time.Since instead of time.Now().Sub
**What this PR does / why we need it**:
should use time.Since instead of time.Now().Sub
**Special notes for your reviewer**:
We expect lots of tools to be able to install on provider=gce - the
cluster API, kops, kube-up etc.
We introduce a new optional flag to e2e ('tooling') to enable switching
on the tooling, not just the cloud.
This will prove useful for upgrade tests, for example, where the
mechanism will likely vary by tooling, but is currently tightly bound to
the provider (i.e. cloud)
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
remove versioning interface
Builds on prior restmapping/converter separation to completely remove the versioning interface which isn't needed.
intersection of @kubernetes/sig-api-machinery-pr-reviews and @kubernetes/sig-cli-maintainers again
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 61962, 58972, 62509, 62606). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Allow a test suite reusing framework to register namespaces to delete
If the suite bypasses CreateNamespace (because it wants to create more
specialized namespaces) it has no way to register deletes.
@liggitt this was something that came up in an older rebase and was missed (for things creating projects)
- Add PodStartShortTimeout and ClaimProvisionShortTimeout constants.
- Change framework.PodStartTimeout to framework.PodStartShortTimeout in
persistent_volumes-local.go. Busybox image is very small, no need to
wait for a long time.
Automatic merge from submit-queue (batch tested with PRs 62694, 62569, 62646, 61633, 62433). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add GCE-PD CSI Driver test to E2E test suite
Fixes: #60462
/sig storage
/kind technical-debt
/assign @saad-ali @msau42
**What this PR does / why we need it**:
This PR adds an E2E test for the GCE-PD CSI driver that deploys the driver in a production-like setting and tests whether dynamic provisioning with the driver is possible.
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 62694, 62569, 62646, 61633, 62433). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Report events to apiserver in local volume plugin.
**What this PR does / why we need it**:
See https://github.com/kubernetes/kubernetes/pull/62102#discussion_r179238429.
Report events to apiserver in local volume plugin.
- Add VolumeHost.GetEventRecorder() method
- Add related e2e tests
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#62248
**Special notes for your reviewer**:
Example output of `kubectl describe pods`:
```
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 7s default-scheduler Successfully assigned e2e-tests-persistent-local-volumes-test-x4h5x/security-context-670da435-4174-11e8-9098-000c29bb0377 to 127.0.0.1
Warning AlreadyMountedVolume 7s kubelet, 127.0.0.1 The requested fsGroup is 4321, but the volume local-pvfbb76 has GID 1234. The volume may not be shareable.
Normal SuccessfulMountVolume 7s kubelet, 127.0.0.1 MountVolume.SetUp succeeded for volume "default-token-996xr"
Normal SuccessfulMountVolume 7s kubelet, 127.0.0.1 MountVolume.SetUp succeeded for volume "local-pvfbb76"
Normal Pulled 6s kubelet, 127.0.0.1 Container image "k8s.gcr.io/busybox:1.24" already present on machine
Normal Created 6s kubelet, 127.0.0.1 Created container
Normal Started 6s kubelet, 127.0.0.1 Started container
```
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add prometheus cluster monitoring addon.
This PR adds new cluster monitoring addon based on prometheus.
It adds prometheus deployment with e2e tests.
Additional components will be added iterativly in future.
Manifests based on current Helm chart.
At current state it's not intended for production use.
cc @piosz @kawych @miekg
```release-note
Add prometheus cluster monitoring addon to kube-up
```
/sig instrumentation
/kind feature
/priority important-soon
Automatic merge from submit-queue (batch tested with PRs 62324, 61459, 62475, 62476, 61914). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Bump image in ingress downgrade test
**Release note**:
```release-note
None
```
/assign @nicksardo
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Reimplement migrate-if-needed.sh in go
The `migrate-if-needed.sh` script was already partially implemented in go (see the attachlease and rollback sub-dirs), but was still unnecessarily difficult to understand and test. This closely reimplements the original logic but with improved code structure, error handling and testing.
Where possible, go code that was previously executed as separate binaries is now statically linked into a single 'migrate' go cobra CLI app, which is then thinly wrapped by`migrate-if-needed.sh`.
There are numerous additional improvements that need to be made, but will be submitted in future PRs. This PR is focused on achieving parity with the pre-existing functionality and introducing some much needed test coverage, in particular HA cluster upgrade test coverage.
It appears that the `attachlease` and `rollback` go binaries are no longer needed as standalones and so I have consolidated them into the new `migrate` go binary. Other than that, this change aims to be 100% backward compatible.
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 46903, 61721, 62317). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Double container probe timeout
in some environments, we see a combination of start latency
and the corresponding effect on sync pod latency causing status
manager to fail to report within the 2 minute window.
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 62192, 61866, 62206, 62360). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Remove rkt references in the codebase
```release-note
None
```
Automatic merge from submit-queue (batch tested with PRs 62209, 62281, 61792, 62285, 61973). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add ingress e2e test for multiple TLS (SNI) support
**What this PR does / why we need it**:
Add an e2e test for multiple TLS support in ingress-gce.
**Release note**:
```release-note
None
```
/assign @MrHohn
/hold
Automatic merge from submit-queue (batch tested with PRs 62208, 62114, 62144, 60460, 62214). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
removes job scaler, continued
Builds on https://github.com/kubernetes/kubernetes/pull/61912 (original commit is there for credit/blame)
This keeps all the updates to the scaler building and all the test and reaper cleanup. It just keeps a fake job scaler around for a different command path and the reaper.
/assign @p0lyn0mial
/assign @soltysh
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 60102, 59970, 60021, 62011, 62080). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
[GCE] Ingress HTTP2 e2e test
**What this PR does / why we need it**:
- Adds e2e test for bringing up an HTTP2 Ingress, converting it to HTTPS, then back to HTTP2
- Update echoserver image to 1.10
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 62063, 62169, 62155, 62139, 61445). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Wait longer in pod cleanup
I've checked the [garbage collector tests](827b3d77cd/test/e2e/apimachinery/garbage_collector.go (L557)) and usually they're waiting up to a minute to clean the resources. So I'm proposing to use that same values when waiting for the pod cleanup. I was not able to reproduce the problem locally, but from reading the logs it looks like the problem might be with just the timing nothing else.
Related to #61484
/assign @Liujingfang1 @mengqiy
Automatic merge from submit-queue (batch tested with PRs 54997, 61869, 61816, 61909, 60525). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Updating multicluster e2e test to ensure that controller only creates instance groups
Ref https://github.com/kubernetes/ingress-gce/issues/182
Updating the multicluster ingress e2e test to also verify that the controller does not create any other resource than instance group.
I was planning to do that using annotations first, but turns out that we have the right check for annotations: 61d574a2c7/pkg/controller/controller.go (L317) :)
Am using `length(ListResource) == 0` now but have still kept the annotation check.
cc @nicksardo @MrHohn @G-Harmon
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
[e2e ingress-gce] Run preshared-cert and backend HTTPS tests with kubemci
**What this PR does / why we need it**:
Make preshared-cert and backside-reencryption tests compatible with kubemci.
Test is currently failing with the symptom described on https://github.com/GoogleCloudPlatform/k8s-multicluster-ingress/issues/131#issuecomment-377098544.
@nikhiljindal @g-harmon
/hold
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #NONE
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 61904, 61565, 61401, 61432, 61772). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add retry to AssertCleanup
**What this PR does / why we need it**:
Add retry in AssertCleanup for e2e kubectl test to remove the flakeness
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#61484
**Special notes for your reviewer**:
This function is only used in e2e kubectl test. It doesn't affect other tests' behavior.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 61848, 61188, 56363, 61357, 61838). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Affinity test
**What this PR does / why we need it**:
Add e2e test for service session affinity, including all three types of services and externalTravicPolicy=Local.
**Which issue(s) this PR fixes**:
Fixes#55520
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
fix todo: use the ServiceTestJig replace of service in e2e/network/service.go
**What this PR does / why we need it**:
fix todo: use the ServiceTestJig replace of service in e2e/network/service.go
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 61378, 60915, 61499, 61507, 61478). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Capture pod startup phases as metrics
Learning from https://github.com/kubernetes/kubernetes/issues/60589, we should also start collecting and graphing sub-parts of pod-startup latency.
/sig scalability
/kind feature
/priority important-soon
/cc @wojtek-t
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 61378, 60915, 61499, 61507, 61478). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Bump image for ingress downgrade test
**What this PR does / why we need it**:
Bumps the image to the latest released version. This should also fix the failing test.
/assign @bowei
**Release note**:
```release-note
None
```
Automatic merge from submit-queue (batch tested with PRs 61453, 61393, 61379, 61373, 61494). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fixing kubemci conformance tests
Ref https://github.com/GoogleCloudPlatform/k8s-multicluster-ingress/issues/131
Changes:
* Add a static ip annotation while running the tests for kubemci. kubemci requires the IP to be preallocated.
* Add a default backend service. I have added it in the spec directly, so the change will be for ingress controller as well which should be fine. kubemci requires users to specify a default backend service that they need to ensure exists in all clusters.
* Disabled update SSL cert test for kubemci since it does not support that.
* Minor logging fixes.
Verified by running the tests locally that they now pass.
```
$ KUBECONFIG=~/.kube/config KUBE_MASTER_IP="<IP>" go run hack/e2e.go -- --test --test_args="--ginkgo.focus=kubemci"
• [SLOW TEST:629.179 seconds]
Loadbalancing: L7
/usr/local/google/home/nikhiljindal/code/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/framework.go:22
GCE [Slow] [Feature:kubemci]
/usr/local/google/home/nikhiljindal/code/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/ingress.go:604
should conform to Ingress spec
/usr/local/google/home/nikhiljindal/code/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/ingress.go:637
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSMar 19 19:45:59.242: INFO: Running AfterSuite actions on all node
Mar 19 19:45:59.242: INFO: Running AfterSuite actions on node 1
Ran 1 of 820 Specs in 631.245 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 819 Skipped PASS
Ginkgo ran 1 suite in 10m32.602848558s
Test Suite Passed
```
cc @G-Harmon @MrHohn
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 60980, 61273, 60811, 61021, 61367). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Use apps/v1 ReplicaSet in controller and tests.
This updates the RS/RC controller and RS integration/e2e tests to use apps/v1 ReplicaSet, as part of #55714.
It does *not* update the Deployment controller, nor its integration/e2e tests, to use apps/v1 ReplicaSet. That will be done in a separate PR (#61419) because Deployment has many more tendrils embedded throughout the system.
```release-note
Conformance: ReplicaSet must be supported in the `apps/v1` version.
```
/assign @janetkuo
Automatic merge from submit-queue (batch tested with PRs 60793, 61181, 61267, 61252, 61334). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Refactor disruptive tests to use more volume types
Refactor storage disruptive tests to use different volume types. Also mark existing tests to be NFS specific. Fixes https://github.com/kubernetes/kubernetes/issues/61150
cc @jeffvance @jingxu97
/sig storage
```release-note
None
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix test images
These commits fix volume_io tests for iSCSI and Ceph RBD. Both server images were quite old (Fedora 22), so I updated them to ~~something more stable (CentOS 7) and to newer Ceph (Jewel, 10.2.7).~~ something newer (Fedora 26).
The most important fix is that the test volumes have 120 MB so volume_io test can actually run - the tests put 100MB file to the volume to check its persistence.
When mount containers in #53440 are merged I'll try to run the tests regularly with every PR (or merge) so we catch regressions quickly.
```release-note
NONE
```
/sig testing
/sig storage
/assign @jeffvance
Fixes: #56725
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Bump to etcd 3.1.12 to pick up critical fix
etcd [3.1.12](https://github.com/coreos/etcd/releases/tag/v3.1.12) (as well as 3.2.17 and 3.3.2) was released yesterday to fix a bug critical to kubernetes:
Fix [mvcc "unsynced" watcher restore operation](https://github.com/coreos/etcd/pull/9297).
- "unsynced" watcher is watcher that needs to be in sync with events that have happened.
- That is, "unsynced" watcher is the slow watcher that was requested on old revision.
- "unsynced" watcher restore operation was not correctly populating its underlying watcher group.
- Which possibly causes [missing events from "unsynced" watchers](https://github.com/coreos/etcd/issues/9086).
This will be backported to 1.9 as well.
Release note:
```release-note
Upgrade the default etcd server version to 3.1.12 to pick up critical etcd "mvcc "unsynced" watcher restore operation" fix.
```
cc @gyuho @wojtek-t @shyamjvs @timothysc @jdumars
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
[e2e service] Fix CleanupGCEResources for regional test
*What this PR does / why we need it**:
From https://k8s-testgrid.appspot.com/google-gke-staging#gke-staging-1-8-1-9-upgrade-regional-cluster&width=20, regional cluster test is failing because the GCE resource cleanup function attempts to parse region from `--gcp-zone` while regional cluster only set `--gcp-region`.
This PR pipes region into the cleanup function as well. This will need to be cherrypicked to 1.8 and 1.9.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #NONE
**Special notes for your reviewer**:
/assign @bowei @wojtek-t
cc @nikhiljindal to see if there is anything should be fixed for federation.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
[e2e service] Move apiserver restart validation logic into util
**What this PR does / why we need it**:
Follow up of #60906, on GKE apiserver pod is invisible on k8s, hence test is failing.
This PR bakes the restart validation logic into the util function instead so it could be env-awared.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#60761
**Special notes for your reviewer**:
Sorry for the noise.
/assign @rramkumar1 @bowei
cc @krzyzacy
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
[e2e service] Refine apiserver restart logic
**What this PR does / why we need it**:
Ref https://github.com/kubernetes/kubernetes/issues/60761#issuecomment-371308569, wait for apiserver's restart count increases before proceeding the test.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes (hopefully) #60761
**Special notes for your reviewer**:
/assign @rramkumar1 @bowei
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Bump etcd server patch version to 3.2.16
etcd 3.2.16 contains a critical fix for HA clusters: https://github.com/coreos/etcd/pull/9281
Also, update newly added tests to use `REGISTRY` make variable.
Release note:
```release-note
Upgrade the default etcd server version to 3.2.16
```
After client pod is stopped we need to wait for a while for kubelet to
unmount and clean up the client pod before we can destroy the server pod.
Killing e.g. iSCSI server too early may result in stale attached device
that cannot be removed.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Volume deletion should be idempotent
- Describe* calls should return `aws.Error` so caller can handle individual errors. `aws.Error` already has enough context (`"InvalidVolume.NotFound: The volume 'vol-0a06cc096e989c5a2' does not exist"`)
- Deletion of already deleted volume should succeed.
**Release note**:
Fixes: #60778
```release-note
NONE
```
/sig storage
/sig aws
/assign @justinsb @gnufied
Maciej Borsz