Automatic merge from submit-queue (batch tested with PRs 66064, 66040). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Re-enable write-read pv check in volume provisioning tests
**What this PR does / why we need it**: Except for the recently-added CSI tests, every test that uses testDynamicProvisioning in volume_provisioning has been skipping the
`By("checking the created volume is writable and has the PV's mount options")`
and
`By("checking the created volume is readable and retains data")`
steps due to an unintentional change in https://github.com/kubernetes/kubernetes/pull/59879
**Special notes for your reviewer**: The 'attach' variable name is not descriptive at all, default behaviour is unclear. Tests will take longer of course, but that was always the intention, to not only test provisioning of PV is according to PVC but that you can actually r/w it (hence e2e).
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add local volume pod affinity tests.
**What this PR does / why we need it**:
Add local volume pod affinity tests.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#56100 (with already merged prs #59682, #60260)
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 64593, 65117, 65629, 65827, 65686). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add e2e tests for volumeMode of persistent volume
This set of e2e tests is to confirm that persistent volume works well for all volumeModes.
Coverage of the tests are shown in the figure of [Test cases], below.
Once implementation policy is confirmed to be good, we can add plugins and test cases to this.
[Test cases]
```
# plugin volumeMode Test case Expectation
--- ---------- -------------- ------------------------------------------------------ ------------
1 iSCSI Block (a) Create Pod with PV and confirm Read/Write to PV Success
2 iSCSI Block (b) Create Pod and restart kubelet and confirm PV Success
3 iSCSI Block (c) Create Pod and restart kubelet while deleting PV Success
4 iSCSI FileSystem (a) Create Pod with PV and confirm Read/Write to PV Success
5 iSCSI FileSystem (b) Create Pod and restart kubelet and confirm PV Success
6 iSCSI FileSystem (c) Create Pod and restart kubelet while deleting PV Success
7 RBD Block (a) Create Pod with PV and confirm Read/Write to PV Success
8 RBD Block (b) Create Pod and restart kubelet and confirm PV Success
9 RBD Block (c) Create Pod and restart kubelet while deleting PV Success
10 RBD FileSystem (a) Create Pod with PV and confirm Read/Write to PV Success
11 RBD FileSystem (b) Create Pod and restart kubelet and confirm PV Success
12 RBD FileSystem (c) Create Pod and restart kubelet while deleting PV Success
13 CephFS Block (a) Create Pod with PV and confirm Read/Write to PV Fail
14 CephFS FileSystem (a) Create Pod with PV and confirm Read/Write to PV Success
15 CephFS FileSystem (b) Create Pod and restart kubelet and confirm PV Success
16 CephFS FileSystem (c) Create Pod and restart kubelet while deleting PV Success
17 NFS Block (a) Create Pod with PV and confirm Read/Write to PV Fail
18 NFS FileSystem (a) Create Pod with PV and confirm Read/Write to PV Success
19 NFS FileSystem (b) Create Pod and restart kubelet and confirm PV Success
20 NFS FileSystem (c) Create Pod and restart kubelet while deleting PV Success
```
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#56803
**Special notes for your reviewer**:
/sig storage
@gnufied @rootfs @jsafrane @jeffvance
This PR is intend to reflect the comments for the below PR.
https://github.com/kubernetes/kubernetes/pull/56804
**Release note**:
```release-note
NONE
```
This set of e2e tests is to confirm that persistent volume works well for all volumeModes.
Coverage of the tests are shown in the figure of [Test cases], below.
Once implementation policy is confirmed to be good, we can add plugins and test cases to this.
[Test cases]
# plugin volumeMode Test case Expectation
--- ---------- -------------- ------------------------------------------------------ ------------
1 iSCSI Block (a) Create Pod with PV and confirm Read/Write to PV Success
2 iSCSI FileSystem (a) Create Pod with PV and confirm Read/Write to PV Success
3 RBD Block (a) Create Pod with PV and confirm Read/Write to PV Success
4 RBD FileSystem (a) Create Pod with PV and confirm Read/Write to PV Success
5 CephFS Block (a) Create Pod with PV and confirm Read/Write to PV Fail
6 CephFS FileSystem (a) Create Pod with PV and confirm Read/Write to PV Success
7 NFS Block (a) Create Pod with PV and confirm Read/Write to PV Fail
8 NFS FileSystem (a) Create Pod with PV and confirm Read/Write to PV Success
fixes: #56803
Automatic merge from submit-queue (batch tested with PRs 65064, 65218, 65260, 65241, 64372). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Added attach/mount/check steps to CSI Driver E2E tests
This PR makes the CSI Volume E2E tests actually go through the entire dynamic provisioning pipeline and test attach/mount/check file etc.
Fixes#64927
```release-note
None
```
Automatic merge from submit-queue (batch tested with PRs 65187, 65206, 65223, 64752, 65238). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fixed detection of inaccessible AWS encryption key.
AWS provisioner now checks if created encrypted volume gets "Available" or it gets silently deleted by AWS because StorageClass referenced invalid (e.g. non-existing) KMS key for encryption.
This seems to be the only way how to detect such invalid key, because Kubernetes may not have enough permission to check if the key exists.
**Which issue(s) this PR fixes**
Fixes#62171
**Special notes for your reviewer**:
**Release note**:
```release-note
AWS now checks for validity of ecryption key when creating encrypted volumes. Dynamic provisioning of encrypted volume may get slower due to these checks.
```
/sig aws
/sig storage
@kubernetes/sig-aws-misc
Automatic merge from submit-queue (batch tested with PRs 65123, 65176, 65139, 65084, 65056). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Enable dynamic provisioning tests on AWS.
The tests are not slow, they finish on GCE in ~46 seconds and on AWS
in ~103 seconds.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 64416, 63625, 60967, 64767, 64588). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Remove Feature:Volume from tests
These tests don't depend on any alpha/beta feature and they have correct `SkipUnlessProviderIs` tests so they run only on proper cloud.
/kind feature
/sig storage
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 60699, 63780). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
e2e/storage: parameterize container images
**What this PR does / why we need it**:
The CSI integration test for hostpath was hard-coded to use the latest
stable release of the sidecar and hostpath container images. This
makes sense for regression testing of changes made in Kubernetes
itself, but the same test is also useful for testing the "canary"
images on quay.io before tagging them as a new release or for testing
locally produced images. Both is now possible via command line
parameters.
**Which issue(s) this PR fixes**:
Related-to: kubernetes-csi/docs#23
**Special notes for your reviewer**:
The commit message has usage instructions.
```release-note
NONE
```
/sig storage
Automatic merge from submit-queue (batch tested with PRs 63348, 63839, 63143, 64447, 64567). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add block volume support to internal provisioners.
**What this PR does / why we need it**:
Internal provisioners now create filesystem PVs when block PVs are requested. This leads to unbindable PVCs.
In this PR, volume plugins that support block volumes provision block PVs when block is requested. All the other provisioners return clear error in `kubectl describe pvc`:
```
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ProvisioningFailed 7s (x2 over 18s) persistentvolume-controller Failed to provision volume with StorageClass "standard": kubernetes.io/cinder does not support block volume provisioning
```
```release-note
NONE
```
cc @kubernetes/vmware for vsphere changes
cc @andyzhangx for Azure changes
/assign @copejon @mtanino
Automatic merge from submit-queue (batch tested with PRs 62460, 64480, 63774, 64540, 64337). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Modified regional PD test to fetch template name from GCE
**What this PR does / why we need it**: Previously, the regional PD failover e2e test assumes a specific relationship between the names of an instance group and its corresponding template. It turns out to not always hold true for different types of clusters. Instead, the test should fetch the correct template name by calling out to GCE.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Part of #59988
Need to cherry pick this back to 1.10 along with #64223
**Release note**:
```release-note
NONE
```
/assign @saad-ali @wojtek-t
/sig storage
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Move Ceph server secret creation to common code.
The secret should be created only on one place.
**Release note**:
```release-note
NONE
```
@jeffvance @copejon @rootfs @msau42 PTAL
Putting the command line argument handling into the central test
context seems like the better solution, in particular considering that
argument handling might get changed in the future to use Viper.
Automatic merge from submit-queue (batch tested with PRs 64174, 64187, 64216, 63265, 64223). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Moving Regional PD e2e tests to regular test suites
**What this PR does / why we need it**: Regional PD e2e tests previously only run on alpha suites. Need to move them to the regular test suites
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Part of #59988
**Special notes for your reviewer**: Will be cherry-picked to 1.10 once merged.
**Release note**:
```release-note
NONE
```
/sig gcp
/sig storage
/assign @msau42 @saad-ali
Automatic merge from submit-queue (batch tested with PRs 64013, 63896, 64139, 57527, 62102). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Setup fsGroup for local volumes correctly
**What this PR does / why we need it**:
This pr fixes fsGroup check in local volume in containerized kubelet. Except this, it also fixes fsGroup check when volume source is a normal directory whether kubelet is running on the host or in a container.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#61741
**Special notes for your reviewer**:
Bind mounts are detected in `/proc/mounts`, but it does not contain root of mount for bind mounts. So `mount.GetMountRefsByDev()` cannot get all references if source is a normal directory. e.g.
```
# mkdir /tmp/src /mnt/dst
# mount --bind /tmp/src /tmp/src # required by local-volume-provisioner, see https://github.com/kubernetes-incubator/external-storage/pull/499
# mount --bind /tmp/src /mnt/dst
# grep -P 'src|dst' /proc/mounts
tmpfs /tmp/src tmpfs rw,nosuid,nodev,noatime,size=4194304k 0 0
tmpfs /mnt/dst tmpfs rw,nosuid,nodev,noatime,size=4194304k 0 0
# grep -P 'src|dst' /proc/self/mountinfo
234 409 0:42 /src /tmp/src rw,nosuid,nodev,noatime shared:30 - tmpfs tmpfs rw,size=4194304k
235 24 0:42 /src /mnt/dst rw,nosuid,nodev,noatime shared:30 - tmpfs tmpfs rw,size=4194304k
```
We need to compare root of mount and device in this case.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 62756, 63862, 61419, 64015, 64063). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Use apps/v1 Deployment/ReplicaSet in controller and kubectl
This updates the Deployment controller and integration/e2e tests to use apps/v1, as part of #55714.
This also requires updating any other components that use the `deployment/util` package, most notably `kubectl`. That means client versions 1.11 and above will only work with server versions 1.9 and above. This is well within our client-server version skew policy of +/-1 minor version.
However, this PR *only* updates the parts of `kubectl` that used `deployment/util`. So although kubectl now requires apps/v1, it still also depends on extensions/v1beta1. Migrating other parts of kubectl to apps/v1 is beyond the scope of this PR, which was just to change the Deployment controller and fix all the fallout.
```release-note
kubectl: This client version requires the `apps/v1` APIs, so it will not work against a cluster version older than v1.9.0. Note that kubectl only guarantees compatibility with clusters that are +/-1 minor version away.
```
The CSI integration test for hostpath was hard-coded to use the latest
stable release of the sidecar and hostpath container images. This
makes sense for regression testing of changes made in Kubernetes
itself, but the same test is also useful for testing the "canary"
images on quay.io before tagging them as a new release or for testing
locally produced images. Both is now possible via command line
parameters.
Testing "canary" images on quay.io:
go run hack/e2e.go -- --provider=local --test \
--test_args="--ginkgo.focus=CSI.plugin.test.using.CSI.driver..hostPath -csiImageVersion=canary"
Testing local container images:
# https://docs.docker.com/registry/deploying/
docker run -d -p 5000:5000 --restart=always --name registry registry:2
for i in driver-registrar drivers external-attacher external-provisioner; do
make -C $i REGISTRY_NAME=localhost:5000 push
done
go run hack/e2e.go -- --provider=local --test \
--test_args="--ginkgo.focus=CSI.plugin.test.using.CSI.driver..hostPath -csiImageVersion=canary -csiImageRegistry=localhost:5000"
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
passthrough readOnly to subpath
**What this PR does / why we need it**:
If a volume is mounted as readonly, or subpath volumeMount is configured as readonly, then the subpath bind mount should be readonly.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#62752
**Special notes for your reviewer**:
**Release note**:
```release-note
Fixes issue where subpath readOnly mounts failed
```
Automatic merge from submit-queue (batch tested with PRs 63373, 63338). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Enable GCE/GKE e2e tests for GlusterFS Dynamic Provisioner.
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue (batch tested with PRs 63421, 63432, 63333). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
update tests to be specific about the versions they are testing
When setting up tests, you want to rely on your own scheme. This eliminates coupling to floating versions which gives unnecessary flexibility in most cases and prevents testing all the versions you need.
@liggitt scrubs unnecessary deps.
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 62657, 63278, 62903, 63375). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add more volume types in e2e and fix part of them.
**What this PR does / why we need it**:
- Add dir-link/dir-bindmounted/dir-link-bindmounted/bockfs volume types for e2e tests.
- Fix fsGroup related e2e tests partially.
- Return error if we cannot resolve volume path.
- Because we should not fallback to volume path, if it's a symbolic link, we may get wrong results.
To safely set fsGroup on local volume, we need to implement these two methods correctly for all volume types both on the host and in container:
- get volume path kubelet can access
- paths on the host and in container are different
- get mount references
- for directories, we cannot use its mount source (device field) to identify mount references, because directories on same filesystem have same mount source (e.g. tmpfs), we need to check filesystem's major:minor and directory root path on it
Here is current status:
| | (A) volume-path (host) | (B) volume-path (container) | (C) mount-refs (host) | (D) mount-refs (container) |
| --- | --- | --- | --- | --- |
| (1) dir | OK | FAIL | FAIL | FAIL |
| (2) dir-link | OK | FAIL | FAIL | FAIL |
| (3) dir-bindmounted | OK | FAIL | FAIL | FAIL |
| (4) dir-link-bindmounted | OK | FAIL | FAIL | FAIL |
| (5) tmpfs| OK | FAIL | FAIL | FAIL |
| (6) blockfs| OK | FAIL | OK | FAIL |
| (7) block| NOTNEEDED | NOTNEEDED | NOTNEEDED | NOTNEEDED |
| (8) gce-localssd-scsi-fs| NOTTESTED | NOTTESTED | NOTTESTED | NOTTESTED |
- This PR uses `nsenter ... readlink` to resolve path in container as @msau42 @jsafrane [suggested](https://github.com/kubernetes/kubernetes/pull/61489#pullrequestreview-110032850). This fixes B1:B6 and D6, , the rest will be addressed in https://github.com/kubernetes/kubernetes/pull/62102.
- C5:D5 marked `FAIL` because `tmpfs` filesystems can share same mount source, we cannot rely on it to check mount references. e2e tests passes due to we use unique mount source string in tests.
- A7:D7 marked `NOTNEEDED` because we don't set fsGroup on block devices in local plugin. (TODO: Should we set fsGroup on block device?)
- A8:D8 marked `NOTTESTED` because I didn't test it, I leave it to `pull-kubernetes-e2e-gce`. I think it should be same as `blockfs`.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add tests for GlusterFS dynamic provisioner.
This bring up a heketi server pod and the server will be running
in mock mode, the PVC creation should work, however the volume
attachment to a pod and read/write is not part of this test. Due
to the same reason the tests are marked as [fast].
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
```
in mock mode, the PVC creation should work, however the volume
attachment to a pod and read/write is not part of this test. Due
to the same reason the tests are marked as [fast].
Signed-off-by: Humble Chirammal <hchiramm@redhat.com>
- Add PodStartShortTimeout and ClaimProvisionShortTimeout constants.
- Change framework.PodStartTimeout to framework.PodStartShortTimeout in
persistent_volumes-local.go. Busybox image is very small, no need to
wait for a long time.
Automatic merge from submit-queue (batch tested with PRs 62694, 62569, 62646, 61633, 62433). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add GCE-PD CSI Driver test to E2E test suite
Fixes: #60462
/sig storage
/kind technical-debt
/assign @saad-ali @msau42
**What this PR does / why we need it**:
This PR adds an E2E test for the GCE-PD CSI driver that deploys the driver in a production-like setting and tests whether dynamic provisioning with the driver is possible.
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 62694, 62569, 62646, 61633, 62433). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Report events to apiserver in local volume plugin.
**What this PR does / why we need it**:
See https://github.com/kubernetes/kubernetes/pull/62102#discussion_r179238429.
Report events to apiserver in local volume plugin.
- Add VolumeHost.GetEventRecorder() method
- Add related e2e tests
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#62248
**Special notes for your reviewer**:
Example output of `kubectl describe pods`:
```
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 7s default-scheduler Successfully assigned e2e-tests-persistent-local-volumes-test-x4h5x/security-context-670da435-4174-11e8-9098-000c29bb0377 to 127.0.0.1
Warning AlreadyMountedVolume 7s kubelet, 127.0.0.1 The requested fsGroup is 4321, but the volume local-pvfbb76 has GID 1234. The volume may not be shareable.
Normal SuccessfulMountVolume 7s kubelet, 127.0.0.1 MountVolume.SetUp succeeded for volume "default-token-996xr"
Normal SuccessfulMountVolume 7s kubelet, 127.0.0.1 MountVolume.SetUp succeeded for volume "local-pvfbb76"
Normal Pulled 6s kubelet, 127.0.0.1 Container image "k8s.gcr.io/busybox:1.24" already present on machine
Normal Created 6s kubelet, 127.0.0.1 Created container
Normal Started 6s kubelet, 127.0.0.1 Started container
```
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 62650, 62303, 62545, 62375). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Refactor subpath reconstruction tests to use util test
**What this PR does / why we need it**:
Use common util method for reconstruction tests so that test updates will be reflected in all e2es. Also increase termination grace period to allow more time for volume manager to cleanup before the tests check for leftover mounts
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#62293
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 62455, 62465, 62427, 62416, 62411). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Set slave mount propagation for local provisioner
**What this PR does / why we need it**:
Explicitly set slave mount propagation instead of assuming it's the default mode.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 46903, 61721, 62317). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Remove the use of storage class beta annotations in e2e tests.
**What this PR does / why we need it**:
Since we are going to remove storage class beta annotations, we'd like to remove the use of it in e2e tests to avoid upgrade test issues when it's finally removed.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
ref: https://github.com/kubernetes/kubernetes/pull/51440#discussion_r180290591
**Special notes for your reviewer**:
This should be cherry-picked to 1.10 as well. https://github.com/kubernetes/kubernetes/pull/51440#discussion_r178919464
/cc @msau42
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 62192, 61866, 62206, 62360). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add CSI External Components ClusterRole to bootstrapped roles
Added CSI External Components ClusterRole to bootstrapped roles and removed creation from failing e2e test
Fixes: #61781
/sig storage
/kind bug
/assign @liggitt @saad-ali
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 60197, 61614, 62074, 62071, 62301). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Adds e2e test for vpxd restart scenario in vSphere Cloud Provider
This PR adds a test to verify Volume access in a situation where VMware vCenter's `vpxd` service is down. The test mainly verifies Volume/file access before, during, and after restarting the `vpxd` service on the vCenter host.
See vmware/kubernetes#373 for more details.
**Reviewers note:** This PR was internally reviewed at vmware/kubernetes#468.
/cc @kubernetes/vmware
Automatic merge from submit-queue (batch tested with PRs 60465, 61773, 61371, 61146). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add more e2e subpath tests
**What this PR does / why we need it**:
Adds e2e tests for detecting subpath cleanup failures:
* Add wait for pod deletion to the end of subpath tests
* Add subpath as file test case
* Add subpath as file with container restart test case
* Refactor atomic volume tests
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#61178
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 61402, 61143, 61427, 60592). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
delete some unused code
**What this PR does / why we need it**:
delete some unused code
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 61644, 61624, 61743, 61019, 61287). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Added e2e test for local-volume provisioner that does not create PV for discovered non-bind-mounted filesystem.
**What this PR does / why we need it**:
For v2+ of the [local volume provisioner](https://github.com/kubernetes-incubator/external-storage/tree/master/local-volume) non-bind mounted filesystems in a discovery directory will no longer result in local PVs. This change was put in place to handle the non-atomic nature of other methods of adding directories (e.g. https://github.com/kubernetes-incubator/external-storage/issues/482).
This PR tests this change in behavior, and it validates that non-bind mounted directories within a discovery directory do NOT result in a local PV.
**Which issue(s) this PR fixes**:
Fixes https://github.com/kubernetes/kubernetes/issues/61020
**Special notes for your reviewer**:
This test can be executed using the following commands:
```
KUBE_FEATURE_GATES="BlockVolume=true" NUM_NODES=1 go run hack/e2e.go -- --up
go run hack/e2e.go -- --test --test_args='--ginkgo.focus=PersistentVolumes-local.*Local.*volume.*provisioner'
```
If you get the logs from a local volume provisioner pod, you will see the following log messages:
```
$ kubectl logs local-volume-provisioner-94ddb -n e2e-tests-persistent-local-volumes-test-6ls4z
<snip>
I0311 19:01:30.350504 1 controller.go:73] Controller started
E0311 19:01:30.350849 1 discovery.go:172] Path "/mnt/local-storage/notbindmount" is not an actual mountpoint
```
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 60455, 61365, 61375, 61597, 61491). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Updated lv-provisioner image to v2.1.0 in e2e tests
**What this PR does / why we need it**:
This PR updates the [local volume provisioner](https://github.com/kubernetes-incubator/external-storage/tree/master/local-volume) to v2.1.0 in the e2e tests for k8s v1.10.
**Which issue(s) this PR fixes**:
Fixes#61596
**Special notes for your reviewer**:
The following commands can be used to perform these tests:
```
$ make WHAT=test/e2e/e2e.test
$ KUBE_FEATURE_GATES="BlockVolume=true" NUM_NODES=1 go run hack/e2e.go -- --up
$ go run hack/e2e.go -- --test --test_args='--ginkgo.focus=PersistentVolumes-local.*Local.*volume.*provisioner’
```
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
e2e:Enable CSI tests
**What this PR does / why we need it**:
Enable CSI e2e tests now that CSI objects in Kubernetes are Beta.
Inspect daemonset provisioner logs for validation.
The logs are examined for the log lines showing that the provisioner discovered the directories and did NOT create a local PV.
This commit also adds SkipUnlessProvider supports SSH, which is required for local PV e2e tests.
This commit adds a test to verify Volume access in a situation where
VMware vCenter's vpxd service is down. The test mainly verifies
Volume/file access before, during, and after restarting the vpxd
service on the vCenter host.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Replace package "golang.org/x/net/context" with "context"
**What this PR does / why we need it**:
Replace package "golang.org/x/net/context" with "context"
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#60560
**Special notes for your reviewer**:
As of Go 1.7 this package(golang.org/x/net/context) is available in the standard library under the name context. see (https://godoc.org/golang.org/x/net/context)
It is almost machinery replace.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 61453, 61393, 61379, 61373, 61494). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Use inner volume name instead of outer volume name for subpath directory
**What this PR does / why we need it**:
Fixes volume reconstruction for PVCs with subpath
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#61372
**Special notes for your reviewer**:
**Release note**:
```release-note
ACTION REQUIRED: In-place node upgrades to this release from versions 1.7.14, 1.8.9, and 1.9.4 are not supported if using subpath volumes with PVCs. Such pods should be drained from the node first.
```
Automatic merge from submit-queue (batch tested with PRs 60980, 61273, 60811, 61021, 61367). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Added e2e test for local volume provisioner discovery of new mountpoints while running.
**What this PR does / why we need it**:
This PR adds an e2e test for the local volume provisioner leveraging of the MountPropogation feature (https://github.com/kubernetes/kubernetes/pull/46444) to discover newly created mountpoints. The MountPropogation feature is now in beta (https://github.com/kubernetes/kubernetes/blob/master/pkg/features/kube_features.go#L146).
**Which issue(s) this PR fixes**:
Fixes https://github.com/kubernetes/kubernetes/issues/58416
**Special notes for your reviewer**:
```
KUBE_FEATURE_GATES="BlockVolume=true" NUM_NODES=1 go run hack/e2e.go -- --up
go run hack/e2e.go -- --test --test_args='--ginkgo.focus=PersistentVolumes-local.*Local.*volume.*provisioner'
```
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 60793, 61181, 61267, 61252, 61334). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Refactor disruptive tests to use more volume types
Refactor storage disruptive tests to use different volume types. Also mark existing tests to be NFS specific. Fixes https://github.com/kubernetes/kubernetes/issues/61150
cc @jeffvance @jingxu97
/sig storage
```release-note
None
```
Automatic merge from submit-queue (batch tested with PRs 59740, 59728, 60080, 60086, 58714). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
vSphere: Minimize property collection via Finder
The 'All' parameter of the 'NewFinder' function controls property collection while searching the inventory.
When 'All' is set to 'false', Finder collects the minimal set of object properties required to search inventory.
When 'All' is set to 'true', Finder collects *all* object properties, which are *not* required to search inventory.
Setting 'All' to 'true' is only useful when inspecting all properties of an object,
such as by certain govc commands when the '-json' or '-dump' flags are specified.
Changing All=false in VCP minimizes the SOAP payload size and marshalling required on both sides, without impacting any functionality.
**What this PR does / why we need it**:
Changing All=false in VCP minimizes the SOAP payload size and marshalling required on both sides, without impacting any functionality.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix test images
These commits fix volume_io tests for iSCSI and Ceph RBD. Both server images were quite old (Fedora 22), so I updated them to ~~something more stable (CentOS 7) and to newer Ceph (Jewel, 10.2.7).~~ something newer (Fedora 26).
The most important fix is that the test volumes have 120 MB so volume_io test can actually run - the tests put 100MB file to the volume to check its persistence.
When mount containers in #53440 are merged I'll try to run the tests regularly with every PR (or merge) so we catch regressions quickly.
```release-note
NONE
```
/sig testing
/sig storage
/assign @jeffvance
Fixes: #56725
Automatic merge from submit-queue (batch tested with PRs 61111, 61069). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix subpath e2e tests on multizone cluster.
Use dynamically provisioned PV to run GCE PD tests. This will make sure that the pod is scheduled to the right zone and GCE PD can be attached to a node.
**Which issue(s) this PR fixes**:
Fixes#61101
**Release note**:
```release-note
NONE
```
/sig storage
@msau42 @verult
Use dynamically provisioned PV to run GCE PD tests. This will make sure
that the pod is scheduled to the right zone and GCE PD can be attached
to a node.
The fix for #60803 in commit 2ae33cc324 had a typo, so the "Server
rejected event" error still showed up in the external-provisioner log
of the "Sanity CSI plugin test using hostPath CSI driver" e2e test.
After client pod is stopped we need to wait for a while for kubelet to
unmount and clean up the client pod before we can destroy the server pod.
Killing e.g. iSCSI server too early may result in stale attached device
that cannot be removed.
- create 120MB volume instead of 1MB for volume_io tests
- rebase to Fedora 26
- added compatibility with ext4 and older ceph clients
- unify CephFS and Ceph RBD images.
Automatic merge from submit-queue (batch tested with PRs 60679, 60804). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fixing e2e CSI test
Closes#60803
After switching to CSI 0.2.0 spec, additional RBAC permissions are required. This PR adds missing permissions.
```release-note
None
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Volume deletion should be idempotent
- Describe* calls should return `aws.Error` so caller can handle individual errors. `aws.Error` already has enough context (`"InvalidVolume.NotFound: The volume 'vol-0a06cc096e989c5a2' does not exist"`)
- Deletion of already deleted volume should succeed.
**Release note**:
Fixes: #60778
```release-note
NONE
```
/sig storage
/sig aws
/assign @justinsb @gnufied
Automatic merge from submit-queue (batch tested with PRs 58171, 58036, 60540). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Changing Flexvolume plugin directory on COS in GCE to a durable directory
**What this PR does / why we need it**: The original `/etc/srv/...` directory is in an overlayfs over a path in /tmp, so Flexvolume drivers are erased across node restarts for any reason. Changing it to non-tmpfs location.
Also removing redundant Flexvolume path injection in `config-test.sh` because it's already in `cluster/common.sh`.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#57353
**Release note**:
```release-note
[action required] Default Flexvolume plugin directory for COS images on GCE is changed to `/home/kubernetes/flexvolume`.
```
/assign @roberthbailey @saad-ali
/cc @chakri-nelluri @wongma7
/sig storage
Automatic merge from submit-queue (batch tested with PRs 59310, 60424, 60308, 60436, 60020). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Reduce number of pods created for local PV stress test
**What this PR does / why we need it**:
Local PV stress test is flaking. Failed runs show that test is timing out at 47/50 pods. Reduce the number of pods created by the test so that it's not so close to the max timeout.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
I'll need to investigate further to see why processing the pods is so slow and ways to speed it up. But for now, try to reduce flaking.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
refactor volume util files
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes # https://github.com/kubernetes/kubernetes/issues/44460
**Special notes for your reviewer**:
/assign @jsafrane @msau42
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Enable protection tests
**What this PR does / why we need it**:
- StorageObjectInUseProtection feature is enabled by default so the test can run in regular e2e test suite
- Rename PVC protection test, it tests only PVCs and not whole storage.
**Release note**:
```release-note
NONE
```
VolumeMode block requires alpha feature-gate BlockVolume=true
Added VolumeMode to e2e test framework for pv and pvc
Added volumeDevices to e2e test framework for block volumes
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add Local PV stress test
**What this PR does / why we need it**:
Test heavy usage of local PVs to help test robustness of the volume scheduler, and make sure we don't encounter partial binding failures under load.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Partially addresses #56100
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
@kubernetes/sig-storage-pr-reviews
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
e2e tests for multizone PDs
**What this PR does / why we need it**: e2e tests for multizone PDs.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
partially fixing #59988
/release-note-none
/assign @msau42 @saad-ali
Automatic merge from submit-queue (batch tested with PRs 60302, 57921, 59042, 60126, 59561). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Removing Flexvolume feature tag in e2e tests because Flexvolume is now GA
**What this PR does / why we need it**: Flexvolume e2e tests should be run in serial test suite after the feature was marked GA.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#57466
/release-note-none
/sig storage
/cc @wongma7
- StorageObjectInUseProtection is enabled by default now so the test can run in regular tests.
- Enable StorageObjectInUseProtection admission plugins during tests
The 'All' parameter of the 'NewFinder' function controls property collection while searching the inventory.
When 'All' is set to 'false', Finder collects the minimal set of object properties required to search inventory.
When 'All' is set to 'true', Finder collects *all* object properties, which are *not* required to search inventory.
Setting 'All' to 'true' is only useful when inspecting all properties of an object,
such as by certain govc commands when the '-json' or '-dump' flags are specified.
Changing All=false in VCP minimizes the SOAP payload size and marshalling required on both sides, without impacting any functionality.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Move volume scheduling and local storage to beta
**What this PR does / why we need it**:
* Move the feature gates and APIs for volume scheduling and local storage to beta
* Update tests to use the beta fields
@kubernetes/sig-storage-pr-reviews
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#59390
**Special notes for your reviewer**:
**Release note**:
```release-note
ACTION REQUIRED: VolumeScheduling and LocalPersistentVolume features are beta and enabled by default. The PersistentVolume NodeAffinity alpha annotation is deprecated and will be removed in a future release.
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
CSI 0.2.0 breaking changes
Refactor kubernetes CSI bits to support CSI version 0.2.0
```release-note
Addressing breaking changes introduced by new 0.2.0 release of CSI spec
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add bound/unbound pv/pvc volume metrics
**What this PR does / why we need it**:
Fix two bugs in bound/unbound pv/pvc volumes metric e2e test:
1. Check `HasRegisteredMaster` in `BeforeEach` instead of each tests because we will grab the metrics in `BeforeEach`
2. Fix `calculateRelativeValues` bug, if there are other PV/PVC exist in the cluster, `calculateRelativeValues` will return a map like `map[string]int64{"some-storage-class-name": 0}`, this will lead to test failure. We should skip these unchanged numbers.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#59517
**Release note**:
```release-note
NONE
```
/sig storage
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fixing diskIsAttached function
**What this PR does / why we need it**:
This PR fixes the `diskIsAttached` function.
When disk is detached from the Node, this function was returning true, which results into time out for some vsphere e2e test cases.
**Which issue(s) this PR fixes**
Fixes #
**Special notes for your reviewer**:
Executed E2E test to verify failures are resolved with this change.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Removing Production Code usage from vSphere E2E tests
**What this PR does / why we need it**:
vSphere E2E test cases uses VCP production code for bootstrap logic, example is to get an instance of vSphere struct. Once it gets an instance vSphere struct, it will use this object to call into methods in VCP code like - DiskIsAttached(), DisksAreAttached() etc.
This direct dependency on the VCP production should be removed as any changes to the production code would create a problem in E2E test cases as well.
This PR addressed above stated need.
**Which issue(s) this PR fixes**
Fixes # https://github.com/vmware/kubernetes/issues/434
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add e2e test for PV protection
Add e2e test for PV protection
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
xref: https://github.com/kubernetes/community/pull/1608
**Special notes for your reviewer**:
hold until #58743 gets merged
**Release note**:
```release-note
NONE
```
/sig storage
/hold
/assign @jsafrane
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
fix all the typos across the project
**What this PR does / why we need it**:
There are lots of typos across the project. We should avoid small PRs on fixing those annoying typos, which is time-consuming and low efficient.
This PR does fix all the typos across the project currently. And with #59463, typos could be avoided when a new PR gets merged.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
/sig testing
/area test-infra
/sig release
/cc @ixdy
/assign @fejta
**Release note**:
```release-note
None
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
vSphere test infrastructure improvement and new node-unregister test
**What this PR does / why we need it**:
- Created conf file parsing logic for vSphere tests
- Created NodeMapper to generate node-vsphere map
- Updated bootstrap to parse conf file and generate node-vsphere map, and set it in TestContext
- Moved bootstrap.go and context.go up, in vsphere package to avoid cyclic package dependencies
- Added node register/unregister test, that consumes new test-infra
**Which issue(s) this PR fixes**:
Fixes https://github.com/vmware/kubernetes/issues/437
Fixes https://github.com/vmware/kubernetes/issues/379
**Special notes for your reviewer**:
- Successfully ran vSphere e2e tests to ensure that the bootstrapping is happening only once. More tests in progress
- Successfully ran 'Node Unregister'
```
bash-3.2$ go run hack/e2e.go --check-version-skew=false --v --test --test_args=‘--ginkgo.focus=Node\sUnregister’
flag provided but not defined: -check-version-skew
Usage of /var/folders/97/lnlv1n317xl2ty8hdn7zptxr00b37m/T/go-build743103230/command-line-arguments/_obj/exe/e2e:
-get
go get -u kubetest if old or not installed (default true)
-old duration
Consider kubetest old if it exceeds this (default 24h0m0s)
Will run 1 of 724 specs
Feb 5 22:20:09.890: INFO: >>> kubeConfig: /Users/pshahzeb/kube176.json
Feb 5 22:20:09.903: INFO: Waiting up to 4h0m0s for all (but 0) nodes to be schedulable
Feb 5 22:20:10.036: INFO: Waiting up to 10m0s for all pods (need at least 0) in namespace ‘kube-system’ to be running and ready
Feb 5 22:20:10.182: INFO: 13 / 13 pods in namespace ‘kube-system’ are running and ready (0 seconds elapsed)
Feb 5 22:20:10.182: INFO: expected 4 pod replicas in namespace ‘kube-system’, 4 are Running and Ready.
Feb 5 22:20:10.203: INFO: Waiting for pods to enter Success, but no pods in “kube-system” match label map[name:e2e-image-puller]
Feb 5 22:20:10.203: INFO: Dumping network health container logs from all nodes...
Feb 5 22:20:10.236: INFO: e2e test version: v1.6.0-alpha.0.22494+e66916e052163a-dirty
Feb 5 22:20:10.261: INFO: kube-apiserver version: v1.9.2
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[sig-storage] Node Unregister [Feature:vsphere] [Slow] [Disruptive]
node unregister
/Users/pshahzeb/k8s/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/vsphere/vsphere_volume_node_delete.go:53
[BeforeEach] [sig-storage] Node Unregister [Feature:vsphere] [Slow] [Disruptive]
/Users/pshahzeb/k8s/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:141
STEP: Creating a kubernetes client
Feb 5 22:20:10.268: INFO: >>> kubeConfig: /Users/pshahzeb/kube176.json
STEP: Building a namespace api object
Feb 5 22:20:11.043: INFO: No PodSecurityPolicies found; assuming PodSecurityPolicy is disabled.
STEP: Waiting for a default service account to be provisioned in namespace
[BeforeEach] [sig-storage] Node Unregister [Feature:vsphere] [Slow] [Disruptive]
/Users/pshahzeb/k8s/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/vsphere/vsphere_volume_node_delete.go:41
Feb 5 22:20:11.063: INFO: Initializing vc server 10.160.240.176
Feb 5 22:20:11.063: INFO: ConfigFile &{{administrator@vsphere.local Admin!23 443 true k8s-dc 0} map[10.160.240.176:0xc420babe30] {VM Network} {pvscsi} {10.160.240.176 k8s-dc kubernetes vsanDatastore k8s-cluster}}
vSphere instances map[10.160.240.176:0xc420b08830]
[It] node unregister
/Users/pshahzeb/k8s/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/vsphere/vsphere_volume_node_delete.go:53
STEP: Get total Ready nodes
Feb 5 22:20:11.566: INFO: vmx file path is [vsanDatastore] 2e98735a-cdb9-c3f3-63d8-020010188a6a/kubernetes-node1.vmx
STEP: Unregister a node VM
Feb 5 22:20:11.686: INFO: Powering off node VM kubernetes-node1
Feb 5 22:20:14.148: INFO: Unregistering node VM kubernetes-node1
STEP: Verifying the ready node counts
STEP: Register back the node VM
Feb 5 22:20:49.490: INFO: Registering node VM kubernetes-node1
Feb 5 22:20:51.785: INFO: Powering on node VM kubernetes-node1
STEP: Verifying the ready node counts
Feb 5 22:21:40.600: INFO: Condition Ready of node kubernetes-node1 is false instead of true. Reason: KubeletNotReady, message: container runtime is down
Feb 5 22:21:45.625: INFO: Condition Ready of node kubernetes-node1 is false instead of true. Reason: KubeletNotReady, message: container runtime is down
STEP: Sanity check for volume lifecycle
STEP: Creating Storage Class With storage policy params
STEP: Creating PVC using the Storage Class
STEP: Waiting for claim to be in bound phase
Feb 5 22:21:50.718: INFO: Waiting up to 5m0s for PersistentVolumeClaim pvc-ztj7g to have phase Bound
Feb 5 22:22:15.053: INFO: PersistentVolumeClaim pvc-ztj7g found and phase=Bound (24.334875493s)
STEP: Creating pod to attach PV to the node
STEP: Verify the volume is accessible and available in the pod
Feb 5 22:22:25.976: INFO: Running ‘/Users/pshahzeb/k8s/kubernetes/_output/bin/kubectl --server=https://10.160.241.49 --kubeconfig=/Users/pshahzeb/kube176.json exec pvc-tester-q7q2w --namespace=e2e-tests-node-unregister-csdrc -- /bin/touch /mnt/volume1/emptyFile.txt’
Feb 5 22:22:26.740: INFO: stderr: “”
Feb 5 22:22:26.740: INFO: stdout: “”
STEP: Deleting pod
Feb 5 22:22:26.740: INFO: Deleting pod “pvc-tester-q7q2w” in namespace “e2e-tests-node-unregister-csdrc”
Feb 5 22:22:26.799: INFO: Wait up to 5m0s for pod “pvc-tester-q7q2w” to be fully deleted
STEP: Waiting for volumes to be detached from the node
Feb 5 2223:16.966: INFO: Volume “[vsanDatastore] f0c55f5a-7349-1aad-2464-02001067f24e/kubernetes-dynamic-pvc-04775fe5-0b06-11e8-9872-005056809c8d.vmdk” has successfully detached from “kubernetes-node1"
Feb 5 2223:16.966: INFO: Deleting PersistentVolumeClaim “pvc-ztj7g”
[AfterEach] [sig-storage] Node Unregister [Feature:vsphere] [Slow] [Disruptive]
/Users/pshahzeb/k8s/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142
Feb 5 2223:17.026: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace “e2e-tests-node-unregister-csdrc” for this suite.
Feb 5 2223:23.158: INFO: Waiting up to 30s for server preferred namespaced resources to be successfully discovered
Feb 5 2223:24.421: INFO: namespace: e2e-tests-node-unregister-csdrc, resource: bindings, ignored listing per whitelist
Feb 5 2223:24.795: INFO: namespace e2e-tests-node-unregister-csdrc deletion completed in 7.715803086s
• [SLOW TEST:194.521 seconds]
[sig-storage] Node Unregister [Feature:vsphere] [Slow] [Disruptive]
/Users/pshahzeb/k8s/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/utils/framework.go:22
node unregister
/Users/pshahzeb/k8s/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/vsphere/vsphere_volume_node_delete.go:53
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSFeb 5 2223:24.797: INFO: Running AfterSuite actions on all node
Feb 5 2223:24.798: INFO: Running AfterSuite actions on node 1
Ran 1 of 724 Specs in 194.905 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 723 Skipped PASS
Ginkgo ran 1 suite in 3m15.529747133s
Test Suite Passed
2018/02/05 2223:24 util.go:174: Step ‘./hack/ginkgo-e2e.sh --ginkgo.focus=Node\sUnregister’ finished in 3m16.095671615s
2018/02/05 2223:24 e2e.go:81: Done
```
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 59466, 58912, 59605, 59548). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Rename and restructure local PV tests
**What this PR does / why we need it**:
Reorganizes the local PV tests to have a more consistent structure.
@kubernetes/sig-storage-pr-reviews
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Redesign and implement volume reconstruction work
This PR is the first part of redesign of volume reconstruction work. The detailed design information is https://github.com/kubernetes/community/pull/1601
The changes include
1. Remove dependency on volume spec stored in actual state for volume
cleanup process (UnmountVolume and UnmountDevice)
Modify AttachedVolume struct to add DeviceMountPath so that volume
unmount operation can use this information instead of constructing from
volume spec
2. Modify reconciler's volume reconstruction process (syncState). Currently workflow
is when kubelet restarts, syncState() is only called once before
reconciler starts its loop.
a. If volume plugin supports reconstruction, it will use the
reconstructed volume spec information to update actual state as before.
b. If volume plugin cannot support reconstruction, it will use the
scanned mount path information to clean up the mounts.
In this PR, all the plugins still support reconstruction (except
glusterfs), so reconstruction of some plugins will still have issues.
The next PR will modify those plugins that cannot support reconstruction
well.
This PR addresses issue #52683
Automatic merge from submit-queue (batch tested with PRs 59276, 51042, 58973, 59377, 59472). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix local PV node affinity tests and only run once
**What this PR does / why we need it**:
* Don't look for specific scheduling error messages for the NodeAffinity tests. Unit/integration will cover that.
* Move PV NodeAffinity tests outside the local volume loop. Mounts are not involved so don't need to be tested per volume type.
* Move mount failure tests outside the local volume loop.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#59369
**Release note**:
```release-note
NONE
```
@kubernetes/sig-storage-pr-reviews
Automatic merge from submit-queue (batch tested with PRs 52942, 58415). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Improve messaging on volume expansion
- we now provide clear message to user what to do when cloudprovider resizing is finished
and file system resizing is needed.
- add a event when resizing is successful
- Use PATCH both in controller-manager and kubelet for updating PVC status
- Remove code duplication between controller-manager and kubelet for updating PVC status
- Only remove conditions that are managed by resize controller
```release-note
Improve messages user gets during and after volume resizing is done.
```
This PR is the first part of redesign of volume reconstruction work. The
changes include
1. Remove dependency on volume spec stored in actual state for volume
cleanup process (UnmountVolume and UnmountDevice)
Modify AttachedVolume struct to add DeviceMountPath so that volume
unmount operation can use this information instead of constructing from
volume spec
2. Modify reconciler's volume reconstruction process (syncState). Currently workflow
is when kubelet restarts, syncState() is only called once before
reconciler starts its loop.
a. If volume plugin supports reconstruction, it will use the
reconstructed volume spec information to update actual state as before.
b. If volume plugin cannot support reconstruction, it will use the
scanned mount path information to clean up the mounts.
In this PR, all the plugins still support reconstruction (except
glusterfs), so reconstruction of some plugins will still have issues.
The next PR will modify those plugins that cannot support reconstruction
well.
This PR addresses issue #52683, #54108 (This PR includes the changes to
update devicePath after local attach finishes)
Automatic merge from submit-queue (batch tested with PRs 58645, 59211). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fixing node labels assignment for random order tests execution
**What this PR does / why we need it**:
For some vsphere cloud provider e2e test cases, we are setting node labels to control pod scheduling on specific node.
When these e2e tests are executed in random order from testsuite, they are overwriting node labels, so some tests were failing to schedule pod on desired node. Tests are failing with following error.
```
“FailedScheduling: No nodes are available that match all of the predicates: MatchNodeSelector (5), NodeUnschedulable (1).
```
This PR is fixing the above issue with setting distinct node label key for each test group within test suite.
**Which issue(s) this PR fixes**
Fixes #
**Special notes for your reviewer**:
Executed testsuites containing tests from both files using following script.
```
GINKGO_FOCUS[0]="Volume\sPlacement"
GINKGO_FOCUS[1]="Volume\sDisk\sFormat"
REGEX="--ginkgo.focus="$(IFS='|' ; echo "${GINKGO_FOCUS[*]}")
go run hack/e2e.go --check-version-skew=false --v --test --test_args="${REGEX}"
```
All test passed.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Perform resize of mounted volume if necessary
Under certain conditions - we must perform resize of volume even when it is mounted. This enables us to get around problem of resizing volumes used by deployments etc.
```release-note
Allow expanding mounted volumes
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Postpone PV deletion with finalizer when it is being used
Postpone PV deletion if it is bound to a PVC
xref: https://github.com/kubernetes/community/pull/1608
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#33355
**Special notes for your reviewer**:
**Release note**:
```release-note
Postpone PV deletion when it is being bound to a PVC
```
WIP, assign to myself first
/assign @NickrenREN
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
PVC Protection E2E Tests for Failed Scheduling
**What this PR does / why we need it**:
Change in scheduler that causes that scheduling of a pod that uses PVC that is being deleted fails was introduced in:
- https://github.com/kubernetes/kubernetes/pull/55957
This PR adds an E2E test for the above merged PR.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
N/A
**Special notes for your reviewer**:
N/A
**Release note**:
```release-note
NONE
```
- we now provide clear message to user what to do when cloudprovider resizing is finished
and file system resizing is needed.
- add a event when resizing is successful.
- Use Patch for updating PVCs in both kubelet and controller-manager
- Extract updating pvc util function in one place.
- Only update resize conditions on progress
Automatic merge from submit-queue (batch tested with PRs 55792, 58342). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Promote Statefulset controller and its e2e tests to use apps/v1
**What this PR does / why we need it**:
Promotes the statefulset controller to use to use the latest apps group [apps/v1](https://github.com/kubernetes/kubernetes/pull/53679)
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes # https://github.com/kubernetes/kubernetes/issues/55714
**Special notes for your reviewer**:
* Listerexpansion for v1 `k8s.io/client-go/listers/apps/v1` (was recently done for v1beta2)
* `v1beta2` && `v1` had `ObservedGeneration` as `int64` where as `v1beta1` and rest of the code (including conversion) is expecting `ObservedGeneration` to be `*int64`
```
type StatefulSetStatus struct {
// observedGeneration is the most recent generation observed for this StatefulSet. It corresponds to the
// StatefulSet's generation, which is updated on mutation by the API Server.
// +optional
ObservedGeneration int64 `json:"observedGeneration,omitempty" protobuf:"varint,1,opt,name=observedGeneration"`
```
* for kubectl's `rollback` and `history` commands a couple functions have been duplicated to allow us to use `v1` version instead of `v1beta1` for statefulsets, while the older functions are still used by other controllers.
We should be able to remove these duplicates once all the controllers are moved.
If this aligns with the plan then i could move other controllers too.
cc: @kow3ns
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Allow Ceph server some time to start
Ceph server needs to create our "foo" volume on startup. It keeps the image small, however it makes the server container start slow.
Add sleep before the server is usable. Without this PR, all pods that use Ceph fail to start for couple of seconds with cryptic "image foo not found" error and it clutters logs and pod logs and makes it harder to spot real errors.
```release-note
NONE
```
/sig storage
/sig testing
/assign @jeffvance @copejon
The local-volume bootstrapper/provisioner configuration format changed in
https://github.com/kubernetes-incubator/external-storage/pull/352
This format is exposed in v2.0.0 of the provisioner and boostratpper images.
This PR updates the e2e test config, so that the existing tests continue
to work.
Automatic merge from submit-queue (batch tested with PRs 56600, 56814). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix volume size allocation in gcd_pd
**What this PR does / why we need it**:
GCE PDs are allocated in chunks of GBs not GiB but CreateVolume function incorrectly creates volume in chunks of GiB.
1 GiB = 1024 * 1024 * 1024 Bytes
1 GB = 1000 * 1000 * 1000 Bytes
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#56081
**Special notes for your reviewer**:
**Release note**:
```release-note
Fixed dynamic provisioning of GCE PDs to round to the next GB instead of GiB
```
utils/image/manifest has an additional `arch` parameter, which determines
whether an image ends in `-$ARCH` (like `-amd64`).
All locations that previously had gcr.io urls referenced in costants or inline
have been updated to refere test/utils/image.
This e2e test tests the CSI volume plugin in kubernetes with
a CSI hostPath driver. It is also setup to be able to be
tested with more drivers in the future.
Automatic merge from submit-queue (batch tested with PRs 56217, 56268, 56263, 56328, 56200). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Use batch client with explicit version
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** :
Fixes partially #55993
**Special notes for your reviewer**:
/cc @caesarxuchao @sttts
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 55900, 55995, 55913, 55467, 55376). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Use extensions client with explicit version
**What this PR does / why we need it**:
Extensions client without explicit version has been deprecated, change them to the one with explicit version.
**Which issue(s) this PR fixes**:
Fixes partially #55993
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 51321, 55969, 55039, 56183, 55976). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
use local `diskNames` slice to fix issue 56041
Fixes#56041
```release-note
NONE
```
@msau42
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fixes issue where PVCs using `standard` StorageClass create PDs in disks in wrong zone in multi-zone GKE clusters
Fixes#50115
Changed GetAllZones to only get zones with nodes that are currently running (renamed to GetAllCurrentZones). Added E2E test to confirm this behavior.
Automatic merge from submit-queue (batch tested with PRs 55112, 56029, 55740, 56095, 55845). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Updating vsphere cloud provider to support k8s cluster spread across multiple vCenters
**What this PR does / why we need it**:
vSphere cloud provider in Kubernetes 1.8 was designed to work only if all the nodes of the cluster are in one single datacenter folder. This is a hard restriction that makes the cluster not span across different folders/datacenter/vCenters. Users have use-cases to span the cluster across datacenters/vCenters.
**Which issue(s) this PR fixes**
Fixes # https://github.com/vmware/kubernetes/issues/255
**Special notes for your reviewer**:
This is a change purely in vsphere cloud provider and no changes in kubernetes core are needed.
**Release note**:
```release-note
With this change
- User should be able to create k8s cluster which spans across multiple ESXi clusters, datacenters or even vCenters.
- vSphere cloud provider (VCP) uses OS hostname and not vSphere Inventory VM Name.
That means, now VCP can handle cases where user changes VM inventory name.
- VCP can handle cases where VM migrates to other ESXi cluster or datacenter or vCenter.
The only requirement is the shared storage. VCP needs shared storage on all Node VMs.
```
Internally tested and reviewed the code.
@tthole, @shaominchen, @abrarshivani
Automatic merge from submit-queue (batch tested with PRs 54824, 55911, 55730, 55979, 55961). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add options for mounting SCSI or NVMe local SSD though Block or Filesystem and do all of that with UUID
Fixes: #51431
Fixed version of: #53466
Mount SCSI local SSD by UUID in /mnt/disks/by-uuid/, also allows for users to request and mount NVMe disks. Both types of disks will be accessible either through block or file-system.
I have confirmed that it is no longer crashing when nodes are initialized on GKE.
running (renamed to GetAllCurrentZones). Added E2E test to confirm this
behavior.
Added node informer to cloud-provider controller to keep track of zones
with k8s nodes in them.
- vsphere.conf (cloud-config) is now needed only on master node
- VCP uses OS hostname and not vSphere inventory name
- VCP is now resilient to VM inventory name change and VM migration
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add options for mounting SCSI or NVMe local SSD though Block or Filesystem and do all of that with UUID
Fixes: #51431
Mount SCSI local SSD by UUID in /mnt/disks/by-uuid/, also allows for users to request and mount NVMe disks. Both types of disks will be accessable either through block or filesystem
To see code in progress for NVMe and block support see working branch: https://github.com/davidz627/kubernetes/tree/localExt
Automatic merge from submit-queue (batch tested with PRs 55301, 55319, 54018, 55322, 55125). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
E2E scale test for vSphere Cloud Provider Volume lifecycle operations
This PR adds an E2E test for vSphere Cloud Provider which will create/attach/detach/detach the volumes at scale with multiple threads based on user configurable values for number of volumes, volumes per pod and number of threads. (Since this is a scale test, number of threads would be low. This is only used to speed up the operation)
Test performs following tasks.
1. Create Storage Classes of 4 Categories (Default, SC with Non Default Datastore, SC with SPBM Policy, SC with VSAN Storage Capalibilies.)
2. Read VCP_SCALE_VOLUME_COUNT from System Environment.
3. Launch VCP_SCALE_INSTANCES go routines for creating VCP_SCALE_VOLUME_COUNT volumes. Each go routine is responsible for create/attach of VCP_SCALE_VOLUME_COUNT/VCP_SCALE_INSTANCES volumes.
4. Read VCP_SCALE_VOLUMES_PER_POD from System Environment. Each pod will be have VCP_SCALE_VOLUMES_PER_POD attached to it.
5. Once all the go routines are completed, we delete all the pods and volumes.
Which issue this PR fixes
fixes # vmware#291
```release-note
None
```
Automatic merge from submit-queue (batch tested with PRs 53747, 54528, 55279, 55251, 55311). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Adding e2e test to verify volume attach status after master kubelet restart
**What this PR does / why we need it**:
This PR adds test to verify volume remains attached after the kubelet is restarted on master node.
**Which issue this PR fixes** :
fixes vmware#274
**Special notes for your reviewer**:
This test does not run as part of existing sig-storage test grid. It has been tested internally at VMware.
Test logs
```
root@k8s-dev-vm-01:~/shahzeb/k8s/kubernetes# go run hack/e2e.go --check-version-skew=false -v -test --test_args='--ginkgo.focus=Volume\sAttach\sVerify'
flag provided but not defined: -check-version-skew
Usage of /tmp/go-build395888807/command-line-arguments/_obj/exe/e2e:
-get
go get -u kubetest if old or not installed (default true)
-old duration
Consider kubetest old if it exceeds this (default 24h0m0s)
2017/10/11 12:14:05 e2e.go:55: NOTICE: go run hack/e2e.go is now a shim for test-infra/kubetest
2017/10/11 12:14:05 e2e.go:56: Usage: go run hack/e2e.go [--get=true] [--old=24h0m0s] -- [KUBETEST_ARGS]
2017/10/11 12:14:05 e2e.go:57: The separator is required to use --get or --old flags
2017/10/11 12:14:05 e2e.go:58: The -- flag separator also suppresses this message
2017/10/11 12:14:05 e2e.go:151: The kubetest binary is older than 24h0m0s.
2017/10/11 12:14:05 e2e.go:156: Updating kubetest binary...
2017/10/11 12:14:13 e2e.go:77: Calling kubetest --check-version-skew=false -v -test --test_args=--ginkgo.focus=Volume\sAttach\sVerify...
2017/10/11 12:14:13 util.go:154: Running: ./cluster/kubectl.sh --match-server-version=false version
2017/10/11 12:14:13 util.go:156: Step './cluster/kubectl.sh --match-server-version=false version' finished in 493.364761ms
2017/10/11 12:14:13 util.go:154: Running: ./hack/e2e-internal/e2e-status.sh
Skeleton Provider: prepare-e2e not implemented
Client Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.17307+d274c30f81d1c2", GitCommit:"d274c30f81d1c2d966dc950014ac90f8fad140f7", GitTreeState:"clean", BuildDate:"2017-10-11T18:57:31Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.5", GitCommit:"490c6f13df1cb6612e0993c4c14f2ff90f8cdbf3", GitTreeState:"clean", BuildDate:"2017-06-14T20:03:38Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}
2017/10/11 12:14:14 util.go:156: Step './hack/e2e-internal/e2e-status.sh' finished in 352.041653ms
2017/10/11 12:14:14 util.go:154: Running: ./hack/ginkgo-e2e.sh --ginkgo.focus=Volume\sAttach\sVerify
Conformance test: not doing test setup.
Oct 11 12:14:15.478: INFO: Overriding default scale value of zero to 1
Oct 11 12:14:15.478: INFO: Overriding default milliseconds value of zero to 5000
I1011 12:14:15.692022 29999 e2e.go:383] Starting e2e run "5f33ad5b-aeb8-11e7-9f17-0050569c27f6" on Ginkgo node 1
Running Suite: Kubernetes e2e suite
===================================
Random Seed: 1507749254 - Will randomize all specs
Will run 1 of 709 specs
Oct 11 12:14:15.744: INFO: >>> kubeConfig: /tmp/kube204.json
Oct 11 12:14:15.751: INFO: Waiting up to 4h0m0s for all (but 0) nodes to be schedulable
Oct 11 12:14:15.861: INFO: Waiting up to 10m0s for all pods (need at least 0) in namespace 'kube-system' to be running and ready
Oct 11 12:14:16.067: INFO: 4 / 4 pods in namespace 'kube-system' are running and ready (0 seconds elapsed)
Oct 11 12:14:16.067: INFO: expected 0 pod replicas in namespace 'kube-system', 0 are Running and Ready.
Oct 11 12:14:16.077: INFO: Waiting for pods to enter Success, but no pods in "kube-system" match label map[name:e2e-image-puller]
Oct 11 12:14:16.077: INFO: Dumping network health container logs from all nodes...
Oct 11 12:14:16.083: INFO: Client version: v1.6.0-alpha.0.17307+d274c30f81d1c2
Oct 11 12:14:16.086: INFO: Server version: v1.6.5
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[sig-storage] Volume Attach Verify [Feature:vsphere]
verify volume remains attached after master kubelet restart
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/vsphere_volume_master_restart.go:144
[BeforeEach] [sig-storage] Volume Attach Verify [Feature:vsphere]
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:133
STEP: Creating a kubernetes client
Oct 11 12:14:16.087: INFO: >>> kubeConfig: /tmp/kube204.json
STEP: Building a namespace api object
STEP: Waiting for a default service account to be provisioned in namespace
[BeforeEach] [sig-storage] Volume Attach Verify [Feature:vsphere]
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/vsphere_volume_master_restart.go:81
Oct 11 12:14:16.265: INFO: Waiting up to 4h0m0s for all (but 0) nodes to be schedulable
[It] verify volume remains attached after master kubelet restart
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/vsphere_volume_master_restart.go:144
STEP: Creating a test vsphere volume 0
STEP: Creating pod 0 on node kubernetes-node1
STEP: Waiting for pod to be ready
STEP: Verify volume [vsanDatastore] 8c95d659-46fa-b9a6-5e19-02002f28e688/e2e-vmdk-1507749256431387056.vmdk is attached to the pod kubernetes-node1
STEP: Creating a test vsphere volume 1
STEP: Creating pod 1 on node kubernetes-node2
STEP: Waiting for pod to be ready
STEP: Verify volume [vsanDatastore] 8c95d659-46fa-b9a6-5e19-02002f28e688/e2e-vmdk-1507749281940603428.vmdk is attached to the pod kubernetes-node2
STEP: Creating a test vsphere volume 2
STEP: Creating pod 2 on node kubernetes-node3
STEP: Waiting for pod to be ready
STEP: Verify volume [vsanDatastore] 8c95d659-46fa-b9a6-5e19-02002f28e688/e2e-vmdk-1507749305162880964.vmdk is attached to the pod kubernetes-node3
STEP: Creating a test vsphere volume 3
STEP: Creating pod 3 on node kubernetes-node4
STEP: Waiting for pod to be ready
STEP: Verify volume [vsanDatastore] 8c95d659-46fa-b9a6-5e19-02002f28e688/e2e-vmdk-1507749330788801099.vmdk is attached to the pod kubernetes-node4
STEP: Restarting kubelet on master node
Oct 11 12:16:12.239: INFO: Restarting kubelet via ssh on host 10.192.113.70:22 with command systemctl restart kubelet
STEP: Verifying the kubelet on master node is up
Oct 11 12:16:13.318: INFO: ssh root@10.192.113.70:22: command: curl http://localhost:10255/healthz
Oct 11 12:16:13.318: INFO: ssh root@10.192.113.70:22: stdout: ""
Oct 11 12:16:13.318: INFO: ssh root@10.192.113.70:22: stderr: " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 10255: Connection refused\n"
Oct 11 12:16:13.318: INFO: ssh root@10.192.113.70:22: exit code: 7
STEP: After master restart, verify volume [vsanDatastore] 8c95d659-46fa-b9a6-5e19-02002f28e688/e2e-vmdk-1507749256431387056.vmdk is attached to the pod kubernetes-node1
STEP: Deleting pod on node kubernetes-node1
Oct 11 12:16:18.538: INFO: Deleting pod "vsphere-e2e-pwjr1" in namespace "e2e-tests-restart-master-j9x0f"
Oct 11 12:16:18.559: INFO: Wait up to 5m0s for pod "vsphere-e2e-pwjr1" to be fully deleted
STEP: Waiting for volume [vsanDatastore] 8c95d659-46fa-b9a6-5e19-02002f28e688/e2e-vmdk-1507749256431387056.vmdk to be detached from the node kubernetes-node1
Oct 11 12:17:10.686: INFO: Volume "[vsanDatastore] 8c95d659-46fa-b9a6-5e19-02002f28e688/e2e-vmdk-1507749256431387056.vmdk" appears to have successfully detached from "kubernetes-node1".
STEP: Deleting volume [vsanDatastore] 8c95d659-46fa-b9a6-5e19-02002f28e688/e2e-vmdk-1507749256431387056.vmdk
STEP: After master restart, verify volume [vsanDatastore] 8c95d659-46fa-b9a6-5e19-02002f28e688/e2e-vmdk-1507749281940603428.vmdk is attached to the pod kubernetes-node2
STEP: Deleting pod on node kubernetes-node2
Oct 11 12:17:11.614: INFO: Deleting pod "vsphere-e2e-vqkbp" in namespace "e2e-tests-restart-master-j9x0f"
Oct 11 12:17:11.624: INFO: Wait up to 5m0s for pod "vsphere-e2e-vqkbp" to be fully deleted
STEP: Waiting for volume [vsanDatastore] 8c95d659-46fa-b9a6-5e19-02002f28e688/e2e-vmdk-1507749281940603428.vmdk to be detached from the node kubernetes-node2
Oct 11 12:17:55.748: INFO: Volume "[vsanDatastore] 8c95d659-46fa-b9a6-5e19-02002f28e688/e2e-vmdk-1507749281940603428.vmdk" appears to have successfully detached from "kubernetes-node2".
STEP: Deleting volume [vsanDatastore] 8c95d659-46fa-b9a6-5e19-02002f28e688/e2e-vmdk-1507749281940603428.vmdk
STEP: After master restart, verify volume [vsanDatastore] 8c95d659-46fa-b9a6-5e19-02002f28e688/e2e-vmdk-1507749305162880964.vmdk is attached to the pod kubernetes-node3
STEP: Deleting pod on node kubernetes-node3
Oct 11 12:17:56.051: INFO: Deleting pod "vsphere-e2e-fkrzb" in namespace "e2e-tests-restart-master-j9x0f"
Oct 11 12:17:56.069: INFO: Wait up to 5m0s for pod "vsphere-e2e-fkrzb" to be fully deleted
STEP: Waiting for volume [vsanDatastore] 8c95d659-46fa-b9a6-5e19-02002f28e688/e2e-vmdk-1507749305162880964.vmdk to be detached from the node kubernetes-node3
Oct 11 12:18:38.199: INFO: Volume "[vsanDatastore] 8c95d659-46fa-b9a6-5e19-02002f28e688/e2e-vmdk-1507749305162880964.vmdk" appears to have successfully detached from "kubernetes-node3".
STEP: Deleting volume [vsanDatastore] 8c95d659-46fa-b9a6-5e19-02002f28e688/e2e-vmdk-1507749305162880964.vmdk
STEP: After master restart, verify volume [vsanDatastore] 8c95d659-46fa-b9a6-5e19-02002f28e688/e2e-vmdk-1507749330788801099.vmdk is attached to the pod kubernetes-node4
STEP: Deleting pod on node kubernetes-node4
Oct 11 12:18:38.541: INFO: Deleting pod "vsphere-e2e-4cb0d" in namespace "e2e-tests-restart-master-j9x0f"
Oct 11 12:18:38.556: INFO: Wait up to 5m0s for pod "vsphere-e2e-4cb0d" to be fully deleted
STEP: Waiting for volume [vsanDatastore] 8c95d659-46fa-b9a6-5e19-02002f28e688/e2e-vmdk-1507749330788801099.vmdk to be detached from the node kubernetes-node4
Oct 11 12:19:22.672: INFO: Volume "[vsanDatastore] 8c95d659-46fa-b9a6-5e19-02002f28e688/e2e-vmdk-1507749330788801099.vmdk" appears to have successfully detached from "kubernetes-node4".
STEP: Deleting volume [vsanDatastore] 8c95d659-46fa-b9a6-5e19-02002f28e688/e2e-vmdk-1507749330788801099.vmdk
[AfterEach] [sig-storage] Volume Attach Verify [Feature:vsphere]
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134
Oct 11 12:19:23.460: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "e2e-tests-restart-master-j9x0f" for this suite.
Oct 11 12:19:29.544: INFO: namespace: e2e-tests-restart-master-j9x0f, resource: bindings, ignored listing per whitelist
Oct 11 12:19:29.622: INFO: namespace e2e-tests-restart-master-j9x0f deletion completed in 6.156220683s
• [SLOW TEST:313.535 seconds]
[sig-storage] Volume Attach Verify [Feature:vsphere]
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework.go:22
verify volume remains attached after master kubelet restart
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/vsphere_volume_master_restart.go:144
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSOct 11 12:19:29.666: INFO: Running AfterSuite actions on all node
Oct 11 12:19:29.666: INFO: Running AfterSuite actions on node 1
Ran 1 of 709 Specs in 313.923 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 708 Skipped PASS
```
Internally reviewed by VMware reviewers @divyenpatel @BaluDontu @tusharnt
**Release note**:
```
None
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Update volume OWNERS to reflect active sig-storage reviewers
**What this PR does / why we need it**:
Update sig-storage reviewers to add new members and remove those that don't have as much time to review storage PRs. Approvers are unchanged.
**Special notes for your reviewer**:
For all those that have been removed, please approve. If you want to remain as a reviewer, let me know and I will add you back.
**Release note**:
NONE
Automatic merge from submit-queue (batch tested with PRs 52868, 53196, 54207). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
eviction/detach test
**What this PR does / why we need it**:
e2e test for detach after a pod is evicted.
**Which issue this PR fixes** : fixes#52676
**Release note**:
```release-note
NONE
```
cc @jingxu97 @copejon
Automatic merge from submit-queue (batch tested with PRs 54107, 54184, 54377, 54094, 54111). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix detach metric flake by not using exact equals
Also poll for detach value increase.
Fixes https://github.com/kubernetes/kubernetes/issues/52871
I have ran these tests for more than 3 hours in a tight loop and did not see it flake. The changes here include dropping exact equality test and making sure we poll for increase in detach metric count.
```release-note
None
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Updating E2E test for deleting PVC when PVC is in use
**What this PR does / why we need it**:
This test updates an existing e2e test and adds extra verification.
Updated workflow of the test is as below
1. Create PVC, Wait until PV is provisioned. Create POD using PVC.
2. Verify POD is running and PV is attached to the node.
3. Delete PVC.
4. Verify Volume remains attached to the pod after deleting claim.
5. Verify Volume is accessible in the pod after deleting claim.
6. Verify associated PV is present and its status should be failed.
7. Delete Pod and wait until PV is unmounted and detached from the Node.
6. Wait and Verify PV is deleted after POD is deleted.
**Which issue this PR fixes**
fixes # https://github.com/vmware/kubernetes/issues/279
**Special notes for your reviewer**:
Test logs
```
# go run hack/e2e.go --check-version-skew=false -v -test --test_args='--ginkgo.focus=should\snot\sdetach\sand\sunmount\sPV\swhen\sassociated\spvc\swith\sdelete\sas\sreclaimPolicy\sis\sdeleted\swhen\sit\sis\sin\suse\sby\sthe\spod'
flag provided but not defined: -check-version-skew
Usage of /tmp/go-build371606839/command-line-arguments/_obj/exe/e2e:
-get
go get -u kubetest if old or not installed (default true)
-old duration
Consider kubetest old if it exceeds this (default 24h0m0s)
2017/10/16 15:42:40 e2e.go:55: NOTICE: go run hack/e2e.go is now a shim for test-infra/kubetest
2017/10/16 15:42:40 e2e.go:56: Usage: go run hack/e2e.go [--get=true] [--old=24h0m0s] -- [KUBETEST_ARGS]
2017/10/16 15:42:40 e2e.go:57: The separator is required to use --get or --old flags
2017/10/16 15:42:40 e2e.go:58: The -- flag separator also suppresses this message
2017/10/16 15:42:40 e2e.go:77: Calling kubetest --check-version-skew=false -v -test --test_args=--ginkgo.focus=should\snot\sdetach\sand\sunmount\sPV\swhen\sassociated\spvc\swith\sdelete\sas\sreclaimPolicy\sis\sdeleted\swhen\sit\sis\sin\suse\sby\sthe\spod...
2017/10/16 15:42:40 util.go:154: Running: ./cluster/kubectl.sh --match-server-version=false version
2017/10/16 15:42:40 util.go:156: Step './cluster/kubectl.sh --match-server-version=false version' finished in 293.775296ms
2017/10/16 15:42:40 util.go:154: Running: ./hack/e2e-internal/e2e-status.sh
Skeleton Provider: prepare-e2e not implemented
Client Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.0-alpha.1.913+297ab03890a6a7-dirty", GitCommit:"297ab03890a6a76f268eb5415e0fb16f20b2309e", GitTreeState:"dirty", BuildDate:"2017-10-16T20:50:38Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.0-alpha.1.1181+77b83e446b4e65", GitCommit:"77b83e446b4e655a71c315ad3f3890dc2a220ccf", GitTreeState:"clean", BuildDate:"2017-10-16T07:07:02Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
2017/10/16 15:42:40 util.go:156: Step './hack/e2e-internal/e2e-status.sh' finished in 317.940582ms
2017/10/16 15:42:40 util.go:154: Running: ./hack/ginkgo-e2e.sh --ginkgo.focus=should\snot\sdetach\sand\sunmount\sPV\swhen\sassociated\spvc\swith\sdelete\sas\sreclaimPolicy\sis\sdeleted\swhen\sit\sis\sin\suse\sby\sthe\spod
Conformance test: not doing test setup.
Oct 16 15:42:42.327: INFO: Overriding default scale value of zero to 1
Oct 16 15:42:42.327: INFO: Overriding default milliseconds value of zero to 5000
I1016 15:42:42.577720 8325 e2e.go:369] Starting e2e run "51f11717-b2c3-11e7-bd54-0050569c26b8" on Ginkgo node 1
Running Suite: Kubernetes e2e suite
===================================
Random Seed: 1508193761 - Will randomize all specs
Will run 1 of 706 specs
Oct 16 15:42:42.678: INFO: >>> kubeConfig: /root/.kube/config
Oct 16 15:42:42.686: INFO: Waiting up to 4h0m0s for all (but 0) nodes to be schedulable
Oct 16 15:42:42.724: INFO: Waiting up to 10m0s for all pods (need at least 0) in namespace 'kube-system' to be running and ready
Oct 16 15:42:42.883: INFO: 13 / 13 pods in namespace 'kube-system' are running and ready (0 seconds elapsed)
Oct 16 15:42:42.883: INFO: expected 4 pod replicas in namespace 'kube-system', 4 are Running and Ready.
Oct 16 15:42:42.891: INFO: Waiting for pods to enter Success, but no pods in "kube-system" match label map[name:e2e-image-puller]
Oct 16 15:42:42.891: INFO: Dumping network health container logs from all nodes...
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[sig-storage] PersistentVolumes [Feature:ReclaimPolicy] [sig-storage] persistentvolumereclaim:vsphere
should not detach and unmount PV when associated pvc with delete as reclaimPolicy is deleted when it is in use by the pod
/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/pv_reclaimpolicy.go:136
[BeforeEach] [sig-storage] PersistentVolumes [Feature:ReclaimPolicy]
/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:133
STEP: Creating a kubernetes client
Oct 16 15:42:42.904: INFO: >>> kubeConfig: /root/.kube/config
STEP: Building a namespace api object
STEP: Waiting for a default service account to be provisioned in namespace
[BeforeEach] [sig-storage] PersistentVolumes [Feature:ReclaimPolicy]
/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/pv_reclaimpolicy.go:48
Oct 16 15:42:42.994: INFO: Waiting up to 4h0m0s for all (but 0) nodes to be schedulable
[BeforeEach] [sig-storage] persistentvolumereclaim:vsphere
/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/pv_reclaimpolicy.go:56
[It] should not detach and unmount PV when associated pvc with delete as reclaimPolicy is deleted when it is in use by the pod
/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/pv_reclaimpolicy.go:136
STEP: running testSetupVSpherePersistentVolumeReclaim
STEP: creating vmdk
STEP: creating the pv
STEP: creating the pvc
Oct 16 15:42:44.595: INFO: Waiting for PV vspherepv-ksccp to bind to PVC pvc-n4rq7
Oct 16 15:42:44.595: INFO: Waiting up to 5m0s for PersistentVolumeClaim pvc-n4rq7 to have phase Bound
Oct 16 15:42:44.606: INFO: PersistentVolumeClaim pvc-n4rq7 found but phase is Pending instead of Bound.
Oct 16 15:42:47.625: INFO: PersistentVolumeClaim pvc-n4rq7 found and phase=Bound (3.029926391s)
Oct 16 15:42:47.625: INFO: Waiting up to 5m0s for PersistentVolume vspherepv-ksccp to have phase Bound
Oct 16 15:42:47.632: INFO: PersistentVolume vspherepv-ksccp found and phase=Bound (6.598243ms)
STEP: Creating the Pod
STEP: Deleting the Claim
Oct 16 15:42:59.709: INFO: Deleting PersistentVolumeClaim "pvc-n4rq7"
STEP: Verify the volume is attached to the node
STEP: Verify the volume is accessible and available in the pod
Oct 16 15:43:00.076: INFO: Running '/root/divyenp/kubernetes/_output/bin/kubectl --server=https://10.192.55.64 --kubeconfig=/root/.kube/config exec pvc-tester-r9ww9 --namespace=e2e-tests-persistentvolumereclaim-6pfpf -- /bin/touch /mnt/volume1/emptyFile.txt'
Oct 16 15:43:00.604: INFO: stderr: ""
Oct 16 15:43:00.604: INFO: stdout: ""
Oct 16 15:43:00.604: INFO: Verified that Volume is accessible in the POD after deleting PV claim
Oct 16 15:43:00.610: INFO: Waiting up to 1m0s for PersistentVolume vspherepv-ksccp to have phase Failed
Oct 16 15:43:00.619: INFO: PersistentVolume vspherepv-ksccp found and phase=Failed (9.016306ms)
STEP: Deleting the Pod
Oct 16 15:43:00.619: INFO: Deleting pod pvc-tester-r9ww9
Oct 16 15:43:00.650: INFO: Waiting up to 5m0s for pod "pvc-tester-r9ww9" in namespace "e2e-tests-persistentvolumereclaim-6pfpf" to be "terminated due to deadline exceeded"
Oct 16 15:43:00.668: INFO: Pod "pvc-tester-r9ww9": Phase="Running", Reason="", readiness=true. Elapsed: 18.507993ms
Oct 16 15:43:02.675: INFO: Pod "pvc-tester-r9ww9": Phase="Running", Reason="", readiness=true. Elapsed: 2.024854663s
Oct 16 15:43:04.682: INFO: Pod "pvc-tester-r9ww9": Phase="Running", Reason="", readiness=true. Elapsed: 4.03197856s
Oct 16 15:43:06.688: INFO: Pod "pvc-tester-r9ww9": Phase="Running", Reason="", readiness=true. Elapsed: 6.037718623s
Oct 16 15:43:08.697: INFO: Pod "pvc-tester-r9ww9": Phase="Running", Reason="", readiness=true. Elapsed: 8.047192574s
Oct 16 15:43:10.703: INFO: Pod "pvc-tester-r9ww9": Phase="Running", Reason="", readiness=true. Elapsed: 10.052754761s
Oct 16 15:43:12.708: INFO: Pod "pvc-tester-r9ww9": Phase="Running", Reason="", readiness=true. Elapsed: 12.057876018s
Oct 16 15:43:14.714: INFO: Pod "pvc-tester-r9ww9": Phase="Running", Reason="", readiness=true. Elapsed: 14.063962712s
Oct 16 15:43:16.719: INFO: Pod "pvc-tester-r9ww9": Phase="Running", Reason="", readiness=true. Elapsed: 16.068826626s
Oct 16 15:43:18.725: INFO: Pod "pvc-tester-r9ww9": Phase="Running", Reason="", readiness=true. Elapsed: 18.074735397s
Oct 16 15:43:20.730: INFO: Pod "pvc-tester-r9ww9": Phase="Running", Reason="", readiness=true. Elapsed: 20.080498293s
Oct 16 15:43:22.736: INFO: Pod "pvc-tester-r9ww9": Phase="Running", Reason="", readiness=true. Elapsed: 22.086586123s
Oct 16 15:43:24.742: INFO: Pod "pvc-tester-r9ww9": Phase="Running", Reason="", readiness=true. Elapsed: 24.092219324s
Oct 16 15:43:26.747: INFO: Pod "pvc-tester-r9ww9": Phase="Running", Reason="", readiness=true. Elapsed: 26.097385301s
Oct 16 15:43:28.753: INFO: Pod "pvc-tester-r9ww9": Phase="Running", Reason="", readiness=true. Elapsed: 28.103127591s
Oct 16 15:43:30.758: INFO: Pod "pvc-tester-r9ww9": Phase="Running", Reason="", readiness=true. Elapsed: 30.108014823s
Oct 16 15:43:32.764: INFO: Pod "pvc-tester-r9ww9": Phase="Pending", Reason="", readiness=false. Elapsed: 32.113847674s
Oct 16 15:43:34.772: INFO: Pod "pvc-tester-r9ww9": Phase="Pending", Reason="", readiness=false. Elapsed: 34.122010171s
Oct 16 15:43:36.787: INFO: Pod "pvc-tester-r9ww9" in namespace "e2e-tests-persistentvolumereclaim-6pfpf" not found. Error: pods "pvc-tester-r9ww9" not found
Oct 16 15:43:36.787: INFO: Ignore "not found" error above. Pod "pvc-tester-r9ww9" successfully deleted
STEP: Verify PV is detached from the node after Pod is deleted
Oct 16 15:43:46.913: INFO: Waiting for Volume "[vsanDatastore] 1874c359-f300-a0cc-fd7e-02002a623c85/e2e-vmdk-1508193763110460154.vmdk" to detach from "kubernetes-node2".
Oct 16 15:43:56.918: INFO: Waiting for Volume "[vsanDatastore] 1874c359-f300-a0cc-fd7e-02002a623c85/e2e-vmdk-1508193763110460154.vmdk" to detach from "kubernetes-node2".
Oct 16 15:44:06.905: INFO: Volume "[vsanDatastore] 1874c359-f300-a0cc-fd7e-02002a623c85/e2e-vmdk-1508193763110460154.vmdk" appears to have successfully detached from "kubernetes-node2".
STEP: Verify PV should be deleted automatically
Oct 16 15:44:06.905: INFO: Waiting up to 30s for PersistentVolume vspherepv-ksccp to get deleted
Oct 16 15:44:06.909: INFO: PersistentVolume vspherepv-ksccp was removed
[AfterEach] [sig-storage] persistentvolumereclaim:vsphere
/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/pv_reclaimpolicy.go:62
STEP: running testCleanupVSpherePersistentVolumeReclaim
Oct 16 15:44:06.962: INFO: Deleting PersistentVolume "vspherepv-ksccp"
[AfterEach] [sig-storage] PersistentVolumes [Feature:ReclaimPolicy]
/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134
Oct 16 15:44:06.967: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "e2e-tests-persistentvolumereclaim-6pfpf" for this suite.
Oct 16 15:44:15.325: INFO: namespace: e2e-tests-persistentvolumereclaim-6pfpf, resource: bindings, ignored listing per whitelist
Oct 16 15:44:15.638: INFO: namespace e2e-tests-persistentvolumereclaim-6pfpf deletion completed in 8.651759385s
• [SLOW TEST:92.734 seconds]
[sig-storage] PersistentVolumes [Feature:ReclaimPolicy]
/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/framework.go:22
[sig-storage] persistentvolumereclaim:vsphere
/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/framework.go:22
should not detach and unmount PV when associated pvc with delete as reclaimPolicy is deleted when it is in use by the pod
/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/pv_reclaimpolicy.go:136
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSOct 16 15:44:15.651: INFO: Running AfterSuite actions on all node
Oct 16 15:44:15.651: INFO: Running AfterSuite actions on node 1
Ran 1 of 706 Specs in 92.974 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 705 Skipped PASS
Ginkgo ran 1 suite in 1m33.830856163s
Test Suite Passed
2017/10/16 15:44:15 util.go:156: Step './hack/ginkgo-e2e.sh --ginkgo.focus=should\snot\sdetach\sand\sunmount\sPV\swhen\sassociated\spvc\swith\sdelete\sas\sreclaimPolicy\sis\sdeleted\swhen\sit\sis\sin\suse\sby\sthe\spod' finished in 1m34.75838192s
2017/10/16 15:44:15 e2e.go:81: Done
```
VVMware Reviewers: @rohitjogvmw @BaluDontu @tusharnt
**Release note**:
```release-note
None
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Adding e2e test for statefulsets for vsphere cloud provider
**What this PR does / why we need it**:
This PR adds a new e2e test for statefulsets for vSphere cloud Provider.
Test does following tasks.
- Create a storage class with thin diskformat.
- Create nginx service.
- Create nginx statefulsets with 3 replicas.
- Wait until all Pods are ready and PVCs are bounded with PV.
- Verify volumes are accessible in all statefulsets pods with creating empty file.
- Scale down statefulsets to 2 replicas.
- Scale up statefulsets to 3 replicas.
- Scale down statefulsets to 0 replicas and delete all pods.
- Delete all PVCs from the test namespace.
- Delete the storage class.
**Which issue this PR fixes**
fixes # https://github.com/vmware/kubernetes/issues/275
**Special notes for your reviewer**:
Test Logs
```
root@k8s-dev-vm-02:~/divyenp/kubernetes# go run hack/e2e.go --check-version-skew=false --v --test --test_args='--ginkgo.focus=vsphere\sstatefulset\stesting'
flag provided but not defined: -check-version-skew
Usage of /tmp/go-build247641121/command-line-arguments/_obj/exe/e2e:
-get
go get -u kubetest if old or not installed (default true)
-old duration
Consider kubetest old if it exceeds this (default 24h0m0s)
2017/10/18 19:24:33 e2e.go:55: NOTICE: go run hack/e2e.go is now a shim for test-infra/kubetest
2017/10/18 19:24:33 e2e.go:56: Usage: go run hack/e2e.go [--get=true] [--old=24h0m0s] -- [KUBETEST_ARGS]
2017/10/18 19:24:33 e2e.go:57: The separator is required to use --get or --old flags
2017/10/18 19:24:33 e2e.go:58: The -- flag separator also suppresses this message
2017/10/18 19:24:33 e2e.go:77: Calling kubetest --check-version-skew=false --v --test --test_args=--ginkgo.focus=vsphere\sstatefulset\stesting...
2017/10/18 19:24:33 util.go:154: Running: ./cluster/kubectl.sh --match-server-version=false version
2017/10/18 19:24:34 util.go:156: Step './cluster/kubectl.sh --match-server-version=false version' finished in 290.682219ms
2017/10/18 19:24:34 util.go:154: Running: ./hack/e2e-internal/e2e-status.sh
Skeleton Provider: prepare-e2e not implemented
Client Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.0-alpha.1.1217+8b041da0f996c1-dirty", GitCommit:"8b041da0f996c185438a7ed8282f92734a2ed0e7", GitTreeState:"dirty", BuildDate:"2017-10-19T00:46:00Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.0-alpha.1.1293+d462bac7805f53", GitCommit:"d462bac7805f536a43c7d5fb98aca138ba1237eb", GitTreeState:"clean", BuildDate:"2017-10-18T07:07:08Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
2017/10/18 19:24:34 util.go:156: Step './hack/e2e-internal/e2e-status.sh' finished in 305.965323ms
2017/10/18 19:24:34 util.go:154: Running: ./hack/ginkgo-e2e.sh --ginkgo.focus=vsphere\sstatefulset\stesting
Conformance test: not doing test setup.
Oct 18 19:24:35.808: INFO: Overriding default scale value of zero to 1
Oct 18 19:24:35.808: INFO: Overriding default milliseconds value of zero to 5000
I1018 19:24:36.073718 7768 e2e.go:383] Starting e2e run "a63561de-b474-11e7-8f6b-0050569c26b8" on Ginkgo node 1
Running Suite: Kubernetes e2e suite
===================================
Random Seed: 1508379875 - Will randomize all specs
Will run 1 of 713 specs
Oct 18 19:24:36.132: INFO: >>> kubeConfig: /root/.kube/config
Oct 18 19:24:36.139: INFO: Waiting up to 4h0m0s for all (but 0) nodes to be schedulable
Oct 18 19:24:36.177: INFO: Waiting up to 10m0s for all pods (need at least 0) in namespace 'kube-system' to be running and ready
Oct 18 19:24:36.321: INFO: 13 / 13 pods in namespace 'kube-system' are running and ready (0 seconds elapsed)
Oct 18 19:24:36.321: INFO: expected 4 pod replicas in namespace 'kube-system', 4 are Running and Ready.
Oct 18 19:24:36.326: INFO: Waiting for pods to enter Success, but no pods in "kube-system" match label map[name:e2e-image-puller]
Oct 18 19:24:36.326: INFO: Dumping network health container logs from all nodes...
Oct 18 19:24:36.338: INFO: Client version: v1.9.0-alpha.1.1217+8b041da0f996c1-dirty
Oct 18 19:24:36.340: INFO: Server version: v1.9.0-alpha.1.1293+d462bac7805f53
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
------------------------------
[sig-storage] vsphere statefulset
vsphere statefulset testing
/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/vsphere_statefulsets.go:155
[BeforeEach] [sig-storage] vsphere statefulset
/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:133
STEP: Creating a kubernetes client
Oct 18 19:24:36.349: INFO: >>> kubeConfig: /root/.kube/config
STEP: Building a namespace api object
STEP: Waiting for a default service account to be provisioned in namespace
[BeforeEach] [sig-storage] vsphere statefulset
/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/vsphere_statefulsets.go:63
[It] vsphere statefulset testing
/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/vsphere_statefulsets.go:155
STEP: Creating StorageClass for Statefulset
STEP: Creating statefulset
Oct 18 19:24:36.489: INFO: Parsing statefulset from test/e2e/testing-manifests/statefulset/nginx/statefulset.yaml
Oct 18 19:24:36.503: INFO: Parsing service from test/e2e/testing-manifests/statefulset/nginx/service.yaml
Oct 18 19:24:36.514: INFO: creating web service
Oct 18 19:24:36.527: INFO: creating statefulset e2e-tests-vsphere-statefulset-gnfmp/web with 3 replicas and selector &LabelSelector{MatchLabels:map[string]string{app: nginx,},MatchExpressions:[],}
Oct 18 19:24:36.561: INFO: Found 0 stateful pods, waiting for 3
Oct 18 19:24:46.567: INFO: Found 1 stateful pods, waiting for 3
Oct 18 19:24:56.568: INFO: Found 1 stateful pods, waiting for 3
Oct 18 19:25:06.568: INFO: Found 1 stateful pods, waiting for 3
Oct 18 19:25:16.566: INFO: Found 1 stateful pods, waiting for 3
Oct 18 19:25:26.567: INFO: Found 1 stateful pods, waiting for 3
Oct 18 19:25:36.567: INFO: Found 1 stateful pods, waiting for 3
Oct 18 19:25:46.567: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:25:56.568: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:26:06.567: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:26:16.568: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:26:26.567: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:26:36.568: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:26:46.568: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:26:56.571: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:27:06.568: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:27:16.569: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:27:26.568: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:27:36.569: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:27:46.569: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:27:56.567: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:28:06.567: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:28:16.567: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:28:26.567: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:28:36.574: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:28:46.567: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:28:56.571: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:29:06.569: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:29:16.568: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:29:26.566: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:29:36.568: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:29:46.566: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:29:56.567: INFO: Found 2 stateful pods, waiting for 3
Oct 18 19:30:06.568: INFO: Waiting for pod web-0 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:30:06.568: INFO: Waiting for pod web-1 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:30:06.568: INFO: Waiting for pod web-2 to enter Running - Ready=true, currently Pending - Ready=false
Oct 18 19:30:16.567: INFO: Waiting for pod web-0 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:30:16.567: INFO: Waiting for pod web-1 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:30:16.567: INFO: Waiting for pod web-2 to enter Running - Ready=true, currently Pending - Ready=false
Oct 18 19:30:26.567: INFO: Waiting for pod web-0 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:30:26.567: INFO: Waiting for pod web-1 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:30:26.567: INFO: Waiting for pod web-2 to enter Running - Ready=true, currently Pending - Ready=false
Oct 18 19:30:36.567: INFO: Waiting for pod web-0 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:30:36.567: INFO: Waiting for pod web-1 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:30:36.567: INFO: Waiting for pod web-2 to enter Running - Ready=true, currently Pending - Ready=false
Oct 18 19:30:46.567: INFO: Waiting for pod web-0 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:30:46.567: INFO: Waiting for pod web-1 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:30:46.567: INFO: Waiting for pod web-2 to enter Running - Ready=true, currently Pending - Ready=false
Oct 18 19:30:56.566: INFO: Waiting for pod web-0 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:30:56.567: INFO: Waiting for pod web-1 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:30:56.567: INFO: Waiting for pod web-2 to enter Running - Ready=true, currently Pending - Ready=false
Oct 18 19:31:06.567: INFO: Waiting for pod web-0 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:31:06.568: INFO: Waiting for pod web-1 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:31:06.568: INFO: Waiting for pod web-2 to enter Running - Ready=true, currently Pending - Ready=false
Oct 18 19:31:16.567: INFO: Waiting for pod web-0 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:31:16.567: INFO: Waiting for pod web-1 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:31:16.567: INFO: Waiting for pod web-2 to enter Running - Ready=true, currently Pending - Ready=false
Oct 18 19:31:26.567: INFO: Waiting for pod web-0 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:31:26.568: INFO: Waiting for pod web-1 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:31:26.568: INFO: Waiting for pod web-2 to enter Running - Ready=true, currently Pending - Ready=false
Oct 18 19:31:36.568: INFO: Waiting for pod web-0 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:31:36.568: INFO: Waiting for pod web-1 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:31:36.568: INFO: Waiting for pod web-2 to enter Running - Ready=true, currently Pending - Ready=false
Oct 18 19:31:46.568: INFO: Waiting for pod web-0 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:31:46.568: INFO: Waiting for pod web-1 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:31:46.568: INFO: Waiting for pod web-2 to enter Running - Ready=true, currently Pending - Ready=false
Oct 18 19:31:56.568: INFO: Waiting for pod web-0 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:31:56.568: INFO: Waiting for pod web-1 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:31:56.568: INFO: Waiting for pod web-2 to enter Running - Ready=true, currently Pending - Ready=false
Oct 18 19:32:06.567: INFO: Waiting for pod web-0 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:32:06.567: INFO: Waiting for pod web-1 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:32:06.567: INFO: Waiting for pod web-2 to enter Running - Ready=true, currently Pending - Ready=false
Oct 18 19:32:16.571: INFO: Waiting for pod web-0 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:32:16.571: INFO: Waiting for pod web-1 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:32:16.571: INFO: Waiting for pod web-2 to enter Running - Ready=true, currently Pending - Ready=false
Oct 18 19:32:26.567: INFO: Waiting for pod web-0 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:32:26.567: INFO: Waiting for pod web-1 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:32:26.567: INFO: Waiting for pod web-2 to enter Running - Ready=true, currently Running - Ready=true
Oct 18 19:32:26.567: INFO: Waiting for statefulset status.replicas updated to 3
Oct 18 19:32:26.605: INFO: Running '/root/divyenp/kubernetes/_output/bin/kubectl --server=https://10.192.38.85 --kubeconfig=/root/.kube/config exec --namespace=e2e-tests-vsphere-statefulset-gnfmp web-0 -- /bin/sh -c ls -idlh /usr/share/nginx/html'
Oct 18 19:32:27.170: INFO: stderr: ""
Oct 18 19:32:27.170: INFO: stdout of ls -idlh /usr/share/nginx/html on web-0: 2 drwxr-xr-x 3 root root 4.0K Oct 19 02:25 /usr/share/nginx/html
Oct 18 19:32:27.171: INFO: Running '/root/divyenp/kubernetes/_output/bin/kubectl --server=https://10.192.38.85 --kubeconfig=/root/.kube/config exec --namespace=e2e-tests-vsphere-statefulset-gnfmp web-1 -- /bin/sh -c ls -idlh /usr/share/nginx/html'
Oct 18 19:32:27.687: INFO: stderr: ""
Oct 18 19:32:27.688: INFO: stdout of ls -idlh /usr/share/nginx/html on web-1: 2 drwxr-xr-x 3 root root 4.0K Oct 19 02:29 /usr/share/nginx/html
Oct 18 19:32:27.688: INFO: Running '/root/divyenp/kubernetes/_output/bin/kubectl --server=https://10.192.38.85 --kubeconfig=/root/.kube/config exec --namespace=e2e-tests-vsphere-statefulset-gnfmp web-2 -- /bin/sh -c ls -idlh /usr/share/nginx/html'
Oct 18 19:32:28.177: INFO: stderr: ""
Oct 18 19:32:28.177: INFO: stdout of ls -idlh /usr/share/nginx/html on web-2: 2 drwxr-xr-x 3 root root 4.0K Oct 19 02:32 /usr/share/nginx/html
Oct 18 19:32:28.183: INFO: Running '/root/divyenp/kubernetes/_output/bin/kubectl --server=https://10.192.38.85 --kubeconfig=/root/.kube/config exec --namespace=e2e-tests-vsphere-statefulset-gnfmp web-0 -- /bin/sh -c find /usr/share/nginx/html'
Oct 18 19:32:28.690: INFO: stderr: ""
Oct 18 19:32:28.690: INFO: stdout of find /usr/share/nginx/html on web-0: /usr/share/nginx/html
/usr/share/nginx/html/lost+found
Oct 18 19:32:28.690: INFO: Running '/root/divyenp/kubernetes/_output/bin/kubectl --server=https://10.192.38.85 --kubeconfig=/root/.kube/config exec --namespace=e2e-tests-vsphere-statefulset-gnfmp web-1 -- /bin/sh -c find /usr/share/nginx/html'
Oct 18 19:32:29.166: INFO: stderr: ""
Oct 18 19:32:29.166: INFO: stdout of find /usr/share/nginx/html on web-1: /usr/share/nginx/html
/usr/share/nginx/html/lost+found
Oct 18 19:32:29.166: INFO: Running '/root/divyenp/kubernetes/_output/bin/kubectl --server=https://10.192.38.85 --kubeconfig=/root/.kube/config exec --namespace=e2e-tests-vsphere-statefulset-gnfmp web-2 -- /bin/sh -c find /usr/share/nginx/html'
Oct 18 19:32:29.696: INFO: stderr: ""
Oct 18 19:32:29.696: INFO: stdout of find /usr/share/nginx/html on web-2: /usr/share/nginx/html
/usr/share/nginx/html/lost+found
Oct 18 19:32:29.707: INFO: Running '/root/divyenp/kubernetes/_output/bin/kubectl --server=https://10.192.38.85 --kubeconfig=/root/.kube/config exec --namespace=e2e-tests-vsphere-statefulset-gnfmp web-0 -- /bin/sh -c touch /usr/share/nginx/html/1508380346587629054'
Oct 18 19:32:30.171: INFO: stderr: ""
Oct 18 19:32:30.171: INFO: stdout of touch /usr/share/nginx/html/1508380346587629054 on web-0:
Oct 18 19:32:30.171: INFO: Running '/root/divyenp/kubernetes/_output/bin/kubectl --server=https://10.192.38.85 --kubeconfig=/root/.kube/config exec --namespace=e2e-tests-vsphere-statefulset-gnfmp web-1 -- /bin/sh -c touch /usr/share/nginx/html/1508380346587629054'
Oct 18 19:32:30.653: INFO: stderr: ""
Oct 18 19:32:30.653: INFO: stdout of touch /usr/share/nginx/html/1508380346587629054 on web-1:
Oct 18 19:32:30.654: INFO: Running '/root/divyenp/kubernetes/_output/bin/kubectl --server=https://10.192.38.85 --kubeconfig=/root/.kube/config exec --namespace=e2e-tests-vsphere-statefulset-gnfmp web-2 -- /bin/sh -c touch /usr/share/nginx/html/1508380346587629054'
Oct 18 19:32:31.149: INFO: stderr: ""
Oct 18 19:32:31.150: INFO: stdout of touch /usr/share/nginx/html/1508380346587629054 on web-2:
STEP: Scaling down statefulsets to number of Replica: 2
Oct 18 19:32:31.263: INFO: Scaling statefulset web to 2
Oct 18 19:32:51.314: INFO: Waiting for statefulset status.replicas updated to 2
STEP: Verify Volumes are detached from Nodes after Statefulsets is scaled down
Oct 18 19:32:51.524: INFO: Waiting for Volume: "[vsanDatastore] 1874c359-f300-a0cc-fd7e-02002a623c85/kubernetes-dynamic-pvc-67b7e88c-b475-11e7-a38c-0050569c555f.vmdk" to detach from Node: "kubernetes-node2"
Oct 18 19:33:01.657: INFO: Volume "[vsanDatastore] 1874c359-f300-a0cc-fd7e-02002a623c85/kubernetes-dynamic-pvc-67b7e88c-b475-11e7-a38c-0050569c555f.vmdk" appears to have successfully detached from "kubernetes-node2".
STEP: Scaling up statefulsets to number of Replica: 3
Oct 18 19:33:01.657: INFO: Scaling statefulset web to 3
Oct 18 19:33:11.731: INFO: Waiting for statefulset status.replicas updated to 3
Oct 18 19:33:11.747: INFO: Waiting for statefulset status.replicas updated to 3
STEP: Verify all volumes are attached to Nodes after Statefulsets is scaled up
Oct 18 19:33:13.823: INFO: Verify Volume: "[vsanDatastore] 1874c359-f300-a0cc-fd7e-02002a623c85/kubernetes-dynamic-pvc-a6cf15ef-b474-11e7-a38c-0050569c555f.vmdk" is attached to the Node: "kubernetes-node4"
Oct 18 19:33:15.990: INFO: Verify Volume: "[vsanDatastore] 1874c359-f300-a0cc-fd7e-02002a623c85/kubernetes-dynamic-pvc-cfb65f92-b474-11e7-a38c-0050569c555f.vmdk" is attached to the Node: "kubernetes-node3"
Oct 18 19:33:18.154: INFO: Verify Volume: "[vsanDatastore] 1874c359-f300-a0cc-fd7e-02002a623c85/kubernetes-dynamic-pvc-67b7e88c-b475-11e7-a38c-0050569c555f.vmdk" is attached to the Node: "kubernetes-node2"
[AfterEach] [sig-storage] vsphere statefulset
/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134
Oct 18 19:33:18.323: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "e2e-tests-vsphere-statefulset-gnfmp" for this suite.
Oct 18 19:33:44.960: INFO: namespace: e2e-tests-vsphere-statefulset-gnfmp, resource: bindings, ignored listing per whitelist
Oct 18 19:33:44.960: INFO: namespace e2e-tests-vsphere-statefulset-gnfmp deletion completed in 26.620223678s
[AfterEach] [sig-storage] vsphere statefulset
/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/vsphere_statefulsets.go:67
Oct 18 19:33:44.960: INFO: Deleting all statefulset in namespace: e2e-tests-vsphere-statefulset-gnfmp
• [SLOW TEST:548.654 seconds]
[sig-storage] vsphere statefulset
/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/framework.go:22
vsphere statefulset testing
/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/vsphere_statefulsets.go:155
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSOct 18 19:33:45.006: INFO: Running AfterSuite actions on all node
Oct 18 19:33:45.006: INFO: Running AfterSuite actions on node 1
Ran 1 of 713 Specs in 548.875 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 712 Skipped PASS
Ginkgo ran 1 suite in 9m9.728218415s
Test Suite Passed
2017/10/18 19:33:45 util.go:156: Step './hack/ginkgo-e2e.sh --ginkgo.focus=vsphere\sstatefulset\stesting' finished in 9m10.656371481s
2017/10/18 19:33:45 e2e.go:81: Done
```
VMware Reviewers: @rohitjogvmw @BaluDontu @tusharnt
**Release note**:
```release-note
NONE
```
Ceph server needs to create our "foo" volume on startup. It keeps the image
small, however it makes the server container start slow.
Add sleep before the server is usable. Without this PR, all pods that use Ceph
fail to start for couple of seconds with cryptic "image foo not found" error
and it clutters logs and pod logs and makes it harder to spot real errors.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
refactor pd.go for future tests
**What this PR does / why we need it**:
Refactored _test/e2e/storage/pd.go_ so that it will be easier to add new tests, which I plan on doing to address issue 52676
1. Condenses 8 `It` blocks into 3 table driven tests.
2. Adds several `By` descriptions and `Logf` messages.
3. provides more consistent formatting and messages.
**Special notes for your reviewer**:
The diff is large but mostly I've not altered any test. The one semantic change I made was to remove the call to verify a write to a PD when, in fact, nothing had been written yet. This was essentially a no-op since the verify code returned immediately if the passed-in map was empty (which it was since nothing had been written).
```release-note
NONE
```
cc @jingxu97 @copejon
Automatic merge from submit-queue (batch tested with PRs 53507, 53772, 52903, 53543). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Adding e2e tests to verify vsphere volume lifecycle on a clustered datastore
**What this PR does / why we need it**:
This PR introduces tests for volume provisioning on a clustered datastore. It does so in three ways
1. Static provisioning (create vsphere volume and then create a pod with it)
2. Dynamic provisioning (specify clustered datastore in storage class parameters)
3. Dynamic provisioning with spbm policy (specify storage policy name in storage class parameters. This policy is a tag based policy and tagged to a clustered datastore)
**Which issue this PR fixes** :
fixes vmware#278
**Special notes for your reviewer**:
Set env as per following example due to the need mentioned in description
```
export CLUSTER_DATASTORE="dscl1/sharedVmfs-1"
export VSPHERE_SPBM_POLICY_DS_CLUSTER="gold_cluster"
```
Internally reviewed by VMware reviewers @divyenpatel @BaluDontu @tusharnt
**Release note**:
```
None
```
Automatic merge from submit-queue (batch tested with PRs 53668, 53624, 52639, 53581, 51215). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Local e2e test fixes
**What this PR does / why we need it**:
1. Remove tests using TestContainerOutput because they don't wait for unmount
2. Fix scheduling error test to handle updated event msgs.
@kubernetes/sig-storage-pr-reviews
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#53597
**Release note**:
NONE
This test has been flaking. The current working theory is that
volume stats collection didn't run in time to grab the metrics
from the newly created pod.
Made the following changes:
- Added more logs to help debug future failures
- Poll metrics a few additional times before failing the test
Automatic merge from submit-queue (batch tested with PRs 52990, 53064, 52686, 52221, 53069). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Allow kubelet metrics tests to run on gke
**What this PR does / why we need it**:
On GKE, you can still access kubelet metrics, so allow the kubelet metrics test.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
NONE
Automatic merge from submit-queue (batch tested with PRs 52469, 52574, 52330, 52689, 52829). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Fixing E2E Test - After restarting kubelet test expects node's status to be NotReady
**What this PR does / why we need it**:
This PR is fixing the e2e tests involves restarting the kubelets. After the kubelet is restarted, test expect the desired state to be NotReady.
After restarting the kubelet we should wait for some time and then check nodes status to be Ready.
Node should not be checked for NotReady state, after restarting kubelet.
**Which issue this PR fixes**
fixes # https://github.com/vmware/kubernetes/issues/285
**Special notes for your reviewer**:
@BaluDontu @rohitjogvmw @tusharnt
Test logs before fix
-----
STEP: Restarting kubelet
Sep 15 11:26:32.768: INFO: Attempting sudo systemctl restart kubelet
Sep 15 11:26:33.001: INFO: ssh root@10.162.22.205:22: command: sudo systemctl restart kubelet
Sep 15 11:26:33.001: INFO: ssh root@10.162.22.205:22: stdout: ""
Sep 15 11:26:33.001: INFO: ssh root@10.162.22.205:22: stderr: ""
Sep 15 11:26:33.001: INFO: ssh root@10.162.22.205:22: exit code: 0
Sep 15 11:26:33.002: INFO: Waiting up to 1m0s for node kubernetes-node2 condition Ready to be false
Sep 15 11:26:33.012: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:26:35.023: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:26:37.032: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:26:39.041: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:26:41.051: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:26:43.061: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:26:45.070: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:26:47.080: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:26:49.093: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:26:51.105: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:26:53.117: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:26:55.128: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:26:57.140: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:26:59.151: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:27:01.158: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:27:03.167: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:27:05.180: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:27:07.188: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:27:09.210: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:27:11.221: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:27:13.231: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:27:15.240: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:27:17.249: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:27:19.263: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:27:21.272: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:27:23.283: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:27:25.309: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:27:27.317: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:27:29.327: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:27:31.342: INFO: Condition Ready of node kubernetes-node2 is true instead of false. Reason: KubeletReady, message: kubelet is posting ready status
Sep 15 11:27:33.343: INFO: Node kubernetes-node2 didn't reach desired Ready condition status (false) within 1m0s
Sep 15 11:27:33.343: INFO: Node kubernetes-node2 failed to enter NotReady state
[AfterEach] [sig-storage] PersistentVolumes:vsphere
Test logs after fix
-----
STEP: Restarting kubelet
Sep 18 15:40:49.066: INFO: Checking if sudo command is present
Sep 18 15:40:49.342: INFO: Checking if systemctl command is present
Sep 18 15:40:49.573: INFO: Attempting `sudo systemctl status kubelet | grep 'Main PID'`
Sep 18 15:40:49.733: INFO: ssh root@10.162.16.97:22: command: sudo systemctl status kubelet | grep 'Main PID'
Sep 18 15:40:49.733: INFO: ssh root@10.162.16.97:22: stdout: " Main PID: 19715 (docker)\n"
Sep 18 15:40:49.733: INFO: ssh root@10.162.16.97:22: stderr: ""
Sep 18 15:40:49.733: INFO: ssh root@10.162.16.97:22: exit code: 0
Sep 18 15:40:49.733: INFO: Attempting `sudo systemctl restart kubelet`
Sep 18 15:40:49.986: INFO: ssh root@10.162.16.97:22: command: sudo systemctl restart kubelet
Sep 18 15:40:49.986: INFO: ssh root@10.162.16.97:22: stdout: ""
Sep 18 15:40:49.986: INFO: ssh root@10.162.16.97:22: stderr: ""
Sep 18 15:40:49.986: INFO: ssh root@10.162.16.97:22: exit code: 0
Sep 18 15:40:49.988: INFO: Attempting `sudo systemctl status kubelet | grep 'Main PID'`
Sep 18 15:40:50.158: INFO: ssh root@10.162.16.97:22: command: sudo systemctl status kubelet | grep 'Main PID'
Sep 18 15:40:50.158: INFO: ssh root@10.162.16.97:22: stdout: " Main PID: 25021 (docker)\n"
Sep 18 15:40:50.158: INFO: ssh root@10.162.16.97:22: stderr: ""
Sep 18 15:40:50.158: INFO: ssh root@10.162.16.97:22: exit code: 0
Sep 18 15:40:50.158: INFO: Noticed that kubelet PID is changed. Waiting for 30 Seconds for Kubelet to come back
Sep 18 15:41:20.159: INFO: Waiting up to 1m0s for node kubernetes-node4 condition Ready to be true
STEP: Testing that written file is accessible.
Sep 18 15:41:20.191: INFO: Running '/Users/divyenp/github/vmware/kubernetes/_output/dockerized/bin/darwin/amd64/kubectl --server=https://10.162.0.45 --kubeconfig=/Users/divyenp/.kube/config exec --namespace=e2e-tests-pv-9j8j0 pvc-tester-3t9ds -- /bin/sh -c cat /mnt/_SUCCESS'
Sep 18 15:41:20.855: INFO: stderr: ""
Sep 18 15:41:20.855: INFO:
Sep 18 15:41:20.855: INFO: Volume mount detected on pod pvc-tester-3t9ds and written file /mnt/_SUCCESS is readable post-restart.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 48406, 52819). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..
Fixed nil dereference in dynamic provisioning e2e tests
**What this PR does / why we need it**: Fixed nil dereference in dynamic provisioning e2e tests.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#52815
**Release note**:
```release-note-none
NONE
```
/sig storage
/assign @saad-ali
/cc @wongma7
/release-note-none
Automatic merge from submit-queue (batch tested with PRs 51833, 51936)
Changed volume IO e2e test to verify file hash instead of content.
**What this PR does / why we need it**: The existing way of verifying file content takes too much memory, causing processes to be OOM killed.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes https://github.com/kubernetes/kubernetes/issues/51717
**Release note**:
```release-note
NONE
```
/sig storage
/release-note-none
/assign @jeffvance @rootfs
/cc @msau42
Automatic merge from submit-queue (batch tested with PRs 51805, 51725, 50925, 51474, 51638)
Flexvolume dynamic plugin discovery: Prober unit tests and basic e2e test.
**What this PR does / why we need it**: Tests for changes introduced in PR #50031 .
As part of the prober unit test, I mocked filesystem, filesystem watch, and Flexvolume plugin initialization.
Moved the filesystem event goroutine to watcher implementation.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#51147
**Special notes for your reviewer**:
First commit contains added functionality of the mock filesystem.
Second commit is the refactor for moving mock filesystem into a common util directory.
Third commit is the unit and e2e tests.
**Release note**:
```release-note
NONE
```
/release-note-none
/sig storage
/assign @saad-ali @liggitt
/cc @mtaufen @chakri-nelluri @wongma7
Automatic merge from submit-queue (batch tested with PRs 50670, 50332)
e2e test for local storage mount point
**What this PR does / why we need it**:
We discovered that kubernetes can treat local directories and actual mountpoints differently. For example, https://github.com/kubernetes/kubernetes/issues/48331. The current local storage e2e tests use directories.
This PR introduces a test that creates a tmpfs and mounts it, and runs one of the local storage e2e tests.
**Which issue this PR fixes**: fixes https://github.com/kubernetes/kubernetes/issues/49126
**Special notes for your reviewer**:
I cherrypicked PR https://github.com/kubernetes/kubernetes/pull/50177, since local storage e2e tests are broken in master on 2017-08-08 due to "no such host" error. This PR replaces NodeExec with SSH commands.
You can run the tests using the following commands:
```
$ NUM_NODES=1 KUBE_FEATURE_GATES="PersistentLocalVolumes=true" go run hack/e2e.go -- -v --up
$ go run hack/e2e.go -- -v --test --test_args="--ginkgo.focus=\[Feature:LocalPersistentVolumes\]"
```
Here are the summary of results from my test run:
```
Ran 9 of 651 Specs in 387.905 seconds
SUCCESS! -- 9 Passed | 0 Failed | 0 Pending | 642 Skipped PASS
Ginkgo ran 1 suite in 6m29.369318483s
Test Suite Passed
2017/08/08 11:54:01 util.go:133: Step './hack/ginkgo-e2e.sh --ginkgo.focus=\[Feature:LocalPersistentVolumes\]' finished in 6m32.077462612s
```
**Release note**:
`NONE`
LocalVolumeType tmpfs added
Added checks to ensure tha volume created during setup contains expected testFileContent
Refactored tests out to avoid code duplication
Two different tests are performed with tmpfs:
-serial write and read in two different pods
-write and read in two different pods mounted at the same time
Fixed local storage test failures by integrating https://github.com/kubernetes/kubernetes/pull/50177
Switched NodeExec to SSH
Automatic merge from submit-queue (batch tested with PRs 49642, 50335, 50390, 49283, 46582)
Add rbac.authorization.k8s.io/v1
xref https://github.com/kubernetes/features/issues/2
Promotes the rbac.authorization.k8s.io/v1beta1 API to v1 with no changes
```release-note
The `rbac.authorization.k8s.io/v1beta1` API has been promoted to `rbac.authorization.k8s.io/v1` with no changes.
The `rbac.authorization.k8s.io/v1alpha1` version is deprecated and will be removed in a future release.
```
Automatic merge from submit-queue
VSphere cloud provider code refactoring
The current PR tracks the vSphere Cloud Provider code refactoring which includes the following changes.
- VCLib Package - A framework used by vSphere cloud provider for managing the vSphere entities. VCLib package mainly does the following:
- Volume management on datastore (Create/Delete)
- Volume management on Virtual Machines (Attach/Detach)
- Storage Policy Management
- vSphere Cloud Provider changes to implement the cloud provider interfaces by calling into VCLib package.
- Modifications to e2e tests to accomodate the latest design changes.
@divyenpatel @rohitjogvmw @luomiao
```release-note
vSphere cloud provider: vSphere cloud provider code refactoring
```
Automatic merge from submit-queue
Fix local storage test failures
**What this PR does / why we need it**:
Fixed a few issues:
- CI environment on GCE cannot resolve node names, need to use IPs. Use a different SSH wrapper that will get the IPs from the node object.
- Use hostdir instead of containerdir now that commands are executed directly on the host, instead of through a container.
- Get the PVC object again after it is bound so that it has the PV name.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#50128
**Release note**:
NONE
/release-note-none
/sig storage
Automatic merge from submit-queue (batch tested with PRs 48532, 50054, 50082)
Remove [k8s.io] tag and redundant [sig-storage] tags from volume tests
**What this PR does / why we need it**:
Removes redundant tags from storage e2e test names
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#50178
**Release note**:
/release-note-none
Automatic merge from submit-queue
Update OWNERS to correct members' handles
**What this PR does / why we need it**:
Fix some typos of members' handles as per https://github.com/kubernetes/kubernetes/issues/50048#issuecomment-319831957.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
Associated with: #50048
**Special notes for your reviewer**:
/cc @madhusudancs @sebgoa @liggitt @saad-ali
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 50119, 48366, 47181, 41611, 49547)
Add basic install and mount flexvolumes e2e tests
fixes https://github.com/kubernetes/kubernetes/issues/47010
These two tests install a skeleton "dummy" flex driver, attachable and non-attachable respectively, then test that a pod can successfully use the flex driver. They are labeled disruptive because kubelet and controller-manager get restarted as part of the flex install. IMO it's important to keep this install procedure as part of the test to isolate any bugs with the startup plugin probe code.
There is a bit of an ugly dependency on cluster/gce/config-test.sh because --flex-volume-plugin-dir must be set to a dir that's readable from controller-manager container and writable by the flex e2e test. The default path is not writable on GCE masters with read-only root so I picked a location that looks okay.
In the "dummy" drivers I trick kubelet into thinking there is a mount point by doing "mount -t tmpfs none ${MNTPATH} >/dev/null 2>&1", hope that is okay.
I have only tested on GCE and theoretically they may work on AWS but I don't think there is a need to test on multiple cloudproviders.
-->
```release-note
NONE
```
Automatic merge from submit-queue
Add ubuntu to gluster and nfs tests
**What this PR does / why we need it**:
Enable gluster and nfs tests for ubuntu distro
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#50039
**Special notes for your reviewer**:
**Release note**:
/release-note-none
/sig storage
Automatic merge from submit-queue (batch tested with PRs 49898, 49897, 49919, 48860, 49491)
Add basic local volume provisioner e2e tests
**What this PR does / why we need it**:
Adds e2e tests to test local volume provisioner.
**Which issue this PR fixes**: fixes https://github.com/kubernetes/kubernetes/issues/48832
**Special notes for your reviewer**:
- bring up local volume provisioner using bootstrapper
- have provisioner create a volume by creating a directory under discovery path.
- check persistent volume is created
- make a claim on the PV, write some data then delete the claim. Verify volume is cleaned up.
**Release note**:
```release-note
```
@ianchakeres @msau42
Automatic merge from submit-queue (batch tested with PRs 49328, 49285, 49307, 49127, 49163)
Cleanup storage e2e test names
**What this PR does / why we need it**:
Some test names had redundant [sig-storage] tags. Also, some tests still had [Volume] tag. This PR removes those tags.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Release note**:
```release-note
NONE
```
/release-note-none
/sig storage
Automatic merge from submit-queue (batch tested with PRs 47509, 46821, 45319, 49121, 49125)
volume i/o tests for storage plugins
**What this PR does / why we need it**:
Addresses issues [25268](https://github.com/kubernetes/kubernetes/issues/25268) and [28367](https://github.com/kubernetes/kubernetes/issues/28367), though it may be weak re. the streaming i/o issue. @matchstick
**Special notes for your reviewer**:
This is a new file. Plugins other than NFS, GlusterFS, iSCSI, and Ceph-RBD code will need to be supported in a separate PR.
```release-note
NONE
```
Automatic merge from submit-queue
Bump external provisioner image to smaller version
The image is roughly half as big so this should improve speed/flakiness maybe
-->
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 45871, 46498, 46729, 46144, 46804)
PD e2e test: Ready node check now uses the most up-to-date node count.
Follow-up to PR #46746
<!-- Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access)
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`.
Automatic merge from submit-queue (batch tested with PRs 46239, 46627, 46346, 46388, 46524)
move labels to components which own the APIs
During the apimachinery split in 1.6, we accidentally moved several label APIs into apimachinery. They don't belong there, since the individual APIs are not general machinery concerns, but instead are the concern of particular components: most commonly the kubelet. This pull moves the labels into their owning components and out of API machinery.
@kubernetes/sig-api-machinery-misc @kubernetes/api-reviewers @kubernetes/api-approvers
@derekwaynecarr since most of these are related to the kubelet
Automatic merge from submit-queue (batch tested with PRs 46076, 43879, 44897, 46556, 46654)
Local storage plugin
**What this PR does / why we need it**:
Volume plugin implementation for local persistent volumes. Scheduler predicate will direct already-bound PVCs to the node that the local PV is at. PVC binding still happens independently.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*:
Part of #43640
**Release note**:
```
Alpha feature: Local volume plugin allows local directories to be created and consumed as a Persistent Volume. These volumes have node affinity and pods will only be scheduled to the node that the volume is at.
```
Automatic merge from submit-queue (batch tested with PRs 45623, 45241, 45460, 41162)
Revert #45101 Mark PersistentVolumes as [Feature:Volumes]
**What this PR does / why we need it**:
Reverts #45101
`Feature` tag should only be used when a test/suite has dependencies not met by core CI. That is not the case for NFS backed PV tests.
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 43884, 44712, 45124, 43883)
Increase pv test ns deletion timeout to the framework's default.
**What this PR does / why we need it**:
Namespace deletion occasionally exceeds the test's timeout of 3 minutes. The test framework uses a default timeout of 5 minutes, so just making it match here.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#44790
**Special notes for your reviewer**:
**Release note**:
NONE
Automatic merge from submit-queue (batch tested with PRs 44044, 44766, 44930, 45109, 43301)
e2e test: test azure disk volume
**What this PR does / why we need it**:
E2E test Azure disk volume
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
@kubernetes/sig-testing-pr-reviews
**Release note**:
```release-note
NONE
```
Just so that we know that we need a cloud provider that
supports volumes to run this test. This is similar to
the change in 63bc42c872.
Ran into this when i was trying to run e2e tests with
local-up-cluster locally and figured out this test will
not work since we don't support local storage persistent
volumes.
Automatic merge from submit-queue
De-Flake Volume E2E: force GCEPD detach to prevent timeout
**What this PR does / why we need it**:
Fix flake`[k8s.io] Volumes [Volume] [k8s.io] PD should be mountable [Flaky] 5m38s.
Flake occurs as a result of an automated detach taking longer than 5 minutes, which exceeds the timeout limit of the delete function.
This PR adds explicit detach and wait func calls before the deletion. By forcing the detach and giving GCE an appropriate timeout limit, this should squash the timeout flake. This also significantly shortens cleanup time.
This PR does not remove the [Flaky] tag. Once this PR is merged, I'll keep an eye on the test grid for ~1 week. If no flakes surface, I'll submit a PR to pull the tag off.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #43977
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue
adding test for volume fstype validation
**What this PR does / why we need it**:
This PR is adding a test for volume fstype validation. Test verifies fstype specified in storage-class is being honored after volume creation.
Steps:
1. Create StorageClass with fstype set to valid type (default case included).
2. Create PVC which uses the StorageClass created in step 1.
3. Wait for PV to be provisioned.
4. Wait for PVC's status to become Bound.
5. Create pod using PVC on specific node.
6. Wait for Disk to be attached to the node.
7. Execute command in the pod to get fstype.
8. Delete pod and Wait for Volume Disk to be detached from the Node.
9. Delete PVC, PV and Storage Class.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
None
```
cc: @jeffvance @tusharnt
Automatic merge from submit-queue (batch tested with PRs 40055, 42085, 44509, 44568, 43956)
improve error handling in e2e helpers
**What this PR does / why we need it**:
Changes most of the volume related helper funcs to return error rather than calling `Expect`. This is a better programming practice, is consistent with Go and Kubernetes, and allows helper funcs that create multiple resources to perform cleanup.
```release-note
NONE
```
Krunal Jain