* FindPodVolumes
* Prebound PVCs are treated like unbound immediate PVCs and will error
* Always check for fully bound PVCs and cache bindings for not fully
bound PVCs
* BindPodVolumes
* Retry API updates for not fully bound PVCs even if the assume cache
already marked it
* Wait for PVCs to be fully bound after making the API updates
* Error when detecting binding/provisioning failure conditions
If PV is bound by external PV binder (e.g. kube-scheduler), it's
possible on heavy load that corresponding PVC is not synced to
controller local cache yet.
Automatic merge from submit-queue (batch tested with PRs 65570, 65616). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Retry scheduling on StorageClass events
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#56163
**Special notes for your reviewer**:
I have taken over #60006.
It's hard to test in e2e, because we cannot know reschedule of pod is triggered by which event (periodically service/node events will move pods to active queue too). ~~I'll add integration tests for this functionality after [this PR](https://github.com/kubernetes/kubernetes/pull/65296) get merged.~~ (already added)
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add lichuqiang as reviewer of persistentvolume controller (for volume scheduling)
Now that I've been working on the storage topology-aware feature for quite a time. Really hope that I can help do some review.
```release-note
NONE
```
/assign @msau42
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Volume topology aware dynamic provisioning: work based on new API
**What this PR does / why we need it**:
The PR has been split to 3 parts:
Part1: https://github.com/kubernetes/kubernetes/pull/63232 for basic scheduler and PV controller plumbing
Part2: https://github.com/kubernetes/kubernetes/pull/63233 for API change
and the PR itself includes work based on the API change:
- Dynamic provisioning allowed topologies scheduler work
- Update provisioning interface to be aware of selected node and topology
**Which issue(s) this PR fixes**
Feature: https://github.com/kubernetes/features/issues/561
Design: https://github.com/kubernetes/community/issues/2168
**Special notes for your reviewer**:
/sig storage
/sig scheduling
/assign @msau42 @jsafrane @saad-ali @bsalamat
@kubernetes/sig-storage-pr-reviews
@kubernetes/sig-scheduling-pr-reviews
**Release note**:
```release-note
Volume topology aware dynamic provisioning
```
Automatic merge from submit-queue (batch tested with PRs 62694, 62569, 62646, 61633, 62433). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Report events to apiserver in local volume plugin.
**What this PR does / why we need it**:
See https://github.com/kubernetes/kubernetes/pull/62102#discussion_r179238429.
Report events to apiserver in local volume plugin.
- Add VolumeHost.GetEventRecorder() method
- Add related e2e tests
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#62248
**Special notes for your reviewer**:
Example output of `kubectl describe pods`:
```
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 7s default-scheduler Successfully assigned e2e-tests-persistent-local-volumes-test-x4h5x/security-context-670da435-4174-11e8-9098-000c29bb0377 to 127.0.0.1
Warning AlreadyMountedVolume 7s kubelet, 127.0.0.1 The requested fsGroup is 4321, but the volume local-pvfbb76 has GID 1234. The volume may not be shareable.
Normal SuccessfulMountVolume 7s kubelet, 127.0.0.1 MountVolume.SetUp succeeded for volume "default-token-996xr"
Normal SuccessfulMountVolume 7s kubelet, 127.0.0.1 MountVolume.SetUp succeeded for volume "local-pvfbb76"
Normal Pulled 6s kubelet, 127.0.0.1 Container image "k8s.gcr.io/busybox:1.24" already present on machine
Normal Created 6s kubelet, 127.0.0.1 Created container
Normal Started 6s kubelet, 127.0.0.1 Started container
```
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 62448, 59317, 59947, 62418, 62352). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add binding error message for volumeMode:Block unsupported case
**What this PR does / why we need it**:
This PR adds a binding error message for volumeMode:Block
unsupported case.
With this message, users can understand why PV is not bound to
PVC on dynamic provisioning.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes: #59942
**Special notes for your reviewer**:
/cc @mtanino
/sig storage
**Release note**:
```release-note
NONE
```
This commit adds a binding error message for volumeMode:Block
unsupported case.
With this message, users can understand why PV is not bound to
PVC on both dynamic provisioning and manually creating volume.
This patch works as follows:
- In syncVolume, before adding the claim to claimQueue, check if
there is a volumeMode mismatch, and if there is, record the
event for both pv and pvc and skip adding to the queue.
fixes: #59942
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add msau42 to approvers for volume scheduling
**What this PR does / why we need it**:
Add me as an approver for the volume scheduling code
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 61918, 62180, 62198). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Use provided node object in volume binding predicate
**What this PR does / why we need it**:
Autoscaler creates fake node objects, so we should use the provided node object instead of looking up the node from the informer.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#62178
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 61806, 61508, 62075, 62079, 62052). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
make use of auto generated LoadBalancerStatus.DeepCopy
Remove LoadBalancerStatusDeepCopy in helpers.go, and use the auto generated `LoadBalancerStatus.DeepCopy` in pkg/apis/core/zz_generated.deepcopy.go.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Move volume scheduling and local storage to beta
**What this PR does / why we need it**:
* Move the feature gates and APIs for volume scheduling and local storage to beta
* Update tests to use the beta fields
@kubernetes/sig-storage-pr-reviews
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#59390
**Special notes for your reviewer**:
**Release note**:
```release-note
ACTION REQUIRED: VolumeScheduling and LocalPersistentVolume features are beta and enabled by default. The PersistentVolume NodeAffinity alpha annotation is deprecated and will be removed in a future release.
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Do not recycle volumes that are used by pods
**What this PR does / why we need it**:
Recycler should wait until all pods that use a volume are finished.
Consider this scenario:
1. User creates a PVC that's bound to a NFS PV.
2. User creates a pod that uses the PVC
3. User deletes the PVC.
Now the PV gets `Released` (the PVC does not exists) and recycled, however the PV is still mounted to a running pod. PVC protection won't help us, because it puts finalizers on PVC that is under user's control and user can remove it.
This PR checks that there is no pod that uses a PV before it recycles it.
**Release note**:
```release-note
NONE
```
/sig storage
Automatic merge from submit-queue (batch tested with PRs 55606, 59185, 58763, 59072, 59251). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix a typo in pkg/controller/volume/persistentvolume/pv_controller.go
**What this PR does / why we need it**:
Fix typo (a -> an)
**Special notes for your reviewer**:
none
**Release note**:
```release-note
none
```
When a PVC explicitly requests specific PV and the PV does not match,
we should tell the user what exactly does not match.
From:
Volume's size is smaller than requested or volume's class does not match with claim
To:
Cannot bind to requested volume "<volume name>": %s
where %s is one of:
- requested PV is too small
- storageClasseNames do not match
- incompatible volumeMode
- error checking volumeMode: api defaulting for volumeMode failed (this should not ever happen)
Automatic merge from submit-queue (batch tested with PRs 56375, 56872, 57053, 57165, 57218). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Improved event generation in volume mount,attach, and extend operations
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#57217
refactored mount, attach, extend operation's so that all failures generate events and event generation is more consistent.
```release-note
Improved event generation in volume mount, attach, and extend operations
```
Automatic merge from submit-queue (batch tested with PRs 54604, 55781, 55806, 55935, 55991). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Use core client with explicit version
**What this PR does / why we need it**:
Core client without explicit version has been deprecated, change them to the one with explicit version.
**Which issue(s) this PR fixes**:
Fixes partially #55993
**Special notes for your reviewer**:
/cc @kubernetes/sig-api-machinery-pr-reviews
/cc @caesarxuchao @k82cn @sttts @kevin-wangzefeng
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 51321, 55969, 55039, 56183, 55976). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Topology aware volume scheduler and PV controller changes
**What this PR does / why we need it**:
Scheduler and PV controller changes to support volume topology aware scheduling, as specified in kubernetes/community#1168
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#54435
**Special notes for your reviewer**:
* I've split the PR into logical commits to make it easier to review
* The remaining TODOs I plan to address next release unless you think it needs to be done now
**Release note**:
```release-note
Adds alpha support for volume scheduling, which allows the scheduler to make PersistentVolume binding decisions while respecting the Pod's scheduling requirements. Dynamic provisioning is not supported with this feature yet.
Action required for existing users of the LocalPersistentVolumes alpha feature:
* The VolumeScheduling feature gate also has to be enabled on kube-scheduler and kube-controller-manager.
* The NoVolumeNodeConflict predicate has been removed. For non-default schedulers, update your scheduler policy.
* The CheckVolumeBinding predicate has to be enabled in non-default schedulers.
```
@kubernetes/sig-storage-pr-reviews @kubernetes/sig-scheduling-pr-reviews
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Block volumes Support: CRI, volumemanager and operationexecutor changes
**What this PR does / why we need it**:
This PR contains following items to enable block volumes support feature.
- container runtime interface change
- volumemanager changes
- operationexecuto changes
**Which issue this PR fixes**:
Based on this proposal (kubernetes/community#805) and this feature issue: kubernetes/features#351
**Special notes for your reviewer**:
There are another PRs related to this functionality.
(#50457) API Change
(#53385) VolumeMode PV-PVC Binding change
(#51494) Container runtime interface change, volumemanager changes, operationexecutor changes
(#55112) Block volume: Command line printer update
Plugins
(#51493) Block volumes Support: FC plugin update
(#54752) Block volumes Support: iSCSI plugin update
**Release note**:
```
Adds alpha support for block volume, which allows the users to attach raw block volume to their pod without filesystem on top of the volume.
```
/cc @msau42 @liggitt @jsafrane @saad-ali @erinboyd @screeley44
Automatic merge from submit-queue (batch tested with PRs 54987, 55221, 54099, 55144, 54215). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Should not sync the volume when claim.Spec.VolumeName is null
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
```
PV controller should send events to PVCs and not PVs when a PVC requests PV
that's either too small or has mismatching StorageClass.
Regular users can't see events on PVs so `kubectl describe pvc` is the only
way how to tell user that something is wrong.
Automatic merge from submit-queue
Add storageClass.mountOptions and use it in all applicable plugins
split off from https://github.com/kubernetes/kubernetes/pull/50919 and still dependent on it. cc @gnufied
issue: https://github.com/kubernetes/features/issues/168
```release-note
Add mount options field to StorageClass. The options listed there are automatically added to PVs provisioned using the class.
```
When API server crashes *after* saving a provisioned PV and before sending
200 OK, the controller tries to save the PV again. In this case, it gets
AlreadyExists error, which should be interpreted as success and not as error.
Especially, a volume that corresponds to the PV should not be deleted in the
underlying storage.
Automatic merge from submit-queue (batch tested with PRs 51441, 51356, 51460)
Don't update pvc.status.capacity if pvc is already Bound
As discussed here https://github.com/kubernetes/community/pull/657#discussion_r128008128, in order for `pvc.status.Capacity < pv.Spec.Capcity` to be the mechanism for volume filesystem* resize, the pv controller should stop updating pvc.status.Capacity every resync period.
/assign @jsafrane
/sig storage
```release-note
NONE
```
Automatic merge from submit-queue
Add volume operation metrics to operation executor and PV controller
This PR implements the proposal for high level volume metrics https://github.com/kubernetes/community/pull/809
**Special notes for your reviewer**:
~Differences from proposal:~ all resolved
~"verify_volume" is now "verify_volumes_are_attached" + "verify_volumes_are_attached_per_node" + "verify_controller_attached_volume." Which of them do we want?~
~There is no "mount_device" metric because the MountVolume operation combines MountDevice and mount (plugin.Setup). Do we want to extract the mount_device metric or is it okay to keep mountvolume as one? For attachable volumes, MountDevice is the actual mount and Setup is a bindmount + setvolumeownership. For unattachable, mountDevice does not occur and Setup is an actual mount + setvolumeownership.~
~PV controller metrics I did not implement following the proposal at all. I did not change goroutinemap nor scheduleOperation. Because provisionClaimOperation does not return an error, so it's impossible for the caller to know if there is actually a failure worth reporting. So I manually create a new metric inside the function according to some conditions.~
@gnufied
I have tested the operationexecutor metrics but not provision & delete. Sample:
**Release note**:
```release-note
Add error count and time-taken metrics for storage operations such as mount and attach, per-volume-plugin.
```
Automatic merge from submit-queue (batch tested with PRs 46458, 50934, 50766, 50970, 47698)
Prepare VolumeHost for running mount tools in containers
This is the first part of implementation of https://github.com/kubernetes/features/issues/278 - running mount utilities in containers.
It updates `VolumeHost` interface:
* `GetMounter()` now requires volume plugin name, as it is going to return different mounter to different volume plugings, because mount utilities for these plugins can be on different places.
* New `GetExec()` method that should volume plugins use to execute any utilities. This new `Exec` interface will execute them on proper place.
* `SafeFormatAndMount` is updated to the new `Exec` interface.
This is just a preparation, `GetExec` right now leads to simple `os.Exec` and mount utilities are executed on the same place as before. Also, the volume plugins will be updated in subsequent PRs (split into separate PRs, some plugins required lot of changes).
```release-note
NONE
```
@kubernetes/sig-storage-pr-reviews
@rootfs @gnufied
Yecheng Fu
