Automatic merge from submit-queue (batch tested with PRs 62441, 66702, 67254, 67421, 65309). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
document expectations for stopped/shutdown instances in cloudprovider.Instances interface
**What this PR does / why we need it**:
Add comments which outline expectations for how cloud providers should handle nodes that are stopped/shutdown/sleeping.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
It doesn't fix https://github.com/kubernetes/kubernetes/issues/46442 since some providers still delete node resources for stopped instances (though there are PRs already open to fix this) but we've (mostly?) agreed that not deleting node resources for stopped instances is the correct behavior, or at least the behavior that we want consistent across providers at the moment.
**Special notes for your reviewer**:
```release-note
NONE
```
/sig cloud-provider
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Update `pkg/cloudprovider/providers/azure/OWNERS`
* Remove Jaice
* Remove Cole
* Add Stephen as reviewer
Signed-off-by: Stephen Augustus <foo@agst.us>
**Release note**:
```release-note
NONE
```
/sig azure
Automatic merge from submit-queue (batch tested with PRs 67026, 62945, 66917). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Cloud Provider Zones doc fixups
**What this PR does / why we need it**:
A few godoc fixups for Cloud Provider Zones.
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 67061, 66589, 67121, 67149). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add DynamicProvisioningScheduling and VolumeScheduling support for Azure managed disks
**What this PR does / why we need it**:
Continue of [Azure Availability Zone feature](https://github.com/kubernetes/features/issues/586).
This PR adds `VolumeScheduling` and `DynamicProvisioningScheduling` support to Azure managed disks.
When feature gate `VolumeScheduling` disabled, no NodeAffinity set for PV:
```yaml
kubectl describe pv
Name: pvc-d30dad05-9ad8-11e8-94f2-000d3a07de8c
Labels: failure-domain.beta.kubernetes.io/region=southeastasia
failure-domain.beta.kubernetes.io/zone=southeastasia-2
Annotations: pv.kubernetes.io/bound-by-controller=yes
pv.kubernetes.io/provisioned-by=kubernetes.io/azure-disk
volumehelper.VolumeDynamicallyCreatedByKey=azure-disk-dynamic-provisioner
Finalizers: [kubernetes.io/pv-protection]
StorageClass: default
Status: Bound
Claim: default/pvc-azuredisk
Reclaim Policy: Delete
Access Modes: RWO
Capacity: 5Gi
Node Affinity:
Required Terms:
Term 0: failure-domain.beta.kubernetes.io/region in [southeastasia]
failure-domain.beta.kubernetes.io/zone in [southeastasia-2]
Message:
Source:
Type: AzureDisk (an Azure Data Disk mount on the host and bind mount to the pod)
DiskName: k8s-5b3d7b8f-dynamic-pvc-d30dad05-9ad8-11e8-94f2-000d3a07de8c
DiskURI: /subscriptions/<subscription-id>/resourceGroups/<rg-name>/providers/Microsoft.Compute/disks/k8s-5b3d7b8f-dynamic-pvc-d30dad05-9ad8-11e8-94f2-000d3a07de8c
Kind: Managed
FSType:
CachingMode: None
ReadOnly: false
Events: <none>
```
When feature gate `VolumeScheduling` enabled, NodeAffinity will be populated for PV:
```yaml
kubectl describe pv
Name: pvc-0284337b-9ada-11e8-a7f6-000d3a07de8c
Labels: failure-domain.beta.kubernetes.io/region=southeastasia
failure-domain.beta.kubernetes.io/zone=southeastasia-2
Annotations: pv.kubernetes.io/bound-by-controller=yes
pv.kubernetes.io/provisioned-by=kubernetes.io/azure-disk
volumehelper.VolumeDynamicallyCreatedByKey=azure-disk-dynamic-provisioner
Finalizers: [kubernetes.io/pv-protection]
StorageClass: default
Status: Bound
Claim: default/pvc-azuredisk
Reclaim Policy: Delete
Access Modes: RWO
Capacity: 5Gi
Node Affinity:
Required Terms:
Term 0: failure-domain.beta.kubernetes.io/region in [southeastasia]
failure-domain.beta.kubernetes.io/zone in [southeastasia-2]
Message:
Source:
Type: AzureDisk (an Azure Data Disk mount on the host and bind mount to the pod)
DiskName: k8s-5b3d7b8f-dynamic-pvc-0284337b-9ada-11e8-a7f6-000d3a07de8c
DiskURI: /subscriptions/<subscription-id>/resourceGroups/<rg-name>/providers/Microsoft.Compute/disks/k8s-5b3d7b8f-dynamic-pvc-0284337b-9ada-11e8-a7f6-000d3a07de8c
Kind: Managed
FSType:
CachingMode: None
ReadOnly: false
Events: <none>
```
When both `VolumeScheduling` and `DynamicProvisioningScheduling` are enabled, storage class also supports `allowedTopologies` and `volumeBindingMode: WaitForFirstConsumer` for volume topology aware dynamic provisioning:
```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
name: managed-disk-dynamic
parameters:
cachingmode: None
kind: Managed
storageaccounttype: Standard_LRS
provisioner: kubernetes.io/azure-disk
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
- key: failure-domain.beta.kubernetes.io/zone
values:
- southeastasia-2
- southeastasia-1
```
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
DynamicProvisioningScheduling and VolumeScheduling is not supported for Azure managed disks. Feature gates DynamicProvisioningScheduling and VolumeScheduling should be enabled before using this feature.
```
/kind feature
/sig azure
/cc @brendandburns @khenidak @andyzhangx
/cc @ddebroy @msau42 @justaugustus
Automatic merge from submit-queue (batch tested with PRs 67061, 66589, 67121, 67149). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Get load balancer name per provider
**What this PR does / why we need it**:
GetLoadBalancerName() should be implemented per cloud provider as opposed to one neutral implementation.
This PR will address this by moving `cloudprovider.GetLoadBalancerName()` to the `LoadBalancer interface` and then provide an implementation for each cloud provider, while maintaining previously expected functionality.
**Which issue(s) this PR fixes**:
Fixes [#43173](https://github.com/kubernetes/kubernetes/issues/43173)
**Special notes for your reviewer**:
This is a work in progress. Looking for feedback as I work on this, from any interested parties.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
scope AWS LoadBalancer security group ICMP rules to spec.loadBalancerSourceRanges
/sig aws
**What this PR does / why we need it**:
Make the client CIDR ranges for MTU consistent with what [the documentation appears to describe](https://kubernetes.io/docs/concepts/services-networking/service/#type-loadbalancer), where the ranges should be equal to `spec.loadBalancerSourceRanges` if supplied.
**Which issue(s) this PR fixes**:
Fixes#63564
**Release note**:
```release-note
scope AWS LoadBalancer security group ICMP rules to spec.loadBalancerSourceRanges
```
Automatic merge from submit-queue (batch tested with PRs 67052, 67094, 66795). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add zones support for vSphere cloud provider(in-tree)
**What this PR does / why we need it**:
This PR added zones(built-in node labels) support for vSphere cloud provider(in-tree). More details can be found in the issue as below.
**Which issue(s) this PR fixes** :
Partially fixes phase 1 of issue #64021
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
use func WaitForCompletionRef replace of deprecated func WaitForCompletion
**What this PR does / why we need it**:
use func WaitForCompletionRef replace of deprecated func WaitForCompletion
```
// WaitForCompletion will return when one of the following conditions is met: the long
// running operation has completed, the provided context is cancelled, or the client's
// polling duration has been exceeded. It will retry failed polling attempts based on
// the retry value defined in the client up to the maximum retry attempts.
// Deprecated: Please use WaitForCompletionRef() instead.
func (f Future) WaitForCompletion(ctx context.Context, client autorest.Client) error {
return f.WaitForCompletionRef(ctx, client)
}
```
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add availability zones support to Azure managed disks
**What this PR does / why we need it**:
Continue of [Azure Availability Zone feature](https://github.com/kubernetes/features/issues/586).
This PR adds availability zone support for Azure managed disks and its storage class. Zoned managed disks is enabled by default if there are zoned nodes in the cluster.
The zone could also be customized by `zone` or `zones` parameter, e.g.
```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
name: managed-disk-zone-1
parameters:
zone: "southeastasia-1"
# zones: "southeastasia-1,"southeastasia-2"
cachingmode: None
kind: Managed
storageaccounttype: Standard_LRS
provisioner: kubernetes.io/azure-disk
reclaimPolicy: Delete
volumeBindingMode: Immediate
```
All zoned AzureDisk PV will also be labeled with its availability zone, e.g.
```sh
$ kubectl get pvc pvc-azuredisk-az-1
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc-azuredisk-az-1 Bound pvc-5ad0c7b8-8f0b-11e8-94f2-000d3a07de8c 5Gi RWO managed-disk-zone-1 2h
$ kubectl get pv pvc-5ad0c7b8-8f0b-11e8-94f2-000d3a07de8c --show-labels
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE LABELS
pvc-5ad0c7b8-8f0b-11e8-94f2-000d3a07de8c 5Gi RWO Delete Bound default/pvc-azuredisk-az-1 managed-disk-zone-1 2h failure-domain.beta.kubernetes.io/region=southeastasia,failure-domain.beta.kubernetes.io/zone=southeastasia-1
```
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
See also the [KEP](https://github.com/kubernetes/community/pull/2364).
DynamicProvisioningScheduling feature would be added in a following PR.
**Release note**:
```release-note
Azure managed disks now support availability zones and new parameters `zoned`, `zone` and `zones` are added for AzureDisk storage class.
```
/kind feature
/sig azure
/assign @brendandburns @khenidak @andyzhangx
Zones implementation for vSphere cloud provider needs dependencies
which are not included in current vmware/govmomi vendor. So this
update added "vapi" package to support zones.
Automatic merge from submit-queue (batch tested with PRs 65730, 66615, 66684, 66519, 66510). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add DynamicProvisioningScheduling support for EBS
**What this PR does / why we need it**:
This PR adds support for the DynamicProvisioningScheduling feature in EBS. With this in place, if VolumeBindingMode: WaitForFirstConsumer is specified in a EBS storageclass and DynamicProvisioningScheduling is enabled, EBS provisioner will use the selected node's LabelZoneFailureDomain as the zone to provision the EBS volume in.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
Related to #63232
Sample `describe pv` output with NodeAffinity populated:
```
~$ kubectl describe pv pvc-f9d2138b-7e3e-11e8-a4ea-064124617820
Name: pvc-f9d2138b-7e3e-11e8-a4ea-064124617820
Labels: failure-domain.beta.kubernetes.io/region=us-west-2
failure-domain.beta.kubernetes.io/zone=us-west-2a
Annotations: kubernetes.io/createdby=aws-ebs-dynamic-provisioner
pv.kubernetes.io/bound-by-controller=yes
pv.kubernetes.io/provisioned-by=kubernetes.io/aws-ebs
Finalizers: [kubernetes.io/pv-protection]
StorageClass: slow3
Status: Bound
Claim: default/pvc3
Reclaim Policy: Delete
Access Modes: RWO
Capacity: 6Gi
Node Affinity:
Required Terms:
Term 0: failure-domain.beta.kubernetes.io/zone in [us-west-2a]
failure-domain.beta.kubernetes.io/region in [us-west-2]
Message:
Source:
Type: AWSElasticBlockStore (a Persistent Disk resource in AWS)
VolumeID: aws://us-west-2a/vol-0fc1cdae7d10860f6
FSType: ext4
Partition: 0
ReadOnly: false
Events: <none>
```
**Release note**:
```release-note
none
```
/sig storage
/assign @msau42 @jsafrane
Automatic merge from submit-queue (batch tested with PRs 66489, 66728, 66739). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Speed up volume modifications on AWS
Volume becomes reusable after it reached optimizing state.
/sig storage
/sig aws
```release-note
Make EBS volume expansion faster
```
cc @d-nishi @kokhang
Automatic merge from submit-queue (batch tested with PRs 66225, 66648, 65799, 66630, 66619). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Implement InstanceShutdownByProviderID for vSphere CP
**What this PR does / why we need it**:
To detach volumes from shutdown nodes #66181
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 66225, 66648, 65799, 66630, 66619). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Update Azure Go SDK to v19.0.0 and get availability zone for VirtualMachineScaleSetVM
**What this PR does / why we need it**:
Continue of #66242. This PR updates Azure Go SDK to v19.0.0 (with compute API 2018-04-01) and gets availability zones for VirtualMachineScaleSetVM.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
Azure Go SDK has been upgraded to v19.0.0 and VirtualMachineScaleSetVM now supports availability zones.
```
/sig azure
/assign @brendandburns @khenidak @andyzhangx
Automatic merge from submit-queue (batch tested with PRs 66464, 66488). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Avoid overflowing int64 in RoundUpSize and return error if overflow int
**What this PR does / why we need it**:
There are many places in plugins (some I may have missed) that we naively convert a resource.Quantity.Value() which is an int64, to an int, which may be only 32 bits long.
Background, optional to read :): Kubernetes canonicalizes resource.Quantities, and from what I have seen testing creating PVCs, decimalSI is the default. If a quantity is in `decimalSI` format and its value in bytes would overflow an int64, e.g. `10E`, nothing happens. If it is in binarySI and its value in bytes would overflow an int64, e.g. `10Ei`, it is set down to 2^63-1 and there's no overflow of the field value. But there may be overflow later in the code which is what this PR is addressing.
* Change `RoundUpSize` implementation to avoid overflowing `int64`
* Add `RoundUp*Int` functions for use when an `int` is expected instead of an `int64`, because `int` may be 32bits and naively doing `int($INT64_VALUE)` can lead to silent overflow. These functions return an error if overflow has occurred.
* Rename `*GB` variables to `*GiB` where appropriate for maximum clarity
* Use `RoundUpToGiB` instead of `RoundUpSize` where possible
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**: please review carefully as we don't have e2e tests for most plugins!
**Release note**:
```release-note
NONE
```
edit: remove 'we do not need to worry about...'. yes we do, i worded that badly :))
Automatic merge from submit-queue (batch tested with PRs 62423, 66180, 66492, 66506, 65242). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add user assigned MSI support
**What this PR does / why we need it**:
Adds the support for generating tokens via user assigned MSI.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
Add support for using User Assigned MSI (https://docs.microsoft.com/en-us/azure/active-directory/managed-service-identity/overview) with Kubernetes cluster on Azure.
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add myself, Micah to reviewers
**Release note**:
```release-note
NONE
```
Signed-off-by: Nick Turner <nic@amazon.com>
Automatic merge from submit-queue (batch tested with PRs 66098, 66389, 66400, 66413, 66378). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
GCE: Return correct error type and HTTP Status code for operation errors
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#66399
**Special notes for your reviewer**:
/assign bowei, zihongz, rramkumar
/cc bowei
**Release note**:
```release-note
GCE: Fixes loadbalancer creation and deletion issues appearing in 1.10.5.
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add initial availability zones support for Azure nodes
**What this PR does / why we need it**:
The first part of [Azure Availability Zone feature](https://github.com/kubernetes/features/issues/586).
This PR adds initial availability zone (AZ) support for Azure nodes. With this PR, Azure nodes with AZ will have label `failure-domain.beta.kubernetes.io/zone=<region>-<zoneID>`, e.g. `southeastasia-1`.
It also updates instance metadata api-version to 2017-12-01, which is required for AZ.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
VirtualMachineScaleSetVM doesn't have AZ info yet. It will be supported later after new Azure Go SDK releases.
**Release note**:
```release-note
Azure nodes with availability zone now will have label `failure-domain.beta.kubernetes.io/zone=<region>-<zoneID>`.
```
/kind feature
/sig azure
/assign @brendandburns @khenidak @andyzhangx
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add error check and ignore unused variable (SA4006)
**What this PR does / why we need it**:
Fix some bugs in cloud provider vsphere, issue can be found here #66303
```pkg/cloudprovider/providers/vsphere/nodemanager.go:176:5: defers in this range loop won't run unless the channel gets closed (SA9001)
pkg/cloudprovider/providers/vsphere/vclib/diskmanagers/vmdm.go:129:8: this value of err is never used (SA4006)
pkg/cloudprovider/providers/vsphere/vsphere.go:596:34: argument ctx is overwritten before first use (SA4009)
pkg/cloudprovider/providers/vsphere/vsphere_test.go:360:2: this value of instanceID is never used (SA4006)
pkg/cloudprovider/providers/vsphere/vsphere_util.go:301:3: defers in this infinite loop will never run (SA5003)
```
**Special notes for your reviewer**:
I fixed ```SA4006``` report in that issue, but there are still other code needed to discuss and fix.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix locating resourcepool-path specified in the vsphere.conf file
**What this PR does / why we need it**:
When volume is provisioned using the vsphere storage policy, `resourcepool-path` specified in the `vsphere.conf` file is used for creating a shadow/dummy VM. Dummy VM is temporarily created and then deleted once volume is created on the compatible Datastore.
At present If user specifies `resourcepool-path` in the `vsphere.conf` file, volume provisioner is not able to locate the compute resource for the given path. This is because look up is made using `finder.DefaultComputeResource(ctx)` and `finder.ComputeResource(ctx, computePath)`, which is not correct. If user specifies name of the cluster or cluster path then provisioning works.
This is resolved with using correct govmomi method - `func (f *Finder) ResourcePoolOrDefault(ctx context.Context, path string) (*object.ResourcePool, error)`
**Which issue(s) this PR fixes**
Fixes # https://github.com/vmware/kubernetes/issues/493
**Special notes for your reviewer**:
Following testing is performed for this change.
1) specified resource-pool path in the `vsphere.conf` file and verified VM is created under the specified resource pool.
```
resourcepool-path="ClusterFolder-1/cluster-vsan-1/Resources/ShadowVMPool"
```
2) If resource pool is not available, specified cluster's default resource pool path in the `vsphere.conf` file and verified volume provisioning works. For this case, VM is directly created under cluster.
```
resourcepool-path="ClusterFolder-1/cluster-vsan-1/Resources"
```
3) Verified above with having multiple clusters with the same name in one datacenter.
4) Verified with empty resource pool path in the vsphere.conf file.
```
resourcepool-path=""
```
As expected, provisioning is failing with `Failed to provision volume with StorageClass "vsan-gold-policy": no default resource pool found`.
Refer to this datacenter inventory for the path specified in the `resourcepool-path` configuration.
Current documentation describes `resourcepool-path`configuration is optional, which needs to be corrected once PR is merged. For policy based provisioning this is not an optional parameter.
Documentation link: https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/existing.html
**Release note**:
```release-note
Fix for resourcepool-path configuration in the vsphere.conf file.
```
cc: @kubernetes/vmware
Automatic merge from submit-queue (batch tested with PRs 66121, 66140, 66045). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Support setting azure LB idle timeout
**What this PR does / why we need it**:
Adds a new annotation to allow users to configure the idle timeout of
the Azure LB.
**Release note**:
```release-note
Support configuring the Azure load balancer idle connection timeout for services
```
Automatic merge from submit-queue (batch tested with PRs 66122, 66007). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Return vmUUID when renewing nodeinfo in VCP
**What this PR does / why we need it**:
This PR fixes an issue that VM UUID is removed when renewing node information in vSphere cloud provider
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
/sig vmware
Automatic merge from submit-queue (batch tested with PRs 66076, 65792, 65649). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
kubernetes: fix printf format errors
These are all flagged by Go 1.11's
more accurate printf checking in go vet,
which runs as part of go test.
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add region label to dynamic provisioned cinder PVs
**What this PR does / why we need it**:
This PR adds region label to dynamic provisioned Cinder PVs at the time of the PV creation.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#65977
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
These are all flagged by Go 1.11's
more accurate printf checking in go vet,
which runs as part of go test.
Lubomir I. Ivanov <neolit123@gmail.com>
applied ammend for:
pkg/cloudprovider/provivers/vsphere/nodemanager.go
Automatic merge from submit-queue (batch tested with PRs 65902, 65781). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
if loadbalancer section is not defined in cloudconfig, do not init support
**What this PR does / why we need it**: if LoadBalancer section is not defined in cloudconfig, we should not initialize loadbalancer support for openstack cloudprovider.
**Which issue(s) this PR fixes**:
Fixes#65775
**Special notes for your reviewer**:
**Release note**:
```release-note
If LoadBalancer is not defined in cloud config, the loadbalancer is not initialized anymore in openstack. All setups must have some setting under that section
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Store the latest cloud provider node addresses
**What this PR does / why we need it**:
Buffer the recently retrieved node address so they can be used as soon as the next node status update is run.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#65814
**Special notes for your reviewer**:
**Release note**:
```release-note
None
```
Setting the client UserAgent makes it easier to identify vCenter sessions
used by the vSphere Cloud Provider. This is useful to remove sessions that
have leaked, such as when a VCP process goes away without calling Logout().
And to test that VCP properly re-authenticates when a session is removed.
Example use:
govc session.ls | grep kubernetes-cloudprovider | awk '{print $1}' | xargs -n1 govc session.rm
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
skip nic that are in failing state
**What this PR does / why we need it**: this fixes partially #65025. Currently when getting primary NIC for VMSS the provisioning state isn't returned.
**Which issue(s) this PR fixes** : Fixes partially (for VMAS) #65025
**Special notes for your reviewer**:
/assign @feiskyer
**Release note**:
```release-note
skip nodes that have a primary NIC in a 'Failed' provisioningState
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Setup TLS with CA Cert for vsphere cloud provider
- Extend config to take a path to a CA Certificate
- Use the CA Cert when establishing a connection with the SOAP client
Testing
We provide certs and keys for tests as fixtures, `vclib/fixtures`.
Those were created (and can be regenerated) using `vclib/fixtures/createCerts.sh`.
At the moment it's possible to configure a CA path and at the same time allow insecure
communication between vsphere cloud provider and vcenter. This may
change in the future; we might opt for overwriting the insecure
communication if a CA is configured / log and transparently pass the
arguments to the vcenter command / other. To be discussed.
At the moment the CA is a global level configuration. In other
words, all vcenter servers need to use certificates signed by the same
CA. There might be use cases for different CA per vcenter server; to be
discussed.
**What this PR does / why we need it**:
This PR adds the option of configuring a trusted CA for the communication between the vsphere cloud provider and the vcenter control plane.
**Which issue(s) this PR fixes**:
Fixes#64222
**Special notes for your reviewer**:
**Release note**:
```release-note
- Can configure the vsphere cloud provider with a trusted Root-CA
```
Automatic merge from submit-queue (batch tested with PRs 64575, 65120, 65463, 65434, 65522). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Revendor GCE Go Client
Revendor GCE API go client and switch to use beta neg api in gce cloud provider.
```release-note
None
```
Automatic merge from submit-queue (batch tested with PRs 65492, 65516, 65447). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
fix azure disk creation issue when specifying external resource group
**What this PR does / why we need it**:
fix azure disk creation issue when specifying external resource group, after azure disk creation succeeded, it fails to get azure disk state since it's still using original resource group
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#65515
**Special notes for your reviewer**:
Together with https://github.com/kubernetes/kubernetes/pull/65443, this feature has been done, I will cherry-pick to prior versions later.
So in the end, we have two ways to make azure disk dynamic provision under an external resource group
- specify `resourcegroup` parameter in azure disk storage class
```
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: hdd
provisioner: kubernetes.io/azure-disk
parameters:
skuname: Standard_LRS
kind: managed
cachingmode: None
resourcegroup: USER-SPECIFIED-RG
```
- specify `volume.beta.kubernetes.io/resource-group` in PVC annotations
```
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvc-azuredisk
annotations:
volume.beta.kubernetes.io/resource-group: "USER-SPECIFIED-RG"
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: hdd
```
**Release note**:
```
fix azure disk issue when specifying external resource group
```
/kind bug
/sig azure
@jsafrane @rootfs
Just FYI @khenidak @brendandburns @feiskyer
Automatic merge from submit-queue (batch tested with PRs 65507, 65508, 65486). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix typo in vsphere cloud provider comment
**What this PR does / why we need it**:
Fix typo in code of vsphere cloud provider
As far as I know, it's not purchased by Google right..?
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 65449, 65373, 49410). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
OpenStack LBaaS fix: must use ID, not name, of the node security group
This is a bugfix for the OpenStack LBaaS cloud provider security group management.
A bit of context: When creating a load balancer for a given `type: LoadBalancer` service, the provider will try to:
(see `pkg/cloudprovider/providers/openstack/openstack_loadbalancer.go`/`EnsureLoadBalancer`)
1. create a load balancer (LB) in Openstack with listeners corresponding to the service's ports
2. attach a floating IP to the LB's network port
If `manage-security-groups` is enabled in controller-manager's cloud.conf:
3. create a security group with ingress rules corresponding to the LB's listeners, and attach it to the LB's network port
4. for all nodes of the cluster, pick an existing security group for the nodes ("node security group") and add ingress rules to it exposing the service's NodePorts to the security group created in step 3.
In the current upstream master, steps 1 through 3 work fine, step 4 fails, leading to a service that's not accessible via the LB without further manual intervention.
The bug is in the "pick an existing security group" operation (func `getNodeSecurityGroupIDForLB`), which, contrary to its name, will return the security group's name rather than its ID (actually it returns a list of names rather than IDs, apparently to cover some corner cases where you might have more than one node security group, but anyway). This will then be used when trying to add the ingress rules to the group, which the Openstack API will reject with a 404 (at least on our (fairly standard) Openstack Ocata installation) because we're giving it a name where it expects an ID.
The PR adds a "get ID given a name" lookup to the `getNodeSecurityGroupIDForLB` function, so it actually returns IDs. That's it. I'm not sure if the upstream code wasn't really tested, or maybe other people use other Openstacks with more lenient APIs. The bug and the fix is always reproducible on our installation.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#58145
**Special notes for your reviewer:**
Should we turn `getNodeSecurityGroupIDForLB` into a method with the lbaas as its receiver because it now requires two of the lbaas's attributes? I'm not sure what the conventions are here, if any.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 65156, 65448). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
[GCE] move NEG out of featuregate
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes # https://github.com/kubernetes/ingress-gce/issues/274
**Release note**:
-->
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 65164, 65258). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Query candidate zones for EBS when zone/zones not passed
**What this PR does / why we need it**:
This PR skips invoking `getCandidateZonesForDynamicVolume` to query EC2 zones of instances when zone/zones is present.
/sig storage
**Release note**:
```
none
```
Automatic merge from submit-queue (batch tested with PRs 61330, 64793, 64675, 65059, 65368). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
[gce provider ] Wrapper for beta backend service create/update
**What this PR does / why we need it**:
Feel free to wait after 1.11 code freeze.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #NONE
**Special notes for your reviewer**:
/assign @rramkumar1
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 65024, 65287, 65345, 64693, 64941). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Change signature of SetUrlMapForTargetHTTPProxy
**Special notes for your reviewer**:
/assign MrHohn
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Change Azure ARM Rate limiting error message
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#65284
Change Azure ARM Rate limiting error message
**Special notes for your reviewer**:
@khenidak
**Release note**:
```release-note
Change Azure ARM Rate limiting error message.
```
Automatic merge from submit-queue (batch tested with PRs 65187, 65206, 65223, 64752, 65238). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fixed detection of inaccessible AWS encryption key.
AWS provisioner now checks if created encrypted volume gets "Available" or it gets silently deleted by AWS because StorageClass referenced invalid (e.g. non-existing) KMS key for encryption.
This seems to be the only way how to detect such invalid key, because Kubernetes may not have enough permission to check if the key exists.
**Which issue(s) this PR fixes**
Fixes#62171
**Special notes for your reviewer**:
**Release note**:
```release-note
AWS now checks for validity of ecryption key when creating encrypted volumes. Dynamic provisioning of encrypted volume may get slower due to these checks.
```
/sig aws
/sig storage
@kubernetes/sig-aws-misc
Automatic merge from submit-queue (batch tested with PRs 65152, 65199, 65179, 64598, 65216). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Do not query for VMUUID in vsphere cloud provider if it was explicitly passed
**What this PR does / why we need it**:
It allows running the kube controller-manager with the vsphere cloudprovider enabled on a machine that is not a vmware VM
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#65145
**Special notes for your reviewer**:
**Release note**:
```release-note
In the vSphere cloud provider the `Global.vm-uuid` configuration option is not deprecated anymore, it can be used to overwrite the VMUUID on the controller-manager
```
/sig vmware
Automatic merge from submit-queue (batch tested with PRs 64140, 64898, 65022, 65037, 65027). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Handle empty cluster UID in instance group name
**What this PR does / why we need it**:
This PR handles an empty cluster UID when generating InstanceGroupName. The current implementation will create a name ending in "--" which is invalid.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 64895, 64938, 63700, 65050, 64957). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
set EnableHTTPSTrafficOnly in azure storage account creation
**What this PR does / why we need it**:
Enforce azure storage account creation with https traffic only, this PR will apply for both azure disk & azure file features.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#64956
**Special notes for your reviewer**:
Tests with azure disk & azure file all pass
**Release note**:
```
Enforce azure storage account creation with https traffic only
```
/sig azure
/kind feature
/assign @khenidak
Automatic merge from submit-queue (batch tested with PRs 65032, 63471, 64104, 64672, 64427). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
add external resource group support for azure disk
**What this PR does / why we need it**:
add external resource group support for azure disk,
- without this PR, user could only create dynamic azure disk in the same resource group as cluster
- with this PR, user could specify external resource group in PVC:
```
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvc-azuredisk
annotations:
volume.beta.kubernetes.io/resource-group: "USER-SPECIFIED-RG"
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: hdd
```
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#64388
**Special notes for your reviewer**:
Pls note above config won't change resource group for azure disk forever, next time if user don't specify resource group, only default resource group will be used.
**Release note**:
```
add external resource group support for azure disk
```
/sig azure
/assign @feiskyer @karataliu
/cc @khenidak
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
add @andrewsykim as pkg/cloudprovider approver
**What this PR does / why we need it**:
adds myself as an approver for pkg/cloudprovider
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Move out azure_loadbalancer.md to cloud provider repository
**What this PR does / why we need it**:
Move out doc to cloud provider repository.
The file 'azure_loadbalancer.md' has been moved to cloud provider repository in https://github.com/kubernetes/cloud-provider-azure/pull/23 , this PR is to remove it from upstream.
**Which issue(s) this PR fixes**
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
cc @feiskyer
- Extend config to take a path to a CA Certificate
- Use the CA Cert when establishing a connection with the SOAP client
Testing
We provide certs and keys for tests as fixtures, `vclib/fixtures`.
Those were created (and can be regenerated) using `vclib/fixtures/createCerts.sh`.
At the moment it's possible to configure a CA path and at the same time allow insecure
communication between vsphere cloud provider and vcenter. This may
change in the future; we might opt for overwriting the insecure
communication if a CA is configured / log and transparently pass the
arguments to the vcenter command / other. To be discussed.
At the moment the CA is a global level configuration. In other
words, all vcenter servers need to use certificates signed by the same
CA. There might be use cases for different CA per vcenter server; to be
discussed.
Automatic merge from submit-queue (batch tested with PRs 63386, 64624, 62297, 64847). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Refactoring GCE Disk APIs to use generated client
**What this PR does / why we need it**: Improves maintainability and testing of GCE disks code.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#62296
Note that `gen.go` and `gen_test.go` are generated files.
I'm planning to do a more extensive refactor that takes advantage of the generated cloud provider mocks, but that'll be in a separate PR and will be a larger change.
/cc @davidz627
/assign @saad-ali
/release-note-none
/sig storage
Automatic merge from submit-queue (batch tested with PRs 64009, 64780, 64354, 64727, 63650). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
GCE: Update cloud provider to use TPU v1 API
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
GCE: Update cloud provider to use TPU v1 API
```
/assign @vishh
/assign @cheftako
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
GCE PD plugin now prevents attaching a regional PD PV with pdName of …
…a regular PD
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#63317
/sig storage
/sig gcp
/assign @msau42 @saad-ali
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix panic while provisioning Azure security group rules
**What this PR does / why we need it**:
kube-controller-manager panic when provisioning Azure security group rules, especially when securityGroupName is wrong configured. This PR fixes the issue.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#64716
**Special notes for your reviewer**:
Should also cherry pick to release-1.10.
**Release note**:
```release-note
Fix kube-controller-manager panic while provisioning Azure security group rules
```
Automatic merge from submit-queue (batch tested with PRs 64364, 64369, 63819, 64528). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
[gce provider] More wrappers for alpha/beta backend service
**What this PR does / why we need it**:
Sorry, some more wrapper functions I missed last time.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #NONE
**Special notes for your reviewer**:
/assign @rramkumar1
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
[GCE] use fakeGCECloud instead of gce address fakes
**What this PR does / why we need it**:
Use the fakeGCECloud mock instead of FakeCloudAddressService.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 57082, 64325, 64016, 64443, 64403). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Allow AWS EBS volumes to be attached as ReadOnly.
**Which issue(s) this PR fixes**
Fixes#64402
**Special notes for your reviewer**:
This follows logic e.g. in Cinder volume plugin.
**Release note**:
```release-note
AWS EBS volumes can be now used as ReadOnly in pods.
```
/sig storage
/sig aws
Automatic merge from submit-queue (batch tested with PRs 64318, 64269, 64438, 64516, 64311). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Ensure that only IPs are used as node addresses in OpenStack LBs
**What this PR does / why we need it**:
ATM, when no InternalIP can be found for a node, the openstack cloud provider tries to create a LB with whatever is the first address it can find for the node. This could also be the hostname or a dns name.
However, LBaaS will reject anything that is not an IP address for pool members. Therefore a meaningful error should be returned instead of just returning the first address of the node, even if it is clear that this will lead to an error in LBaaS.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#64317
**Special notes for your reviewer**:
**Release note**:
```release-note
Provide a meaningful error message in openstack cloud provider when no valid IP address can be found for a node
```
Automatic merge from submit-queue (batch tested with PRs 64288, 64343). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
[gce provider] Add more wrappers for security policy
**What this PR does / why we need it**:
Adding more wrappers for security policy, mostly for implementing the e2e test.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #NONE
**Special notes for your reviewer**:
/assign @nicksardo
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Update function hasClusterTag to fix issue #64230
**What this PR does / why we need it**:
Fixes issue #64230, by changing function hasClusterTag, in aws/tags.go, to ensure that, when called with a list of tags containing a tag with a key which matches clusterTagKey, function will return true even if a tag with key TagNameKubernetesClusterLegacy also exists in the list with a value other than the ClusterID.
**Which issue(s) this PR fixes**:
Fixes#64230
**Special notes for your reviewer**:
Notes are in issue
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 63319, 64248, 64250, 63890, 64233). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add reason message logs for non-exist Azure resources
**What this PR does / why we need it**:
In azure_wrap.go the function checkResourceExistsFromError() looks at a received error and determines it to be a ResourceNotFound if the HTTP status is http.StatusNotFound. However, the HTTP status also equals http.StatusNotFound in case of SubscriptionNotFound.
this PR adds logs to indict the real error messages for such case.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#64220
**Special notes for your reviewer**:
**Release note**:
```release-note
Add reason message logs for non-exist Azure resources
```
Fixes issue #64230, by changing function hasClusterTag, in aws/tags.go, to ensure that a list of tags containing a tag with a key which matches clusterTagKey will return true even if a TagNameKubernetesClusterLegacy tag also exists in the list with a value other than the ClusterID.
/sig aws
Automatic merge from submit-queue (batch tested with PRs 63969, 63902, 63689, 63973, 63978). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Adds a mechanism in vSphere Cloud Provider to get credentials from Kubernetes secrets
**What this PR does / why we need it**:
Currently, vCenter credentials are stored in plain text in vsphere.conf. This PR adds a mechanism in vSphere Cloud Provider to get vCenter credentials from Kubernetes secrets.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
Internally review here: https://github.com/vmware/kubernetes/pull/484
**Workflow:**
1. Create vsphere.conf file with ```secret-name``` and ```secret-namespace```.
```
[Global]
insecure-flag = 1
secret-name = "vcconf"
secret-namespace = "kube-system"
[VirtualCenter "10.160.45.119"]
port = 443
datacenters = k8s-dc-1
[Workspace]
server = 10.160.45.119
datacenter = k8s-dc-1
default-datastore = sharedVMFS-0
folder = Discovered virtual machine
```
2. Launch Kubernetes cluster with vSphere Cloud Provider Configured.
3. Create secret with vCenter credentials.
a. Create base64 encoding for username and password:
username:
```
> echo -n 'admin' | base64
YWRtaW4=
```
password:
```
> echo -n 'vsphere' | base64
dnNwaGVyZQ==
```
b. kubectl create -f vccredentials.yaml
```
#vccredentials.yaml
apiVersion: v1
kind: Secret
metadata:
name: vcconf
type: Opaque
data:
10.192.44.199.username: YWRtaW4=
10.192.44.199.password: dnNwaGVyZQ==
```
4. vSphere Cloud Provider can be used now.
**Note:**
Secrets info can be provided with both (old and new) vSphere Cloud provider configuration formats.
**Tests Done:**
- [x] vSphere Cloud Provider unit test.
- [x] Volume lifecyle with Username and Password in vsphere.conf (for backward compability)
- [x] Volume lifecyle with secrets information in vsphere.conf.
- [x] Update secrets workflow
**Release note**:
```release-note
Adds a mechanism in vSphere Cloud Provider to get credentials from Kubernetes secrets
```
Automatic merge from submit-queue (batch tested with PRs 63569, 63918, 63980, 63295, 63989). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
GCE provider: add wrapper for security policy
**What this PR does / why we need it**:
Generated mock interface for beta backend service and security policy as well.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #NONE
**Special notes for your reviewer**:
/assign @nicksardo
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 63886, 63857, 63824). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
vSphere Cloud Provider: add SAML token authentication support
**What this PR does / why we need it**:
The vSphere cloud provider currently supports username+password based authentication, this PR adds an option to use token based authentication.
**Which issue(s) this PR fixes**:
Fixes#63209
**Special notes for your reviewer**:
For now the config structs and validation are left as-is and
the LoginByToken method is used if the username value is PEM encoded.
In this case of username field configured with the public key, the password
field is expected to be configured with the private key.
In a follow-up PR we can look at collapsing the auth related fields into
a common struct to avoid duplication of field merging and validation.
And then add separate fields for the public and private keys.
**Release note**:
```release-note
vSphere Cloud Provider: add SAML token authentication support
```
For now the config structs and validation are left as-is and
the LoginByToken method is used if the username value is PEM encoded.
In this case of username field configured with the public key, the password
field is expected to be configured with the private key.
In a follow-up PR we can look at collapsing the auth related fields into
a common struct to avoid duplication of field merging and validation.
And then add separate fields for the public and private keys.
Fixes#63209
Automatic merge from submit-queue (batch tested with PRs 63686, 63736). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Deprecate photon cloud provider
vmware/photon-controller is no longer maintained, as of Oct 2017.
Adds a detail field to deprecatedCloudProviders, meant for pointing to external provider url or other reason for deprecation.
**What this PR does / why we need it**:
The photon cloud provider is no longer supported, need to let users know.
Photon controller support has already been removed from kube-up in PR #58096
**Release note**:
```release-note
NONE
```
vmware/photon-controller is no longer maintained, as of Oct 2017.
Adds a detail field to deprecatedCloudProviders, meant for pointing to external provider url or other reason for deprecation.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
should use time.Since instead of time.Now().Sub
**What this PR does / why we need it**:
should use time.Since instead of time.Now().Sub
**Special notes for your reviewer**:
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Include more information when multiple security groups are tagged
**What this PR does / why we need it**:
When trying to create ELB we can sometime fail if there is more then one AWS
security group tagged. It very useful to get the list of security groups printed in
the error message.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
Include the list of security groups when failing with the errors that more then one is tagged
```
Automatic merge from submit-queue (batch tested with PRs 63563, 63541). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Do not check vmSetName when getting Azure node's IP
**What this PR does / why we need it**:
Do not check vmSetName when getting Azure node's IP.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#63538
**Special notes for your reviewer**:
**Release note**:
```release-note
Do not check vmSetName when getting Azure node's IP
```
Automatic merge from submit-queue (batch tested with PRs 63364, 63464). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
vsphere: use vim25.Client directly to support token authentication
**What this PR does / why we need it**:
This refactor is in support of SAML token authentication: #63209
Avoid use of govmomi.Client as it only supports username+password authentication via SessionManager.Login().
Using vim25.Client directly will allow VCP to add other authentication methods,
such as SessionManager.LoginByToken().
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
adding support for VM name with extra Separator String
**What this PR does / why we need it**:
Allows the Azure VM name to have within their name the `_` character
**Special notes for your reviewer**:
This is the error I got before testing
```
May 07 21:55:14 kn-infra000000.<domainname> kubelet[39465]: I0507 21:55:14.462125 39465 azure_vmss.go:108] getVmssVM gets scaleSetName ("kn-infra_ss") and instanceID ("0") for node "kn-infra000000"
May 07 21:55:14 kn-infra000000.<domainname> kubelet[39465]: E0507 21:55:14.462147 39465 azure_vmss_cache.go:52] Failed to extract vmssVMName "kn-infra_ss_0"
May 07 21:55:14 kn-infra000000.<domainname> kubelet[39465]: F0507 21:55:14.462160 39465 kubelet.go:1349] Kubelet failed to get node info: failed to get external ID from cloud provider: not a vmss instance
```
**Release note**:
```release-note
Azure VMSS: support VM names to contain the `_` character
```
The TestVSphereLogin method still defaults to testing against a real vCenter,
but if the required environment variables are not set, it can test against vcsim.
More tests can be converted to use configFromEnvOrSim(), but can be in follow up PRs.
This refactor is in support of SAML token authentication: #63209
Avoid use of govmomi.Client as it only supports username+password authentication via SessionManager.Login().
Using vim25.Client directly will allow VCP to add other authentication methods,
such as SessionManager.LoginByToken().
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Clean up Azure clients
**What this PR does / why we need it**:
Follow up of #63278 (Use new Azure SDK APIs for load balancer and public IP operations), clean up all other clients.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Follow up of #63278
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 61725, 63261). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Use cloudprovider.NotImplemented in AddSSHKeyToAllInstances
**What this PR does / why we need it**:
Minor cleanup: In looking at implementations of this method,
noticed a few providers created their own errors instead of using cloudprovider.NotImplemented
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 62657, 63278, 62903, 63375). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Use new Azure SDK APIs for load balancer and public IP operations
**What this PR does / why we need it**:
#63063 updated Azure SDK to a stable version. After that, we should also update existing clients to use new SDK APIs.
Without this, public IP listing will be blocked forever in some case.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
A following PR will be sent for other interfaces, e.g. routes and NSGs.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 62060, 62516). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
[GCE] Add new formats to resource parser and self link creator
**What this PR does / why we need it**:
- Expands the set of parse-able strings for resource IDs, while also simplifying the code. Note that these are acceptable values for some fields in GCP API.
- global/networks/my-network
- regions/us-central1/subnetworks/my-sub
- zones/us-central1-a/instances/my-hacker-instance
- Fixes the SelfLink function to return links for regions and zones:
- https://www.googleapis.com/compute/v1/projects/proj4/regions/us-central1
- Generates helper functions to create a ResourceID for each resource
- Generates a unit test that ensures all links can be generated and all generated links can be parsed.
- Fixes an ILB test which creates a malformed URL.
**Special notes for your reviewer**:
/assign rramkumar1
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 63349, 63294). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add metric for throttled requests in AWS
**What this PR does / why we need it**:
This PR adds a metric for request throttling in AWS.
**Special notes for your reviewer**:
* Added metric.
* Moved metrics-related code to `aws_metrics.go`.
* Capitalized acronyms, e.g., `recordAwsMetric` to `recordAWSMetric`.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 63138, 63091, 63201, 63341). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Set names for OpenStack loadbalancer members and monitors
**What this PR does / why we need it**:
This PR sets names for OpenStack loadbalancer members and monitors.
ATM names for members and monitors are empty, making it difficult to identify where they belong.
Healthmonitors will be named `monitor_<lb_name>_<index>` and members
will be named `member_<lb_name>_<index>_<node_name>`.
This naming scheme should in sync with the naming of other LB resources, e.g. pools: `pool_<lb_name>_<index>`.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 59879, 62729). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Openstack: fix orphaned route deletion
This is a follow-up to #56258 which only got half of the work done.
The OpenStack cloud providers DeleteRoute method fails to delete routes when it can’t find the corresponding instance in OpenStack.
```release-note
OpenStack cloudprovider: Fix deletion of orphaned routes
```
Minor cleanup: In looking at implementations of this method,
noticed a few providers created their own errors instead of using cloudprovider.NotImplemented
Automatic merge from submit-queue (batch tested with PRs 63129, 63066, 60009, 63136, 63086). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
add node shutdown taint
**What this PR does / why we need it**: we need node stopped taint in order to detach volumes immediately without waiting timeout. More info in issue ticket #58635
**Which issue(s) this PR fixes**
Fixes#58635
**Special notes for your reviewer**: this was reverted, original PR https://github.com/kubernetes/kubernetes/pull/59323 Hopefully now bugs are fixed. However, I will execute more tests manually today.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 63007, 62919, 62669, 62860). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix vSphere Cloud Provider to handle upgrade from k8s version less than v1.9.4 to v1.9.4+
**What this PR does / why we need it**:
vSphere Cloud Provider in kubernetes master v1.9.4+ is not able to identify the kubernetes nodes of version less than 1.9.4. Hence, volume operations fail in this case. This PR fixes this.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#62435
**Special notes for your reviewer**:
Internally reviewed here: https://github.com/vmware/kubernetes/pull/477
**Release note**:
```release-note
None
```
Automatic merge from submit-queue (batch tested with PRs 62857, 62707). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add support of standard LB to Azure vmss
**What this PR does / why we need it**:
Add support of standard LB to Azure vmss.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#60485
**Special notes for your reviewer**:
**Release note**:
```release-note
Add support of standard LB to Azure vmss
```
/sig azure
morrislaw
