Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
update cloud provider deprecation notice
**What this PR does / why we need it**:
Updates cloud provider deprecation warning to clearly outline intentions set by sig cloud provider.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes # https://github.com/kubernetes/kubernetes/issues/68234
**Special notes for your reviewer**:
also removed some extra white spaces
```release-note
NONE
```
/sig cloud-provider
cc @tpepper
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Deprecate cloudstack and ovirt controller projects
Change-Id: Icca9142940269ad1cd28f1f3491684a1bc626c55
**What this PR does / why we need it**:
Do we have folks invested in these providers trying to work on the external controllers for these providers? Is there a future for these providers? If not can we deprecate and eventually remove them?
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
cc @ngtuna @sebgoa @svanharmelen (for cloudstack)
cc @simon3z
**Release note**:
```release-note
Deprecate cloudstack and ovirt controllers
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
implement InstanceShutdownByProviderID for azure
**What this PR does / why we need it**: implements #66265
**Which issue(s) this PR fixes**: Fixes#66265
**Special notes for your reviewer**:
**Release note**:
```release-note
Support NodeShutdown taint for azure
```
Automatic merge from submit-queue (batch tested with PRs 67736, 68123, 68138). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Added support to get clusters in gce cloud provider.
**What this PR does / why we need it**:
Implemented the call to get all cluster objects in a zone for a project.
Also added code to allow the container api to be set in the gce.conf
file.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 67986, 68210, 67817). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Fix panic when processing http response
**What this PR does / why we need it**:
When Azure ARM API gets something wrong, kube-controller-manager may panic because of azure cloud provider:
```
/usr/local/go/src/runtime/asm_amd64.s:2361
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1d4cad9]
goroutine 1386 [running]:
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107
panic(0x44468c0, 0x8b76a30)
/usr/local/go/src/runtime/panic.go:502 +0x229
k8s.io/kubernetes/pkg/cloudprovider/providers/azure.processHTTPRetryResponse(0x0, 0x64ffec0, 0xc4229fd1f0, 0xc422ed05b0, 0x2, 0x2)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_backoff.go:364 +0x69
k8s.io/kubernetes/pkg/cloudprovider/providers/azure.(*Cloud).CreateOrUpdatePIPWithRetry.func1(0xc422ed0600, 0x0, 0x0)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_backoff.go:205 +0x298
```
This PR fixes that.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#68209
**Special notes for your reviewer**:
Should cherry pick to old releases.
**Release note**:
```release-note
Fix panic when processing Azure HTTP response.
```
Automatic merge from submit-queue (batch tested with PRs 67986, 68210, 67817). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
add mixed protocol support for azure load balancer
**What this PR does / why we need it**:
If user specify `service.beta.kubernetes.io/azure-load-balancer-mixed-protocols: "true"`, azure cloud provider will create both TCP and UDP lb rules, for more details, could refer to https://github.com/kubernetes/kubernetes/issues/66887
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#66887
**Special notes for your reviewer**:
original `reconcileLoadBalancer` func is too big, I move part of code implementation to a standalone func `createLoadBalancerRule`
example service config:
```
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/azure-load-balancer-mixed-protocols: "true"
name: web
namespace: default
spec:
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: web
sessionAffinity: None
type: LoadBalancer
```
**Release note**:
```
add mixed protocol support for azure load balancer
```
/kind feature
/sig azure
/assign @feiskyer @khenidak
This change enables ```getLoadBalancer``` to return the loadbalancer even if no floating ip is associated to the VIP port of the loadbalancer.
Signed-off-by: Eunsoo Park <esevan.park@gmail.com>
Automatic merge from submit-queue (batch tested with PRs 65566, 67959, 68029, 68017, 67263). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
add test to verify vsphere cloud provider report node hostname
**What this PR does / why we need it**:
as in pull #67922 has modify vsphere cloud provider to report
node hostname, this patch is to add the test for it.
also fix an issue at InstanceID(), it suppose to return
cloudprovider.InstanceNotFound when vm not found,
but it did double error type translation with a retry() logic,
after removing the retry() (which is not necessary), the logic is simpler, and easier to understand how double translation happened, after the fix,
test TestInstance() can pass
**Which issue(s) this PR fixes** :
Fixes#67714
**Special notes for your reviewer**:
as in the process of create pull to fix https://github.com/crosscloudci/crosscloudci/issues/65#,
i saw #67922 already merged, so this pull is just to submit the test part.
**Release note**:
```release-note
none
```
/cc @abrarshivani @BaluDontu @divyenpatel @imkin @kerneltime @luomiao @frapposelli @dougm @liggitt
/sig cloudprovider
Automatic merge from submit-queue (batch tested with PRs 67368, 59930, 68074). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
implement InstanceShutdownByProviderID to aws cloudprovider
**What this PR does / why we need it**: implement InstanceShutdownByProviderID to aws cloudprovider
**Which issue(s) this PR fixes**:
Fixes#59925
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Add support for unmanaged nodes for Azure cloud provider
**What this PR does / why we need it**:
Continue of [Azure cross resource groups feature](https://github.com/kubernetes/features/issues/604).
This PR adds support for unmanaged nodes (such as on-prem or on other clouds) that are labeled with `alpha.service-controller.kubernetes.io/exclude-balancer=true` and `kubernetes.azure.com/managed=false`. Azure cloud provider would exclude such nodes from LoadBalancer backends and always assumes they are existing.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
See KEP [here](https://github.com/kubernetes/community/blob/master/keps/sig-azure/0025-20180809-cross-resource-group-nodes.md).
**Special notes for your reviewer**:
Azure cloud provider won't provision network routes for on-prem nodes, so cluster admins should ensure the network (including pod-to-pod, pod-to-node and node-to-node connectivity) has been set up properly.
**Release note**:
```release-note
Azure cloud provider now supports unmanaged nodes (such as on-prem) that are labeled with `kubernetes.azure.com/managed=false` and `alpha.service-controller.kubernetes.io/exclude-balancer=true`
```
/assign @khenidak @andyzhangx
/sig azure
/kind feature
/milestone v1.12
Automatic merge from submit-queue (batch tested with PRs 66577, 67948, 68001, 67982). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
implement InstanceShutdownByProviderID to openstack
**What this PR does / why we need it**: without this openstack cannot get shutdown taint when instance is shutdown (original pr where this feature was added https://github.com/kubernetes/kubernetes/pull/60009)
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
Openstack supports now node shutdown taint. Taint is added when instance is shutdown in openstack.
```
as in pull #67922 has modify vsphere cloud provider to report
node hostname, this patch is to add the test for it.
also fix an issue at InstanceID(), it suppose to return
cloudprovider.InstanceNotFound when vm not found, after the fix,
test TestInstance() can pass
On-prem nodes should register themselves with required labels, e.g.
kubelet --node-labels=alpha.service-controller.kubernetes.io/exclude-balancer=true,kubernetes.azure.com/managed=false ...
Automatic merge from submit-queue (batch tested with PRs 67694, 64973, 67902). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
SCTP support implementation for Kubernetes
**What this PR does / why we need it**: This PR adds SCTP support to Kubernetes, including Service, Endpoint, and NetworkPolicy.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#44485
**Special notes for your reviewer**:
**Release note**:
```release-note
SCTP is now supported as additional protocol (alpha) alongside TCP and UDP in Pod, Service, Endpoint, and NetworkPolicy.
```
Automatic merge from submit-queue (batch tested with PRs 67447, 67719). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Report cloudstack hostname address
Cloud providers are now authoritative for the addresses reported by the kubelet. Cloud providers that have hostname information available via metadata should report it for use by the apiserver
```release-note
The cloudstack cloud provider now reports a `Hostname` address type for nodes based on the `local-hostname` metadata key.
```
Automatic merge from submit-queue (batch tested with PRs 54935, 67768, 67896, 67787). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
vsphere: support zone tags at any level in the hierarchy
**What this PR does / why we need it**:
Rather than just looking for zone tags at the VM's Host level, traverse up the hierarchy.
This allows zone tags to be attached at host level, along with cluster, datacenter, root folder
and any inventory folders in between.
Issue #64021
Example log output from the tests, with tags attached at host level:
```console
Found "k8s-region" tag (k8s-region-US) for e85df495-93b9-4b0e-96f1-dc9d56e97263 attached to HostSystem:host-19
Found "k8s-zone" tag (k8s-zone-US-CA1) for e85df495-93b9-4b0e-96f1-dc9d56e97263 attached to HostSystem:host-19
```
And region tag at Datacenter level and zone tag at Cluster level:
```console
Found "k8s-zone" tag (k8s-zone-US-CA1) for e85df495-93b9-4b0e-96f1-dc9d56e97263 attached to ComputeResource:computeresource-21
Found "k8s-region" tag (k8s-region-US) for e85df495-93b9-4b0e-96f1-dc9d56e97263 attached to Datacenter:datacenter-2
```
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 67766, 67642, 67772). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Enable dynamic azure disk volume limits
**What this PR does / why we need it**:
Enable dynamic azure disk volume limits,
This is an azure cloud provider implementation related to feature: [Dynamic Maximum volume count](https://github.com/kubernetes/features/issues/554)
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#66269
**Special notes for your reviewer**:
This PR use `az.VirtualMachineSizesClient.List` to list all vm sizes under region, match vm size with current node size, and then got `MaxDataDiskCount`, the `GetVolumeLimits` happens in kubelet and will return `attachable-volumes-azure-disk` in node status as following example:
```
agentpool-22082114-0
...
allocatable:
attachable-volumes-azure-disk: "8"
cpu: "2"
ephemeral-storage: "28043041951"
hugepages-1Gi: "0"
hugepages-2Mi: "0"
memory: 7034772Ki
pods: "30"
```
**Release note**:
```
Enable dynamic azure disk volume limits
```
/sig azure
/kind feature
The requested Service Protocol is checked against the supported protocols of GCE Internal LB. The supported protocols are TCP and UDP.
SCTP is not supported by OpenStack LBaaS. If SCTP is requested in a Service with type=LoadBalancer, the request is rejected. Comment style is also corrected.
SCTP is not allowed for LoadBalancer Service and for HostPort. Kube-proxy can be configured not to start listening on the host port for SCTP: see the new SCTPUserSpaceNode parameter
changed the vendor github.com/nokia/sctp to github.com/ishidawataru/sctp. I.e. from now on we use the upstream version.
netexec.go compilation fixed. Various test cases fixed
SCTP related conformance tests removed. Netexec's pod definition and Dockerfile are updated to expose the new SCTP port(8082)
SCTP related e2e test cases are removed as the e2e test systems do not support SCTP
sctp related firewall config is removed from cluster/gce/util.sh. Variable name sctp_addr is corrected to sctpAddr in pkg/proxy/ipvs/proxier.go
cluster/gce/util.sh is copied from master
Implemented the call to get all cluster objects in a zone for a project.
Also added code to allow the container api to be set in the gce.conf
file.
Requested fix for @lavalamp. Fixed GetClusters to be GetManagedClusters.
Leaving ListClusters as ListClusters as it is part of the Cloud Clusters
interface, despite also being a "managed" call.
Remove copy pasta :D
Fixed method variable name.
Rather than just looking for zone tags at the VM's Host level, traverse up the hierarchy.
This allows zone tags to be attached at host level, along with cluster, datacenter, root folder
and any inventory folders in between.
Issue #64021
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Make OpenStack cloud provider report a node hostname address
Related to: #67714
Cloud-provider-reported addresses are authoritative in 1.12, this
preserves default behavior that used the internal dns name as
the "Hostname" address on Node status.
**Release note**:
```release-note
The OpenStack cloud provider now reports a `Hostname` address type for nodes
```
Automatic merge from submit-queue (batch tested with PRs 66980, 67604, 67741, 67715). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Make aws cloud provider report a node hostname address
Cloud-provider-reported addresses are authoritative in 1.12, this preserves default behavior that used the internal dns name as the "Hostname" address on Node status
/sig aws
/sig cloud-provider
```release-note
The aws cloud provider now reports a `Hostname` address type for nodes based on the `local-hostname` metadata key.
```
Automatic merge from submit-queue (batch tested with PRs 66980, 67604, 67741, 67715). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
vsphere: add tests for Cloud Provider Zones implementation
**What this PR does / why we need it**:
- Add tests for GetZones()
- Fix bug where a host tag other than region or zone caused an error
- Fix bug where GetZones() errored if zone tag was set, but region was not
Follow up to PR #66795 / towards #64021
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 66980, 67604, 67741, 67715). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add support of Azure cross resource group nodes
**What this PR does / why we need it**:
Part of feature [Cross resource group nodes](https://github.com/kubernetes/features/issues/604).
This PR adds support of Azure cross resource group nodes that are labeled with `kubernetes.azure.com/resource-group=<rg-name>` and `alpha.service-controller.kubernetes.io/exclude-balancer=true`
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
See designs [here](https://github.com/kubernetes/community/pull/2479).
**Release note**:
```release-note
Azure cloud provider now supports cross resource group nodes that are labeled with `kubernetes.azure.com/resource-group=<rg-name>` and `alpha.service-controller.kubernetes.io/exclude-balancer=true`
```
/sig azure
/kind feature
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Reduce API calls for Azure instance metadata
**What this PR does / why we need it**:
Azure cloud provider gets a lot of `"Too many requests"` error when getting availability zones, instance types and node addresses. Hence kubelet won't be able to initialize itself sometimes.
This PR reduces such calls and alos switches to json API which is more stable.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes https://github.com/Azure/acs-engine/issues/3681
**Special notes for your reviewer**:
**Release note**:
```release-note
Reduce API calls for Azure instance metadata.
```
cc @ritazh @khenidak @andyzhangx
Related to: #67714
Cloud-provider-reported addresses are authoritative in 1.12, this
preserves default behavior that used the internal dns name as
the "Hostname" address on Node status.
- Add tests for GetZones()
- Fix bug where a host tag other than region or zone caused an error
- Fix bug where GetZones() errored if zone tag was set, but region was not
Follow up to PR #66795 / towards #64021
Automatic merge from submit-queue (batch tested with PRs 66973, 67704, 67722, 67723, 63512). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Populate internal DNS names in GCE provider
Both VM name and hostname are internally routable.
**What this PR does / why we need it**: GCE cloud provider only populated IP addresses for instances. This PR adds internal DNS names. DNS names are used e.g. in kubelet server certificates (5b77996433/pkg/kubelet/kubelet.go (L385))
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 67332, 66737, 67281, 67173). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
use aws.StringSlice replace of deprecated func stringPointerArray
**What this PR does / why we need it**:
use aws.StringSlice replace of deprecated func stringPointerArray
```
// stringPointerArray creates a slice of string pointers from a slice of strings
// Deprecated: consider using aws.StringSlice - but note the slightly different behaviour with a nil input
func stringPointerArray(orig []string) []*string {
if orig == nil {
return nil
}
return aws.StringSlice(orig)
}
```
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 66793, 67405, 67068, 67501, 67484). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Updated comment for DefaultLoadBalancerName to provide further context
**What this PR does / why we need it**:
Updates the comment for DefaultLoadBalancerName to provide better context and also as a reminder that it should eventually be removed.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 67375, 67353). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Allow passing timeout on TPU API operations via context
The caller will provide a context with timeout attached.
**Release note**:
```release-note
None
```
/assign @vishh
Automatic merge from submit-queue (batch tested with PRs 62441, 66702, 67254, 67421, 65309). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
document expectations for stopped/shutdown instances in cloudprovider.Instances interface
**What this PR does / why we need it**:
Add comments which outline expectations for how cloud providers should handle nodes that are stopped/shutdown/sleeping.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
It doesn't fix https://github.com/kubernetes/kubernetes/issues/46442 since some providers still delete node resources for stopped instances (though there are PRs already open to fix this) but we've (mostly?) agreed that not deleting node resources for stopped instances is the correct behavior, or at least the behavior that we want consistent across providers at the moment.
**Special notes for your reviewer**:
```release-note
NONE
```
/sig cloud-provider
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Update `pkg/cloudprovider/providers/azure/OWNERS`
* Remove Jaice
* Remove Cole
* Add Stephen as reviewer
Signed-off-by: Stephen Augustus <foo@agst.us>
**Release note**:
```release-note
NONE
```
/sig azure
Automatic merge from submit-queue (batch tested with PRs 67026, 62945, 66917). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Cloud Provider Zones doc fixups
**What this PR does / why we need it**:
A few godoc fixups for Cloud Provider Zones.
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 67061, 66589, 67121, 67149). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add DynamicProvisioningScheduling and VolumeScheduling support for Azure managed disks
**What this PR does / why we need it**:
Continue of [Azure Availability Zone feature](https://github.com/kubernetes/features/issues/586).
This PR adds `VolumeScheduling` and `DynamicProvisioningScheduling` support to Azure managed disks.
When feature gate `VolumeScheduling` disabled, no NodeAffinity set for PV:
```yaml
kubectl describe pv
Name: pvc-d30dad05-9ad8-11e8-94f2-000d3a07de8c
Labels: failure-domain.beta.kubernetes.io/region=southeastasia
failure-domain.beta.kubernetes.io/zone=southeastasia-2
Annotations: pv.kubernetes.io/bound-by-controller=yes
pv.kubernetes.io/provisioned-by=kubernetes.io/azure-disk
volumehelper.VolumeDynamicallyCreatedByKey=azure-disk-dynamic-provisioner
Finalizers: [kubernetes.io/pv-protection]
StorageClass: default
Status: Bound
Claim: default/pvc-azuredisk
Reclaim Policy: Delete
Access Modes: RWO
Capacity: 5Gi
Node Affinity:
Required Terms:
Term 0: failure-domain.beta.kubernetes.io/region in [southeastasia]
failure-domain.beta.kubernetes.io/zone in [southeastasia-2]
Message:
Source:
Type: AzureDisk (an Azure Data Disk mount on the host and bind mount to the pod)
DiskName: k8s-5b3d7b8f-dynamic-pvc-d30dad05-9ad8-11e8-94f2-000d3a07de8c
DiskURI: /subscriptions/<subscription-id>/resourceGroups/<rg-name>/providers/Microsoft.Compute/disks/k8s-5b3d7b8f-dynamic-pvc-d30dad05-9ad8-11e8-94f2-000d3a07de8c
Kind: Managed
FSType:
CachingMode: None
ReadOnly: false
Events: <none>
```
When feature gate `VolumeScheduling` enabled, NodeAffinity will be populated for PV:
```yaml
kubectl describe pv
Name: pvc-0284337b-9ada-11e8-a7f6-000d3a07de8c
Labels: failure-domain.beta.kubernetes.io/region=southeastasia
failure-domain.beta.kubernetes.io/zone=southeastasia-2
Annotations: pv.kubernetes.io/bound-by-controller=yes
pv.kubernetes.io/provisioned-by=kubernetes.io/azure-disk
volumehelper.VolumeDynamicallyCreatedByKey=azure-disk-dynamic-provisioner
Finalizers: [kubernetes.io/pv-protection]
StorageClass: default
Status: Bound
Claim: default/pvc-azuredisk
Reclaim Policy: Delete
Access Modes: RWO
Capacity: 5Gi
Node Affinity:
Required Terms:
Term 0: failure-domain.beta.kubernetes.io/region in [southeastasia]
failure-domain.beta.kubernetes.io/zone in [southeastasia-2]
Message:
Source:
Type: AzureDisk (an Azure Data Disk mount on the host and bind mount to the pod)
DiskName: k8s-5b3d7b8f-dynamic-pvc-0284337b-9ada-11e8-a7f6-000d3a07de8c
DiskURI: /subscriptions/<subscription-id>/resourceGroups/<rg-name>/providers/Microsoft.Compute/disks/k8s-5b3d7b8f-dynamic-pvc-0284337b-9ada-11e8-a7f6-000d3a07de8c
Kind: Managed
FSType:
CachingMode: None
ReadOnly: false
Events: <none>
```
When both `VolumeScheduling` and `DynamicProvisioningScheduling` are enabled, storage class also supports `allowedTopologies` and `volumeBindingMode: WaitForFirstConsumer` for volume topology aware dynamic provisioning:
```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
name: managed-disk-dynamic
parameters:
cachingmode: None
kind: Managed
storageaccounttype: Standard_LRS
provisioner: kubernetes.io/azure-disk
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
- key: failure-domain.beta.kubernetes.io/zone
values:
- southeastasia-2
- southeastasia-1
```
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
DynamicProvisioningScheduling and VolumeScheduling is not supported for Azure managed disks. Feature gates DynamicProvisioningScheduling and VolumeScheduling should be enabled before using this feature.
```
/kind feature
/sig azure
/cc @brendandburns @khenidak @andyzhangx
/cc @ddebroy @msau42 @justaugustus
Automatic merge from submit-queue (batch tested with PRs 67061, 66589, 67121, 67149). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Get load balancer name per provider
**What this PR does / why we need it**:
GetLoadBalancerName() should be implemented per cloud provider as opposed to one neutral implementation.
This PR will address this by moving `cloudprovider.GetLoadBalancerName()` to the `LoadBalancer interface` and then provide an implementation for each cloud provider, while maintaining previously expected functionality.
**Which issue(s) this PR fixes**:
Fixes [#43173](https://github.com/kubernetes/kubernetes/issues/43173)
**Special notes for your reviewer**:
This is a work in progress. Looking for feedback as I work on this, from any interested parties.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
scope AWS LoadBalancer security group ICMP rules to spec.loadBalancerSourceRanges
/sig aws
**What this PR does / why we need it**:
Make the client CIDR ranges for MTU consistent with what [the documentation appears to describe](https://kubernetes.io/docs/concepts/services-networking/service/#type-loadbalancer), where the ranges should be equal to `spec.loadBalancerSourceRanges` if supplied.
**Which issue(s) this PR fixes**:
Fixes#63564
**Release note**:
```release-note
scope AWS LoadBalancer security group ICMP rules to spec.loadBalancerSourceRanges
```
Automatic merge from submit-queue (batch tested with PRs 67052, 67094, 66795). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add zones support for vSphere cloud provider(in-tree)
**What this PR does / why we need it**:
This PR added zones(built-in node labels) support for vSphere cloud provider(in-tree). More details can be found in the issue as below.
**Which issue(s) this PR fixes** :
Partially fixes phase 1 of issue #64021
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
use func WaitForCompletionRef replace of deprecated func WaitForCompletion
**What this PR does / why we need it**:
use func WaitForCompletionRef replace of deprecated func WaitForCompletion
```
// WaitForCompletion will return when one of the following conditions is met: the long
// running operation has completed, the provided context is cancelled, or the client's
// polling duration has been exceeded. It will retry failed polling attempts based on
// the retry value defined in the client up to the maximum retry attempts.
// Deprecated: Please use WaitForCompletionRef() instead.
func (f Future) WaitForCompletion(ctx context.Context, client autorest.Client) error {
return f.WaitForCompletionRef(ctx, client)
}
```
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add availability zones support to Azure managed disks
**What this PR does / why we need it**:
Continue of [Azure Availability Zone feature](https://github.com/kubernetes/features/issues/586).
This PR adds availability zone support for Azure managed disks and its storage class. Zoned managed disks is enabled by default if there are zoned nodes in the cluster.
The zone could also be customized by `zone` or `zones` parameter, e.g.
```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
name: managed-disk-zone-1
parameters:
zone: "southeastasia-1"
# zones: "southeastasia-1,"southeastasia-2"
cachingmode: None
kind: Managed
storageaccounttype: Standard_LRS
provisioner: kubernetes.io/azure-disk
reclaimPolicy: Delete
volumeBindingMode: Immediate
```
All zoned AzureDisk PV will also be labeled with its availability zone, e.g.
```sh
$ kubectl get pvc pvc-azuredisk-az-1
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc-azuredisk-az-1 Bound pvc-5ad0c7b8-8f0b-11e8-94f2-000d3a07de8c 5Gi RWO managed-disk-zone-1 2h
$ kubectl get pv pvc-5ad0c7b8-8f0b-11e8-94f2-000d3a07de8c --show-labels
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE LABELS
pvc-5ad0c7b8-8f0b-11e8-94f2-000d3a07de8c 5Gi RWO Delete Bound default/pvc-azuredisk-az-1 managed-disk-zone-1 2h failure-domain.beta.kubernetes.io/region=southeastasia,failure-domain.beta.kubernetes.io/zone=southeastasia-1
```
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
See also the [KEP](https://github.com/kubernetes/community/pull/2364).
DynamicProvisioningScheduling feature would be added in a following PR.
**Release note**:
```release-note
Azure managed disks now support availability zones and new parameters `zoned`, `zone` and `zones` are added for AzureDisk storage class.
```
/kind feature
/sig azure
/assign @brendandburns @khenidak @andyzhangx
Zones implementation for vSphere cloud provider needs dependencies
which are not included in current vmware/govmomi vendor. So this
update added "vapi" package to support zones.
Automatic merge from submit-queue (batch tested with PRs 65730, 66615, 66684, 66519, 66510). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add DynamicProvisioningScheduling support for EBS
**What this PR does / why we need it**:
This PR adds support for the DynamicProvisioningScheduling feature in EBS. With this in place, if VolumeBindingMode: WaitForFirstConsumer is specified in a EBS storageclass and DynamicProvisioningScheduling is enabled, EBS provisioner will use the selected node's LabelZoneFailureDomain as the zone to provision the EBS volume in.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
Related to #63232
Sample `describe pv` output with NodeAffinity populated:
```
~$ kubectl describe pv pvc-f9d2138b-7e3e-11e8-a4ea-064124617820
Name: pvc-f9d2138b-7e3e-11e8-a4ea-064124617820
Labels: failure-domain.beta.kubernetes.io/region=us-west-2
failure-domain.beta.kubernetes.io/zone=us-west-2a
Annotations: kubernetes.io/createdby=aws-ebs-dynamic-provisioner
pv.kubernetes.io/bound-by-controller=yes
pv.kubernetes.io/provisioned-by=kubernetes.io/aws-ebs
Finalizers: [kubernetes.io/pv-protection]
StorageClass: slow3
Status: Bound
Claim: default/pvc3
Reclaim Policy: Delete
Access Modes: RWO
Capacity: 6Gi
Node Affinity:
Required Terms:
Term 0: failure-domain.beta.kubernetes.io/zone in [us-west-2a]
failure-domain.beta.kubernetes.io/region in [us-west-2]
Message:
Source:
Type: AWSElasticBlockStore (a Persistent Disk resource in AWS)
VolumeID: aws://us-west-2a/vol-0fc1cdae7d10860f6
FSType: ext4
Partition: 0
ReadOnly: false
Events: <none>
```
**Release note**:
```release-note
none
```
/sig storage
/assign @msau42 @jsafrane
Automatic merge from submit-queue (batch tested with PRs 66489, 66728, 66739). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Speed up volume modifications on AWS
Volume becomes reusable after it reached optimizing state.
/sig storage
/sig aws
```release-note
Make EBS volume expansion faster
```
cc @d-nishi @kokhang
Automatic merge from submit-queue (batch tested with PRs 66225, 66648, 65799, 66630, 66619). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Implement InstanceShutdownByProviderID for vSphere CP
**What this PR does / why we need it**:
To detach volumes from shutdown nodes #66181
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 66225, 66648, 65799, 66630, 66619). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Update Azure Go SDK to v19.0.0 and get availability zone for VirtualMachineScaleSetVM
**What this PR does / why we need it**:
Continue of #66242. This PR updates Azure Go SDK to v19.0.0 (with compute API 2018-04-01) and gets availability zones for VirtualMachineScaleSetVM.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
Azure Go SDK has been upgraded to v19.0.0 and VirtualMachineScaleSetVM now supports availability zones.
```
/sig azure
/assign @brendandburns @khenidak @andyzhangx
Automatic merge from submit-queue (batch tested with PRs 66464, 66488). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Avoid overflowing int64 in RoundUpSize and return error if overflow int
**What this PR does / why we need it**:
There are many places in plugins (some I may have missed) that we naively convert a resource.Quantity.Value() which is an int64, to an int, which may be only 32 bits long.
Background, optional to read :): Kubernetes canonicalizes resource.Quantities, and from what I have seen testing creating PVCs, decimalSI is the default. If a quantity is in `decimalSI` format and its value in bytes would overflow an int64, e.g. `10E`, nothing happens. If it is in binarySI and its value in bytes would overflow an int64, e.g. `10Ei`, it is set down to 2^63-1 and there's no overflow of the field value. But there may be overflow later in the code which is what this PR is addressing.
* Change `RoundUpSize` implementation to avoid overflowing `int64`
* Add `RoundUp*Int` functions for use when an `int` is expected instead of an `int64`, because `int` may be 32bits and naively doing `int($INT64_VALUE)` can lead to silent overflow. These functions return an error if overflow has occurred.
* Rename `*GB` variables to `*GiB` where appropriate for maximum clarity
* Use `RoundUpToGiB` instead of `RoundUpSize` where possible
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**: please review carefully as we don't have e2e tests for most plugins!
**Release note**:
```release-note
NONE
```
edit: remove 'we do not need to worry about...'. yes we do, i worded that badly :))
Automatic merge from submit-queue (batch tested with PRs 62423, 66180, 66492, 66506, 65242). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add user assigned MSI support
**What this PR does / why we need it**:
Adds the support for generating tokens via user assigned MSI.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
Add support for using User Assigned MSI (https://docs.microsoft.com/en-us/azure/active-directory/managed-service-identity/overview) with Kubernetes cluster on Azure.
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add myself, Micah to reviewers
**Release note**:
```release-note
NONE
```
Signed-off-by: Nick Turner <nic@amazon.com>
Automatic merge from submit-queue (batch tested with PRs 66098, 66389, 66400, 66413, 66378). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
GCE: Return correct error type and HTTP Status code for operation errors
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#66399
**Special notes for your reviewer**:
/assign bowei, zihongz, rramkumar
/cc bowei
**Release note**:
```release-note
GCE: Fixes loadbalancer creation and deletion issues appearing in 1.10.5.
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add initial availability zones support for Azure nodes
**What this PR does / why we need it**:
The first part of [Azure Availability Zone feature](https://github.com/kubernetes/features/issues/586).
This PR adds initial availability zone (AZ) support for Azure nodes. With this PR, Azure nodes with AZ will have label `failure-domain.beta.kubernetes.io/zone=<region>-<zoneID>`, e.g. `southeastasia-1`.
It also updates instance metadata api-version to 2017-12-01, which is required for AZ.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
VirtualMachineScaleSetVM doesn't have AZ info yet. It will be supported later after new Azure Go SDK releases.
**Release note**:
```release-note
Azure nodes with availability zone now will have label `failure-domain.beta.kubernetes.io/zone=<region>-<zoneID>`.
```
/kind feature
/sig azure
/assign @brendandburns @khenidak @andyzhangx
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add error check and ignore unused variable (SA4006)
**What this PR does / why we need it**:
Fix some bugs in cloud provider vsphere, issue can be found here #66303
```pkg/cloudprovider/providers/vsphere/nodemanager.go:176:5: defers in this range loop won't run unless the channel gets closed (SA9001)
pkg/cloudprovider/providers/vsphere/vclib/diskmanagers/vmdm.go:129:8: this value of err is never used (SA4006)
pkg/cloudprovider/providers/vsphere/vsphere.go:596:34: argument ctx is overwritten before first use (SA4009)
pkg/cloudprovider/providers/vsphere/vsphere_test.go:360:2: this value of instanceID is never used (SA4006)
pkg/cloudprovider/providers/vsphere/vsphere_util.go:301:3: defers in this infinite loop will never run (SA5003)
```
**Special notes for your reviewer**:
I fixed ```SA4006``` report in that issue, but there are still other code needed to discuss and fix.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix locating resourcepool-path specified in the vsphere.conf file
**What this PR does / why we need it**:
When volume is provisioned using the vsphere storage policy, `resourcepool-path` specified in the `vsphere.conf` file is used for creating a shadow/dummy VM. Dummy VM is temporarily created and then deleted once volume is created on the compatible Datastore.
At present If user specifies `resourcepool-path` in the `vsphere.conf` file, volume provisioner is not able to locate the compute resource for the given path. This is because look up is made using `finder.DefaultComputeResource(ctx)` and `finder.ComputeResource(ctx, computePath)`, which is not correct. If user specifies name of the cluster or cluster path then provisioning works.
This is resolved with using correct govmomi method - `func (f *Finder) ResourcePoolOrDefault(ctx context.Context, path string) (*object.ResourcePool, error)`
**Which issue(s) this PR fixes**
Fixes # https://github.com/vmware/kubernetes/issues/493
**Special notes for your reviewer**:
Following testing is performed for this change.
1) specified resource-pool path in the `vsphere.conf` file and verified VM is created under the specified resource pool.
```
resourcepool-path="ClusterFolder-1/cluster-vsan-1/Resources/ShadowVMPool"
```
2) If resource pool is not available, specified cluster's default resource pool path in the `vsphere.conf` file and verified volume provisioning works. For this case, VM is directly created under cluster.
```
resourcepool-path="ClusterFolder-1/cluster-vsan-1/Resources"
```
3) Verified above with having multiple clusters with the same name in one datacenter.
4) Verified with empty resource pool path in the vsphere.conf file.
```
resourcepool-path=""
```
As expected, provisioning is failing with `Failed to provision volume with StorageClass "vsan-gold-policy": no default resource pool found`.
Refer to this datacenter inventory for the path specified in the `resourcepool-path` configuration.
Current documentation describes `resourcepool-path`configuration is optional, which needs to be corrected once PR is merged. For policy based provisioning this is not an optional parameter.
Documentation link: https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/existing.html
**Release note**:
```release-note
Fix for resourcepool-path configuration in the vsphere.conf file.
```
cc: @kubernetes/vmware
k8s-ci-robot
