Automatic merge from submit-queue (batch tested with PRs 67739, 65222). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Honor --hostname-override, report compatible hostname addresses with cloud provider
xref #677147828e5d made cloud providers authoritative for the addresses reported on Node objects, so that the addresses used by the node (and requested as SANs in serving certs) could be verified via cloud provider metadata.
This had the effect of no longer reporting addresses of type Hostname for Node objects for some cloud providers. Cloud providers that have the instance hostname available in metadata should add a `type: Hostname` address to node status. This is being tracked in #67714
This PR does a couple other things to ease the transition to authoritative cloud providers:
* if `--hostname-override` is set on the kubelet, make the kubelet report that `Hostname` address. if it can't be verified via cloud-provider metadata (for cert approval, etc), the kubelet deployer is responsible for fixing the situation by adjusting the kubelet configuration (as they were in 1.11 and previously)
* if `--hostname-override` is not set, *and* the cloud provider didn't report a Hostname address, *and* the auto-detected hostname matches one of the addresses the cloud provider *did* report, make the kubelet report that as a Hostname address. That lets the addresses remain verifiable via cloud provider metadata, while still including a `Hostname` address whenever possible.
/sig node
/sig cloud-provider
/cc @mikedanese
fyi @hh
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 67938, 66719, 67883). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Remove incorrect glog error from Horizontal Pod Autoscaler Controller.
**What this PR does / why we need it**:
Pro removes incorrect glog error from Horizontal Pod Autoscaler Controller.
**Release note:**
```release-note
none
```
Automatic merge from submit-queue (batch tested with PRs 67938, 66719, 67883). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
return err when Unmarshal failed
**What this PR does / why we need it**:
return err when Unmarshal failed
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 67694, 64973, 67902). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
SCTP support implementation for Kubernetes
**What this PR does / why we need it**: This PR adds SCTP support to Kubernetes, including Service, Endpoint, and NetworkPolicy.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#44485
**Special notes for your reviewer**:
**Release note**:
```release-note
SCTP is now supported as additional protocol (alpha) alongside TCP and UDP in Pod, Service, Endpoint, and NetworkPolicy.
```
Automatic merge from submit-queue (batch tested with PRs 67694, 64973, 67902). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
kube-apiserver: always create configmap/extension-apiserver-authentication
Other components (aggregated apiservers) read the configmap and fail hard if it does not exist. But they work without all fields being set (https://github.com/kubernetes/kubernetes/pull/66394). In the future, components like ctrl-manager and scheduler won't need kube-apiserver to authenticate with them at all. So, consequently we should always create the file, even if it is empty.
```release-note
Always create configmaps/extensions-apiserver-authentication from kube-apiserver.
```
Automatic merge from submit-queue (batch tested with PRs 64597, 67854, 67734, 67917, 67688). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Move kubeproxy ComponentConfig external types to `k8s.io/kube-proxy`
**What this PR does / why we need it**:
This PR implements most of kubernetes/community#2354 for the kube-proxy.
The PR:
- Moves k8s.io/kubernetes/pkg/proxy/apis/kubeproxyconfig as-is to k8s.io/kubernetes/pkg/proxy/apis/config as agreed
- Moves the external types to the new staging repo k8s.io/kube-proxy, in the k8s.io/kube-proxy/config/v1beta1 package.
- Makes k8s.io/kubernetes/pkg/proxy/apis/config/v1beta1 source the types from k8s.io/kube-proxy/config/v1beta1. The defaulting and conversion code is kept in this package as before.
- All references to these packages have been updated.
Ref #67233
**Special notes for your reviewer**:
**Release note**:
```release-note
kube-proxy v1beta1 external ComponentConfig types are now available in the `k8s.io/kube-proxy` repo
```
Automatic merge from submit-queue (batch tested with PRs 64597, 67854, 67734, 67917, 67688). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
GCE: Add ListLocations to Cloud TPU API
Add ListLocations to Cloud TPU API.
**Release note**:
```release-note
None
```
/assign @vishh
/assign @MrHohn
for approval.
Automatic merge from submit-queue (batch tested with PRs 64597, 67854, 67734, 67917, 67688). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
fix an issue that scheduling doesn't respect NodeLost status of a node
**What this PR does / why we need it**:
- if Node is in UnknowStatus, apply unreachable taint with NoSchedule effect
- some internal data structure refactoring
- update unit test
**Which issue(s) this PR fixes**:
Fixes#67733, and very likely #67536
**Special notes for your reviewer**:
See detailed reproducing steps in #67733.
**Release note**:
```release-note
Apply unreachable taint to a node when it lost network connection.
```
Automatic merge from submit-queue (batch tested with PRs 64597, 67854, 67734, 67917, 67688). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Remove unused kubelet dependency
**What this PR does / why we need it**:
Cleanup unused `ExternalKubeClient`. I think it's a vestige of when the Kubelet still used the internal types.
**Release note**:
```release-note
NONE
```
/kind cleanup
/sig node
/assign @mtaufen
Automatic merge from submit-queue (batch tested with PRs 64597, 67854, 67734, 67917, 67688). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Allow ImageReview backend to add audit annotations.
**What this PR does / why we need it**:
This can be used to create annotations that will allow auditing of the created
pods.
The change also introduces "fail open" audit annotations in addition to the
previously existing pod annotation for fail open. The pod annotations for
fail open will be deprecated soon.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
Allow ImageReview backend to return annotations to be added to the created pod.
```
Automatic merge from submit-queue (batch tested with PRs 67323, 66717, 67038). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Prevent side effects on dryrun in service registry
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Added set and map structural validation for AllowedTopologies
**What this PR does / why we need it**: Adding structural validation to AllowedTopologies field in StorageClass.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#66184
**Release note**:
```release-note
AllowedTopologies field inside StorageClass is now validated against set and map semantics. Specifically, there cannot be duplicate TopologySelectorTerms, MatchLabelExpressions keys, and TopologySelectorLabelRequirement Values.
```
Automatic merge from submit-queue (batch tested with PRs 67894, 64097). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
HPA metrics specificity improvements
**What this PR does / why we need it**:
Improves available specificity for HPA metrics by adding metric selector fields for metrics of Pods and Objects.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Implements this KEP: https://github.com/kubernetes/community/pull/2055
**Special notes for your reviewer**:
Need to add/update tests?
**Release note**:
```release-note
Introduces autoscaling/v2beta2 and custom_metrics/v1beta2, which implement metric selectors for Object and Pods metrics, as well as allowing AverageValue targets on Objects, similar to External metrics.
```
/assign @DirectXMan12
Automatic merge from submit-queue (batch tested with PRs 67447, 67719). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Report cloudstack hostname address
Cloud providers are now authoritative for the addresses reported by the kubelet. Cloud providers that have hostname information available via metadata should report it for use by the apiserver
```release-note
The cloudstack cloud provider now reports a `Hostname` address type for nodes based on the `local-hostname` metadata key.
```
Automatic merge from submit-queue (batch tested with PRs 54935, 67768, 67896, 67787). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
vsphere: support zone tags at any level in the hierarchy
**What this PR does / why we need it**:
Rather than just looking for zone tags at the VM's Host level, traverse up the hierarchy.
This allows zone tags to be attached at host level, along with cluster, datacenter, root folder
and any inventory folders in between.
Issue #64021
Example log output from the tests, with tags attached at host level:
```console
Found "k8s-region" tag (k8s-region-US) for e85df495-93b9-4b0e-96f1-dc9d56e97263 attached to HostSystem:host-19
Found "k8s-zone" tag (k8s-zone-US-CA1) for e85df495-93b9-4b0e-96f1-dc9d56e97263 attached to HostSystem:host-19
```
And region tag at Datacenter level and zone tag at Cluster level:
```console
Found "k8s-zone" tag (k8s-zone-US-CA1) for e85df495-93b9-4b0e-96f1-dc9d56e97263 attached to ComputeResource:computeresource-21
Found "k8s-region" tag (k8s-region-US) for e85df495-93b9-4b0e-96f1-dc9d56e97263 attached to Datacenter:datacenter-2
```
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 67766, 67642, 67772). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Enable dynamic azure disk volume limits
**What this PR does / why we need it**:
Enable dynamic azure disk volume limits,
This is an azure cloud provider implementation related to feature: [Dynamic Maximum volume count](https://github.com/kubernetes/features/issues/554)
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#66269
**Special notes for your reviewer**:
This PR use `az.VirtualMachineSizesClient.List` to list all vm sizes under region, match vm size with current node size, and then got `MaxDataDiskCount`, the `GetVolumeLimits` happens in kubelet and will return `attachable-volumes-azure-disk` in node status as following example:
```
agentpool-22082114-0
...
allocatable:
attachable-volumes-azure-disk: "8"
cpu: "2"
ephemeral-storage: "28043041951"
hugepages-1Gi: "0"
hugepages-2Mi: "0"
memory: 7034772Ki
pods: "30"
```
**Release note**:
```
Enable dynamic azure disk volume limits
```
/sig azure
/kind feature
Automatic merge from submit-queue (batch tested with PRs 67766, 67642, 67772). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add unit test cases for scheduler/algorithm/predicates.
**What this PR does / why we need it**:
Add unit test cases for scheduler/algorithm/predicates for more code coverage.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
NONE
**Special notes for your reviewer**:
NONE
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 66257, 67750). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add unit test cases for scheduler/util.
**What this PR does / why we need it**:
Add unit test cases for scheduler/util for more code coverage.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
NONE
**Special notes for your reviewer**:
NONE
**Release note**:
```release-note
NONE
```
The requested Service Protocol is checked against the supported protocols of GCE Internal LB. The supported protocols are TCP and UDP.
SCTP is not supported by OpenStack LBaaS. If SCTP is requested in a Service with type=LoadBalancer, the request is rejected. Comment style is also corrected.
SCTP is not allowed for LoadBalancer Service and for HostPort. Kube-proxy can be configured not to start listening on the host port for SCTP: see the new SCTPUserSpaceNode parameter
changed the vendor github.com/nokia/sctp to github.com/ishidawataru/sctp. I.e. from now on we use the upstream version.
netexec.go compilation fixed. Various test cases fixed
SCTP related conformance tests removed. Netexec's pod definition and Dockerfile are updated to expose the new SCTP port(8082)
SCTP related e2e test cases are removed as the e2e test systems do not support SCTP
sctp related firewall config is removed from cluster/gce/util.sh. Variable name sctp_addr is corrected to sctpAddr in pkg/proxy/ipvs/proxier.go
cluster/gce/util.sh is copied from master
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Kubelet creates and manages node leases
This extends the Kubelet to create and periodically update leases in a
new kube-node-lease namespace. Based on [KEP-0009](https://github.com/kubernetes/community/blob/master/keps/sig-node/0009-node-heartbeat.md),
these leases can be used as a node health signal, and will allow us to
reduce the load caused by over-frequent node status reporting.
- add NodeLease feature gate
- add kube-node-lease system namespace for node leases
- add Kubelet option for lease duration
- add Kubelet-internal lease controller to create and update lease
- add e2e test for NodeLease feature
I would like to determine a standard policy for lease renewal frequency, based on the configured lease duration, so that we don't need to expose frequency as an additional knob. The renew interval is currently calculated as 1/3 of the lease duration.
```release-note
kubelet: Users can now enable the alpha NodeLease feature gate to have the Kubelet create and periodically renew a Lease in the kube-node-lease namespace. The lease duration defaults to 40s, and can be configured via the kubelet.config.k8s.io/v1beta1.KubeletConfiguration's NodeLeaseDurationSeconds field.
```
/cc @wojtek-t @liggitt
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Complement unit test case TestNodesWherePreemptionMightHelp for scheduler/core
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
```
This extends the Kubelet to create and periodically update leases in a
new kube-node-lease namespace. Based on [KEP-0009](https://github.com/kubernetes/community/blob/master/keps/sig-node/0009-node-heartbeat.md),
these leases can be used as a node health signal, and will allow us to
reduce the load caused by over-frequent node status reporting.
- add NodeLease feature gate
- add kube-node-lease system namespace for node leases
- add Kubelet option for lease duration
- add Kubelet-internal lease controller to create and update lease
- add e2e test for NodeLease feature
- modify node authorizer and node restriction admission controller
to allow Kubelets access to corresponding leases
Automatic merge from submit-queue (batch tested with PRs 64564, 65043). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Make CredentialProvider config loading deterministic.
@mattmoor fixed an old TODO :)
xing-yang