Writable was added to HostPathMount in v1alpha1 in order to control if an extra
volume is mounted in read only or writable mode.
Usually, in Kubernetes, this option is referred to as ReadOnly, instead of
Writable and is defaulted to `false`. However, at the time, all extra volumes
to pods were defaulted to read-only. Therefore, to avoid changes to existing
v1alpha1 configs, this option had to be added with reversed meaning.
Hence, it's called `writable`.
Now, with the migration towards v1beta1, we can safely change this to ReadOnly
and get it in sync with the reset of Kubernetes.
Signed-off-by: Rostislav M. Georgiev <rostislavg@vmware.com>
In v1alpha3's, control plane component config options were nested directly into
the ClusterConfiguration structure. This is cluttering the config structure and
makes it hard to maintain. Therefore the control plane config options must be
separated into different substructures in order to graduate the format to beta.
This change does the following:
- Introduces a new structure called ControlPlaneComponent, that contains fields
common to all control plane component types. These are currently extra args
and extra volumes.
- Introduce a new structure called APIServer that contains
ControlPlaneComponent and APIServerCertSANs field (from ClusterConfiguration)
- Replace all API Server, Scheduler and Controller Manager options in
ClusterConfiguration with APIServer, ControllerManager and Scheduler fields
of APIServer and ControlPlaneComponent types.
Signed-off-by: Rostislav M. Georgiev <rostislavg@vmware.com>
Remove custom flags that were previously available per
sub-phase. Rely on the config passed to 'kubeadm init' for that.
Remove redundant functions in manifests.go.
Move the audit policy settings to the control plane phase (under
API server).
Remove the test based on these points:
- it's apparently flaky by more than one reports and it's hard to debug.
- it's not a CLI unit test.
- it's testing API objects using a fake API server which is a bad hack...
- existing e2e tests already provide coverage for all
the `expectedError: false` cases.
In the past the discovery configuration expected, that we can support multiple
API server endpoints. In practice, we always end up with a single API server
endpoint, because, even in HA setups, we use a load balancer scheme for API
servers.
Therefore, to reduce complexity and improve readability of the config, the
multiple API server endpoints support is removed from the bootstrap token
discovery join method and configuration.
Signed-off-by: Rostislav M. Georgiev <rostislavg@vmware.com>
The service account authenticator isn't the only authenticator that
should respect API audience. The authentication config structure should
reflect that.
Order of Volumes and VolumeMounts in the pod objects created by
kubeadm is undefined as they're represended as maps in the
controlPlaneHostPathMounts struct.
This influences 'kubeadm upgrade' logic in a way that even when
manifest of the component is not changed kubeadm tries to upgrade
it because most of the time current and new pods are not equal
due to the different order of Volumes and VolumeMounts.
For example 'kubeadm apply diff' almost always shows difference
in Volumes and VolumeMounts because of this:
volumeMounts:
+ - mountPath: /etc/kubernetes/pki
+ name: k8s-certs
+ readOnly: true
- mountPath: /etc/ssl/certs
name: ca-certs
+ readOnly: true
+ - mountPath: /etc/pki
+ name: etc-pki
+ readOnly: true
+ - mountPath: /usr/share/ca-certificates
+ name: usr-share-ca-certificates
readOnly: true
- mountPath: /etc/ca-certificates
name: etc-ca-certificates
readOnly: true
- - mountPath: /etc/pki
- name: etc-pki
- readOnly: true
- - mountPath: /etc/kubernetes/pki
- name: k8s-certs
- readOnly: true
- - mountPath: /usr/share/ca-certificates
- name: usr-share-ca-certificates
- readOnly: true
Sorting Volumes and VolumeMounts should fix this issue and help
to avoid unnecessary upgrades.
In order to improve the UX of kubeadm, it was decided to introduce the
following subcommands:
- `kubeadm config print` - this is currently only a placeholder for subcommands
that deal printing of some kind of configuration.
- `kubeadm config print init-defaults` - prints the default combination of
InitConfiguration and ClusterConfiguration. Selected component configs can be
printed too if the `--component-configs` command line switch is used.
- `kubeadm config print join-defaults` - prints the default JoinConfiguration.
This command also supports the use of `--component-configs`.
- `kubeadm config print-defaults` is deprecated in favor of
`kubeadm config print init/join-defaults`.
Signed-off-by: Rostislav M. Georgiev <rostislavg@vmware.com>
When doing upgrades kubeadm generates new manifest and
waits until kubelet restarts correspondent pod.
However, kubelet won't restart pod if there are no changes
in the manifest. That makes kubeadm stuck waiting for
restarted pod.
Skipping upgrade if new component manifest is the same as
current manifest should solve this.
Fixes: kubernetes/kubeadm#1054
This change splits out discovery fields from JoinConfiguration by performing
the following changes:
- Introduce a BootstrapTokenDiscovery structure, that houses configuration
options needed for bootstrap token based discovery.
- Introduce a FileDiscovery structure, that houses configuration options
(currently only a single option) needed for KubeConfig based discovery.
- Introduce a Discovery structure, that houses common options (such as
discovery timeout and TLS bootstrap token) as well as pointer to an instance
of either BootstrapTokenDiscovery or FileDiscovery structures.
- Replace the old discovery related JoinConfiguration members with a single
Discovery member.
This change is required in order to cleanup the code of unnecessary logic and
make the serialized JoinConfiguration more structured (and therefore, more
intuitive).
Signed-off-by: Rostislav M. Georgiev <rostislavg@vmware.com>
It makes more sense to use advertised etcd endpoints instead of using
listened client URLS.
I'm solving this use case: Listen on localhost and public IP but I don't
want to add multiple endpoints to kube-apiservers because it's targeting
same etcd server.
Without this patch SetKubernetesVersion does not set
a version in all locations where it's called, because
the passed config to it always has a default version
stable-1.
This always triggers the != "" check and the function
returns without setting a version.
Validate against DefaultKubernetesVersion and "" instead.
This fixes all cases where fetching a version from the internet
is not needed at all - e.g. "kubeadm token create".
Also make SetKubernetesVersion default to version.Get().String()
and add unit tests for the function.
kubeadm token create --print-join-command --config <path>
fails with Error: can not mix '--config' with arguments [print-join-command]
As print-joint command can't be put into the configuration file it
should be possible to use those 2 options at the same time.
Added print-join-command options to the list of exceptions in
ValidateMixedArguments check to allow its usage with --config option.
Fixes: kubernetes/kubeadm#1166
This change allows mixing InitConfiguration/ClusterConfiguration with
JoinConfiguration in a single YAML file, by performing the following changes:
- Replace the explicit error in `DetectUnsupportedVersion` with a warning.
- Modify `NodeConfigFileAndDefaultsToInternalConfig` to use only
`JoinConfiguration`.
Signed-off-by: Rostislav M. Georgiev <rostislavg@vmware.com>
Individual implementations are not yet being moved.
Fixed all dependencies which call the interface.
Fixed golint exceptions to reflect the move.
Added project info as per @dims and
https://github.com/kubernetes/kubernetes-template-project.
Added dims to the security contacts.
Fixed minor issues.
Added missing template files.
Copied ControllerClientBuilder interface to cp.
This allows us to break the only dependency on K8s/K8s.
Added TODO to ControllerClientBuilder.
Fixed GoDeps.
Factored in feedback from JustinSB.
The default version in kubeadm is now `stable-1`. This will
pull a version from the `stable-1.txt` endpoint which might
end up being newer than the version of the client by a magnitude
of MINOR or even a MAJOR release.
To be able to prevent this scenario add the new helper function:
validateStableVersion()
This function determines if the remote version is newer than the
local client version and if that's the case it returns `stable-X.xx`
that conforms with the version of the client. If not it returns
the remote version.
If 'kubeadm join' is run with --discovery-file option it still
prompts for bootstrap token. This makes it impossible to use
existing client certificates mentioned in the discovery file.
Fixed by checking for empty bootstrap token only if discovery file
is not specified.
Fixes: kubernetes/kubeadm#1142
A previous commit updated 17.xx -> 18.xx without considering
that some systems still use 17.xx.
Add handling for 17.0[3,6,9] as validated versions.
Also re-format the error message because the version is not validated
per maximum basis, but rather based on existing validation.
This will allow for kube-proxy to be run without `privileged` and
with only adding the capability `NET_ADMIN`.
Signed-off-by: Jess Frazelle <acidburn@microsoft.com>
As kube-dns transitioned to fat manifests, it's no longer required to use arch
suffixed images. This change makes use of fat manifests for kube-dns and
removes the last few calls to the GetGenericArchImage function, thus removing
GetGenericArchImage too.
Signed-off-by: Rostislav M. Georgiev <rostislavg@vmware.com>
This fixes an issue where SRV records were incorrectly being compressed.
Also updated kubedns version for kubeadm
Upgrade to 1.14.12 with manifest support. Runs dnsmasq version 2.78
Remove the test called TestNewCmdReset in cmd/reset_test.go, since it
can prompt for root password and it then calls an actual reset
and shuts down the kubelet.
Adding blank line between comment tag and package name in doc.go. So
that the comment tags such as '+k8s:deepcopy-gen=package' do not show up
in GoDoc.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
kubeadm: add mandatory phase "annotate-cri"
**What this PR does / why we need it**:
In v1.12, after the changing how kubeadm-config upload/fetch, AnnotateCRISocket is now a necessary step for getting a cluster that can be upgraded using phases. This PR adds a new subcommand for letting users to invoke AnnotateCRISocket as a phase
**Which issue(s) this PR fixes**:
Fixes # https://github.com/kubernetes/kubeadm/issues/925
**Special notes for your reviewer**:
When testing this PR, I found that one of the tests on component config defaulting was broken (didn't have time to investigate why yet). This PR fixes this test as well
**Release note**:
```release-note
kubeadm: added phase command "alpha phase kubelet config annotate-cri"
```
@kubernetes/sig-cluster-lifecycle-pr-reviews
/sig cluster-lifecycle
/area kubeadm
/assign @timothysc @neolit123
/kind bug
Add the:
- mandatory flag "--config" to the preflight phase
and parse the specified config file for either "master" or "node".
- flag "--ignore-preflight-errors" to the preflight phase to
allow skipping errors.
- the function AddIgnorePreflightErrorsFlag()
to "options/generic.go", because the flag is used in multiple commands.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Add a completely populated yaml InitConfig to the v1lapha3 dodoc
**What this PR does / why we need it**:
The existing docs are located in [k/website][website], which means it's difficult to keep them up to date. Having an example next to the code will make it much easier to update the docs in lockstep with spec changes.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixeskubernetes/kubeadm#1085
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 65250, 68241). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Use informer cache instead of active pod gets in HPA controller.
**What this PR does / why we need it**:
Use informer cache instead of active pod gets in HPA controller.
**Which issue(s) this PR fixes**:
Fixes#68217
**Release note**:
```release-note
kube-controller-manager: use informer cache instead of active pod gets in HPA controller
```
Automatic merge from submit-queue (batch tested with PRs 68341, 68385). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Fix kubeadm token list
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #https://github.com/kubernetes/kubeadm/issues/1094
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue (batch tested with PRs 67950, 68195). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Consolidate componentconfig code standards
**What this PR does / why we need it**:
This PR fixes a bunch of very small misalignments in ComponentConfig packages:
- Add sane comments to all functions/variables in componentconfig `register.go` files
- Make the `register.go` files of componentconfig pkgs follow the same pattern and not differ from each other like they do today.
- Register the `openapi-gen` tag in all `doc.go` files where the pkg contains _external_ types.
- Add the `groupName` tag where missing
- Fix cases where `addKnownTypes` was registered twice in the `SchemeBuilder`
- Add `Readme` and `OWNERS` files to `Godeps` directories if missing.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
/assign @sttts @thockin
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Update default etcd server to 3.2.24 for kubernetes 1.12
**What this PR does / why we need it**:
Update default etcd server to 3.2.24 for kubernetes 1.12
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
xref #68147
**Special notes for your reviewer**:
NONE
**Release note**:
```
Update default etcd server to 3.2.24 for kubernetes 1.12
```
/assign @wojtek-t @jpbetz @dims
/cc @kubernetes/sig-cluster-lifecycle-pr-reviews @gyuho
Automatic merge from submit-queue (batch tested with PRs 68087, 68256, 64621, 68299, 68296). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Fixes using externally managed certs for kubeadm
**What this PR does / why we need it**:
The certificates overhaul caused a regression when using external certificates. This fixes that issue so external CAs no longer require a key if all certificates exist.
Walk the certificate tree, at each step checking for a CACert.
If the CACert is found, try to use it to generate certificates.
Otherwise, generate a new CA cert.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixeskubernetes/kubeadm#918
**Special notes for your reviewer**:
**Release note**:
```release-note
External CAs can now be used for kubeadm with only a certificate, as long as all required certificates already exist.
```
This changes the custom metrics client logic over to support multiple versions
of the custom metrics API by checking discovery to find the appropriate versions.
Fixes#68011
Co-authored-by: Solly Ross <sross@redhat.com>
Walk the certificate tree, at each step checking for a CACert.
If the CACert is found, try to use it to generate certificates.
Otherwise, generate a new CA cert.
Technically we don't need this. the instruction below:
```
// +k8s:conversion-gen=k8s.io/kubernetes/pkg/controller/apis/config/v1alpha1
```
registers the apiserver/apimachinery packages in the "package universe"
of the conversion-gen program per comment from lucas in PR 68233
However it looks like some files that use both packages run into trouble
and causes failures in CI harness. Attempting here to see if we fix the
order by specifying them explicitly helps.
Change-Id: I20e9c9256f0b7ffdf4e2101d0ca1fe5090e51344
Automatic merge from submit-queue (batch tested with PRs 68171, 67945, 68233). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Move the CloudControllerManagerConfiguration to an API group in `cmd/`
**What this PR does / why we need it**:
This PR is the last piece of https://github.com/kubernetes/kubernetes/issues/67233.
It moves the `CloudControllerManagerConfiguration` to its own `cloudcontrollermanager.config.k8s.io` config API group, but unlike the other components this API group is "private" (only available in `k8s.io/kubernetes`, which limits consumer base), as it's located entirely in `cmd/` vs a staging repo.
This decision was made for now as we're not sure what the story for the ccm loading ComponentConfig files is, and probably a "real" file-loading ccm will never exist in core, only helper libraries. Eventually the ccm will only be a library in any case, and implementors will/can use the base types the ccm library API group provides. It's probably good to note that there is no practical implication of this change as the ccm **cannot** read ComponentConfig files. Hencec the code move isn't user-facing.
With this change, we're able to remove `pkg/apis/componentconfig`, as this was the last consumer. That is hence done in this PR as well (so the move is easily visible in git, vs first one "big add" then a "big remove"). The only piece of code that was used was the flag helper structs, so I moved them to `pkg/util/flag` that I think makes sense for now.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
ref: kubernetes/community#2354
**Special notes for your reviewer**:
This PR builds on top of (first two commits, marked as `Co-authored by: @stewart-yu`) https://github.com/kubernetes/kubernetes/pull/67689
**Release note**:
```release-note
NONE
```
/assign @liggitt @sttts @thockin @stewart-yu
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Add yue9944882 as subproject reviewer (core admission/apiserver)
extend reviewer bandwidth 😃am a super careful reviewer
i had contributed series of refactors for core admission controllers and apiserver launch flow.
/assign @deads2k
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 67709, 67556). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Fix volume scheduling issue with pod affinity and anti-affinity
**What this PR does / why we need it**:
The previous design of the volume scheduler had volume assume + bind done before pod assume + bind. This causes issues when trying to evaluate future pods with pod affinity/anti-affinity because the pod has not been assumed while the volumes have been decided.
This PR changes the design so that volume and pod are assumed first, followed by volume and pod binding. Volume binding waits (asynchronously) for the operations to complete or error. This eliminates the subsequent passes through the scheduler to wait for volume binding to complete (although pod events or resyncs may still cause the pod to run through scheduling while binding is still in progress). This design also aligns better with the scheduler framework design, so will make it easier to migrate in the future.
Many changes had to be made in the volume scheduler to handle this new design, mostly around:
* How we cache pending binding operations. Now, any delayed binding PVC that is not fully bound must have a cached binding operation. This also means bind API updates may be repeated.
* Waiting for the bind operation to fully complete, and detecting failure conditions to abort the bind and retry scheduling.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#65131
**Special notes for your reviewer**:
**Release note**:
```release-note
Fixes issue where pod scheduling may fail when using local PVs and pod affinity and anti-affinity without the default StatefulSet OrderedReady pod management policy
```
Automatic merge from submit-queue (batch tested with PRs 63011, 68089, 67944, 68132). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Kubeadm upload and fetch of kubeam config v1alpha3
**What this PR does / why we need it**:
This PR implements upload and fetch of kubeam config v1alpha3 from cluster.
More in detail:
In upload, `kubeadm-config` gets
- `ClusterConfiguration` (without components config which are already stored in separated ConfigMaps)
- `ClusterStatus`(initialised or updated with the API endpoint of the current node)
During fetch `InitConfiguration` is composed with:
- `ClusterConfiguration` from `kubeadm-config`
- The `APIEndpoint` of the current node from `ClusterStatus` in `kubeadm-config`
- Component configs from corresponding ConfigMaps
**Which issue(s) this PR fixes** :
refs https://github.com/kubernetes/kubeadm/issues/911, refs https://github.com/kubernetes/kubeadm/issues/963
**Special notes for your reviewer**:
In order to implement this it was necessary to extend current component config management with a new GetFromConfigMap operation. This is implemented in a separated commit "
implement component configs GetFromConfigMap".
The real change build on this (commi "upload and fetch kubeadm v1alpha3")
**Release note**:
```release-note
NONE
```
/cc @kubernetes/sig-cluster-lifecycle-pr-reviews
/sig cluster-lifecycle
/area kubeadm
/kind enhancement
/assign @luxas
/assign @timothysc
/cc @chuckha @rosti @neolit123 @liztio
Automatic merge from submit-queue (batch tested with PRs 63011, 68089, 67944, 68132). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Mark HPA REST Clients Flag as Deprecated
This marks the option to switch to Heapster for the HPA as deprecated.
It'll be removed next release when Heapster is retired.
**Release note**:
```release-note
Using the Horizontal Pod Autoscaler with metrics from Heapster is now deprecated.
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Replace scale down window
**What this PR does / why we need it**:
Replace scale down forbidden window with scale down stabilization window.
This allows scale down based on more than one sample, to avoid rapidly changing size up and down for controllers with fluctuating load.
A bit more in https://docs.google.com/document/d/1IdG3sqgCEaRV3urPLA29IDudCufD89RYCohfBPNeWIM
This PR is copy of #67771 with resolved comments.
**Release note**:
```release-note
Replace scale down forbidden window with scale down stabilization window. Rather than waiting a fixed period of time between scale downs HPA now scales down to the highest recommendation it during the scale down stabilization window.
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Add validation for kube-scheduler configuration options
**What this PR does / why we need it**: This adds validation to the kube-scheduler so that we're not accepting bogus values to the kube-scheduler. As requested by @bsalamat in issue https://github.com/kubernetes/kubernetes/issues/66743
**Which issue(s) this PR fixes**:
Fixes#66743
**Special notes for your reviewer**:
- Not sure if this validation is too heavy handed. Would love some feedback.
- I started working on this before I realized @islinwb was also working on this same problem... https://github.com/kubernetes/kubernetes/pull/66787 I put this PR up anyways since I'm sure good code exists in both. I wasn't aware of the /assign command so didn't assign myself before starting work.
- I didn't have time to work on adding validation to deprecated cli options. If the rest of this looks ok, I can finish that up.
- I hope the location of IsValidSocketAddr is correct. Lmk if it isn't.
**Release note**:
```release-note
Adding validation to kube-scheduler at the API level
```
Automatic merge from submit-queue (batch tested with PRs 67397, 68019). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
kubeadm: fix offline and air-gapped support
**What this PR does / why we need it**:
1.
Change the error output of getAllDefaultRoutes() so that it includes
information on which files were probed for the IP routing tables
even if such files are obvious.
Introduce a new error type which can be used to figure out of this
error is exactly of the "no routes" type.
2.
If netutil.ChooseBindAddress() fails looking up IP route tables
it will fail with an error in which case the kubeadm config
code will hard stop.
This scenario is possible if the Linux user intentionally disables
the WiFi from the distribution settings. In such a case the distro
could empty files such files as /proc/net/route and ChooseBindAddress()
will return an error.
For improved offline support, don't error on such scenarios but instead
show a warning. This is done by using the NoRoutesError type.
Also default the address to 0.0.0.0.
While doing that, prevent some commands like `init`, `join` and also
phases like `controlplane` and `certs` from using such an invalid
address.
3.
If there is no internet, label versions fail and this breaks
air-gapped setups unless the users pass an explicit version.
To work around that:
- Remain using 'release/stable-x.xx' as the default version.
- On timeout or any error different from status 404 return error
- On status 404 fallback to using the version of the client via
kubeadmVersion()
Add unit tests for kubeadmVersion().
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
refs kubernetes/kubeadm#1041
**Special notes for your reviewer**:
1st and second commits fix offline support.
3rd commit fixes air-gabbed support (as discussed in the linked issue)
the api-machinery change is only fmt.Errorf() related.
**Release note**:
```release-note
kubeadm: fix air-gapped support and also allow some kubeadm commands to work without an available networking interface
```
/cc @kubernetes/sig-cluster-lifecycle-pr-reviews
/cc @kubernetes/sig-api-machinery-pr-reviews
/assign @kad
/assign @xiangpengzhao
/area UX
/area kubeadm
/kind bug
Automatic merge from submit-queue (batch tested with PRs 65566, 67959, 68029, 68017, 67263). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Move kubelet ComponentConfig external types to `k8s.io/kubelet`
**What this PR does / why we need it**:
This PR implements most of kubernetes/community#2354 for the kubelet.
The PR:
- Moves `k8s.io/kubernetes/pkg/apis/kubeletconfig` as-is to `k8s.io/kubernetes/pkg/apis/config` as agreed
- Moves the external types to the new staging repo `k8s.io/kubelet`, in the `k8s.io/kubelet/config/v1beta1` package.
- Makes `k8s.io/kubernetes/pkg/apis/config/v1beta1` source the types from `k8s.io/kubelet/config/v1beta1`. The defaulting and conversion code is kept in this package as before.
- All references to these packages have been updated.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
ref: kubernetes/community#2354
**Special notes for your reviewer**:
This PR depends on getting https://github.com/kubernetes/kubernetes/pull/67780 merged first.
**Release note**:
```release-note
kubelet v1beta1 external ComponentConfig types are now available in the `k8s.io/kubelet` repo
```
/assign @sttts @mtaufen @liggitt
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
cloud-ctrl-mgr: enable secure port 10258
This PR enables authn+authz (delegated to the kube-apiserver) and the secure port 10258 for the cloud-controller-manager. In addition, the insecure port is disabled.
This is the counterpart PR to https://github.com/kubernetes/kubernetes/pull/64149.
Moreover, it adds integration test coverage for the `--port` and `--secure-port` flags, plus the testserver infrastructure to tests flags in general inside integration tests.
```release-note
Enable secure serving on port 10258 to cloud-controller-manager (configurable via `--secure-port`). Delegated authentication and authorization have to be configured like for aggregated API servers.
```
adding validation for componentconfig
adding validation to cmd kube-scheduler
Add support for ipv6 in IsValidSocketAddr function
updating copyright date in componentconfig/validation/validation.go
updating copyright date in componentconfig/validation/validation_test.go
adding validation for cli options
adding BUILD files
updating validate function to return []errors in cmd/kube-scheduler
ok, really returning []error this time
adding comments for exported componentconfig Validation functions
silly me, not checking structs along the way :'(
refactor to avoid else statement
moving policy nil check up one function
rejigging some deprecated cmd validations
stumbling my way around validation slowly but surely
updating according to review from @bsalamat
- not validating leader election config unless leader election is enabled
- leader election time values cannot be zero
- removing validation for KubeConfigFile
- removing validation for scheduler policy
leader elect options should be non-negative
adding test cases for renewDeadline and leaseDuration being zero
fixing logic in componentconfig validation 😅
removing KubeConfigFile reference from tests as it was removed in master
2ff9bd6699
removing bogus space after var assignment
adding more tests for componentconfig based on feedback
making updates to validation because types were moved on master
update bazel build
adding validation for staging/apimachinery
adding validation for staging/apiserver
adding fieldPaths for staging validations
moving staging validations out of componentconfig
updating test case scenario for staging/apimachinery
./hack/update-bazel.sh
moving kube-scheduler validations from componentconfig
./hack/update-bazel.sh
removing non-negative check for QPS
resourceLock required
adding HardPodAffinitySymmetricWeight 0-100 range to cmd flag help section
Automatic merge from submit-queue (batch tested with PRs 64283, 67910, 67803, 68100). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
CSI Cluster Registry and Node Info CRDs
**What this PR does / why we need it**:
Introduces the new `CSIDriver` and `CSINodeInfo` API Object as proposed in https://github.com/kubernetes/community/pull/2514 and https://github.com/kubernetes/community/pull/2034
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes https://github.com/kubernetes/features/issues/594
**Special notes for your reviewer**:
Per the discussion in https://groups.google.com/d/msg/kubernetes-sig-storage-wg-csi/x5CchIP9qiI/D_TyOrn2CwAJ the API is being added to the staging directory of the `kubernetes/kubernetes` repo because the consumers will be attach/detach controller and possibly kubelet, but it will be installed as a CRD (because we want to move in the direction where the API server is Kubernetes agnostic, and all Kubernetes specific types are installed).
**Release note**:
```release-note
Introduce CSI Cluster Registration mechanism to ease CSI plugin discovery and allow CSI drivers to customize Kubernetes' interaction with them.
```
CC @jsafrane
Automatic merge from submit-queue (batch tested with PRs 64283, 67910, 67803, 68100). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Kubeadm Cert Renewal
**What this PR does / why we need it**:
adds explicit support for renewal of certificates via command
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixeskubernetes/kubeadm#206
**Special notes for your reviewer**:
The targeted documentation is at kubernetes/website#9712
**Release note**:
```release-note
Adds the commands `kubeadm alpha phases renew <cert-name>`
```
This is the old behaviour and we did not intent to change it due to enabled authn/z in general.
As the kube-apiserver this sets the "system:unsecured" user info.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Change CPU sample sanitization in HPA.
**What this PR does / why we need it**:
Change CPU sample sanitization in HPA.
Ignore samples if:
- Pod is beeing initalized - 5 minutes from start defined by flag
- pod is unready
- pod is ready but full window of metric hasn't been colected since
transition
- Pod is initialized - 5 minutes from start defined by flag:
- Pod has never been ready after initial readiness period.
**Release notes:**
```release-note
Improve CPU sample sanitization in HPA by taking metric's freshness into account.
```
Automatic merge from submit-queue (batch tested with PRs 67756, 64149, 68076, 68131, 68120). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Update manifest and version for CoreDNS
**What this PR does / why we need it**:
Updates the manifest of CoreDNS and also bumps the version of CoreDNS to 1.2.2
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes https://github.com/kubernetes/kubernetes/issues/68020
**Special notes for your reviewer**:
**Release note**:
```release-note
CoreDNS is now v1.2.2 for Kubernetes 1.12
```
Automatic merge from submit-queue (batch tested with PRs 67756, 64149, 68076, 68131, 68120). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
kube-ctrl-mgr: enable secure port 10257
This PR enables authn+authz (delegated to the kube-apiserver) and the secure port 10257 for the kube-controller-manager. In addition, the insecure port is disabled.
Moreover, it adds integration test coverage for the `--port` and `--secure-port` flags, plus the testserver infrastructure to tests flags in general inside integration tests.
```release-note
Enable secure serving on port 10257 to kube-controller-manager (configurable via `--secure-port`). Delegated authentication and authorization have to be configured like for aggregated API servers.
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Add function comment to fix golint error in cmd/kube-scheduler/app.
**What this PR does / why we need it**:
Add function comment to fix golint error in cmd/kube-scheduler/app.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
NONE
**Special notes for your reviewer**:
NONE
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Refactor admission webhook client code to a apiserver/pkg/util package
As part of #67006 This refactoring enable us to share code between admission webhooks and CRD conversion webhooks.
@deads2k @lavalamp @sttts @kubernetes/sig-api-machinery-misc
Ignore samples if:
- Pod is beeing initalized - 5 minutes from start defined by flag
- pod is unready
- pod is ready but full window of metric hasn't been colected since
transition
- Pod is initialized - 5 minutes from start defined by flag:
- Pod has never been ready after initial readiness period.
This is the old behaviour and we did not intent to change it due to enabled authn/z in general.
As the kube-apiserver this sets the "system:unsecured" user info.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
kubeadm: Drop arch suffixes
**What this PR does / why we need it**:
This change removes arch suffixes from control plane images (etcd, kube-apiserver, kube-scheduler, kube-proxy, etc.). These are not needed, as almost all control plane images have a fat manifest now. It also adds a missing unit test for `GetGenericImage`
We have arch suffixes only for kube-dns images now.
Sample output of `kubeadm config images pull` with this change:
```
$ ./_output/bin/kubeadm config images pull
[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.11.1
[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.11.1
[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.11.1
[config/images] Pulled k8s.gcr.io/kube-proxy:v1.11.1
[config/images] Pulled k8s.gcr.io/pause:3.1
[config/images] Pulled k8s.gcr.io/etcd:3.2.18
[config/images] Pulled k8s.gcr.io/coredns:1.1.3
$ ./_output/bin/kubeadm config images pull --feature-gates=CoreDNS=false
[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.11.1
[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.11.1
[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.11.1
[config/images] Pulled k8s.gcr.io/kube-proxy:v1.11.1
[config/images] Pulled k8s.gcr.io/pause:3.1
[config/images] Pulled k8s.gcr.io/etcd:3.2.18
[config/images] Pulled k8s.gcr.io/k8s-dns-kube-dns-amd64:1.14.10
[config/images] Pulled k8s.gcr.io/k8s-dns-sidecar-amd64:1.14.10
[config/images] Pulled k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.14.10
```
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Refs kubernetes/kubeadm#1030kubernetes/kubeadm#51
**Special notes for your reviewer**:
/cc @kubernetes/sig-cluster-lifecycle-pr-reviews
/area kubeadm
/kind feature
/assign @luxas
/assign @timothysc
/assign @dims
/assign @neolit123
**Release note**:
```release-note
kubeadm: Control plane images (etcd, kube-apiserver, kube-proxy, etc.) don't use arch suffixes. Arch suffixes are kept for kube-dns only.
```
Automatic merge from submit-queue (batch tested with PRs 67694, 64973, 67902). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
SCTP support implementation for Kubernetes
**What this PR does / why we need it**: This PR adds SCTP support to Kubernetes, including Service, Endpoint, and NetworkPolicy.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#44485
**Special notes for your reviewer**:
**Release note**:
```release-note
SCTP is now supported as additional protocol (alpha) alongside TCP and UDP in Pod, Service, Endpoint, and NetworkPolicy.
```
This change removes arch suffixes from control plane images (etcd,
kube-apiserver, kube-scheduler, kube-proxy, etc.). These are not needed, as
almost all control plane images have a fat manifest now.
We have arch suffixes only for kube-dns images now.
Signed-off-by: Rostislav M. Georgiev <rostislavg@vmware.com>
Automatic merge from submit-queue (batch tested with PRs 64597, 67854, 67734, 67917, 67688). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Move kubeproxy ComponentConfig external types to `k8s.io/kube-proxy`
**What this PR does / why we need it**:
This PR implements most of kubernetes/community#2354 for the kube-proxy.
The PR:
- Moves k8s.io/kubernetes/pkg/proxy/apis/kubeproxyconfig as-is to k8s.io/kubernetes/pkg/proxy/apis/config as agreed
- Moves the external types to the new staging repo k8s.io/kube-proxy, in the k8s.io/kube-proxy/config/v1beta1 package.
- Makes k8s.io/kubernetes/pkg/proxy/apis/config/v1beta1 source the types from k8s.io/kube-proxy/config/v1beta1. The defaulting and conversion code is kept in this package as before.
- All references to these packages have been updated.
Ref #67233
**Special notes for your reviewer**:
**Release note**:
```release-note
kube-proxy v1beta1 external ComponentConfig types are now available in the `k8s.io/kube-proxy` repo
```
Automatic merge from submit-queue (batch tested with PRs 64597, 67854, 67734, 67917, 67688). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Remove unused kubelet dependency
**What this PR does / why we need it**:
Cleanup unused `ExternalKubeClient`. I think it's a vestige of when the Kubelet still used the internal types.
**Release note**:
```release-note
NONE
```
/kind cleanup
/sig node
/assign @mtaufen
Automatic merge from submit-queue (batch tested with PRs 66085, 66052). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
use pause image with fat-manifest
What this PR does / why we need it:
Pause manifest code is merged in #57723, so we should use new image in test.
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #
Special notes for your reviewer:
Release note:
Automatic merge from submit-queue (batch tested with PRs 67323, 66717, 67038). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix kubeadm init kernel validator display message error
Signed-off-by: Yuanbin.Chen <cybing4@gmail.com>
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixeskubernetes/kubeadm#1051
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 67894, 64097). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
HPA metrics specificity improvements
**What this PR does / why we need it**:
Improves available specificity for HPA metrics by adding metric selector fields for metrics of Pods and Objects.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Implements this KEP: https://github.com/kubernetes/community/pull/2055
**Special notes for your reviewer**:
Need to add/update tests?
**Release note**:
```release-note
Introduces autoscaling/v2beta2 and custom_metrics/v1beta2, which implement metric selectors for Object and Pods metrics, as well as allowing AverageValue targets on Objects, similar to External metrics.
```
/assign @DirectXMan12
1) Do not fail in case a bind address cannot be obtained
If netutil.ChooseBindAddress() fails looking up IP route tables
it will fail with an error in which case the kubeadm config
code will hard stop.
This scenario is possible if the Linux user intentionally disables
the WiFi from the distribution settings. In such a case the distro
could empty files such files as /proc/net/route and ChooseBindAddress()
will return an error.
For improved offline support, don't error on such scenarios but instead
show a warning. This is done by using the NoRoutesError type.
Also default the address to 0.0.0.0.
While doing that, prevent some commands like `init`, `join` and also
phases like `controlplane` and `certs` from using such an invalid
address.
Add unit tests for the new function for address verification.
2) Fallback to local client version
If there is no internet, label versions fail and this breaks
air-gapped setups unless the users pass an explicit version.
To work around that:
- Remain using 'release/stable-x.xx' as the default version.
- On timeout or any error different from status 404 return error
- On status 404 fallback to using the version of the client via
kubeadmVersion()
Add unit tests for kubeadmVersion().
Co-authored-by: Alexander Kanevskiy <alexander.kanevskiy@intel.com>
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
kubeadm: chroot to new --rootfs arg
**What this PR does / why we need it**:
This change adds a new --rootfs=path option to kubeadm, and (if
provided) chroot()s to this path before performing file operations.
This makes it possible to run the kubeadm binary from a container, but
perform remaining file operations against the host filesystem using
something like:
```
docker run -v /:/rootfs --net=host --uts=host --pid=host \
kubeadm:latest init ...
```
(Assuming something like the included `examples/kubeadm/Dockerfile` which sets CMD to `kubeadm --rootfs=/rootfs` - Edit: Dockerfile has been removed from this PR, but you get the idea)
Fixeskubernetes/kubeadm#503
**Special notes for your reviewer**:
- I'm not sure where is best to put the Dockerfile, or hook it up to the build process. Advice sought.
- The kubeadm command line arg handling was less unified than I was expecting to find. I've implemented this arg for `init` and `join`. I can add it to all the others too, if we're happy with the approach. An alternative would be to add the arg in the parent `KubeadmCommand`, possibly with a `PersistantFlag` - then it would automatically exist for all kubeadm subcommands.
- It would be slightly preferable if we could order `--rootfs` _before_ the subcommand so we could apply the arg automatically with `ENTRYPOINT ["kubeadm", "--rootfs=/rootfs"]`. This would be the only such flag in `kubeadm` however, so I have not implemented it that way atm. (Another alternative would be an env var)
**Release note**:
```release-note
Adds a new EXPERIMENTAL `--rootfs` flag to kubeadm, which (if specified) causes kubeadm to chroot before performing any file operations. This is expected to be useful when setting up kubernetes on a different filesystem, such as invoking kubeadm from docker.
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
kubeadm config add support for more than one APIEndpoint
**What this PR does / why we need it**:
This PR completes the changes in kubeadm for management of more than one control plane instances introducing the possibility to configure more than one APIEndpoints
**Which issue(s) this PR fixes** :
refs https://github.com/kubernetes/kubeadm/issues/911, refs https://github.com/kubernetes/kubeadm/issues/963
**Special notes for your reviewer**:
Depends on:
- [x] https://github.com/kubernetes/kubernetes/pull/67830
**Release note**:
```release-note
kubeadm: The kubeadm configuration now support definition of more than one control plane instances with their own APIEndpoint. The APIEndpoint for the "bootstrap" control plane instance should be defined using `InitConfiguration.APIEndpoint`, while the APIEndpoints for additional control plane instances should be added using `JoinConfiguration.APIEndpoint`.
```
/cc @kubernetes/sig-cluster-lifecycle-pr-reviews
/sig cluster-lifecycle
/area kubeadm
/kind api-change
/kind enhancement
/assign @luxas
/assign @timothysc
/cc @chuckha @rosti @neolit123 @liztio
The requested Service Protocol is checked against the supported protocols of GCE Internal LB. The supported protocols are TCP and UDP.
SCTP is not supported by OpenStack LBaaS. If SCTP is requested in a Service with type=LoadBalancer, the request is rejected. Comment style is also corrected.
SCTP is not allowed for LoadBalancer Service and for HostPort. Kube-proxy can be configured not to start listening on the host port for SCTP: see the new SCTPUserSpaceNode parameter
changed the vendor github.com/nokia/sctp to github.com/ishidawataru/sctp. I.e. from now on we use the upstream version.
netexec.go compilation fixed. Various test cases fixed
SCTP related conformance tests removed. Netexec's pod definition and Dockerfile are updated to expose the new SCTP port(8082)
SCTP related e2e test cases are removed as the e2e test systems do not support SCTP
sctp related firewall config is removed from cluster/gce/util.sh. Variable name sctp_addr is corrected to sctpAddr in pkg/proxy/ipvs/proxier.go
cluster/gce/util.sh is copied from master
This extends the Kubelet to create and periodically update leases in a
new kube-node-lease namespace. Based on [KEP-0009](https://github.com/kubernetes/community/blob/master/keps/sig-node/0009-node-heartbeat.md),
these leases can be used as a node health signal, and will allow us to
reduce the load caused by over-frequent node status reporting.
- add NodeLease feature gate
- add kube-node-lease system namespace for node leases
- add Kubelet option for lease duration
- add Kubelet-internal lease controller to create and update lease
- add e2e test for NodeLease feature
- modify node authorizer and node restriction admission controller
to allow Kubelets access to corresponding leases
Automatic merge from submit-queue (batch tested with PRs 67776, 67503, 67679, 67786, 67830). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
kubeadm config move ControlPlaneEndpoint to ClusterConfiguration
**What this PR does / why we need it**:
This PR moves `ControlPlaneEndpoint` from the `API` config struct to `ClusterConfiguration`.
This change is required as initial step for enabling management of more than one control plane instances in kubeadm
**Which issue(s) this PR fixes** :
refs https://github.com/kubernetes/kubeadm/issues/911, refs https://github.com/kubernetes/kubeadm/issues/963
**Special notes for your reviewer**:
just an appetizer, the main dish will be the next PR...
**Release note**:
```release-note
kubeadm: ControlPlaneEndpoint was moved from the API config struct to ClusterConfiguration
```
/cc @kubernetes/sig-cluster-lifecycle-pr-reviews
/sig cluster-lifecycle
/area kubeadm
/kind api-change
/kind enhancement
/assign @luxas
/assign @timothysc
/cc @chuckha @rosti @neolit123 @liztio
Automatic merge from submit-queue (batch tested with PRs 67776, 67503, 67679, 67786, 67830). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
kubeadm - deprecate feature-gates HighAvailability, SelfHosting, CertsInSecrets
**What this PR does / why we need it**:
As for sig discussion (see meeting notes - August 22 - 2018) we are going to block usage of feature gates HighAvailability, SelfHosting, CertsInSecrets for new clusters and block updates to v1.12 of existing clusters using such features.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)`
Fixes # https://github.com/kubernetes/kubeadm/issues/1058
**Special notes for your reviewer**:
I'm going to open issue to track code cleanup in v1.13
**Release note**:
```release-note
kubeadm - feature-gates HighAvailability, SelfHosting, CertsInSecrets are now deprecated and can't be used anymore for new clusters. Update of cluster using above feature-gates flag is not supported
```
/sig cluster-lifecycle
/kind feature
/kind cleanup
/assign @timothysc
/cc
Automatic merge from submit-queue (batch tested with PRs 67776, 67503, 67679, 67786, 67830). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
kubeadm: use ClusterConfiguration in images.go
**What this PR does / why we need it**:
This PR is the first in a series, targeting the replacement of InitConfiguration with ClusterConfiguration, when the former is not needed. Please, review only the last commit.
Replace the unnecessary use of InitConfiguration in images.go with ClusterConfiguration. This changes the interfaces of the following functions:
- GetKubeControlPlaneImage
- GetEtcdImage
- GetAllImages
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
refs kubernetes/kubeadm#963
**Special notes for your reviewer**:
/cc @kubernetes/sig-cluster-lifecycle-pr-reviews
/area kubeadm
/kind enhancement
/assign @luxas
/assign @timothysc
/assign @fabriziopandini
Depends on:
- [X] #67441
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 66916, 67252, 67794, 67619, 67328). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix HPA sample sanitization
**What this PR does / why we need it**: @mwielgus pointed out a case when HPA fails as a result of my changes to HPA algorithm:
- Have pods that use a lot of CPU during initilization, become ready right after they initialize,
- Trigger a scale up,
- When new pods become ready will will count their usage (even though it's not related to any work that needs doing),
- This triggers another scale up, even though existing pods can handle work, no problem.
The fix is:
- Use all samples for non-cpu metrics.
- Only use CPU samples if:
- Pod is ready and was started more than 2 minutes ago, or
- Pod is unready and last readiness change happened more than 10s after it was started.
Reasoning behind this in: https://docs.google.com/document/d/1UdtYedhmCxjaJIQi6hwJMY0eHQQKxlVD8lSHZC1BPOA/edit
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
**Special notes for your reviewer**:
**Release note**:
```release-note
Replace scale up forbidden window with disregarding CPU samples collected when pod was initializing.
```
Duration of initialization taint on CPU and window of initial readiness
setting controlled by flags.
Adding API violation exceptions following example of e50340ee23
Automatic merge from submit-queue (batch tested with PRs 67707, 62420). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
svcacct: support jose.OpaqueSigner and push errors to token generator creation
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add labels to kubectl OWNERS files
**What this PR does / why we need it**:
This change makes it possible to automatically add the two labels: `area/kubectl` and `sig/cli` to PRs that touch the paths in question.
this already exists for kubeadm:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/OWNERS#L17-L19
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
refs https://github.com/kubernetes/community/issues/1808
**Special notes for your reviewer**:
none
**Release note**:
```release-note
NONE
```
/area kubectl
@kubernetes/sig-cli-pr-reviews
/cc @cblecker @tpepper
Replace the unnecessary use of InitConfiguration in images.go with
ClusterConfiguration. This changes the interfaces of the following functions:
- GetKubeControlPlaneImage
- GetEtcdImage
- GetAllImages
Signed-off-by: Rostislav M. Georgiev <rostislavg@vmware.com>
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Kubeadm ha upgrade
**What this PR does / why we need it**:
This PR implements one of the actions defined by https://github.com/kubernetes/kubeadm/issues/751 (checklist form implementing HA in kubeadm). see [KEP 0015](https://github.com/kubernetes/community/blob/master/keps/sig-cluster-lifecycle/0015-kubeadm-join-master.md) for more context
With this PR, kubeadm implements a new command `kubeadm upgrade node experimental-control-plane` that managed upgrade of control plane components on a secondary control plane instance.
The entire workflow in case of HA clusters will be:
- Upgrade the control plane
- run `kubeadm upgrade apply` on a first control plane instance
- run `kubeadm upgrade node experimental-control-plane` on secondary control plane instances
- Upgrade nodes
**Special notes for your reviewer**:
/CC @timothysc @luxas @chuckha @kubernetes/sig-cluster-lifecycle-pr-reviews
**Release note**:
```
kubeadm now has the `kubeadm upgrade node experimental-control-plane` command for upgrading secondary control plane instances created with `kubeadm join --experimental-control-plane`.
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
[reissue] kubeadm: Split out ClusterConfiguration from InitConfiguration
As @luxas is not able to take care of #66219, I am reissuing the same change here. There are a few minor things added by me:
- The original PR is rebased on latest master.
- Some broken tests were fixed.
- Some TODOs were added.
- Run update-bazel and update-gofmt
Below is the text of the original PR by Lucas.
-----
**What this PR does / why we need it:**
Splits MasterConfiguration to InitConfiguration and ClusterConfiguration as outlined in the kubeadm Config KEP. InitConfiguration holds init-only information, and ClusterConfiguration holds cluster-wide information. In the internal representation InitConfiguration wraps ClusterConfiguration as a field, but in serialized format they're different YAML documents.
**Which issue(s) this PR fixes** (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
ref: kubernetes/kubeadm#911
Depends on:
- [X] #65776
- [X] #65628
- [X] #65629
- [X] #65631
- [X] #65940
- [X] #65787
- [X] #65945
- [X] #65951
- [X] #65952
**Special notes for your reviewer:**
**Release note**:
```release-note
kubeadm: InitConfiguration now consists of two structs: InitConfiguration and ClusterConfiguration
```
@kubernetes/sig-cluster-lifecycle-pr-reviews
Automatic merge from submit-queue (batch tested with PRs 63757, 67698, 67712, 67494, 67700). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix test case: invalid version should not trigger network operations
**What this PR does / why we need it**: current invalid version value in the test case triggers network operation to check it validity via `https://dl.k8s.io/`. Using incorrect semantic version will achieve same result of test case without possibility to trigger network connection.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
@neolit123
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
remove unused functions in cmd
**What this PR does / why we need it**:
/kind cleanup
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add Labels to various OWNERS files
**What this PR does / why we need it**:
Will reduce the burden of manually adding labels. Information pulled
from:
https://github.com/kubernetes/community/blob/master/sigs.yaml
Change-Id: I17e661e37719f0bccf63e41347b628269cef7c8b
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 67661, 67497, 66523, 67622, 67632). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Externalize node informers for node authz
the pull will completely externalize node authz together with #67194
ref: #66680
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 65788, 67648, 67660). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
[kubeadm] fix panic when node annotation is nil
**What this PR does / why we need it**:
kubeadm will panic, when the node annotation is nil.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
/assign @neolit123
**Release note**:
```release-note
kubeadm: Fix panic when node annotation is nil
```
Automatic merge from submit-queue (batch tested with PRs 66862, 67618). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix bazel
**What this PR does / why we need it**:
`bazel test //cmd/kubeadm/...` has been broken for a while. This is incredibly annoying to me, because I use it constantly during my workflow. This should fix it.
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue (batch tested with PRs 67596, 67520, 67605). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Cert list 2
**What this PR does / why we need it**:
Continuation of #67208. Uses the newly created declarative list of certificates kubeadm requires for the certs phase and upgrade steps.
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
add --dns-loop-detect option to dnsmasq run by kube-dns
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#67299
**Special notes for your reviewer**:
/cc @kubernetes/sig-network-pr-reviews
**Release note**:
```release-note
add --dns-loop-detect option to dnsmasq run by kube-dns
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
kube-{apiserver,ctrl-mgr}: unify into DeprecatedInsecureServingOptions
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue (batch tested with PRs 66793, 67405, 67068, 67501, 67484). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
kubeadm: Remove images from the context of kubeletFailTempl
**What this PR does / why we need it**:
Since #66658 kubeletFailTempl no longer contains any images, thus we don't need to fill them
in its context.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
None
**Special notes for your reviewer**:
/cc @kubernetes/sig-cluster-lifecycle-pr-reviews
/area kubeadm
/kind cleanup
/assign @timothysc
/assign @fabriziopandini
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 66209, 67380, 67499, 67437, 67498). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Use versioned api in kube-proxy
**What this PR does / why we need it**:
Now in kube-proxy someplace still use the internal version api, change to use versioned api.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 66209, 67380, 67499, 67437, 67498). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
[kubeadm] Make kubelet healthz port a constant
**What this PR does / why we need it**:
Make kubelet healthz port a constant
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add test for scheduler config defaults
This test will let us notice if defaults change when loading from a config file or from flags.
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Synchronous & unbatched audit log writes
**What this PR does / why we need it**:
When enabling buffered audit log file writes to reduce latency under high load, we shouldn't be batching the writes, as the large data write can have an inverse (though unpredictable) impact. Additionally, batched audit log writes should not be done asynchronously, as this just creates lock contention on the log writer.
This is a clean-ed up version of https://github.com/kubernetes/kubernetes/pull/61217
**Which issue(s) this PR fixes**
Fixes#61932
**Release note**:
```release-note
NONE
```
/sig auth
/priority important-soon
/kind bug
/milestone v1.12
kubeletFailTempl no longer contains any images, thus we don't need to fill them
in its context.
Signed-off-by: Rostislav M. Georgiev <rostislavg@vmware.com>
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
remove unused code in kubeadm/app/cmd/reset_test.go
**What this PR does / why we need it**:
remove unused code in kubeadm/app/cmd/reset_test.go
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 62441, 66702, 67254, 67421, 65309). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Add backoff for DS's pod deletion to limit fighting with kubelet failing the pod repeatedly
**What this PR does / why we need it**:
Limits consequences of DS controller on hot loop fighting with kubelet.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes https://github.com/kubernetes/kubernetes/issues/65240
**Release note**:
```release-note
DaemonSet controller is now using backoff algorithm to avoid hot loops fighting with kubelet on pod recreation when a particular DaemonSet is misconfigured.
```
TODO:
- [x] Export the backoff settings as args or constants
- [x] Add test a case
/cc @mfojtik
(Will add more folks when it's ready, to avoid spamming them.)
Automatic merge from submit-queue (batch tested with PRs 62441, 66702, 67254, 67421, 65309). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
[kubeadm] Apply means to search for existing KubeConfig files in standard locations across kubeadm
**What this PR does / why we need it**:
Apply means to search for existing KubeConfig files in standard locations across kubeadm.
Tasklist:
- [x] Use the Join of Dir and File name instead of DefaultKubeConfig
- [x] Refactor token.go::findExistingKubeConfig() into a utility in cmd/kubeadm/app/cmd/util
- [x] Use the utility in config, upgrade and some of the phases commands that use --kubeconfig
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
ref [kubernetes/kubeadm/#805](https://github.com/kubernetes/kubeadm/issues/805)
**Special notes for your reviewer**:
/assign @neolit123
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
First pass of cert list
**What this PR does / why we need it**:
Refactors the cert management code in kubeadm to be more extensible and resiliant.
This initial change doesn't change anything on the surface, and in fact appears to add a bunch of complexity. The goal here is to reduce duplication in the certs codebase, which is started in this PR by gutting the New*CertAndKey. Eventually, those functions will be removed altogether. The declarative list will also allow us to build a more explicit renewal function and command line interface and reduce much more duplication in the cert package.
**Special notes for your reviewer**:
**Release note**:
```
```
Automatic merge from submit-queue (batch tested with PRs 66884, 67410, 67229, 67409). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
kubeadm: coverage improvement about ValidateVersion function
**What this PR does / why we need it**:
In commit log :2ef8157644, remove the `MinimumVersion support` for all featuregate, but the `MinimumVersion` still as a parameter in `Feature` struct for `future featuregate`.
However,https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/features/features.go#L79-L87 still have judgement about `MinimumVersion` in `ValidateVersion` function, we also need test it.
This PR make `test coverage` about `ValidateVersion` function from `20%` to `90%`. Details as below.
Before changed, the `test coverage` are:
```
root@shap000101123:/ycj/kubernetes-community/src/k8s.io/kubernetes/cmd/kubeadm/app/features# go tool cover -func=size_coverage.out
k8s.io/kubernetes/cmd/kubeadm/app/features/features.go:71: ValidateVersion 20.0%
k8s.io/kubernetes/cmd/kubeadm/app/features/features.go:92: Enabled 100.0%
k8s.io/kubernetes/cmd/kubeadm/app/features/features.go:101: Supports 100.0%
k8s.io/kubernetes/cmd/kubeadm/app/features/features.go:111: Keys 0.0%
k8s.io/kubernetes/cmd/kubeadm/app/features/features.go:120: KnownFeatures 90.0%
k8s.io/kubernetes/cmd/kubeadm/app/features/features.go:139: NewFeatureGate 94.1%
k8s.io/kubernetes/cmd/kubeadm/app/features/features.go:171: ResolveFeatureGateDependencies 100.0%
total: (statements) 73.1%
```
After change, the `test coverage` are:
```
root@shap000101123:/ycj/kubernetes-community/src/k8s.io/kubernetes/cmd/kubeadm/app/features# go tool cover -func=size_coverage.out
k8s.io/kubernetes/cmd/kubeadm/app/features/features.go:71: ValidateVersion 90.0%
k8s.io/kubernetes/cmd/kubeadm/app/features/features.go:92: Enabled 100.0%
k8s.io/kubernetes/cmd/kubeadm/app/features/features.go:101: Supports 100.0%
k8s.io/kubernetes/cmd/kubeadm/app/features/features.go:111: Keys 0.0%
k8s.io/kubernetes/cmd/kubeadm/app/features/features.go:120: KnownFeatures 90.0%
k8s.io/kubernetes/cmd/kubeadm/app/features/features.go:139: NewFeatureGate 94.1%
k8s.io/kubernetes/cmd/kubeadm/app/features/features.go:171: ResolveFeatureGateDependencies 100.0%
total: (statements) 86.5%
```
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
@luxas @neolit123 @fabriziopandini @dixudx
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 67347, 67307, 67358, 67364, 67385). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
*-controller-manager: remove the redundancy import file
**What this PR does / why we need it**:
remove the redundancy import file
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
remove the redundancy import file @deads2k @mikedanese
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 66491, 66587, 66856, 66657, 66923). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
fix grammar error: the predicate verb should be singular.
**What this PR does / why we need it**:
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 66491, 66587, 66856, 66657, 66923). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
add space for output
**Release note**:
```release-note
NONE
```
fabriziopandini
