Automatic merge from submit-queue (batch tested with PRs 38443, 40145, 40701, 40682)
Move kubeadm etcd SELinux options from container to pod.
**What this PR does / why we need it**:
Works around a bug that surfaces in Docker 1.12+ related to the pause
container's namespace and selinux labels being transferred to the etcd
container when it runs.
At present it appears that applying selinux options to a container may
be broken, or perhaps shouldn't be supported at all. Moving these to the
pod causes all containers (including pause) to run with the correct
labels.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
Related to and partial fix for https://github.com/kubernetes/kubeadm/issues/107
This is one of several selinux related fixes in flight for upcoming releases, and newer versions of Docker. To successfully run kubeadm with selinux enforcing right now would like require a recent container-selinux build as uncovered in https://github.com/kubernetes/kubeadm/issues/107, a bugfix for the format labels in #40179, and finally this fix.
**Release note**:
```release-note
Fixed an SELinux issue in kubeadm on Docker 1.12+ by moving etcd SELinux options from container to pod.
```
Works around a bug that surfaces in Docker 1.12+ related to the pause
container's namespace and selinux labels being transferred to the etcd
container when it runs.
At present it appears that applying selinux options to a container may
be broken, or perhaps shouldn't be supported at all. Moving these to the
pod causes all containers (including pause) to run with the correct
labels.
Automatic merge from submit-queue (batch tested with PRs 40132, 39302, 40194, 40619, 40601)
fix error message in TestGetAddonImage
Signed-off-by: bruceauyeung <ouyang.qinhua@zte.com.cn>
**What this PR does / why we need it**:
1. fix incorrect error message in TestGetAddonImage
2. rename some variables to make them more reasonable
Automatic merge from submit-queue (batch tested with PRs 40132, 39302, 40194, 40619, 40601)
fix nil check and typos
**What this PR does / why we need it**:
1. nil error should be checked before defer statement.
2. fix some typos.
Signed-off-by: bruceauyeung ouyang.qinhua@zte.com.cn
Automatic merge from submit-queue
add test tokens for TestValidateToken
Signed-off-by: bruceauyeung <ouyang.qinhua@zte.com.cn>
**What this PR does / why we need it**:
1. add invalid tokens which token ID and token secret contain special characters such as `*`
2. add valid tokens which token ID and token secret contain mixed lowercase characters and numeric numbers
Automatic merge from submit-queue
fix incorrect parameter passing
Signed-off-by: bruceauyeung <ouyang.qinhua@zte.com.cn>
**What this PR does / why we need it**:
1. fix incorrect parameter passing when creating error
2. fix ineffectual assignment to err variable.
Automatic merge from submit-queue (batch tested with PRs 39538, 40188, 40357, 38214, 40195)
genericapiserver: cut off more dependencies – episode 2
Compare commit subjects.
approved based on #40363
Automatic merge from submit-queue
Wait for the self-hosted control plane during kubeadm init.
**What this PR does / why we need it**:
Currently we are completing kubeadm init while the scheduler and
controller manager are likely still not yet up. In some cases if they
will fail, they won't ever come up.
Instead wait until each pod enters running state before exiting kubeadm.
**Release note**:
```release-note
NONE
```
Currently we are completing kubeadm init while the scheduler and
controller manager are likely still not yet up. In some cases if they
will fail, they won't ever come up.
Instead wait until each pod enters running state before exiting kubeadm.
Automatic merge from submit-queue (batch tested with PRs 40303, 40316)
kubeadm: Fix up the apiserver kubelet client cert
**What this PR does / why we need it**:
I didn't notice that the apiserver serving cert was _only_ a server cert, not both a server and client cert in https://github.com/kubernetes/kubernetes/pull/40292.
This PR fixes that misconception and generates a dedicated client cert for apiserver -> kubelet communication and harmonizes the `ca.crt` path to `/etc/kubernetes/pki/ca.crt` on both master and node.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
```console
$ systemctl cat kubelet
# /lib/systemd/system/kubelet.service
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=http://kubernetes.io/docs/
[Service]
ExecStart=/usr/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10
[Install]
WantedBy=multi-user.target
# /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf **--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt** --require-kubeconfig=true"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_EXTRA_ARGS
$ curl -k https://localhost:10250/containerLogs/kube-system/weave-net-arm64-4t2xv/weave
Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy)
$ curl --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key -k https://localhost:10250/containerLogs/kube-system/weave-net-arm64-4t2xv/weave
[Works!!]
```
**Release note**:
```release-note
NONE
```
@liggitt @deads2k
Automatic merge from submit-queue
make client-go tools/auth and tools/clientcmd authoritative
These moved easily. Pure mechanical. I'll have a couple snip pulls for some of the more complicated packages. `pkg/util/homedir` was a transitive that moved with. It was only used in `kubectl` and you'd need it to be consistent with `clientcmd` loading order.
@sttts
Automatic merge from submit-queue (batch tested with PRs 38445, 40292)
kubeadm: Secure apiserver -> kubelet communication and set storage backend to etcd3
**What this PR does / why we need it**:
Switch storage backend to etcd3
Writes ca.crt down to disk so we can set `--client-ca-file=/etc/kubernetes/ca.crt` for the kubelet.
Adds --kubelet-client-{certificate,key} to the apiserver args and make it allowed to access the kubelets
In some cases the `path` dependency is switched to `filepath`
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*:
fixes https://github.com/kubernetes/kubeadm/issues/118
fixes https://github.com/kubernetes/kubeadm/issues/129
**Special notes for your reviewer**:
This PR is should make it possible to secure the apiserver -> kubelet communication.
**Release note**:
```release-note
NONE
```
@pires @mikedanese @andrewrynhard @liggitt @deads2k
Automatic merge from submit-queue (batch tested with PRs 40187, 40231, 40152)
Update client/server cert generation utilities
Limit generated cert usage to client or server use.
Automatic merge from submit-queue (batch tested with PRs 37228, 40146, 40075, 38789, 40189)
Cleanup temp dirs
So funny story my /tmp ran out of space running the unit tests so I am cleaning up all the temp dirs we create.
Automatic merge from submit-queue (batch tested with PRs 37228, 40146, 40075, 38789, 40189)
kubeadm: add optional self-hosted deployment
**What this PR does / why we need it**: add an optional self-hosted deployment type to `kubeadm`, for master components only, namely `apiserver`, `controller-manager` and `scheduler`.
**Which issue this PR fixes**: closes#38407
**Special notes for your reviewer**: /cc @aaronlevy @luxas @dgoodwin
**Release note**:
```release-note
kubeadm: add optional self-hosted deployment for apiserver, controller-manager and scheduler.
```
Automatic merge from submit-queue (batch tested with PRs 36693, 40154, 40170, 39033)
make client-go authoritative for pkg/client/restclient
Moves client/restclient to client-go and a util/certs, util/testing as transitives.
Automatic merge from submit-queue
kubeadm: init must validate or generate token before anything else.
**What this PR does / why we need it**: `kubeadm init` must validate or generate a token before anything else. Otherwise, if token validation or generation fail, one will need to run `kubeadm reset && systemctl restart kubelet` before re-running `kubeadm init`.
**Which issue this PR fixes**: fixeskubernetes/kubeadm#112
**Special notes for your reviewer**: /cc @luxas
Tested manually.
### With no token
```
$ sudo ./kubeadm init
[kubeadm] WARNING: kubeadm is in alpha, please do not use it for production clusters.
[preflight] Running pre-flight checks
[init] Using Kubernetes version: v1.5.2
[token-discovery] A token has not been provided, generating one
[certificates] Generated Certificate Authority key and certificate.
[certificates] Generated API Server key and certificate
[certificates] Generated Service Account signing keys
[certificates] Created keys and certificates in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[apiclient] Created API client, waiting for the control plane to become ready
[apiclient] All control plane components are healthy after 7.762803 seconds
[apiclient] Waiting for at least one node to register and become ready
[apiclient] First node is ready after 1.003148 seconds
[apiclient] Creating a test deployment
[apiclient] Test deployment succeeded
[token-discovery] Using token: 8321b6:a535ba541af7623c
[token-discovery] Created the kube-discovery deployment, waiting for it to become ready
[token-discovery] kube-discovery is ready after 1.003423 seconds
[addons] Created essential addon: kube-proxy
[addons] Created essential addon: kube-dns
Your Kubernetes master has initialized successfully!
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
http://kubernetes.io/docs/admin/addons/
You can now join any number of machines by running the following on each node:
kubeadm join --discovery token://8321b6:a535ba541af7623c@10.142.0.6:9898
```
### With invalid token
```
$ sudo ./kubeadm init --discovery token://12345:12345
[kubeadm] WARNING: kubeadm is in alpha, please do not use it for production clusters.
[preflight] Running pre-flight checks
[init] Using Kubernetes version: v1.5.2
[token-discovery] A token has been provided, validating [&{ID:12345 Secret:12345 Addresses:[]}]
token ["12345:12345"] was not of form ["^([a-z0-9]{6})\\:([a-z0-9]{16})$"]
```
### With valid token
```
$ sudo ./kubeadm ex token generate
cd540e:c0e0318e2f4a63b1
$ sudo ./kubeadm init --discovery token://cd540e:c0e0318e2f4a63b1
[kubeadm] WARNING: kubeadm is in alpha, please do not use it for production clusters.
[preflight] Running pre-flight checks
[init] Using Kubernetes version: v1.5.2
[token-discovery] A token has been provided, validating [&{ID:cd540e Secret:c0e0318e2f4a63b1 Addresses:[]}]
[certificates] Generated Certificate Authority key and certificate.
[certificates] Generated API Server key and certificate
[certificates] Generated Service Account signing keys
[certificates] Created keys and certificates in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[apiclient] Created API client, waiting for the control plane to become ready
[apiclient] All control plane components are healthy after 13.513305 seconds
[apiclient] Waiting for at least one node to register and become ready
[apiclient] First node is ready after 0.502656 seconds
[apiclient] Creating a test deployment
[apiclient] Test deployment succeeded
[token-discovery] Using token: cd540e:c0e0318e2f4a63b1
[token-discovery] Created the kube-discovery deployment, waiting for it to become ready
[token-discovery] kube-discovery is ready after 2.002457 seconds
[addons] Created essential addon: kube-proxy
[addons] Created essential addon: kube-dns
Your Kubernetes master has initialized successfully!
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
http://kubernetes.io/docs/admin/addons/
You can now join any number of machines by running the following on each node:
kubeadm join --discovery token://cd540e:c0e0318e2f4a63b1@10.142.0.6:9898
```
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 39911, 40002, 39969, 40012, 40009)
kubeadm: upgrade kube-dns to 1.11.0.
**What this PR does / why we need it**: See kubernetes/dns#25
**Which issue this PR fixes**: fixeskubernetes/kubeadm#121
**Special notes for your reviewer**: /cc @luxas
I know this is not the template solution you are looking for but seems to me it's important enough to do this now because of the issues it fixes.
Tested manually and it works.
`NONE`
Automatic merge from submit-queue
[kubeadm] resetting cluster should check whether docker service is active
Signed-off-by: bruceauyeung <ouyang.qinhua@zte.com.cn>
**What this PR does / why we need it**:
if not, `kubeadm reset` will fail to remove kubernetes-managed containers
Automatic merge from submit-queue
kubeadm: must lower-case token portion used in DNS label.
**What this PR does / why we need it**: In Kubernetes, DNS labels must be lower-case. `kubeadm` doesn't care when creating certain objects through the API. This PR fixes that erroneous behavior.
**Which issue this PR fixes**: fixes https://github.com/kubernetes/kubeadm/issues/104
**Special notes for your reviewer**: /cc @luxas @mikedanese @dgoodwin
Automatic merge from submit-queue (batch tested with PRs 39495, 39547)
kubeadm: updated preflight types to avoid stutter
Small change to kubeadm preflight pkg to remove stutter from preflight types PreFlightError and PreFlightCheck (now names Error and Checker).
**Release note**:
`NONE`
Automatic merge from submit-queue (batch tested with PRs 39684, 39577, 38989, 39534, 39702)
kubeadm: refactored token discovery.
**What this PR does / why we need it**: refactored `kubeadm` token discovery to follow the discovery interface and use new TLS bootstrap mechanism.
/cc @luxas @mikedanese
Automatic merge from submit-queue (batch tested with PRs 39673, 39536, 39617, 39540, 39686)
kubeadm: updated DNS deployment.
**What this PR does / why we need it**: Updates KubeDNS deployment to match upstream.
**Special notes for your reviewer**: It was tested manually by bootstrapping a new cluster, running a busybox container and making sure one could `nslookup` from within the container to find `kubernetes` and other services.
Automatic merge from submit-queue
Implemented file and HTTPS based discovery for kubeadm
**What this PR does / why we need it**:
This PR implements both file and HTTPS based discovery for `kubeadm`.
**Which issue this PR fixes**:
fixes https://github.com/kubernetes/kubeadm/issues/93
fixes https://github.com/kubernetes/kubeadm/issues/94
**Special notes for your reviewer**:
I'd like to add some tests but 'm sure `kubeconfig` loading is already covered by other tests in `clientcmd` package.
/cc @luxas @pipejakob
Automatic merge from submit-queue
kubeadm token discovery URL may not have valid input for url.Parse
**What this PR does / why we need it**:
**Which issue this PR fixes**: fixes https://github.com/kubernetes/kubeadm/issues/95
**Special notes for your reviewer**:
/cc @dgoodwin @luxas @mikedanese this is a WIP! Need a couple things:
* decide token separator, `.` or `:`
* define error handling in `token.go::37`
Automatic merge from submit-queue
kubeadm: change etcd data dir to match host path.
**What this PR does / why we need it**: Trivial change that adopts a well-known path for etcd data, following the `hostPath` defined.
**Which issue this PR fixes**: Fixes https://github.com/kubernetes/kubeadm/issues/80
**Special notes for your reviewer**:
/cc @luxas @kad
Without this change, using a non-token discovery mechanism during a join
triggers a nil pointer panic (instead of an error message about not
being implemented yet).
Automatic merge from submit-queue
Fix cloud-config name in test case
**What this PR does / why we need it**: fixes default cloud-config name in test cases for reset.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: related to kubernetes/kubeadm#75
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue
use bytes.Buffer instead of append for error string concat
**What this PR does / why we need it**:
1. in my benchmark test, `bytes.Buffer` takes much less time ( about 1:1000 ) than string append( `+=` ).
>BenchmarkAppendConcat-4 100000 151438 ns/op 578181 B/op 2 allocs/op
BenchmarkBufferSprintf-4 3000000 487 ns/op 65 B/op 3 allocs/op
BenchmarkBufferConcat-4 5000000 271 ns/op 47 B/op 1 allocs/op
the benchmark codes is here https://play.golang.org/p/LS52zGuwZN
2. in our `RunInitMasterChecks`, `RunJoinNodeChecks` there are lots of preflight checks. they may result in a huge error message. so `bytes.Buffer` can bring considerable performance enhancement in the worst of conditions.
beyond that, this PR
1. fix an exported struct comment,
1. and use `found = append( found, errs...)` instead of for loop for simplicity.
Signed-off-by: bruceauyeung <ouyang.qinhua@zte.com.cn>
Recent changes to support multiple methods for discovery meant that
"kubeadm init" no longer was sufficient and users would need to add
"--discovery token://" to achieve the same results.
Instead lets assume discovery if the user does not specify anything else
to maintain parity and the brevity of our original instructions.
Adds kubeadm subcommands to create, list, and delete bootstrap tokens.
Tokens can be created with a TTL duration, or 0 for tokens that will not
expire. The create command can also be used to specify your own token
(for use when bootstrapping masters and nodes in parallel), or update an
existing token's secret or ttl.
Marked "ex" for experimental for now as the boostrap controllers are not
yet hooked up in core.
Automatic merge from submit-queue
path.Clean paths in GlobalEnvParams and remove unnecessary path.Join
**What this PR does / why we need it**:
1. clean all paths in `GlobalEnvParams`
1. remove unnecessary path.Join call in `pki.go`
2. fix some typos and comment errors
Signed-off-by: bruceauyeung <ouyang.qinhua@zte.com.cn>
Automatic merge from submit-queue
Remove json serialization annotations from internal types
fixes#3933
Internal types should never be serialized, and including json serialization tags on them makes it possible to accidentally do that without realizing it.
fixes in this PR:
* types
* [x] remove json tags from internal types
* [x] fix references from serialized types to internal ObjectMeta
* generation
* [x] remove generated json codecs for internal types (they should never be used)
* kubectl
* [x] fix `apply` to operate on versioned object
* [x] fix sorting by field to operate on versioned object
* [x] fix `--record` to build annotation patch using versioned object
* hpa
* [x] fix unmarshaling to internal CustomMetricTargetList in validation
* thirdpartyresources
* [x] fix encoding API responses using internal ObjectMeta
* tests
* [x] fix tests to use versioned objects when checking encoded content
* [x] fix tests passing internal objects to generic printers
follow ups (will open tracking issues or additional PRs):
- [ ] remove json tags from internal kubeconfig types (`kubectl config set` pathfinding needs to work against external type)
- [ ] HPA should version CustomMetricTargetList serialization in annotations
- [ ] revisit how TPR resthandlers encoding objects
- [ ] audit and add tests for printer use (human-readable printer requires internal versions, generic printers require external versions)
- [ ] add static analysis tests preventing new internal types from adding tags
- [ ] add static analysis tests requiring json tags on external types (and enforcing lower-case first letter)
- [ ] add more tests for `kubectl get` exercising known and unknown types with all output options
Automatic merge from submit-queue (batch tested with PRs 37270, 38309, 37568, 34554)
kubeadm UX improvements for the v1.5 stable release
This PR targets the next stable kubeadm release.
It's work in progress, but please comment on it and review, since there are many changes.
I tried to group the commits logically, so you can review them separately.
Q: Why this large PR? Why not many small?
A: Because of the Submit Queue and the time it takes.
PTAL @kubernetes/sig-cluster-lifecycle
_Edit:_ This work was splitted up in three PRs in total
Automatic merge from submit-queue (batch tested with PRs 38354, 38371)
Add GetOptions parameter to Get() calls in client library
Ref #37473
This PR is super mechanical - the non trivial commits are:
- Update client generator
- Register GetOptions in batch/v2alpha1 group
Fix boilerplates, comments in the code and make the output of kubeadm more user-friendly
Start using HostPKIPath and KubernetesDir everywhere in the code, so they can be changed for real
More robust kubeadm reset code now.
Removed old glog-things from app.Run()
Renamed /etc/kubernetes/cloud-config.json to /etc/kubernetes/cloud-config since it shouldn't be a json file
Simplification of the code
Less verbose output from master/pki.go
Cleaned up dead code
Start a small logging/output framework:
- fmt.Println("[the-stage-here] Capital first letter of this message. Tell the user what the current state is")
- fmt.Printf("[the-stage-here] Capital first letter. Maybe a [%v] in the end if an error should be displayed. Always ends with \n")
- fmt.Errorf("Never starts with []. Includes a short error message plus the underlying error in [%v]. Never ends with \n")
Automatic merge from submit-queue
type HttpProxyCheck should be HTTPProxyCheck
**What this PR does / why we need it**:
Change HttpProxyCheck to HTTPProxyCheck
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
golint
**Release note**:
```release-note
```
Signed-off-by: yupeng <yu.peng36@zte.com.cn>
Automatic merge from submit-queue (batch tested with PRs 38294, 37009, 36778, 38130, 37835)
Add a cloudprovider validator flag to kubeadm and update the DNS spec
Broken out from: https://github.com/kubernetes/kubernetes/pull/37568
This PR creates a flag for `cloud-provider` that validates the value before `RunInit()` is run, which makes it now act as a "real" flag
Then it removes the `k8s.io/kubernetes/pkg/cloudprovider` dependency, which makes the binary ~40MB smaller! That's _really_ worth it!
In the second commit, the DNS addon is updated to the latest version: https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/skydns-rc.yaml.base
@kubernetes/sig-cluster-lifecycle
Automatic merge from submit-queue (batch tested with PRs 38194, 37594, 38123, 37831, 37084)
[kubeadm] Fail fast if there already is a node in the cluster with that name
kubeadm Fail fast if there is another node with the same name already in the cluster.
Fixes#36255
cc @kubernetes/sig-cluster-federation
Automatic merge from submit-queue (batch tested with PRs 35300, 36709, 37643, 37813, 37697)
Revert "[kubeadm] use iteration instead of recursion in function"
Reverts kubernetes/kubernetes#36625
Removing the recursive call means that `n` is never updated, so you never succeed in the update, and you've creating an infinite loop.
Also, this entire bit of functionality should be a patch and you won't have to worry about conflicts.
@luxas
Automatic merge from submit-queue
Remove ExportOptions from api/internal and use unversioned
Should only have one internal object in use
Part of #37530
Automatic merge from submit-queue (batch tested with PRs 37945, 37498, 37391, 37209, 37169)
Warn if firewalld service is enabled.
Fixes https://github.com/kubernetes/kubeadm/issues/21
Output will be:
```
(root@centos1 ~) $ kubeadm init
Running pre-flight checks
WARNING: firewalld is active, please ensure ports [6443 9898 10250] are open
```
I went with the port list from @errordeveloper 's ansible playbook here but it's possible there should be others listed.
Automatic merge from submit-queue (batch tested with PRs 36263, 36755, 37357, 37222, 37524)
kubeadm: Skip etcd related preflight checks and reset actions for external etcd
**What this PR does / why we need it**:
Skip etcd related preflight checks and reset actions for external etcd
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes # https://github.com/kubernetes/kubeadm/issues/69#issuecomment-262988388
**Special notes for your reviewer**:
**Release note**:
<!-- Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access)
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`.
-->
```release-note
kubeadm: Skip etcd related preflight checks and reset actions for external etcd
```
Automatic merge from submit-queue (batch tested with PRs 36263, 36755, 37357, 37222, 37524)
kubeadm: Implement support for symbolic labels in --use-kubernetes-version
**What this PR does / why we need it**: This patch adds "stable", "latest" and other labels as valid versions in "kubeadm init --use-kubernetes-version" flag.
Now, defaults can be pointing to "stable" and users will always get
latest available stable build of Kubernetes via kubeadm.
There is no need anymore to hardcode version string inside kubeadm
binary.
It is also possible to use labels like "latest" or point to exact
branch: "stable-1.4"
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
- kubeadm init --use-kubernetes-version now understands "stable","latest", "stable-1.4" and other labels as valid version. It will fetch actual build number from release servers.
```
Automatic merge from submit-queue
add failure check on umount when kubeadm reset, and on service stop
**What this PR does / why we need it**:
before this PR, `umount` will exit with code `123` if `grep` does not match anything
`xargs` has an option:
>-r, --no-run-if-empty
If the standard input does not contain any nonblanks, do not run the command. Normally, the command is run once even if
there is no input. This option is a GNU extension.
1. this PR add `-r` option to `xargs` , so `umount` will not execute and exit with code `0` correctly while `grep` does not match anything.
2. this PR add failure check on umount. for example, if the directory to be umount is busy, a error message will be printed:
>failed to unmount directories in /var/lib/kubelet, umount: /var/lib/kubelet/foo/bar: target is busy
(In some cases useful info about processes that
use the device is found by lsof(8) or fuser(1).)
3. add failure check on kubelet service stop.
Signed-off-by: bruceauyeung <ouyang.qinhua@zte.com.cn>
Automatic merge from submit-queue
Kubeadm unit tests pkg node
Added unit tests for the kubeadm/app/node package testing functionality of bootstrap.go, csr.go, and discovery.go.
This PR is part of the ongoing effort to add tests (#35025)
/cc @pires @jbeda
Automatic merge from submit-queue
change unconfined_t to spc_t
**What this PR does / why we need it**:
When installing kube via kubeadm on a system w/ selinux enabled, it's necessary to disable selinux in order for the etcd and kube-discovery containers to run.
The kube etcd and discovery pods are currently set to unconfined_t in order to avoid disabling selinux, but the correct type for an unconfined container is spc_t. For more information, see http://danwalsh.livejournal.com/2016/10/03/.
updated pkg "k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset" to "k8s.io/kubernetes/pkg/client/clientset_generated/release_1_5" because the type that bootstrap.go:126 checkAPIEndpoint requires changed as well as *internalversion.CertificatesClient in csr_test.go:69
Automatic merge from submit-queue
Add the system verification test to the kubeadm preflight checks
And refactor the system verification test to accept to write to a specific writer in order to customize the output
This PR is targeting v1.5, PTAL
cc @Random-Liu @dchen1107 @kubernetes/sig-cluster-lifecycle
Automatic merge from submit-queue
[kubeadm] use iteration instead of recursion in function
**What this PR does / why we need it**:
before this PR, function `attemptToUpdateMasterRoleLabelsAndTaints` recursively call itself. there are some defeats in it:
1. potential stack overflow.
2. unnecessary extra `json.Marshal` calls.
3. unnecessary extra `client.Nodes().List` calls.
this PR rewrite `attemptToUpdateMasterRoleLabelsAndTaints` function, use iterate instead of recursion.
so these 3 defeats the metioned above are gone.
Signed-off-by: bruceauyeung <ouyang.qinhua@zte.com.cn>
Automatic merge from submit-queue
kubeadm: changed tests to be table driven
Small change to migrate tests in kubeadm/app/util/tokens_test.go to be table driven which should make adding more tests in the future easier and also have them match how other tests are being written.
This PR is part of the ongoing effort to add tests (#35025)
Automatic merge from submit-queue
support customize repository prefix of image through environment KUBE…
## Problem
kubeadm does not support customize repository prefix of image. this prevent us from using our own image repository to deploy k8s.
## Fix
make ```gcr.io/google_containers/ ``` be configurable.
ADD environment variable KUBE_REPO_PREFIX
Signed-off-by: yaoyao.xyy <yaoyao.xyy@alibaba-inc.com>
Hardcoded known stable version will be returned if user
didn't request specific version and kubeadm for some reason
not able to fetch latest stable information from release servers.
For now, fallback version is v1.4.6
Now, defaults can be pointing to "stable" and users will always get
latest available stable build of Kubernetes via kubeadm.
There is no need anymore to hardcode version string inside kubeadm
binary.
It is also possible to use labels like "latest" or point to exact
branch: "stable-1.4"
Automatic merge from submit-queue
Update kubeadm etcd to 3.0.13 in order to switch to the etcd3 storage format
ref: https://github.com/kubernetes/kubernetes/issues/35723
I think we should switch as soon as possible, but run it in etcd2 mode until the full etcd3 mode is stable
@kubernetes/sig-cluster-lifecycle @wojtek-t @xiang90 @lavalamp
In future we might try to verify ports are actually exposed in firewalld
policy, but this can be quite complex. Instead lets just warn the user
if we see firewalld is running.
The kube etcd and discovery pods are set to unconfined_t in
order to avoid disabling selinux, but the correct type for
an unconfined container is spc_t. For more information, see
http://danwalsh.livejournal.com/2016/10/03/.
Automatic merge from submit-queue
add master address into kubeadm join help message and some validations
**What this PR does / why we need it**:
1, add master address into kubeadm join help message. looks like :
>Usage:
> kubeadm join <master address> [flags]
2, when user provides more than one master address, return an error.
3, since `kubeadm join` not only support ip addresses but also host names or domain names, so i delete the word `ip` from error message `must specify master ip address (see --help)`
Signed-off-by: bruceauyeung <ouyang.qinhua@zte.com.cn>
Automatic merge from submit-queue
New command: "kubeadm token generate"
As part of #33930, this PR adds a new top-level command to kubeadm to just generate a token for use with the init/join commands. Otherwise, users are left to either figure out how to generate a token on their own, or let `kubeadm init` generate a token, capture and parse the output, and then use that token for `kubeadm join`.
At this point, I was hoping for feedback on the CLI experience, and then I can add tests. I spoke with @mikedanese and he didn't like the original propose of `kubeadm util generate-token`, so here are the runners up:
```
$ kubeadm generate-token # <--- current implementation
$ kubeadm generate token # in case kubeadm might generate other things in the future?
$ kubeadm init --generate-token # possibly as a subcommand of an existing one
```
Currently, the output is simply the token on one line without any padding/formatting:
```
$ kubeadm generate-token
1087fd.722b60cdd39b1a5f
```
CC: @kubernetes/sig-cluster-lifecycle
**Release note**:
<!-- Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access)
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`.
-->
``` release-note
New kubeadm command: generate-token
```
We recently improved this to tolerate existence of things like
/etc/kubernetes/manifests/ as an empty dir, but forgot to do so for the
join pre-flight checks where it is also failing.
Instead ensure only the sub-directories and files we need are available.
Automatic merge from submit-queue
Avoid double decoding all client responses
Fixes#35982
The linked issue uncovered that we were always double decoding the response in restclient for get, list, update, create, and patch. That's fairly expensive, most especially for list. This PR refines the behavior of the rest client to avoid double decoding, and does so while minimizing the changes to rest client consumers.
restclient must be able to deal with multiple types of servers. Alter the behavior of restclient.Result#Raw() to not process the body on error, but instead to return the generic error (which still matches the error checking cases in api/error like IsBadRequest). If the caller uses
.Error(), .Into(), or .Get(), try decoding the body as a Status.
For older servers, continue to default apiVersion "v1" when calling restclient.Result#Error(). This was only for 1.1 servers and the extensions group, which we have since fixed.
This removes a double decode of very large objects (like LIST) - we were trying to DecodeInto status, but that ends up decoding the entire result and then throwing it away. This makes the decode behavior specific to the type of action the user wants.
```release-note
The error handling behavior of `pkg/client/restclient.Result` has changed. Calls to `Result.Raw()` will no longer parse the body, although they will still return errors that react to `pkg/api/errors.Is*()` as in previous releases. Callers of `Get()` and `Into()` will continue to receive errors that are parsed from the body if the kind and apiVersion of the body match the `Status` object.
This more closely aligns rest client as a generic RESTful client, while preserving the special Kube API extended error handling for the `Get` and `Into` methods (which most Kube clients use).
```
Automatic merge from submit-queue
kubeadm preflight checks: Warn user if connections to API or Discovery are going to be over proxy
**What this PR does / why we need it**: Continuing discussion from PR #35044, new version will provide warning if kubeadm run in environment where http connections would go over proxy.
Most of the time, it is not expected behaviour and leads to situations like in #34695
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#34695
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
kubeadm during initialization of master and slave nodes need to make
several API calls directly to the node where it is running or master.
In environments with http/https proxies, user might accidentally
have configuration where connections to API would go over proxy instead
of directly.
User can re-run kubeadm with corrected NO_PROXY variable. Example:
$ NO_PROXY=* kubeadm join ...
kubeadm during initialization of master and slave nodes need to make
several API calls directly to the node where it is running or master.
In environments with http/https proxies, user might accidentally
have configuration where connections to API would go over proxy instead
of directly.
User can re-run kubeadm with corrected NO_PROXY variable. Example:
$ NO_PROXY=* kubeadm join ...
This surfaces the token generation logic so that users can first
generate and store a token, then pass it to kubeadm init/join.
Otherwise, users have to capture and parse the output of "kubeadm init"
to feed the token to "kubeadm join."
Automatic merge from submit-queue
make kubeadm version use kubeadmutil
What this PR does / why we need it:
this PR makes sure `kubeadmutil.CheckErr()` other than `cmdutil.CheckErr()` is called in `kubeadm version` subcommand.
in `version.go`, `RunVersion()` function only returns `nil`, `kubeadmutil.CheckErr()` is enough for this
Signed-off-by: redhatlinux10 <ouyang.qinhua@zte.com.cn>
Automatic merge from submit-queue
kubeadm: added unit test for app/preflight pkg
Added unit test for kubeadm/app/preflight package testing functionality of checks.go.
This PR is part of the ongoing effort to add tests (#35025)
/cc @pires @jbeda
Automatic merge from submit-queue
[kubeadm] pre-flight check hostname to ensure kubelet can launch static pods li…
<!-- Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, read our contributor guidelines https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md and developer guide https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md
2. If you want *faster* PR reviews, read how: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/faster_reviews.md
3. Follow the instructions for writing a release note: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes
-->
**What this PR does / why we need it**: pre-flight check hostname to ensure kubelet can launch static pods like kube-apiserver/kube-controller-manager
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
# what is the influence of this issue?
kubelet will not create api server and kcm pod if your hostname is uncorrect. It complain the config files in "/etc/kubernetes/manifests" are invlid.
# how to reproduce this issue?
change your hostname by `hostnamectl set-hostname vm_81_12_centos`. then run `kubeadm init`. you will get this error log from kubelet:
```log
Oct 27 11:12:57 vm_81_12_centos kubelet: I1027 11:12:57.279458 2695 file.go:123] Can't process config file "/etc/kubernetes/manifests/kube-controller-manager.json": invalid pod: [metadata.name: Invalid value: "kube-controller-manager-vm_81_12_centos": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)* (e.g. 'example.com') spec.nodeName: Invalid value: "vm_81_12_centos": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)* (e.g. 'example.com')]
```
# where the error comes from in the code?
`pkg/kubelet/config/file.go:144 sourceFile:extractFromDir`
```go
func (s *sourceFile) extractFromDir(name string) ([]*api.Pod, error) {
dirents, err := filepath.Glob(filepath.Join(name, "[^.]*"))
if err != nil {
return nil, fmt.Errorf("glob failed: %v", err)
}
pods := make([]*api.Pod, 0)
if len(dirents) == 0 {
return pods, nil
}
sort.Strings(dirents)
for _, path := range dirents {
statInfo, err := os.Stat(path)
if err != nil {
glog.V(1).Infof("Can't get metadata for %q: %v", path, err)
continue
}
switch {
case statInfo.Mode().IsDir():
glog.V(1).Infof("Not recursing into config path %q", path)
case statInfo.Mode().IsRegular():
pod, err := s.extractFromFile(path)
if err != nil {
--> glog.V(1).Infof("Can't process config file %q: %v", path, err)
} else {
pods = append(pods, pod)
}
default:
glog.V(1).Infof("Config path %q is not a directory or file: %v", path, statInfo.Mode())
}
}
return pods, nil
}
```
# how to fix it?
1. change hostname by `hostnamectl set-hostname <right host name>` or
2. add `hostnameOverride` config. If hostnameOverride is set, then kubelet will use this value instead of system hostname.
**Release note**:
<!-- Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access)
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`.
-->
```release-note
```
…ke kube-apiserver/kube-controller-manager and so on.
Automatic merge from submit-queue
Kubeadm added unit tests for pkg app/util
Added unit tests for kubeadm/app/util package testing functionality of tokens.go, error.go, and kubeconfig.go.
This PR is part of the ongoing effort to add tests (#35025)
/cc @pires @jbeda
Previously, GetEnvParams (now called SetEnvParams) had no way of being altered unless
it was through enviroment variables. These changes allow for a global
EnvParam to be set and also altered while still initally getting their value from
set enviroment variables. This change is especially helpful for testing
(see kubeadm/app/util/kubeconfig_test.go).
Automatic merge from submit-queue
kubeadm: Stop assuming full ownership of /etc/kubernetes.
Packages may auto-create directories in /etc/kubernetes, and users also
need files such as cloud-config.json to be present and preserved at
their default locations in /etc/kubernetes. As such this modifies
pre-flight checks to only require the absence of the files and
directories we explicitly create in kubeadm.
Reset is similarly modified to not wipe out /etc/kubernetes entirely.
When resetting directories we also now preserve the directory itself,
but delete it's contents.
Also adds tests for reset command logic specifically for /etc/kubernetes
cleanup, to ensure user files are not inadvertently wiped out.
This will allow packages to maintain ownership of config and data
directories, which may carry selinux or other attributes that should be
preserved, but we do not wish to manage within kubeadm itself.
Packages may auto-create directories in /etc/kubernetes, and users also
need files such as cloud-config.json to be present and preserved at
their default locations in /etc/kubernetes. As such this modifies
pre-flight checks to only require the absence of the files and
directories we explicitly create in kubeadm.
Reset is similarly modified to not wipe out /etc/kubernetes entirely.
When resetting directories we also now preserve the directory itself,
but delete it's contents.
Also adds tests for reset command logic specifically for /etc/kubernetes
cleanup, to ensure user files are not inadvertently wiped out.
Automatic merge from submit-queue
kubeadm: added unit tests for app/images pkg
Added unit tests for kubeadm/app/images package testing functionality of images.go.
This PR is part of the ongoing effort to add tests (#35025)
/cc @pires @jbeda
Automatic merge from submit-queue
enhance join arguments generation logic using template
**What this PR does / why we need it**:
this PR enhances kubeadm join arguments generation logic using template, this makes code more readable and adding arguments more easier.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
<!-- Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access)
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`.
-->
```release-note
```
Signed-off-by: 欧阳钦华10079130 <ouyang.qinhua@zte.com.cn>
Automatic merge from submit-queue
kubeadm: Normalized reset command to match init and join commands.
**What this PR does / why we need it**: Overall, improves code structure. Opening single PR in order avoid big PRs in the future, when adding tests and new functionality, i.e. #34404.
```release-note
NONE
```
Automatic merge from submit-queue
kubeadm join: Added support for config file.
As more behavior (#34719, #34807, fix for #33641) is added to `kubeadm join`, this will be eventually very much needed. Makes sense to go in sooner rather than later.
Also references #34501 and #34884.
/cc @luxas @mikedanese
Automatic merge from submit-queue
kubeadm join: wait for API endpoints
**What this PR does / why we need it**: enhance kubeadm to allow for parallel provisioning of API endpoints and slave nodes, continued from https://github.com/kubernetes/kubernetes/pull/33543
**Fixes**: https://github.com/kubernetes/kubernetes/issues/33542
**Special notes for your reviewer**:
* Introduces a concurrent retry mechanism for bootstrapping with a single API endpoint during `kubeadm join` (this was left out in https://github.com/kubernetes/kubernetes/pull/33543 so that it can be implemented in a separate PR). The polling of the discovery service API itself is yet to come.
@errordeveloper @pires
Automatic merge from submit-queue
kubeadm implement preflight checks
Checks that user running kubeamd init and join is root and will only execute
command if user is root. Moved away from using kubectl error handling to
having kubeadm handle its own errors. This should allow kubeadm to have
more meaningful errors, exit codes, and logging for specific kubeadm use
cases.
fixes#33908
Add skip-preflight-checks to known flags.
Fix bug with preflight checks not returning system is-active as errors.
Fix error handling to use correct function.
Includes checks for verifying services exist and are enabled, ports are
open, directories do not exist or are empty, and required binaries are
in the path.
Checks that user running kubeamd init and join is root and will only execute
command if user is root. Moved away from using kubectl error handling to
having kubeadm handle its own errors. This should allow kubeadm to have
more meaningful errors, exit codes, and logging for specific kubeadm use
cases.
Includes checks for verifying services exist and are enabled, ports are
open, directories do not exist or are empty, and required binaries are
in the path.
* `kubeadm join` will now retry to connect to the discovery service
API instead of exit on first failure. Allows for parallel install.
of master and slave nodes.
Automatic merge from submit-queue
Test API more extensivelly before declaring readiness
**What this PR does / why we need it**:
It's possible that first deployment kubeadm creates will hit `deployments.extensions "kube-discovery" is forbidden: not yet ready to handle request`, which comes from NamespaceLifecycle admission controller and has something to do with cache. According to @derekwaynecarr, we need to create a namespace-scoped resource to really check for this. I didn't want to make a check with deployment of whatever comes first right now, and decided to have explicit step for this in `apiclient.go`.
**Which issue this PR fixes**: fixes#34411
**Special notes for your reviewer**: @kubernetes/sig-cluster-lifecycle
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue
kubeadm: fix conversion macros and add kubeadm to round trip testing
Tests are probably broken but I'll fix. @jbeda this probably fixes your change unless we decide we need generated deep copies or conversions.
@kubernetes/sig-cluster-lifecycle
Automatic merge from submit-queue
Do not probe so aggressively which may lead to unnecessary restarts
@errordeveloper @mikedanese PTAL
I came across a case where etcd restarted about 5-10 times because the load was very high on the machine.
The load seems to have lead to that the `etcd` container occasionally didn't respond to the probe, which caused many restart and made the whole thing even worse.
Maybe we should remove the etcd probe totally? I don't know, what do you think?
This is at least a try to loosen the limits here...
Automatic merge from submit-queue
Kubeadm: print information about certificates
Prints basic information about certificates to the user.
Example of `kubeadm init` output:
```
<master/pki> generated Certificate Authority key and certificate:
Issuer: CN=kubernetes | Subject: CN=kubernetes | CA: true
Not before: 2016-09-30 11:19:19 +0000 UTC Not After: 2026-09-28 11:19:19 +0000 UTC
Public: /etc/kubernetes/pki/ca-pub.pem
Private: /etc/kubernetes/pki/ca-key.pem
Cert: /etc/kubernetes/pki/ca.pem
<master/pki> generated API Server key and certificate:
Issuer: CN=kubernetes | Subject: CN=kube-apiserver | CA: false
Not before: 2016-09-30 11:19:19 +0000 UTC Not After: 2017-09-30 11:19:19 +0000 UTC
Alternate Names: [172.18.76.239 10.0.0.1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local]
Public: /etc/kubernetes/pki/apiserver-pub.pem
Private: /etc/kubernetes/pki/apiserver-key.pem
Cert: /etc/kubernetes/pki/apiserver.pem
<master/pki> generated Service Account Signing keys:
Public: /etc/kubernetes/pki/sa-pub.pem
Private: /etc/kubernetes/pki/sa-key.pem
```
Example of `kubeadm join` command:
```
<node/csr> received signed certificate from the API server:
Issuer: CN=kubernetes | Subject: CN=system:node:minion | CA: false
Not before: 2016-09-30 11:28:00 +0000 UTC Not After: 2017-09-30 11:28:00 +0000 UTC
```
Fixes#33642
cc @kubernetes/sig-cluster-lifecycle
Automatic merge from submit-queue
Remove glog added by mistake, start converting phase1+ TODOs to issues
**What this PR does / why we need it**:
Minor cleanup in `cmd/kubeadm/app/node/csr.go`.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue
kubeadm: default to using a private range for service subnet
We are currently using a subnet that is reserved for ISPs. Private network administrators don't control this space. Default to a subnet that private network administrators do control.
@errordeveloper @kubernetes/sig-cluster-lifecycle
Automatic merge from submit-queue
Use strongly-typed types.NodeName for a node name
We had another bug where we confused the hostname with the NodeName.
Also, if we want to use different values for the Node.Name (which is
an important step for making installation easier), we need to keep
better control over this.
A tedious but mechanical commit therefore, to change all uses of the
node name to use types.NodeName
We had another bug where we confused the hostname with the NodeName.
To avoid this happening again, and to make the code more
self-documenting, we use types.NodeName (a typedef alias for string)
whenever we are referring to the Node.Name.
A tedious but mechanical commit therefore, to change all uses of the
node name to use types.NodeName
Also clean up some of the (many) places where the NodeName is referred
to as a hostname (not true on AWS), or an instanceID (not true on GCE),
etc.
Currently the boostrap fails when a token is provided by the user
on `master init` and works when the token is generated. This is
because of a mismatch of how the token string in the kube-discovery
secret is encoded.
- start cleaning up `cmd/manual.go`
- refine progress and error messages
- add missing blank lines after the license headers
- run `gofmt -s -w`
- do not set fake cloud provider
- add a note on why we cannot remove `HostNetwork: true` from `kube-discovery` pod just yet
- taint master and use `role=master`, set tolerations and affinity for `kube-discovery`
- parametrise log-level flag for all components
Kubernetes Submit Queue
