Automatic merge from submit-queue
Build release tars using bazel
**What this PR does / why we need it**: builds equivalents of the various kubernetes release tarballs, solely using bazel.
For example, you can now do
```console
$ make bazel-release
$ hack/e2e.go -v -up -test -down
```
**Special notes for your reviewer**: this is currently dependent on 3b29803eb5, which I have yet to turn into a pull request, since I'm still trying to figure out if this is the best approach.
Basically, the issue comes up with the way we generate the various server docker image tarfiles and load them on nodes:
* we `md5sum` the binary being encapsulated (e.g. kube-proxy) and save that to `$binary.docker_tag` in the server tarball
* we then build the docker image and tag using that md5sum (e.g. `gcr.io/google_containers/kube-proxy:$MD5SUM`)
* we `docker save` this image, which embeds the full tag in the `$binary.tar` file.
* on cluster startup, we `docker load` these tarballs, which are loaded with the tag that we'd created at build time. the nodes then use the `$binary.docker_tag` file to find the right image.
With the current bazel `docker_build` rule, the tag isn't saved in the docker image tar, so the node is unable to find the image after `docker load`ing it.
My changes to the rule save the tag in the docker image tar, though I don't know if there are subtle issues with it. (Maybe we want to only tag when `--stamp` is given?)
Also, the docker images produced by bazel have the timestamp set to the unix epoch, which is not great for debugging. Might be another thing to change with a `--stamp`.
Long story short, we probably need to follow up with bazel folks on the best way to solve this problem.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 37468, 36546, 38713, 38902, 38614)
local-up-cluster additions
**What this PR does / why we need it**:
Changes to local-cluster-up: These include: 1) a simple additional help option. 2) additional error message to not being able to run `docker ps`. 3) fail faster when etcd is not found in path. Hopefully these make developing a bit more pleasant.
**Release note**:
```NONE
```
from etcd.sh split the start process into validate fucntion + start function so that the validate piece can be reused elsewhere. the up-cluster script has been changed to remove duplicate docker logic to the one used in buid-tools/common.sh and the validate etcd function is now used here.
moved docker daemon check function to util.sh and made function name changes and upstream changes.
Automatic merge from submit-queue
goflags must be after subcommand
**What this PR does / why we need it**:
if GOFLAGS is set when calling make, kubernetes will fail to build
as an example, I often have `GOFLAGS=-v` so I can have some idea of progress during compilation
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*:
no known issue
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 38277, 36361, 38452)
Add options for build container rsync optimization
KUBE_RSYNC_COMPRESS env var sets rsync compression level.
KUBE_RSYNC_GENERATED_TO_BUILD_CONTAINER env var disables rsyncing
generated files to build containers.
Why KUBE_RSYNC_COMPRESS is needed -- from rsync manual on `--compress` option (implied by non-zero `--compress-level`):
> Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection.
Use case for `KUBE_RSYNC_GENERATED_TO_BUILD_CONTAINER`: when you sometimes build stuff locally (e.g. `make WHAT=cmd/kubectl`) and sometimes do it on remote docker (`build-tools/run.sh make WHAT=cmd/hyperkube`), local builds touch generated files which causes them to be rsynced to the build data container, which may slow down the builds. Still, I'm not sure whether local->remote rsync of generated files is useful (e.g. someone may want to edit generated files for debugging purposes?), so I made not rsyncing these files an option instead of forcing such behavior.
Automatic merge from submit-queue (batch tested with PRs 37860, 38429, 38451, 36050, 38463)
[Part 2] Adding s390x cross-compilation support for gcr.io images in this repo
<!-- Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, read our contributor guidelines https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md and developer guide https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md
2. If you want *faster* PR reviews, read how: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/faster_reviews.md
3. Follow the instructions for writing a release note: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes
-->
**What this PR does / why we need it**: This PR enables s390x support to kube-dns , pause, addon-manager, etcd, hyperkube, kube-discovery etc. This PR also includes the changes due to which it can be cross compiled on x86 host architecture.
**Which issue this PR fixes#34328
**Special notes for your reviewer**: In existing file "build-tools/build-image/cross/Dockerfile" the repository mentioned for installing cross build tool chains for supporting architecture does not have a tool chain for s390x hence in my PR I am changing the repository so that it will be cross compiled for s390x.
**Release note**:
<!-- Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access)
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`.
-->
```
Allows cross compilation of Kubernetes on x86 host for s390x also enables s390x support to kube-dns , pause, addon-manager, etcd, hyperkube, kube-discovery etc
```
Automatic merge from submit-queue (batch tested with PRs 38260, 32811, 28458, 33570, 37096)
Fix support for DNS in local-up-cluster.sh
[]()
The default appears to be gce now, so this tries to spawn a service in
gce... not the desired outcome for a local test setup.
We also need to sync the kubectl config in the script with the instructions for users, since if people don't test with DNS enabled, it bitrots (which this aims to fix).
To test manually, run something like
```
sudo KUBE_ENABLE_CLUSTER_DNS=true API_HOST_IP=0.0.0.0 ./hack/local-up-cluster.sh
```
The API_HOST_IP=0.0.0.0 is required, otherwise the master is not contactable by kubedns, and the dns pod will fail readychecking on the healthz pod.
Automatic merge from submit-queue (batch tested with PRs 38111, 38121)
Prevent SIGPIPE errors causing the version check line to fail
The way we use pipe in hack/lib/etcd.sh when processing the version command can lead to
141 failues on some systems. Using a prefix of tail can prevent this.
Fixes#38109
Automatic merge from submit-queue (batch tested with PRs 37094, 37663, 37442, 37808, 37826)
Fix the PATH that install-etcd.sh tells you to use
After you run install-etcd.sh, it tells you:
etcd v3.0.14 installed. To use:
export PATH=${PATH}:/home/danw/rh/go/src/k8s.io/kubernetes/third_party/etcd
which doesn't work if you have an older etcd installed in /usr/bin:
danw@w541:kubernetes (master)> PATH=${PATH}:/home/danw/rh/go/src/k8s.io/kubernetes/third_party/etcd etcd --version
etcd Version: 2.2.5
You need to put the local etcd dir first in PATH, not last.
Automatic merge from submit-queue (batch tested with PRs 37094, 37663, 37442, 37808, 37826)
Moved gobindata, refactored ReadOrDie refs
**What this PR does / why we need it**: Having gobindata inside of test/e2e/framework prevents external projects from importing the framework. Moving it out and managing refs fixes this problem.
**Which issue this PR fixes**: fixes#37007
Automatic merge from submit-queue (batch tested with PRs 37997, 37939, 37990, 36700, 37258)
--export flag belongs to kubectl get, not kubectl
**What this PR does / why we need it**:
When running the kubectl via symlink to hyperkube, the ``--export`` flag is not delegated to `kubectl get` but to `kubectl` only. Ending with unrecognized flag.
- Remaining spaghetti untangled
- Missed bazel update and a few hardcoded refs
- New instance of framework.ReadOrDie reference removed post rebase
- Resolve new clientset rebase
- Fixed e2e/generated BUILD dep
- A space
- Missed gobindata ref in golang.sh
Automatic merge from submit-queue
Build vendored copy of go-bindata and use that in go generate step
**What this PR does / why we need it**: as the title says, uses the vendored version of `go-bindata` rather than expecting developers to `go get` it (when building outside docker).
**Which issue this PR fixes**: fixes#34067, partially addresses #36655
**Special notes for your reviewer**: we still call `go generate` far too many times:
```console
~/.../src/k8s.io/kubernetes $ which go-bindata
~/.../src/k8s.io/kubernetes $ make
+++ [1116 17:35:28] Building the toolchain targets:
k8s.io/kubernetes/hack/cmd/teststale
k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [1116 17:35:29] Generating bindata:
test/e2e/framework/gobindata_util.go
+++ [1116 17:35:30] Building go targets for linux/amd64:
cmd/libs/go2idl/deepcopy-gen
+++ [1116 17:35:35] Building the toolchain targets:
k8s.io/kubernetes/hack/cmd/teststale
k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [1116 17:35:35] Generating bindata:
test/e2e/framework/gobindata_util.go
+++ [1116 17:35:36] Building go targets for linux/amd64:
cmd/libs/go2idl/defaulter-gen
+++ [1116 17:35:41] Building the toolchain targets:
k8s.io/kubernetes/hack/cmd/teststale
k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [1116 17:35:41] Generating bindata:
test/e2e/framework/gobindata_util.go
+++ [1116 17:35:42] Building go targets for linux/amd64:
cmd/libs/go2idl/conversion-gen
+++ [1116 17:35:47] Building the toolchain targets:
k8s.io/kubernetes/hack/cmd/teststale
k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [1116 17:35:47] Generating bindata:
test/e2e/framework/gobindata_util.go
+++ [1116 17:35:48] Building go targets for linux/amd64:
cmd/libs/go2idl/openapi-gen
+++ [1116 17:35:56] Building the toolchain targets:
k8s.io/kubernetes/hack/cmd/teststale
k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [1116 17:35:56] Generating bindata:
test/e2e/framework/gobindata_util.go
```
Fixing that is a separate effort, though.
cc @sebgoa @ZhangBanger
- Fix unbound variables in hack/local-up-cluster.sh
- explicitly set KUBERNETES_PROVIDER in local-up-cluster.
The default appears to be gce now, so this tries to spawn a service
in gce... not the desired outcome for a local test setup.
- Remove stale creation of kube-system: the kube-system namespace is
built in these days.
- Pickup third party etcd automatically.
- Retab hack/local-up-cluster.sh for consistency
- Update hack-local-up's internal credentials to match the user
instructions we give (which work).
- Note the CA root requirement in local-up-cluster
- Fix a dangling comment in local-up-cluster
To test manually, run something like:
```sudo KUBE_ENABLE_CLUSTER_DNS+true API_HOST_IP=0.0.0.0
./hack/local-up-cluster.sh```
The API_HOST_IP=0.0.0.0 is required otherwise kubedns cannot contact
the master and the dns pod will fail readychecking on the healthz
container.
When running the kubectl via symlink to hyperkube, the --export is not delegated to 'kubectl get' but to 'kubectl' only.
Ending with unrecognized flag.
Automatic merge from submit-queue
Add --force to kubectl delete and explain force deletion
--force is required for --grace-period=0. --now is == --grace-period=1.
Improve command help to explain what graceful deletion is and warn about
force deletion.
Part of #34160 & #29033
```release-note
In order to bypass graceful deletion of pods (to immediately remove the pod from the API) the user must now provide the `--force` flag in addition to `--grace-period=0`. This prevents users from accidentally force deleting pods without being aware of the consequences of force deletion. Force deleting pods for resources like StatefulSets can result in multiple pods with the same name having running processes in the cluster, which may lead to data corruption or data inconsistency when using shared storage or common API endpoints.
```
Automatic merge from submit-queue
add script to check for updates to the files for generation
Adds a verify script to help people writing APIs remember to wire all the individual pieces up. Eventually it will be better to avoid this altogether, but this is a start.
@ncdc fyi
Automatic merge from submit-queue
Move Statefulset (previously PetSet) to v1beta1
**What this PR does / why we need it**: #28718
**Which issue this PR fixes** _(optional, in `fixes #<issue number>(, #<issue_number>, ...)` format, will close that issue when PR gets merged)_: fixes #
**Special notes for your reviewer**: depends on #35663 (PetSet rename)
cc @erictune @foxish @kubernetes/sig-apps
**Release note**:
``` release-note
v1beta1/StatefulSet replaces v1alpha1/PetSet.
```
--force is required for --grace-period=0. --now is == --grace-period=1.
Improve command help to explain what graceful deletion is and warn about
force deletion.
Automatic merge from submit-queue
Build: Use checksum to rsync results
If dependencies are touched, the Makefile will detect that tools like
deepcopy-gen need to be rebuilt. It will rebuild them, but if Go decides the
dependencies are not ACTUALLY updated (maybe it was a test file - known
limitation of the current Makefile), it does not touch the resulting binary.
The Makefile touches the result explicitly to mark that the dep change has
been handled. But rsync blows away that change with the older file, and
helpfully preserves the timestamp. This repeats on every build.
Now rsync ignores it if the checksum is the same. Result = faster build.
@wojtek-t @gmarek
If dependencies are touched, the Makefile will detect that tools like
deepcopy-gen need to be rebuilt. It will rebuild them, but Go decides the
dependencies are not actually updated (maybe it was a test file - known
limitation of the current Makefile) and does not touch the resulting binary.
Then the Makefile touches the result explicitly to mark thatthe dep change has
been handled. Then rsync blows away that change with the older file, and
helpfully preserves the timestamp.
Now rsync ignores it if the checksum is the same. Result = faster build.
Automatic merge from submit-queue
Retry curl when it failed during download
Fixes#34769
For now I'm adding retry only to etcd installation as I saw it recently to fail in one of my PRs. We may change it also in other places if it also happens.
Generate a kubeconfig for use with controller-manager, kubelet,
scheduler etc. This kubeconfig should use the secure https
port of the api server with appropriate ca cert for the components
to talk to api server.
With this change, one can set API_PORT=0 to completely switch off
insecure access for testing admission controllers etc.
Fixes#33375
Automatic merge from submit-queue
build: Fix version generation.
This PR provides update to version string generation procedure to handle additional case - working on branch delivered from release tag, instead of working on `-(alpha|beta).no` suffixed tag.
Reason of it is that actually if someone would start working on branch forked from release tag (`v1.4.1` for example), will add some commits and then would try to build k8s/run conformance tests - will end up with broken version string in form similar to `v1.4.1.X+YYYY` where `X` is a dictance/number of commits from base tag and `YYYY` will be first 14 characters from commit hash.
Such version - containing four dotted parts is rejected during conformance tests with error similar to:
```
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:843
Oct 11 12:38:33.726: Failed to get server version: Unable to parse server version "v1.4.1.5+somecommithash": Invalid character(s) found in patch number "1.5"
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:413
```
This PR provides a cure for this situation creating version string in form `v1.4.1-X+YYYY` which then should be readable for version validation checks.
Release note:
```release-note
Fix version string generation for local version different from release and not based on `-alpha.no` or `-beta.no` suffixed tag.
```
Automatic merge from submit-queue
Fix a typo in openapi spec generation
A typo in util.sh resulted in the wrong spec stored in source tree. The commit should be explanatory.
We weren't getting incremental builds because of new test only subpackages. Our
voodoo combo of 'go install' and 'go test -c' didn't cache things like
'test/e2e_node/services'. Add the '-i' flag to 'go test' to install test only
dependencies too.
Automatic merge from submit-queue
Use patched golang1.7.1 for cross-builds targeting darwin
This PR extends #32517 to use the patched go1.7.1 introduced by that PR to build all darwin targets (e.g. kubectl).
This is necessary because binaries built with earlier versions of Go regularly segfault on macOS Sierra (see #32999 and #33070).
This solution is somewhat hacky, but we intend to cherry-pick this to 1.4, and switching all of 1.4 to build with go1.7.1 is very high risk.
I haven't yet pushed the cross build image yet, so this will fail to build. Will test locally and update with results.
First step of fixing #33801.
cc @luxas @pwittrock @david-mcmahon @liggitt @smarterclayton @jfrazelle @Starefossen @gerred
Automatic merge from submit-queue
build kube-discovery and kubeadm with release
This will start publishing binaries from our ci builds.
@errordeveloper @dgoodwin
Automatic merge from submit-queue
Automatic generation of man pages
<!-- Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, read our contributor guidelines https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md and developer guide https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md
2. If you want *faster* PR reviews, read how: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/faster_reviews.md
3. Follow the instructions for writing a release note: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes
-->
**What this PR does / why we need it**:
Currently genman supports only kubectl command, so this PR will enable genman to generate man pages for rest of the command like - kube-apiserver, kube-controller-manager, kube-proxy, kube-scheduler, kubelet
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
N/A
**Special notes for your reviewer**:
**Release note**:
<!-- Steps to write your release note:
1. Use the release-note-* labels to set the release note state (if you have access)
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`.
-->
```release-note
```
Automatic merge from submit-queue
Add separate build process for node test.
This PR is part of https://github.com/kubernetes/kubernetes/pull/31093. However, because currently node e2e is built on `KUBE_TEST_PLATFORMS`, which includes linux/amd64, darwin/amd64, windows/amd64 and linux/arm, it caused #32251 to fail.
In fact, node e2e is running on the same node with kubelet, and it also has built-in apiserver, etcd and namespace controller. All of them are only built on `KUBE_SERVER_PLATFORMS`, so node e2e should also only be built on those platforms.
```
KUBE_SERVER_PLATFORMS=(
linux/amd64
linux/arm
linux/arm64
)
```
This PR added a separate build process for node e2e to address this.
@vishh Do you need this for v1.4? because this blocks your #32251. /cc @dchen1107
Jordan Liggitt