Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
Add ability to build with runtime coverage instrumentation
**What this PR does / why we need it**:
This PR adds the ability to instrument a subset of kubernetes binaries to report code coverage information. The specific use-case is to help determine coverage of our end-to-end Conformance tests, as well as provide data that can be used to help determine where to focus. This PR focuses on making it possible to build with instrumentation; collecting and using the generated coverage data will be done in later PRs. For more details as to the intent, see the [design doc](https://docs.google.com/document/d/1FKMBFxz7vtA-6ZgUkA47F8m6yR00fwqLcXMVJqsHt0g/edit?usp=sharing) (google doc; requires kubernetes-dev membership).
Specifically, this PR adds a new `KUBE_BUILD_WITH_COVERAGE` make variable, which when set will cause `kube-apiserver`, `kube-controller-manager`, `kube-scheduler`, `kube-proxy` and `kubelet` to be built with coverage instrumentation. These coverage-instrumented binaries will flush coverage information to disk every five seconds, defaulting to a temporary directory unless the `KUBE_COVERAGE_FILE` environment variable is set at launch, in which case it will write to that file instead.
The mechanism used to achieve coverage instrumentation is to build the targeted binaries as "unit tests" with coverage enabled, and then rigging the unit tests to just execute the binary's usual entry point. This is implemented only for the bash build system.
/sig testing
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 67571, 67284, 66835, 68096, 68152). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.
add make targets for building server images
**What this PR does / why we need it**: Adds `make release-images` and `make quick-release-images`, which allow building the docker-wrapped images without building a full release. Without these you can either use `make {quick}-release` and build test tarballs etc, or hack around in the build system yourself. Using this can be considerably faster if you just want to build the binaries and images, and not the release tarballs etc.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
/sig release
Automatic merge from submit-queue (batch tested with PRs 65247, 63633, 67425). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
test: allow etcd to run on other ports.
running `make test-integration` with the `ETCD_PORT` option set should
work. Numerous integration test expect the correct URL to be populated
in the `KUBE_INTEGRATION_ETCD_URL` environment variable.
**What this PR does / why we need it**:
This makes the ETCD_PORT option work correctly.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
**Special notes for your reviewer**:
I expect nobody is using the ETCD_PORT option, but I noticed that it is broken.
**Release note**:
```release-note
NONE
```
/sig testing
Automatic merge from submit-queue (batch tested with PRs 65561, 67109, 67450, 67456, 67402). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Better error message if etcd not installed
**What this PR does / why we need it**:
Add an error message to install etcd if command check fails instead of just saying to add etcd to PATH.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
First version of this was only standalone, now the code is being
vendored by anyone who wants to use it. So the standalone binary and
container are no longer useful.
Change-Id: Ib9369de66b4ecb3451f73ba2a252526d6615b96f
Automatic merge from submit-queue (batch tested with PRs 66284, 66690). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Exit gce kube-up.sh early if openssl is LibreSSL
**What this PR does / why we need it**:
macOS has an openssl binary, but it's actually LibreSSL, which doesn't play well with the easyrsa script that cluster/gce/util.sh uses to generate certs
Instead of waiting until we generate certs to discover easyrsa doesn't work, consider openssl a prereq for gce, and include a check for the version string starting with OpenSSL
Also, mirror kube-up.sh's "... calling" output in kube-down.sh
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixeskubernetes/community#1954
```release-note
NONE
```
We are not saving the etcd logs and just redirecting the output
to /dev/null. In this change, we set ETCD_LOGFILE to the same
directory where we log other kube relates processes.
Change-Id: I719a941665b749b864f2efdcdfc7ddc7b2a61776
macOS has an openssl binary, but it's actually LibreSSL, which
doesn't play well with the easyrsa script that cluster/gce/util.sh
uses to generate certs
Instead of waiting until we generate certs to discover easyrsa doesn't
work, consider openssl a prereq for gce, and include a check for the
version string starting with OpenSSL
Also, mirror kube-up.sh's "... calling" output in kube-down.sh
running `make test-integration` with the `ETCD_PORT` option set should
work. Numerous integration test expect the correct URL to be populated
in the `KUBE_INTEGRATION_ETCD_URL` environment variable.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Update all script shebangs to use /usr/bin/env interpreter instead of /bin/interpreter
This is required to support systems where bash doesn't reside in /bin (such as NixOS, or the *BSD family) and allow users to specify a different interpreter version through $PATH manipulation.
https://www.cyberciti.biz/tips/finding-bash-perl-python-portably-using-env.html
```release-note
Use /usr/bin/env in all script shebangs to increase portability.
```
Automatic merge from submit-queue (batch tested with PRs 63046, 62925, 63014). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Bump minimum required go version to 1.10.1
**What this PR does / why we need it**:
We have supported go1.10.x for a few weeks now (#60597). CI has all been updated. This enforces it as the new minimum go version required go k8s >=1.11
**Release note**:
No release note, as #60597 already had one.
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 62495, 63003, 62829, 62151, 62002). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Build cleanup with go 1.10
Simplify our build/test scripts now that go 1.10 is in play. It has caching that seems to actually work.
**Release note**:
```release-note
NONE
```
As of go 1.10, we do not need to explicitly build the stdlib for static
any more. That happens automatically. Timing tests show no discernible
difference between a `go install` with and without it.
Also use 'static' suffix instead of 'cgo'.
Automatic merge from submit-queue (batch tested with PRs 61959, 62037). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Bump godep version to v80
**What this PR does / why we need it**:
Update the minimum godep, to v80 (supposed to be the final version).
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Vendor gazelle and kazel
Rather that relying on upstream git repos that can break, vendor it all. These are NOT head of tree, respectively - they are some backrev forms that were previously being used.
```release-note
NONE
```
In nested virutalized environments, the 1 second max-time is too
low. Just bumping up WAIT_FOR_URL_API_SERVER does not work unless
we bump up the max-time too. Let us just make it configurable like
so folks can customize to their environment.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Remove YEAR field of all generated files and fix kubernetes boilerplate checker
**What this PR does / why we need it**:
Remove YEAR field of all generated files and fix kubernetes boilerplate checker
xref: [remove YEAR fileds in gengo #91](https://github.com/kubernetes/gengo/pull/91)
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes [#gengo/issues/24](https://github.com/kubernetes/gengo/issues/24)
**Special notes for your reviewer**:
/cc @thockin @lavalamp @sttts
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 60919, 60953, 61085, 61083, 60971). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
remove unused `pkg/api/unversioned`
**What this PR does / why we need it**:
clean code, see #61084
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes#61084
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Bump to etcd 3.1.12 to pick up critical fix
etcd [3.1.12](https://github.com/coreos/etcd/releases/tag/v3.1.12) (as well as 3.2.17 and 3.3.2) was released yesterday to fix a bug critical to kubernetes:
Fix [mvcc "unsynced" watcher restore operation](https://github.com/coreos/etcd/pull/9297).
- "unsynced" watcher is watcher that needs to be in sync with events that have happened.
- That is, "unsynced" watcher is the slow watcher that was requested on old revision.
- "unsynced" watcher restore operation was not correctly populating its underlying watcher group.
- Which possibly causes [missing events from "unsynced" watchers](https://github.com/coreos/etcd/issues/9086).
This will be backported to 1.9 as well.
Release note:
```release-note
Upgrade the default etcd server version to 3.1.12 to pick up critical etcd "mvcc "unsynced" watcher restore operation" fix.
```
cc @gyuho @wojtek-t @shyamjvs @timothysc @jdumars
Do not bother with pushd/popd construct -- there is no need to go back
to original directory in a subshell. Use "find --exec" instead of "find
| xargs" to handle special file names better.
You can test this change like this:
$ mkdir -p '/tmp/foo bar/x'
$ cd $(dirname "/tmp/foo bar/x")
bash: cd: too many arguments
$ cd "$(dirname "/tmp/foo bar/x")"
$ pwd
/tmp/foo bar
It's admittedly extremely unlikely that the host platform name would
contain special characters, but still use double quotes to pattern
matching.
Consider this script:
#!/bin/bash
bar="foobar"
foo="foo*"
[[ $bar == $foo ]] && echo "first true"
[[ "$bar" == "$foo" ]] && echo "second true"
We get the output:
first true
The plan is to move from first case to the second case to prevent
pattern match where there shouldn't be any.
In https://github.com/kubernetes/kubernetes/issues/52255, it was
determined that if the user has a CDPATH, running `make` will fail. This
is because using the KUBE_ROOT variable results in invalid paths when
the CDPATH environment variable is set.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Build Kubernetes binaries with valid Semantic Version
**What this PR does / why we need it**:
Refuse to build Kubernetes when a version string like `v1.8.3+xx+xx`.
This PR is to restrict the `KUBE_GIT_VERSION`, if the version string does match the rule of Semantic Version, then refuse to build.
Since Kubernetes Conformance test needs the `KUBE_GIT_VERSION` to be a valid Semantic Version, so I think it's better to restrict the version string.
**Release note**:
```release-note
NONE
```
currently the pgrep check does not allow any etcd process to exist
other than the one we intend to start in our scripts. All we need
to know is if the port is free. So let us use netstat to check
if anyone is using that port. We don't really need to know if there
is another instance of etcd is running or not.
This is the 2nd attempt. The previous was reverted while we figured out
the regional mirrors (oops).
New plan: k8s.gcr.io is a read-only facade that auto-detects your source
region (us, eu, or asia for now) and pulls from the closest. To publish
an image, push k8s-staging.gcr.io and it will be synced to the regionals
automatically (similar to today). For now the staging is an alias to
gcr.io/google_containers (the legacy URL).
When we move off of google-owned projects (working on it), then we just
do a one-time sync, and change the google-internal config, and nobody
outside should notice.
We can, in parallel, change the auto-sync into a manual sync - send a PR
to "promote" something from staging, and a bot activates it. Nice and
visible, easy to keep track of.
Automatic merge from submit-queue (batch tested with PRs 59158, 38320, 59059, 55516, 59357). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Promote v1alpha1 meta to v1beta1
No code changes, just renames. We can discuss if there are any field / naming changes here or in a follow-up
Parent #58536Fixes#53224
Prereq to #55637
@kubernetes/sig-api-machinery-pr-reviews @deads2k
```release-note
The `meta.k8s.io/v1alpha1` objects for retrieving tabular responses from the server (`Table`) or fetching just the `ObjectMeta` for an object (as `PartialObjectMetadata`) are now beta as part of `meta.k8s.io/v1beta1`. Clients may request alternate representations of normal Kubernetes objects by passing an `Accept` header like `application/json;as=Table;g=meta.k8s.io;v=v1beta1` or `application/json;as=PartialObjectMetadata;g=meta.k8s.io;v1=v1beta1`. Older servers will ignore this representation or return an error if it is not available. Clients may request fallback to the normal object by adding a non-qualified mime-type to their `Accept` header like `application/json` - the server will then respond with either the alternate representation if it is supported or the fallback mime-type which is the normal object response.
```
Automatic merge from submit-queue (batch tested with PRs 58375, 58306, 58386, 57420, 58035). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
add semver metadata regex
**What this PR does / why we need it**: When building a release tag with optional semver metadata, the minor and major versions are not parsed correctly (empty strings). This fix adds optional support to the regex for the metadata component.
example:
```
~/w/g/s/k/kubernetes ❯❯❯ git tag v1.9.0+stable.0 fixes/semver_metadata ✭
~/w/g/s/k/kubernetes ❯❯❯ hack/print-workspace-status.sh fixes/semver_metadata ✭
STABLE_BUILD_GIT_COMMIT 63e2eacd22
STABLE_BUILD_SCM_STATUS clean
STABLE_BUILD_SCM_REVISION v1.9.0+stable.0
STABLE_BUILD_MAJOR_VERSION 1
STABLE_BUILD_MINOR_VERSION 9
STABLE_DOCKER_TAG v1.9.0_stable.0
gitCommit 63e2eacd22
gitTreeState clean
gitVersion v1.9.0+stable.0
gitMajor 1
gitMinor 9
buildDate 2017-12-19T21:08:48Z
```
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Fix update-swagger-spec.sh to always cleanup etcd
**What this PR does / why we need it**:
This PR fixes `hack/update-swagger-spec.sh` so it always cleanup etcd and to noe leave orphaned process after its execution. This process also doesn't allow to run the script again as it detects existing etcd and won't start.
I also made a minor improvement by adding guard against an empty arguments.
**Release note**:
```release-note
NONE
```
CC @simo5
Prior this change, etcd wasn't cleaned up on my machine that lead to
orphaned etcd process after the update-swagger-spec.sh failed.
This change also adds additional improvements:
- don't kill/wait/rm when argument is empty.
- use kube::util::trap_add.
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Install gazelle from bazelbuild/bazel-gazelle instead of rules_go
**What this PR does / why we need it**: downloads gazelle from its new home; it's being removed from `bazelbuild/rules_go`. It also removes @spiffxp's workaround from a few weeks ago.
**Special notes for your reviewer**: these should really be vendored (https://github.com/kubernetes/kubernetes/pull/57600), but this prevents us from running into issues in the meantime.
**Release note**:
```release-note
NONE
```
/approve no-issue
/assign @BenTheElder
Automatic merge from submit-queue (batch tested with PRs 58207, 58258). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.
Use sort -V to compare golang versions
Change hack/lib/golang.sh to compare golang
version properly with "sort -s -t. -k 1,1 -k 2,2n -k 3,3n",
which sorts key by key and not as strings.
**What this PR does / why we need it**:
trivial fix: hack/lib/golang.sh compare versions like strings, 1.10 is greater than 1.9 as a version but not as a string. Use "sort -s -t. -k 1,1 -k 2,2n -k 3,3n" to compare properly.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
This moves plugin/pkg/scheduler to pkg/scheduler and
plugin/cmd/kube-scheduler to cmd/kube-scheduler.
Bulk of the work was done with gomvpkg, except for kube-scheduler main
package.
k8s-ci-robot