Automatic merge from submit-queue (batch tested with PRs 40971, 41027, 40709, 40903, 39369)
Bump GCI to gci-beta-56-9000-80-0
cc/ @Random-Liu @adityakali
Changelogs since gci-dev-56-8977-0-0 (currently used in Kubernetes):
- "net.ipv4.conf.eth0.forwarding" and "net.ipv4.ip_forward" may get reset to 0
- Track CVE-2016-9962 in Docker in GCI
- Linux kernel CVE-2016-7097
- Linux kernel CVE-2015-8964
- Linux kernel CVE-2016-6828
- Linux kernel CVE-2016-7917
- Linux kernel CVE-2016-7042
- Linux kernel CVE-2016-9793
- Linux kernel CVE-2016-7039 and CVE-2016-8666
- Linux kernel CVE-2016-8655
- Toolbox: allow docker image to be loaded from local tarball
- Update compute-image-package in GCI
- Change the product name on /etc/os-release (to COS)
- Remove 'dogfood' from HWID_OVERRIDE in /etc/lsb-release
- Include Google NVME extensions to optimize LocalSSD performance.
- /proc/<pid>/io missing on GCI (enables process stats accounting)
- Enable BLK_DEV_THROTTLING
cc/ @roberthbailey @fabioy for GKE cluster update
Automatic merge from submit-queue (batch tested with PRs 39169, 40719, 38954, 40808, 40689)
Exports KUBE_TEMP for use in Vagrantfile
In #40147, the logic for setting `KUBE_TEMP` was refactored into `common.sh`. However, it was overlooked that `KUBE_TEMP` [needs to be exported for vagrant to work properly](https://github.com/kubernetes/kubernetes/pull/40147/files#diff-b19d3d93456020e2168c7f304f722969).
This PR restores the `export` so that `Vagrantfile` can use `ENV["KUBE_TEMP"]` properly.
👀 @rthallisey @shyamjvs @timothysc
Automatic merge from submit-queue
Add websocket support for port forwarding
#32880
**Release note**:
```release-note
Port forwarding can forward over websockets or SPDY.
```
Automatic merge from submit-queue (batch tested with PRs 40758, 39145, 40776)
Bumps addon-manager to v6.4-alpha.1 for supporting optional ConfigMap
From #40382. Bumps up addon-manager to use v1.6.0-alpha.1 kubectl for the optional ConfigMap feature. Below images have been pushed:
- gcr.io/google-containers/kube-addon-manager:v6.4-alpha.1
- gcr.io/google-containers/kube-addon-manager-amd64:v6.4-alpha.1
- gcr.io/google-containers/kube-addon-manager-arm:v6.4-alpha.1
- gcr.io/google-containers/kube-addon-manager-arm64:v6.4-alpha.1
- gcr.io/google-containers/kube-addon-manager-ppc64le:v6.4-alpha.1
- gcr.io/google-containers/kube-addon-manager-s390x:v6.4-alpha.1
@liggitt @bowei
- split out port forwarding into its own package
Allow multiple port forwarding ports
- Make it easy to determine which port is tied to which channel
- odd channels are for data
- even channels are for errors
- allow comma separated ports to specify multiple ports
Add portfowardtester 1.2 to whitelist
Automatic merge from submit-queue (batch tested with PRs 40111, 40368, 40342, 40274, 39443)
Libvirt-coreos - Add execute permissions to kubernetes/bin
**What this PR does / why we need it**:
The master node was failing to start for me due to the permission errors on the kubernetes server binaries.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 40111, 40368, 40342, 40274, 39443)
Change OPENSTACK_IMAGE_NAME to be more specific
There may already be other images in the cloud named CentOS7,
but since we are fetching a very specific version (1604) we should
go ahead and make the image name very specific as well.
**What this PR does / why we need it**:
Some clouds already have `Centos7` as an image that is available, however it may not be the *specific* version that openstack-heat looks for and downloads from CentOS.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*:
**Special notes for your reviewer**:
**Release note**:
```release-note
OpenStack-Heat will now look for an image named "CentOS-7-x86_64-GenericCloud-1604". To restore the previous behavior set OPENSTACK_IMAGE_NAME="CentOS7"
```
Automatic merge from submit-queue (batch tested with PRs 38772, 38797, 40732, 40740)
AWS: Deprecate the bash deployment
**What this PR does / why we need it**: Add a strong deprecation warning to the `kube-up.sh` AWS deployment.
**Release note**:
```release-note
The bash AWS deployment via kube-up.sh has been deprecated. See http://kubernetes.io/docs/getting-started-guides/aws/ for alternatives.
```
Automatic merge from submit-queue (batch tested with PRs 40392, 39242, 40579, 40628, 40713)
Add --force-new-cluster when running etcd for migrations.
This is required to avoid etcd trying to create quorum during
migrations.
Might fix#40110
Automatic merge from submit-queue (batch tested with PRs 40691, 40551, 40683, 40700, 40702)
Juju kubernetes-master charm: improve status messages
**What this PR does / why we need it**:
This update to the kubernetes-master charm does the following:
1. Remove "Kubernetes master services ready" status which was occurring too early
2. Add "Waiting for kube-system pods to start" status
3. Replace "Rendering the Kubernetes DNS files." status with "Deploying KubeDNS"
4. Add "Waiting to retry KubeDNS deployment" status
The purpose of this is to give better feedback to the operator during cluster deployment.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*:
Fixes https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/143, which we are tracking in a separate repository
**Special notes for your reviewer**:
This is a rebase of https://github.com/juju-solutions/kubernetes/pull/103, where prior review was done, though it was targeted against a fork.
**Release note**:
```release-note
Juju kubernetes-master charm: improve status messages
```
Automatic merge from submit-queue (batch tested with PRs 40703, 40093, 40618, 40659, 39810)
Change kubemark Makefile to be provider independent
Ref issue #38967
The Kubemark Makefile is defaulted to gcr.io. Instead, make it
provider independent.
The kubemark makefile is set to push the kubemark image to the gcr.io registry. In order to make kubemark not as provider specific, allow the developer to choose a registry.
Automatic merge from submit-queue (batch tested with PRs 40549, 40339)
Invalid node names when deploying with Heat
OpenStack Heat templates create Kubernetes nodes with invalid
hostnames. Capital letters are not allowed in the hostnames:
Unable to register node "kubernetes-node-6s8OizYe" with API server: Node "kubernetes-node-6s8OizYe" is invalid: metadata.name: Invalid value: "kubernetes-node-6s8OizYe": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)* (e.g. 'example.com')
This patch prevents Heat generating hostnames that contain
capital letters.
**What this PR does / why we need it**:
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
```
Automatic merge from submit-queue
Improve the multiarch situation; armel => armhf; reenable pcc64le; remove the patched golang
**What this PR does / why we need it**:
- Improves the multiarch situation as described in #38067
- Tries to bump to go1.8 for arm (and later enable ppc64le)
- GOARM 6 => GOARM 7
- Remove the golang 1.7 patch
- armel => armhf
- Bump QEMU version to v2.7.0
**Release note**:
```release-note
Improve the ARM builds and make hyperkube on ARM working again by upgrading the Go version for ARM to go1.8beta2
```
@kubernetes/sig-testing-misc @jessfraz @ixdy @jbeda @david-mcmahon @pwittrock
Automatic merge from submit-queue
Use a wrapper script to locate kubefed and kubectl binaries instead of directly constructing their paths.
This fixes the e2e failures that is now switched to using kubefed.
cc @kubernetes/sig-federation-pr-reviews
Automatic merge from submit-queue (batch tested with PRs 40497, 39769, 40554, 40569, 40597)
When calling chown, use : instead of . to separate the user and group for cross platform compatibility.
**What this PR does / why we need it**: Makes it possible to build on a Mac, which was broken by #39515.
**Special notes for your reviewer**:
**Release note**:
`NONE`
Automatic merge from submit-queue (batch tested with PRs 39469, 40557)
Refactored kubemark code into provider-specific and provider-independent parts [Part-1]
Applying part of the changes of PR https://github.com/kubernetes/kubernetes/pull/39033 (which refactored kubemark code completely). The changes included in this PR are:
The following are the major changes as part of this refactoring:
- Moved cluster-kubemark/config-default.sh -> cluster-kubemark/gce/config-default.sh (as the config is gce-specific)
- Changed kubernetes/cluster/kubemark/util.sh to source the right scripts based on the cloud-provider
- Added the file test/kubemark/cloud-provider-config.sh which sets the variable CLOUD_PROVIDER that is later picked up by various scripts (run-e2e-tests.sh, common.sh)
- Removed useless code and restructured start-kubemark.sh and stop-kubemark.sh scripts.
@kubernetes/sig-scalability-misc @wojtek-t @gmarek
Automatic merge from submit-queue (batch tested with PRs 40126, 40565, 38777, 40564, 40572)
Bump up glbc version to 0.9.0-beta.1
I plan to bump up the version to 0.9.0 proper in time for the next 1.5.x release, and cherry-pick both this and the future pr.
Previously we were just using a single version, but the "-beta/alpha" is consistent with how we release kube and gives us a convenient revert target. It also forces us to remove the "beta" tag before code freeze, and track the kubernetes release cycle.
Automatic merge from submit-queue (batch tested with PRs 38739, 40480, 40495, 40172, 40393)
Use existing ABAC policy file when upgrading GCE cluster
When upgrading, continue loading an existing ABAC policy file so that existing system components continue working as-is
```
When upgrading an existing 1.5 GCE cluster using `cluster/gce/upgrade.sh`, an existing ABAC policy file located at /etc/srv/kubernetes/abac-authz-policy.jsonl (the default location in 1.5) will enable the ABAC authorizer in addition to the RBAC authorizer. To switch an upgraded 1.5 cluster completely to RBAC, ensure the control plane components and your superuser have been granted sufficient RBAC permissions, move the legacy ABAC policy file to a backup location, and restart the apiserver.
```
Automatic merge from submit-queue
Add support of Keystone v3 'domain-name' to 'openstack-heat' cluster setup
**What this PR does / why we need it**:
Keystone v3 authentication by user name [requires the domain (name or ID)](http://developer.openstack.org/api-ref/identity/v3/index.html?expanded=password-authentication-with-scoped-authorization-detail). If `domain-name` is not provided kubelet fails as seen below:
```
kubelet: error: failed to run Kubelet: could not init cloud provider "openstack": You must provide exactly one of DomainID or DomainName to authenticate by Username
systemd: kubelet.service: main process exited, code=exited, status=1/FAILURE
systemd: Unit kubelet.service entered failed state.
systemd: kubelet.service failed.
```
To solve this I pass a new`OS_USER_DOMAIN_NAME` environment variable through openstack-heat's heat-templates to write it as `domain-name` in `/srv/kubernetes/openstack.conf`.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#39783
**Special notes for your reviewer**:
**Release note**:
```
domain-name support for Keystone v3 added to openstack-heat cluster setup
```
Automatic merge from submit-queue
Able to quick create a HA cluster by kube-up.sh centos provider
Make `kube-up.sh` `centos provider` support quick create a HA cluster, as I said above [#39430](https://github.com/kubernetes/kubernetes/issues/39430), it's more flexible than `kops` or `kubeadm` for some people in a limited network region.
I'm new to k8s dev, so if this pull request need to change, please let me know.
```release-note
Added support for creating HA clusters for centos using kube-up.sh.
```
Automatic merge from submit-queue
[OpenStack-Heat] Add link to the OpenStack CLI install documentation
**What this PR does / why we need it**:
More helpful diagnostic text
**Special notes for your reviewer**:
Trivial patch
**Release note**:
```release-note
NONE
```
Fix: cannot get default master advertise address correctly
Set default value of NUM_MASTERS and NUM_NODES by MASTERS and NODES themself
Code cleanup and documented
Using runtime reconfiguration for etcd cluster instead of etcd discovery
Add exceptions for verify-flags
Automatic merge from submit-queue (batch tested with PRs 40335, 40320, 40324, 39103, 40315)
Use the e2e zone name as the cluster name.
This needs a revamp, but for now e2e zone name is used as the
unique cluster identifier in our e2e tests and we will continue
to use that pattern.
This is a follow up to PR #38638.
cc @kubernetes/sig-federation-pr-reviews @nikhiljindal
Automatic merge from submit-queue (batch tested with PRs 40335, 40320, 40324, 39103, 40315)
Splitting master/node services into separate charm layers
**What this PR does / why we need it**:
This branch includes a roll-up series of commits from a fork of the
Kubernetes repository pre 1.5 release because we didn't make the code freeze.
This additional effort has been fully tested and has results submit into
the gubernator to enhance confidence in this code quality vs. the single
layer, posing as both master/node.
To reference the gubernator results, please see:
https://k8s-gubernator.appspot.com/builds/canonical-kubernetes-tests/logs/kubernetes-gce-e2e-node/
Apologies in advance for the large commit however, we did not want to
submit without having successful upstream automated testing results.
This commit includes:
- Support for CNI networking plugins
- Support for durable storage provided by Ceph
- Building from upstream templates (read: kubedns - no more template
drift!)
- An e2e charm-layer to make running validation tests much simpler/repeatable
- Changes to support the 1.5.x series of Kubernetes
**Special notes for your reviewer**:
Additional note: We will be targeting -all- future work against upstream
so large pull requests of this magnitude will not occur again.
**Release note**:
```release-note
- Splits Juju Charm layers into master/worker roles
- Adds support for 1.5.x series of Kubernetes
- Introduces a tactic for keeping templates in sync with upstream eliminating template drift
- Adds CNI support to the Juju Charms
- Adds durable storage support to the Juju Charms
- Introduces an e2e Charm layer for repeatable testing efforts and validation of clusters
```
Automatic merge from submit-queue (batch tested with PRs 39260, 40216, 40213, 40325, 40333)
Fixed propagation of kube master certs during master replication.
Fixed propagation of kube-master-certs during master replication.
Automatic merge from submit-queue (batch tested with PRs 39275, 40327, 37264)
Fix invalid node name in openstack-heat provider
Cluster node name must follow name syntax in RFC 1123.
But currently, openstack-heat provider generate invalid
node name which contains upper-case characters.
This patch fixes it.
Automatic merge from submit-queue (batch tested with PRs 40299, 40311)
cluster: update default rkt version to 1.23.0
This updates cluster configurations to current stable rkt version.
There may already be other images in the cloud named CentOS7,
but since we are fetching a very specific version (1604) we should
go ahead and make the image name very specific as well.
This branch includes a rollup series of commits from a fork of the
kubernetes repository pre 1.5 release because we didn't make the code freeze.
This additional effort has been fully tested and has results submit into
the gubernator to enhance confidence in this code quality vs. the single
layer, posing as both master/node.
To reference the gubernator results, please see:
https://k8s-gubernator.appspot.com/builds/canonical-kubernetes-tests/logs/kubernetes-gce-e2e-node/
Apologies in advance for the large commit, however we did not want to
submit without having successful upstream automated testing results.
This commit includes:
- Support for CNI networking plugins
- Support for durable storage provided by ceph
- Building from upstream templates (read: kubedns - no more template
drift!)
- An e2e charm-layer to make running validation tests much simpler/repeatable
- Changes to support the 1.5.x series of kubernetes
Additional note: We will be targeting -all- future work against upstream
so large pull requests of this magnitude will not occur again.
OpenStack Heat templates create Kubernetes nodes with invalid
hostnames. Capital letters are not allowed in the hostnames:
Unable to register node "kubernetes-node-6s8OizYe" with API server: Node "kubernetes-node-6s8OizYe" is invalid: metadata.name: Invalid value: "kubernetes-node-6s8OizYe": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)* (e.g. 'example.com')
This patch prevents Heat generating hostnames that contain
capital letters.
Automatic merge from submit-queue
OWNERS: Update latest OWNERS files
These files have been created lately, so we don't have much information
about them anyway, so let's just:
- Remove assignees and make them approvers
- Copy approves as reviewers
Automatic merge from submit-queue (batch tested with PRs 40251, 40171)
Only run gcloud as root if we plan to change something.
Only run gcloud as root if we plan to change something.
Fixes bug introduced in #36292 @jlowdermilk @ixdy
These files have been created lately, so we don't have much information
about them anyway, so let's just:
- Remove assignees and make them approvers
- Copy approves as reviewers
Automatic merge from submit-queue (batch tested with PRs 40011, 40159)
Make CACHEBUST for hyperkube build optional
**What this PR does / why we need it**: It makes CACHEBUST for the hyperkube build optional. Currently, building the hyperkube always results in a full rebuild, including retrieving and installing of all debian packages. This is a good thing for releases, but makes live as a dev hard.
This allows to do something like this:
```
$ REGISTRY=<registry> VERSION=<version> CACHEBUST=0 ./hack/dev-push-hyperkube.sh
```
Probably we should even make CACHEBUST=0 the default when calling dev-xxx.sh scripts.
CC: @aaronlevy
Automatic merge from submit-queue (batch tested with PRs 36693, 40154, 40170, 39033)
Refactored kubemark into cloud-provider independent code and GCE specific code
Ref issue #38967
The following are the major changes as part of this refactoring:
- Moved cluster-kubemark/config-default.sh -> cluster-kubemark/gce/config-default.sh (as the config is gce-specific)
- Changed kubernetes/cluster/kubemark/util.sh to source the right scripts based on the cloud-provider
- Added test/kubemark/skeleton/util.sh which defines a well-commented interface that any cloud-provider should implement to run kubemark. (We have this interface defined only for gce currently)
This includes functions like creating the master machine instance along with its resources, executing a given command on the master (like ssh), scp, deleting the master instance and its resources.
All these functions have to be overrided by each cloud provider inside the file /test/kubemark/$CLOUD_PROVIDER/util.sh
- Added the file test/kubemark/cloud-provider-config.sh which sets the variable CLOUD_PROVIDER that is later picked up by various scripts (start-kubemark.sh, stop-kubemark.sh, run-e2e-tests.sh)
- Removed test/kubemark/common.sh and moved whatever provider-independent code it had into start-kubemark.sh (the only place where the scipt is called) and moved the little gce-specific code
into test/kubemark/gce/util.sh.
- Finally, removed useless code and restructured start-kubemark.sh and stop-kubemark.sh scripts.
@kubernetes/sig-scalability-misc @wojtek-t @gmarek
Automatic merge from submit-queue (batch tested with PRs 36693, 40154, 40170, 39033)
make client-go authoritative for pkg/client/restclient
Moves client/restclient to client-go and a util/certs, util/testing as transitives.
Automatic merge from submit-queue (batch tested with PRs 40168, 40165, 39158, 39966, 40190)
Include system:masters group in the bootstrap admin client certificate
Sets up the bootstrap admin client certificate for new clusters to be in the system:masters group
Removes the need for an explicit grant to the kubecfg user in e2e-bindings
```release-note
The default client certificate generated by kube-up now contains the superuser `system:masters` group
```
Automatic merge from submit-queue
Update root approvers files
Replaces #40040
Update top level OWNERS files mostly to set assignees to approvers. Also remove @bgrant0607 from everywhere but the very top level OWNERS file.
Automatic merge from submit-queue
Use ensure-temp-dir in the common.sh script
Ref issue #38967
Instead of having an ensure-temp-dir function in multiple
places, add it to the common.sh script which is sourced by
all the providers.
Automatic merge from submit-queue (batch tested with PRs 40003, 40017)
Remove library copying from fluentd image
It seems that fluentd can no longer copy systemd libraries from host to be able to read journals.
Automatic merge from submit-queue
Adding cos as an alias for gci.
**What this PR does / why we need it**: Adding COS as an alias for GCI.
cc: @adityakali @wonderfly
Automatic merge from submit-queue (batch tested with PRs 40105, 40095)
[OpenStack-Heat] Fix regex used to get object-store URL
**Release note**:
```release-note
Fixes a bug in the OpenStack-Heat kubernetes provider, in the handling of differences between the Identity v2 and Identity v3 APIs
```
Automatic merge from submit-queue
Build release tars using bazel
**What this PR does / why we need it**: builds equivalents of the various kubernetes release tarballs, solely using bazel.
For example, you can now do
```console
$ make bazel-release
$ hack/e2e.go -v -up -test -down
```
**Special notes for your reviewer**: this is currently dependent on 3b29803eb5, which I have yet to turn into a pull request, since I'm still trying to figure out if this is the best approach.
Basically, the issue comes up with the way we generate the various server docker image tarfiles and load them on nodes:
* we `md5sum` the binary being encapsulated (e.g. kube-proxy) and save that to `$binary.docker_tag` in the server tarball
* we then build the docker image and tag using that md5sum (e.g. `gcr.io/google_containers/kube-proxy:$MD5SUM`)
* we `docker save` this image, which embeds the full tag in the `$binary.tar` file.
* on cluster startup, we `docker load` these tarballs, which are loaded with the tag that we'd created at build time. the nodes then use the `$binary.docker_tag` file to find the right image.
With the current bazel `docker_build` rule, the tag isn't saved in the docker image tar, so the node is unable to find the image after `docker load`ing it.
My changes to the rule save the tag in the docker image tar, though I don't know if there are subtle issues with it. (Maybe we want to only tag when `--stamp` is given?)
Also, the docker images produced by bazel have the timestamp set to the unix epoch, which is not great for debugging. Might be another thing to change with a `--stamp`.
Long story short, we probably need to follow up with bazel folks on the best way to solve this problem.
**Release note**:
```release-note
NONE
```
Wojciech Tyczynski