v1.6.12
Documentation & Examples
Downloads for v1.6.12
| filename |
sha256 hash |
| kubernetes.tar.gz |
d57a05942f581959bc31778def4df4b526df80adaa75942214e9439fdecbe74f |
| kubernetes-src.tar.gz |
c1781161d1b0fa0c8dbd3a644ee42e65968a0a4bb5bec47f2c70796edbc1d396 |
Client Binaries
Server Binaries
Node Binaries
Changelog since v1.6.11
Other notable changes
- Azure cloudprovider: Fix controller manager crash issue on a manually created k8s cluster. (#53694, @andyzhangx)
- Fix kubelet reset liveness probe failure count across pod restart boundaries. (#46371, @sjenning)
v1.6.11
Documentation & Examples
Downloads for v1.6.11
| filename |
sha256 hash |
| kubernetes.tar.gz |
0dacad1c3da0397b6234e474979c4095844733315a853feba5690dbdf8db15dc |
| kubernetes-src.tar.gz |
818fdfc41d7b6d90b9dc37ca12c2fbe1b6fb20f998ee04fddce4a9bb8610351e |
Client Binaries
Server Binaries
Node Binaries
Changelog since v1.6.10
Other notable changes
- Update kube-dns to 1.14.5 (#53112, @bowei)
- Fix panic in ControllerManager on GCE when it has a problem with creating external loadbalancer healthcheck (#52646, @gmarek)
- When performing a GET then PUT, the kube-apiserver must write the canonical representation of the object to etcd if the current value does not match. That allows external agents to migrate content in etcd from one API version to another, across different storage types, or across varying encryption levels. This fixes a bug introduced in 1.5 where we unintentionally stopped writing the newest data. (#48394, @smarterclayton)
- StatefulSet will now fill the
hostname and subdomain fields if they're empty on existing Pods it owns. This allows it to self-correct the issue where StatefulSet Pod DNS entries disappear after upgrading to v1.7.x (#48327). (#51199, @kow3ns)
- Make logdump support kubemark and support gke with 'use_custom_instance_list' (#51834, @shyamjvs)
- Fix credentials providers for docker sandbox image. (#51870, @feiskyer)
v1.6.10
Documentation & Examples
Downloads for v1.6.10
| filename |
sha256 hash |
| kubernetes.tar.gz |
8877359b78950b12a48ea68483f4e4ba2d2521f7e8620efca6f84275cb023428 |
| kubernetes-src.tar.gz |
560d1441b72c670c3d21f838f8d0a94bc75628b1bdd322b18e91df2578a0f84b |
Client Binaries
Server Binaries
Node Binaries
Changelog since v1.6.9
Other notable changes
- Add --request-timeout to kube-apiserver to make global request timeout configurable. (#51415, @jpbetz)
- Fix for Nodes in vSphere lacking an InternalIP. (#48760) (#49202, @cbonte)
- GCE: Bump GLBC version to 0.9.6. (#50096, @nicksardo)
- In GCE with COS, increase TasksMax for Docker service to raise cap on number of threads/processes used by containers. (#51986, @yujuhong)
- Fixed an issue (#47800) where
kubectl logs -f failed with unexpected stream type "". (#51872, @feiskyer)
- Fix for Pod stuck in ContainerCreating with error "Volume is not yet attached according to node". (#50806, @verult)
- Fix initial exec terminal dimensions. (#51126, @chen-anders)
- vSphere: Fix attach volume failing on the first try. (#51218, @BaluDontu)
v1.6.9
Documentation & Examples
Downloads for v1.6.9
| filename |
sha256 hash |
| kubernetes.tar.gz |
08be94c252e7fbdd7c14811ec021818e687c1259e557b70db10aac64c0e8e4b2 |
| kubernetes-src.tar.gz |
519501e26afc341b236c5b46602f010a33fc190e3d1bfb7802969b2e979faaeb |
Client Binaries
Server Binaries
Node Binaries
Changelog since v1.6.8
Other notable changes
- StatefulSet: Set hostname/subdomain fields on new Pods in addition to the deprecated annotations, to allow mitigation of Pod DNS issues upon upgrading to Kubernetes v1.7.x. (#50942, @enisoc)
- Azure: Allow VNet to be in a separate Resource Group. (#49725, @sylr)
- In GCE, add measures to prevent corruption of known_tokens.csv. (#49897, @mikedanese)
- Fixed a bug in the API server watch cache, which could cause a missing watch event immediately after cache initialization. (#49992, @liggitt)
v1.6.8
Documentation & Examples
Downloads for v1.6.8
| filename |
sha256 hash |
| kubernetes.tar.gz |
c87f7826f0b7cf91baddd97ebafb33e99d91dcf6b9019a50bee0689527541ef7 |
| kubernetes-src.tar.gz |
591c43f9624dac351745da35444302cd694ad4953275b8f09016b4654d37b793 |
Client Binaries
Server Binaries
Node Binaries
Changelog since v1.6.7
Other notable changes
- Revert deprecation of vCenter port in vSphere Cloud Provider. (#49689, @divyenpatel)
- kubeadm: Add preflight check for localhost resolution. (#48875, @craigtracey)
- Fix panic when using
kubeadm init with vsphere cloud-provider. (#44661, @xiangpengzhao)
- kubectl: Fix bug that showed terminated/evicted pods even without
--show-all. (#48786, @janetkuo)
- Never prevent deletion of resources as part of namespace lifecycle (#48733, @liggitt)
- AWS cloudprovider plugin: Fix for large clusters (200+ nodes). Also fix bug with volumes not getting detached from a node after reboot. (#48312, @gnufied)
- Fix Pods using Portworx volumes getting stuck in ContainerCreating phase. (#48898, @harsh-px)
- RBAC role and role-binding reconciliation now ensures namespaces exist when reconciling on startup. (#48480, @liggitt)
v1.6.7
Documentation & Examples
Downloads for v1.6.7
| filename |
sha256 hash |
| kubernetes.tar.gz |
6522086d9666543ed4e88a791626953acd1ea843eb024f16f4a4a2390dcbb2b2 |
| kubernetes-src.tar.gz |
b2a73f140966ba0080ce16e3b9a67d5fd9849b36942f3490e9f8daa0fe4511c4 |
Client Binaries
Server Binaries
Node Binaries
Changelog since v1.6.6
Other notable changes
- kubeadm: Expose only the cluster-info ConfigMap in the kube-public ns (#48050, @luxas)
- Fix kubelet request timeout when stopping a container. (#46267, @Random-Liu)
- Add generic NoSchedule toleration to fluentd in gcp config. (#48182, @gmarek)
- Update cluster-proportional-autoscaler, fluentd-gcp, and kube-addon-manager, and kube-dns addons with refreshed base images containing fixes for CVE-2016-9841, CVE-2016-9843, CVE-2017-2616, and CVE-2017-6512. (#47454, @ixdy)
- Fix fluentd-gcp configuration to facilitate JSON parsing (#48139, @crassirostris)
- Bump runc to v1.0.0-rc2-49-gd223e2a - fixes
failed to initialise top level QOS containers kubelet error. (#48117, @sjenning)
kubefed init correctly checks for RBAC API enablement. (#48077, @liggitt)kubectl api-versions now always fetches information about enabled API groups and versions instead of using the local cache. (#48016, @liggitt)- Fix kubelet event recording for selected events. (#46246, @derekwaynecarr)
- Fix
Invalid value: "foregroundDeletion" error when attempting to delete a resource. (#46500, @tnozicka)
v1.6.6
Documentation & Examples
Downloads for v1.6.6
| filename |
sha256 hash |
| kubernetes.tar.gz |
1574d868d43f5d88cfd1af255226e8cd6da72cd65fb9e1285557279c34f8a645 |
| kubernetes-src.tar.gz |
305f372320f78855e298b6caea062c8d1f7db117c7b44943ff5ddd0169424033 |
Client Binaries
Server Binaries
Node Binaries
Changelog since v1.6.5
Action Required
- Azure: Change container permissions to private for provisioned volumes. If you have existing Azure volumes that were created by Kubernetes v1.6.0-v1.6.5, you should change the permissions on them manually. (#47605, @brendandburns)
Other notable changes
v1.6.5
Documentation & Examples
Known Issues for v1.6.5
Downloads for v1.6.5
| filename |
sha256 hash |
| kubernetes.tar.gz |
e497ddd9d0fb03a71babd6c7f879951ba8e2d791c9884613d60794f7df9a5a51 |
| kubernetes-src.tar.gz |
3971e41435f6e22914b603ea56bec72f78673fca9f5a5436a4beda7957dc24e1 |
Client Binaries
Server Binaries
Node Binaries
Changelog since v1.6.4
Other notable changes
- Fix iSCSI iSER mounting. (#47281, @mtanino)
- Added exponential backoff to Azure cloudprovider (#46660, @jackfrancis)
- Update kube-dns to 1.14.2. (#45684, @bowei)
- Fix log spam due to unnecessary status update when node is deleted. (#45923, @verult)
- Poll all active pods periodically for volumes to fix out of order pod & node addition events. Fixes volumes not getting detached after controller restart. (#42033, @NickrenREN)
- iscsi storage plugin: Fix dangling session when using multiple target portal addresses. (#46239, @mtanino)
- The namespace API object no longer supports the deletecollection operation, which was previously allowed by mistake and did not respect expected namespace lifecycle semantics. (#47098, @liggitt)
- Azure: Fix support for multiple
loadBalancerSourceRanges, and add support for UDP ports and the Service spec's sessionAffinity. (#45523, @colemickens)
- Fix the bug where container cannot run as root when SecurityContext.RunAsNonRoot is false. (#47009, @yujuhong)
- Remove broken getvolumename and pass PV or volume name to attach call (#46249, @chakri-nelluri)
- Portworx volume driver no longer has to run on the master. (#45518, @harsh-px)
- Upgrade golang version to 1.7.6 (#46405, @cblecker)
- kube-proxy handling of services with no endpoints now applies to both INPUT and OUTPUT chains, preventing socket leak. (#43972, @thockin)
- Fix kube-apiserver crash when patching TPR data (#44612, @nikhita)
- vSphere cloud provider: Report same Node IP as both internal and external. (#45201, @abrarshivani)
- Kubelet: Fix image garbage collector attempting to remove in-use images. (#46121, @Random-Liu)
- Add metrics exporter to the fluentd-gcp deployment (#45096, @crassirostris)
- Fix the bug where StartedAt time is not reported for exited containers. (#45977, @yujuhong)
- Enable basic auth username rotation for GCI (#44590, @ihmccreery)
- vSphere cloud provider: Filter out IPV6 node addresses. (#45181, @BaluDontu)
- vSphere cloud provider: Fix volume detach on node failure. (#45569, @divyenpatel)
- Job controller: Retry with rate-limiting if a Pod create/delete operation fails. (#45870, @tacy)
- Ensure that autoscaling/v1 is the preferred version for API discovery when autoscaling/v2alpha1 is enabled. (#45741, @DirectXMan12)
- Add support for Azure internal load balancer. (#45690, @jdumars)
- Fix
kubectl delete --cascade=false for resources that don't have legacy overrides to orphan by default. (#45713, @kargakis)
- Fix erroneous FailedSync and FailedMount events being periodically and indefinitely posted on Pods after kubelet is restarted (#44781, @wongma7)
- fluentd will tolerate all NoExecute Taints when run in gcp configuration. (#45715, @gmarek)
- Fix Deployments with Recreate strategy not waiting for Pod termination. (#45639, @ChipmunkV)
- vSphere cloud provider: Remove the dependency of login information on worker nodes for vsphere cloud provider. (#43545, @luomiao)
- vSphere cloud provider: Fix fetching of VM UUID on Ubuntu 16.04 and Fedora. (#45311, @divyenpatel)
v1.6.4
Documentation & Examples
Known Issues for v1.6.4
Downloads for v1.6.4
| filename |
sha256 hash |
| kubernetes.tar.gz |
fef6a97be8195fee1108b40acbd7bea61ef5244b8be23e799d2d01ee365907dd |
| kubernetes-src.tar.gz |
2915465e9b389c5af0fa660f6e7cbc36a416d1286094201e2a2da5b084a37cb3 |
Client Binaries
Server Binaries
Node Binaries
Changelog since v1.6.3
Other notable changes
- Fix kubelet panic during disk eviction introduced in 1.6.3. (#46049, @enisoc)
- Fix pods failing to start if they specify a file as a volume subPath to mount. (#46046, @enisoc)
v1.6.3
Documentation & Examples
Known Issues for v1.6.3
-
This release introduced a regression when using subPath.
If the subPath is a file rather than a directory, Pods may fail to start (#45613).
Do not upgrade to v1.6.3 if your cluster may run Pods with such subPaths.
Downloads for v1.6.3
| filename |
sha256 hash |
| kubernetes.tar.gz |
0af5914fcea36b3c65c8185f31e94b2839eaed52dfdd666d31dfa14534a7d69c |
| kubernetes-src.tar.gz |
0d3cbc716b0d08bf3be779ddd096df965609b5bcb55c8b9ea084c6bb2d6ef1fd |
Client Binaries
Server Binaries
Node Binaries
Changelog since v1.6.2
Other notable changes
- Bump GLBC version to 0.9.3 (#45055, @nicksardo)
- kubeadm: Fix invalid assign statement so it is possible to register the master kubelet with other initial Taints (#45376, @luxas)
- Fix a bug that caused invalid Tolerations to be applied to Pods created by DaemonSets. (#45349, @gmarek)
- Bump cluster autoscaler to v0.5.4, which fixes scale down issues with pods ignoring SIGTERM. (#45483, @mwielgus)
- Fixes and minor cleanups to pod (anti)affinity predicate (#45098, @wojtek-t)
- StatefulSet: Fix to fully preserve restart and termination order when StatefulSet is rapidly scaled up and down. (#45113, @kow3ns)
- Fix some false negatives in detection of meaningful conflicts during strategic merge patch with maps and lists. (#43469, @enisoc)
- cluster-autoscaler: Fix duplicate writing of logs. (#45017, @MaciekPytel)
- Fixes a bug where pods were evicted even after images are successfully deleted. (#44986, @dashpole)
- CRI: respect container's stopping timeout. (#44998, @feiskyer)
- Fix problems with scaling up the cluster when unschedulable pods have some persistent volume claims. (#44860, @mwielgus)
- Exclude nodes labeled as master from LoadBalancer / NodePort; restores documented behaviour. (#44745, @justinsb)
- Fix for scaling down remaining good replicas when a failed Deployment is paused. (#44616, @kargakis)
- kubectl commands run inside a pod using a kubeconfig file now use the namespace specified in the kubeconfig file, instead of using the pod namespace. If no kubeconfig file is used, or the kubeconfig does not specify a namespace, the pod namespace is still used as a fallback. (#44570, @liggitt)
- Restored the ability of kubectl running inside a pod to consume resource files specifying a different namespace than the one the pod is running in. (#44862, @liggitt)
- Fix false positive "meaningful conflict" detection for strategic merge patch with integer values. (#44788, @enisoc)
- Fix insufficient permissions to read/write volumes when mounting them with a subPath. (#43775, @wongma7)
- vSphere cloud provider: Allow specifying fstype in storage class. (#41929, @abrarshivani)
- vSphere cloud provider: Allow specifying VSAN Storage Capabilities during dynamic volume provisioning. (#42974, @BaluDontu)
v1.6.2
Documentation & Examples
Downloads for v1.6.2
| filename |
sha256 hash |
| kubernetes.tar.gz |
240f66a98bf75246b53ef7c1fa3a2be36a92cbc173bc8626e7bc4427bef9ce6a |
| kubernetes-src.tar.gz |
dbf19a8f2e50b3e691eeba0c418fe057f1ea8527b8c0194ba9749c12c801b24b |
Client Binaries
Server Binaries
Changelog since v1.6.1
Other notable changes
- Fix for Windows kubectl sending full path to binary in User Agent. (#44622, @monopole)
- kube-apiserver now drops unneeded path information if an older version of Windows kubectl sends it. (#44421, @mml)
- CRI: Fix kubelet failing to start when using rkt. (#44569, @yujuhong)
- CRI:
kubectl logs -f now stops following when container stops, as it did pre-CRI. (#44406, @Random-Liu)
- Azure cloudprovider: Reduce the polling delay for all azure clients to 5 seconds. This should speed up some operations at the cost of some additional quota. (#43699, @colemickens)
- kubeadm: clean up exited containers and network checkpoints (#43836, @yujuhong)
- Fix corner-case with OnlyLocal Service healthchecks. (#44313, @thockin)
- Fix for disabling federation controllers through override args. (#44341, @irfanurrehman)
- Fix for federation failing to propagate cascading deletion. (#44108, @csbell)
- Fix for failure to delete federation controllers with finalizers. (#44084, @nikhiljindal)
- Fix transition between NotReady and Unreachable taints. (#44042, @gmarek)
- AWS cloud provider: fix support running the master with a different AWS account or even on a different cloud provider than the nodes. (#44235, @mrIncompetent)
- kubeadm: Make
kubeadm reset tolerant of a disabled docker service. (#43951, @luxas)
- Fix broken service accounts when using dedicated service account key. (#44169, @mikedanese)
- Fix for kube-proxy healthcheck handling an update that simultaneously removes one port and adds another. (#44246, @thockin)
- Fix container hostPid settings when CRI is enabled. (#44116, @feiskyer)
- RBAC role and rolebinding auto-reconciliation is now performed only when the RBAC authorization mode is enabled. (#43813, @liggitt)
- Fix incorrect conflict errors applying strategic merge patches to resources. (#43871, @liggitt)
- Fixed an issue mounting the wrong secret into pods as a service account token. (#44102, @ncdc)
- AWS cloud provider: allow to set KubernetesClusterID or KubernetesClusterTag in combination with VPC. (#42512, @scheeles)
- kubeadm: Remove hard-coded default version when fetching stable version from the web fails. Fixes kubeadm 1.6.1 deploying 1.6.0 control plane. (#43999, @mikedanese)
- Fixed recognition of default storage class in DefaultStorageClass admission plugin. (#43945, @mikkeloscar)
- Fix Kubelet panic when Docker is not running. (#44047, @yujuhong)
- Fix vSphere volumes in SELinux environment. (#42973, @gnufied)
- Fix bug with error "Volume has no class annotation" when deleting a PersistentVolume. (#43982, @jsafrane)
v1.6.1
Documentation & Examples
Downloads for v1.6.1
| filename |
sha256 hash |
| kubernetes.tar.gz |
f1634e22ee3fe8af5c355c3f53d1b93f946490addfab029e19acf5317c51cd38 |
| kubernetes-src.tar.gz |
b818f29661dd34db77d1e46c6ba98df6d35906dbc36ac1fdfe45f770b0f695c1 |
Client Binaries
Server Binaries
Changelog since v1.6.0
Other notable changes
- Fix a deadlock in kubeadm master initialization. (#43835, @mikedanese)
- Use Cluster Autoscaler 0.5.1, which fixes an issue in Cluster Autoscaler 0.5 where the cluster may be scaled up unnecessarily. Also the status of Cluster Autoscaler is now exposed in kube-system/cluster-autoscaler-status config map. (#43745, @mwielgus)
v1.6.0
Documentation & Examples
Downloads for v1.6.0
| filename |
sha256 hash |
| kubernetes.tar.gz |
e89318b88ea340e68c427d0aad701e544ce2291195dc1d5901222e7bae48f03b |
| kubernetes-src.tar.gz |
0b03d27e1c7af3be5d94ecd5f679e5f55588d801376cf1ae170d9ec0a229b1e2 |
Client Binaries
Server Binaries
WARNING: etcd backup strongly recommended
Before updating to 1.6, you are strongly recommended to back up your etcd data.
Please consult the installation procedure you are using (kargo, kops, kube-up,
kube-aws, kubeadm etc) for specific advice.
1.6 encourages etcd3, and switching from etcd2 to etcd3 involves a full
migration of data between different storage engines. You must stop the API
from writing to etcd during an etcd2 -> etcd3 migration. HA installations cannot
be migrated at the current time using the official Kubernetes procedure.
1.6 will also default to protobuf encoding if using etcd3. This change is
irreversible. To rollback, you must restore from a backup made before the
protobuf/etcd3 switch, and any changes since the backup will be lost. As 1.5
does not support protobuf encoding, if you roll back to 1.5 after upgrading to
protobuf you will be forced to restore from backup, and you will lose any changes
since you converted to protobuf. After conversion to protobuf, you should
validate the correct operation of your cluster thoroughly before returning it
to normal operation.
Backups are always a good precaution, particularly around upgrades, but this
upgrade has multiple known issues where the only remedy is to restore from
backup.
Also, please note:
-
using application/vnd.kubernetes.protobuf as the media storage type for 1.6 is default but not required
-
the ability to rollback to etcd2 can be preserved by setting the storage media type flag on kube-apiserver
--storage-media-type application/json
to continue to use application/json as the storage media type which can be changed to
application/vnd.kubernetes.protobuf at a later time.
Major updates and release themes
- Kubernetes now supports up to 5,000 nodes via etcd v3, which is enabled by default.
- Role-based access control (RBAC) has graduated to beta, and defines secure default roles for control plane, node, and controller components.
- The
kubeadm cluster bootstrap tool has graduated to beta. Some highlights:
- WARNING: A known issue
in v1.6.0 causes
kubeadm init to hang. Please use v1.6.1, which fixes the issue.
- All communication is now over TLS
- Authorization plugins can be installed by kubeadm, including the new default of RBAC
- The bootstrap token system now allows token management and expiration
- The
kubefed federation bootstrap tool has also graduated to beta.
- Interaction with container runtimes is now through the CRI interface, enabling easier integration of runtimes with the kubelet. Docker remains the default runtime via Docker-CRI (which moves to beta).
- WARNING: A known issue
in v1.6.0 causes
Pod.Spec.HostPid (using the host PID namespace for the pod) to always
be false. Please wait for v1.6.2, which will include a fix for this issue.
- Various scheduling features have graduated to beta:
- You can now specify (per pod) how long a pod should stay bound to a node, when there is a node problem.
- Various storage features have graduated to GA:
- DaemonSets can now be updated by a rolling update.
Action Required
Certificates API
- Users of the alpha certificates API should delete v1alpha1 CSRs from the API before upgrading and recreate them as v1beta1 CSR after upgrading. (#39772, @mikedanese)
Cluster Autoscaler
- If you are using (or planning to use) Cluster Autoscaler please wait for Kubernetes 1.6.1. In 1.6.0 Cluster Autoscaler
may occasionally increase the size of the cluster a bit more than it is actually needed, when there are
unschedulable pods, scale up is required and cloud provider is slow to set up networking for new nodes.
Anyway, the cluster should get back to the proper size after 10 min.
Deployment
Federation
- The --dns-provider argument of 'kubefed init' is now mandatory and does not default to
google-clouddns. To initialize a Federation control plane with Google Cloud DNS, use the following invocation: 'kubefed init --dns-provider=google-clouddns' (#42092, @marun)
- Cluster federation servers have changed the location in etcd where federated services are stored, so existing federated services must be deleted and recreated. Before upgrading, export all federated services from the federation server and delete the services. After upgrading the cluster, recreate the federated services from the exported data. (#37770, @enj)
Internal Storage Layer
- upgrade to etcd3 prior to upgrading to 1.6 OR explicitly specify
--storage-backend=etcd2 --storage-media-type=application/json when starting the apiserver
Node Components
- Kubelet with the Docker-CRI implementation
- The Docker-CRI implementation is enabled by default.
- It is not compatible with containers created by older Kubelets. It is
recommended to drain your node before upgrade. If you choose to perform
an in-place upgrade, the Kubelet will automatically restart all
Kubernetes-managed containers on the node.
- It is not compatible with CNI plugins that do not conform to the
error handling behavior in the spec.
The plugins are being updated to resolve this issue (#43488).
You can disable CRI explicitly (
--enable-cri=false) as a
temporary workaround.
- The standard
bridge plugin have been validated to interoperate with
the new CRI + CNI code path.
- If you are using plugins other than
bridge, make sure you have
updated custom plugins to the latest version that is compatible.
- CNI plugins now affect node readiness
- Kubelet will now block node readiness until a CNI configuration file is
present in
/etc/cni/net.d or a given custom CNI configuration path.
This change ensures
kubelet does not start pods that require networking before
networking is ready.
This change may require changes to clients that gate pod creation and/or
scheduling on the node condition type Ready status being True for pods
that need to run prior to the network being ready.
- Enhance Kubelet QoS:
- Pods are placed under a new cgroup hierarchy by default. This feature requires
draining and restarting the node as part of upgrades. To opt-out set
--cgroups-per-qos=false.
- If
kube-reserved and/or system-reserved are specified, node allocatable
will be enforced on all pods by default. To opt-out set
--enforce-node-allocatable=””
- Hard Eviction Thresholds will be subtracted from Capacity while calculating
Node Allocatable. This will result in a reduction of schedulable capacity in
clusters post upgrade where kubelet hard eviction has been turned on for
memory. To opt-out set
--experimental-allocatable-ignore-eviction=true.
- More details on these feature here:
https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/
- Drop the support for docker 1.9.x. Docker versions 1.10.3, 1.11.2, 1.12.6
have been validated.
- The following deprecated kubelet flags are removed:
--config, --auth-path,
--resource-container, --system-container, --reconcile-cidr
- Remove the temporary fix for pre-1.0 mirror pods. Upgrade directly from
pre-1.0 to 1.6 kubelet is not supported.
- Fluentd was migrated to Daemon Set, which targets nodes with
beta.kubernetes.io/fluentd-ds-ready=true label. If you use fluentd in your
cluster please make sure that the nodes with version 1.6+ contains this
label.
kubectl
- Running
kubectl taint (alpha in 1.5) against a 1.6 server requires upgrading kubectl to version 1.6
- Running
kubectl taint (alpha in 1.5) against a 1.5 server requires a kubectl version of 1.5
- Running
kubectl create secret no longer accepts passing multiple values to a single --from-literal flag using comma separation
- Update command invocations to pass separate --from-literal flags for each value
RBAC
- Default ClusterRole and ClusterRoleBinding objects are automatically updated at server start to add missing permissions and subjects (extra permissions and subjects are left in place). To prevent autoupdating a particular role or rolebinding, annotate it with
rbac.authorization.kubernetes.io/autoupdate=false. (#41155, @liggitt)
v1beta1 RoleBinding/ClusterRoleBinding subjects changed apiVersion to apiGroup to fully-qualify a subject. ServiceAccount subjects default to an apiGroup of "", User and Group subjects default to an apiGroup of "rbac.authorization.k8s.io". (#41184, @liggitt)- To create or update an RBAC RoleBinding or ClusterRoleBinding object, a user must: (#39383, @liggitt)
-
- Be authorized to make the create or update API request
-
- Be allowed to bind the referenced role, either by already having all of the permissions contained in the referenced role, or by having the "bind" permission on the referenced role.
- The
--authorization-rbac-super-user flag (alpha in 1.5) is deprecated; the system:masters group has privileged access (#38121, @deads2k)
- special handling of the user
* in RoleBinding and ClusterRoleBinding objects is removed in v1beta1. To match all users, explicitly bind to the group system:authenticated and/or system:unauthenticated. Existing v1alpha1 bindings to the user * are automatically converted to the group system:authenticated. (#38981, @liggitt)
Scheduling
Service
- The 'endpoints.beta.kubernetes.io/hostnames-map' annotation is no longer supported. Users can use the 'Endpoints.subsets[].addresses[].hostname' field instead. (#39284, @bowei)
StatefulSet
- StatefulSet now respects ControllerRef to avoid fighting over Pods. At the time of upgrade, you must not have StatefulSets with selectors that overlap with any other controllers (such as ReplicaSets), or else ownership of Pods may change. (#42080, @enisoc)
Volumes
- StorageClass pre-installed and set as default on Azure, AWS, GCE, OpenStack, and vSphere.
- This is something to pay close attention to if you’ve been using Kubernetes for a while, because it changes the default behavior of PersistentVolumeClaim objects on these clouds.
- Marking a StorageClass as default makes it so that even a PersistentVolumeClaim without a StorageClass specified will trigger dynamic provisioning (instead of binding to an existing pool of PVs).
- If you depend on the old behavior of volumes binding to existing pool of PersistentVolume objects then modify the StorageClass object and set
storageclass.beta.kubernetes.io/is-default-class to false.
- Flex volume plugin is updated to support attach/detach interfaces. It broke backward compatibility. Please update your drivers and implement the new callouts. (#41804, @chakri-nelluri)
Notable Features
Features for this release were tracked via the use of the kubernetes/features issues repo. Each Feature issue is owned by a Special Interest Group from the kubernetes/community.
Autoscaling
- [alpha] The Horizontal Pod Autoscaler now supports drawing metrics through the API server aggregator.
- [alpha] The Horizontal Pod Autoscaler now supports scaling on multiple, custom metrics.
- Cluster Autoscaler publishes its status to kube-system/cluster-autoscaler-status ConfigMap.
- Cluster Autoscaler can continue operations while some nodes are broken or unready.
- Cluster Autoscaler respects Pod Disruption Budgets when removing a node.
DaemonSet
Deployment
- [beta] Deployments that cannot make progress in rolling out the newest version will now indicate via the API they are blocked (docs)
Federation
- [beta]
kubefed has graduated to beta: supports hosting federation on on-prem clusters, automatically configures kube-dns in joining clusters and allows passing arguments to federation components.
Internal Storage Layer
- [stable] The internal storage layer for kubernetes cluster state has been updated to use etcd v3 by default. Existing clusters will have to plan for a data migration window. (docs)(kubernetes/features#44)
kubeadm
- [beta] Introduces an API for clients to request TLS certificates from the API server. See the tutorial.
- [beta] kubeadm is enhanced and improved with a baseline feature set and command line flags that are now marked as beta. Other parts of kubeadm, including subcommands under
kubeadm alpha, are still in alpha. Using it is considered safe, although note that upgrades and HA are not yet supported. Please try it out and give us feedback!
- [alpha] New Bootstrap Token authentication and management method. Works well with kubeadm. kubeadm now supports managing tokens, including time based expiration, after the cluster is launched. See kubeadm reference docs for details.
- [alpha] Adds a new cloud-controller-manager binary that may be used for testing the new out-of-core cloudprovider flow.
Node Components
- [stable] Init containers have graduated to GA and now appear as a field.
The beta annotation value will still be respected and overrides the field
value.
- Kubelet Container Runtime Interface (CRI) support
- [beta] The Docker-CRI implementation is enabled by default in kubelet.
You can disable it by --enable-cri=false. See
notes on the new implementation
for more details.
- [alpha] Alpha support for other runtimes:
cri-o, frakti, rkt.
- [beta] Node Problem Detector is beta (v0.3.0) now. New
features added into node-problem-detector:v0.3.0:
- Add support for journald.
- The ability to monitor any system daemon logs. Previously only kernel
logs were supported.
- The ability to be deployed and run as a native daemon.
- [alpha] Critical Pods: When feature gate "ExperimentalCriticalPodAnnotation" is
set to true, the system will guarantee scheduling and admission of pods with
the following annotation -
scheduler.alpha.kubernetes.io/critical-pod
- This feature should be used in conjunction with the
rescheduler to guarantee
resource availability for critical system pods.
- [alpha]
--experimental-nvidia-gpus flag is replaced by Accelerators alpha
feature gate along with addition of support for multiple Nvidia GPUs.
- To use GPUs, pass
Accelerators=true as part of --feature-gates flag.
- More information here.
- A pod’s Quality of Service Class is now available in its Status.
- Upgrade cAdvisor library to v0.25.0. Notable changes include,
- Container filesystem usage tracking disabled for device mapper due to excessive
IOPS.
- Ignore
.mount cgroups, fixing disappearing stats.
- A new field
terminationMessagePolicy has been added to containers that allows
a user to request FallbackToLogsOnError, which will read from the container's
logs to populate the termination message if the user does not write to the
termination message log file. The termination message file is now properly
readable for end users and has a maximum size (4k bytes) to prevent abuse.
Each pod may have up to 12k bytes of termination messages before the contents
of each will be truncated.
- Do not delete pod objects until all its compute resource footprint has been
reclaimed.
- This feature prevents the node and scheduler from oversubscribing resources
that are still being used by a pod in the process of being cleaned up.
- This feature can result in increase of Pod Deletion latency especially if the
pod has a large filesystem footprint.
RBAC
- [beta] RBAC API is promoted to v1beta1 (rbac.authorization.k8s.io/v1beta1), and defines default roles for control plane, node, and controller components.
- [beta] The Docker-CRI implementation is Beta and is enabled by default in kubelet. You can disable it by
--enable-cri=false. See notes on the new implementation for more details.
Scheduling
- [beta] The multiple schedulers. This feature allows you to run multiple schedulers in parallel, each responsible for different sets of pods. When using multiple schedulers, the scheduler name is now specified in a new-in-1.6
schedulerName field of the PodSpec rather than using the scheduler.alpha.kubernetes.io/name annotation on the Pod. When you upgrade to 1.6, the Kubernetes default scheduler will start using the schedulerName field of the PodSpec and will ignore the annotation.
- [beta] Node affinity/anti-affinity and [beta] pod affinity/anti-affinity. Node affinity/anti-affinity allow you to specify rules for restricting which node(s) a pod can schedule onto, based on the labels on the node. Pod affinity/anti-affinity allow you to specify rules for spreading and packing pods relative to one another, across arbitrary topologies (node, zone, etc.) These affinity rules are now be specified in a new-in-1.6
affinity field of the PodSpec. Kubernetes 1.6 continues to support the alpha scheduler.alpha.kubernetes.io/affinity annotation on the Pod if you explicitly enable the alpha feature "AffinityInAnnotations", but it will be removed in a future release. When you upgrade to 1.6, if you have not enabled the alpha feature, then the scheduler will use the affinity field in PodSpec and will ignore the annotation. If you have enabled the alpha feature, then the scheduler will use the affinity field in PodSpec if it is present, and otherwise will use the annotation.
- [beta] Taints and tolerations. This feature allows you to specify rules for "repelling" pods from nodes by default, which enables use cases like dedicated nodes and reserving nodes with special features for pods that need those features. We've also added a
NoExecute taint type that evicts already-running pods, and an associated tolerationSeconds field to tolerations to delay the eviction for a specified amount of time. As before, taints are created using kubectl taint (but internally they are now represented as a field taints in the NodeSpec rather than using the scheduler.alpha.kubernetes.io/taints annotation on Node). Tolerations are now specified in a new-in-1.6 tolerations field of the PodSpec rather than using the scheduler.alpha.kubernetes.io/tolerations annotation on the Pod. When you upgrade to 1.6, the scheduler will start using the fields and will ignore the annotations.
- [alpha] Represent node problems "not ready" and "unreachable" using
NoExecute taints. In combination with tolerationSeconds described below, this allows per-pod specification of how long to remain bound to a node that becomes unreachable or not ready, rather than using the default of 5 minutes. You can enable this alpha feature by including TaintBasedEvictions=true in --feature-gates, such as --feature-gates=FooBar=true,TaintBasedEvictions=true. Documentation is here.
Service Catalog
- [alpha] Adds a new API resource
PodPreset and admission controller to enable defining cross-cutting injection of Volumes and Environment into Pods.
Volumes
- [stable] StorageClass API is promoted to v1 (storage.k8s.io/v1).
- [stable] Default storage classes are deployed during installation on Azure, AWS, GCE, OpenStack and vSphere.
- [stable] Added ability to populate environment variables from a configmap or secret.
- [stable] Support for user-written/run dynamic PV provisioners. The Kubernetes Incubator contains a golang library and examples.
- [stable] Volume plugin for ScaleIO enabling pods to seamlessly access and use data stored on Dell EMC ScaleIO volumes.
- [stable] Volume plugin for Portworx added capability to use Portworx as a storage provider for Kubernetes clusters. Portworx pools server capacity and turns servers or cloud instances into converged, highly available compute and storage nodes.
- [stable] Add support to use NFSv3, NFSv4, and GlusterFS on GCE/GKE GCI image based clusters.
- [beta] Added support for mount options in persistent volumes.
- [alpha] All in one volume proposal - a new volume driver capable of projecting secrets, configmaps, and downward API items into the same directory.
Deprecations
- Remove extensions/v1beta1 Jobs resource, and job/v1beta1 generator. (#38614, @soltysh)
federation/deploy/deploy.sh was an interim solution introduced in Kubernetes v1.4 to simplify the federation control plane deployment experience. Now that we have kubefed, we are deprecating deploy.sh scripts. (#38902, @madhusudancs)
Cluster Provisioning Scripts
kubeadm
- Quite a few flags been renamed or removed. Those options that are removed as flags can still be accessed via the config file. Most notably this includes external etcd settings and the option for setting the cloud provider on the API server. The kubeadm reference documentation is up to date with the new flags.
Other Deprecations
- Remove cmd/kube-discovery from the tree since it's not necessary anymore (#42070, @luxas)
Changes to API Resources
ABAC
- ABAC policies using
"user":"*" or "group":"*" to match all users or groups will only match authenticated requests. To match unauthenticated requests, ABAC policies must explicitly specify "group":"system:unauthenticated" (#38968, @liggitt)
Admission Control
- Adds a new API resource
PodPreset and admission controller to enable defining cross-cutting injection of Volumes and Environment into Pods. (#41931, @jessfraz)
- Enable DefaultTolerationSeconds admission controller by default (#41815, @kevin-wangzefeng)
- ResourceQuota ability to support default limited resources (#36765, @derekwaynecarr)
- Add defaultTolerationSeconds admission controller (#41414, @kevin-wangzefeng)
- An
automountServiceAccountToken field was added to ServiceAccount and PodSpec objects. If set to false on a pod spec, no service account token is automounted in the pod. If set to false on a service account, no service account token is automounted for that service account unless explicitly overridden in the pod spec. (#37953, @liggitt)
- Admission control support for versioned configuration files (#39109, @derekwaynecarr)
- Ability to quota storage by storage class (#34554, @derekwaynecarr)
Authentication
- The authentication.k8s.io API group was promoted to v1(#41058, @liggitt)
Authorization
- The authorization.k8s.io API group was promoted to v1 (#40709, @liggitt)
- The SubjectAccessReview API now passes subresource and resource name information to the authorizer to answer authorization queries. (#40935, @liggitt)
Autoscaling
- Introduces an new alpha version of the Horizontal Pod Autoscaler including expanded support for specifying metrics. (#36033, @DirectXMan12
- HorizontalPodAutoscaler is no longer supported in extensions/v1beta1 version. Use autoscaling/v1 instead. (#35782, @piosz)
- Fixes an HPA-related panic due to division-by-zero. (#39694, @DirectXMan12)
Certificates
- The CertificateSigningRequest API added the
extra field to persist all information about the requesting user. This mirrors the fields in the SubjectAccessReview API used to check authorization. (#41755, @liggitt)
- Native support for token based bootstrap flow. This includes signing a well known ConfigMap in the
kube-public namespace and cleaning out expired tokens. (#36101, @jbeda)
ConfigMap
- Volumes and environment variables populated from ConfigMap and Secret objects can now tolerate the named source object or specific keys being missing, by adding
optional: true to the volume or environment variable source specifications. (#39981, @fraenkel)
- Allow pods to define multiple environment variables from a whole ConfigMap (#36245, @fraenkel)
CronJob
- Add configurable limits to CronJob resource to specify how many successful and failed jobs are preserved. (#40932, @peay)
DaemonSet
- DaemonSet now respects ControllerRef to avoid fighting over Pods. (#42173, @enisoc)
- Make DaemonSet respect critical pods annotation when scheduling. (#42028, @janetkuo)
- Implement the update feature for DaemonSet. (#41116, @lukaszo)
- Make DaemonSets survive taint-based evictions when nodes turn unreachable/notReady. (#41896, @kevin-wangzefeng)
- Make DaemonSet controller respect node taints and pod tolerations. (#41172, @janetkuo)
- DaemonSet controller actively kills failed pods (to recreate them) (#40330, @janetkuo)
- DaemonSet ObservedGeneration (#39157, @lukaszo)
Deployment
- Add ready replicas in Deployments (#37959, @kargakis)
- Deployments that cannot make progress in rolling out the newest version will now indicate via the API they are blocked
- Introduce apps/v1beta1.Deployments resource with modified defaults compared to extensions/v1beta1.Deployments. (#39683, @soltysh)
- Introduce new generator for apps/v1beta1 deployments (#42362, @soltysh)
Node
Pod
- Init containers have graduated to GA and now appear as a field. The beta annotation value will still be respected and overrides the field value. (#38382, @hodovska)
- A new field
terminationMessagePolicy has been added to containers that allows a user to request FallbackToLogsOnError, which will read from the container's logs to populate the termination message if the user does not write to the termination message log file. The termination message file is now properly readable for end users and has a maximum size (4k bytes) to prevent abuse. Each pod may have up to 12k bytes of termination messages before the contents of each will be truncated. (#39341, @smarterclayton)
- Fix issue with PodDisruptionBudgets in which
minAvailable specified as a percentage did not work with StatefulSet Pods. (#39454, @foxish)
- Node affinity has moved from annotations to api fields in the pod spec. Node affinity that is defined in the annotations will be ignored. (#37299, @rrati)
- Moved pod affinity and anti-affinity from annotations to api fields #25319 (#39478, @rrati)
- Add QoS pod status field (#37968, @sjenning)
Pod Security Policy
- PodSecurityPolicy resource is now enabled by default in the extensions API group. (#39743, @pweil-)
RBAC
- The
attributeRestrictions field has been removed from the PolicyRule type in the rbac.authorization.k8s.io/v1alpha1 API. The field was not used by the RBAC authorizer. (#39625, @deads2k)
- A user can now be authorized to bind a particular role by having permission to perform the
bind verb on the referenced role (#39383, @liggitt)
ReplicaSet
Secrets
- Populate environment variables from a secrets. (#39446, @fraenkel)
- Added a new secret type "bootstrap.kubernetes.io/token" for dynamically creating TLS bootstrapping bearer tokens. (#41281, @ericchiang)
Service
- Endpoints, that tolerate unready Pods, are now listing Pods in state Terminating as well (#37093, @simonswine)
- Fix Service Update on LoadBalancerSourceRanges Field (#37720, @freehan)
- Bug fix. Incoming UDP packets not reach newly deployed services (#32561, @zreigz)
- Services of type loadbalancer consume both loadbalancer and nodeport quota. (#39364, @zhouhaibing089)
StatefulSet
- Fix zone placement heuristics so that multiple mounts in a StatefulSet pod are created in the same zone (#40910, @justinsb)
- Fixes issue #38418 which, under circumstance, could cause StatefulSet to deadlock. (#40838, @kow3ns)
- Mediates issue #36859. StatefulSet only acts on Pods whose identity matches the StatefulSet, providing a partial mediation for overlapping controllers.
Taints and Tolerations
- Add a manager to NodeController that is responsible for removing Pods from Nodes tainted with NoExecute Taints. This feature is beta (as the rest of taints) and enabled by default. It's gated by controller-manager enable-taint-manager flag. (#40355, @gmarek)
- Add an alpha feature that makes NodeController set Taints instead of deleting Pods from not Ready Nodes. (#41133, @gmarek)
- Make tolerations respect wildcard key (#39914, @kevin-wangzefeng)
- Forgiveness alpha version api definition (#39469, @kevin-wangzefeng)
- Rescheduler uses taints in v1beta1 and will remove old ones (in version v1alpha1) right after its start. (#43106, @piosz)
Volumes
- StorageClassName attribute has been added to PersistentVolume and PersistentVolumeClaim objects and should be used instead of annotation
volume.beta.kubernetes.io/storage-class. The beta annotation is still working in this release, however it will be removed in a future release. (#42128, @jsafrane)
- Parameter keys in a StorageClass
parameters map may not use the kubernetes.io or k8s.io namespaces. (#41837, @liggitt)
- Add storage.k8s.io/v1 API (#40088, @jsafrane)
- Alpha version of dynamic volume provisioning is removed in this release. Annotation (#40000, @jsafrane)
- Reduce verbosity of volume reconciler when attaching volumes (#36900, @codablock)
- We change the default attach_detach_controller sync period to 1 minute to reduce the query frequency through cloud provider to check whether volumes are attached or not. (#41363, @jingxu97)
Changes to Major Components
API Server
--anonymous-auth is enabled by default, unless the API server is started with the AlwaysAllow authorizer. (#38706, @deads2k)- When using OIDC authentication and specifying --oidc-username-claim=email, an
"email_verified":true claim must be returned from the identity provider. (#36087, @ericchiang)
--basic-auth-file supports optionally specifying groups in the fourth column of the file (#39651, @liggitt)
- API server now has two separate limits for read-only and mutating inflight requests. (#36064, @gmarek)
- Restored normalization of custom
--etcd-prefix when --storage-backend is set to etcd3 (#42506, @liggitt)
- Updating apiserver to return http status code 202 for a delete request when the resource is not immediately deleted because of user requesting cascading deletion using DeleteOptions.OrphanDependents=false. (#41165, @nikhiljindal)
- Use full package path for definition name in OpenAPI spec (#40124, @mbohlool)
- Follow redirects for streaming requests (exec/attach/port-forward) in the apiserver by default (alpha -> beta). (#40039, @timstclair)
- Add 'X-Content-Type-Options: nosniff" to some error messages (#37190, @brendandburns)
- Fixes bug in resolving client-requested API versions (#38533, @DirectXMan12)
- Replace glog.Fatals with fmt.Errorfs (#38175, @sttts)
- Pipe get options to storage (#37693, @wojtek-t)
- The --long-running-request-regexp flag to kube-apiserver is deprecated and will be removed in a future release. Long-running requests are now detected based on specific verbs (watch, proxy) or subresources (proxy, portforward, log, exec, attach). (#38119, @liggitt)
- if kube-apiserver is started with
--storage-backend=etcd2, the media type application/json is used. (#43122, @liggitt)
- API fields that previously serialized null arrays as
null and empty arrays as [] no longer distinguish between those values and always output [] when serializing to JSON. (#43422, @liggitt)
- Generate OpenAPI definition for inlined types (#39466, @mbohlool)
API Server Aggregator
Generic API Server
- Move pkg/api/rest into genericapiserver (#39948, @sttts)
- Move non-generic apiserver code out of the generic packages (#38191, @sttts)
- Fixes API compatibility issue with empty lists incorrectly returning a null
items field instead of an empty array. (#39834, @liggitt)
- Re-add /healthz/ping handler in genericapiserver (#38603, @sttts)
- Remove genericapiserver.Options.MasterServiceNamespace (#38186, @sttts)
- genericapiserver: cut off more dependencies – episode 3 (#40426, @sttts)
- genericapiserver: more dependency cutoffs (#40216, @sttts)
- genericapiserver: cut off kube pkg/version dependency (#39943, @sttts)
- genericapiserver: cut off pkg/serviceaccount dependency (#39945, @sttts)
- genericapiserver: cut off pkg/apis/extensions and pkg/storage dependencies (#39946, @sttts)
- genericapiserver: cut off certificates api dependency (#39947, @sttts)
- genericapiserver: extract CA cert from server cert and SNI cert chains (#39022, @sttts)
- genericapiserver: turn APIContainer.SecretRoutes into a real ServeMux (#38826, @sttts)
- genericapiserver: unify swagger and openapi in config (#38690, @sttts)
Client
- Use Prometheus instrumentation conventions (#36704, @fabxc)
- Clients now use the
?watch=true parameter to make watch API calls, instead of the /watch/ path prefix (#41722, @liggitt)
- Move private key parsing from serviceaccount/jwt.go to client-go/util/cert (#40907, @cblecker)
- Caching added to the OIDC client auth plugin to fix races and reduce the time kubectl commands using this plugin take by several seconds. (#38167, @ericchiang)
- Fix resync goroutine leak in ListAndWatch (#35672, @tatsuhiro-t)
client-go
Cloud Provider
AWS
- Allow to running the master with a different AWS account or even on a different cloud provider than the nodes. (#39996, @scheeles)
- Support shared tag
kubernetes.io/cluster/<clusterid> (#41695, @justinsb)
- Do not consider master instance zones for dynamic volume creation (#41702, @justinsb)
- Fix AWS device allocator to only use valid device names (#41455, @gnufied)
- Trust region if found from AWS metadata (#38880, @justinsb)
- Remove duplicate calls to DescribeInstance during volume operations (#39842, @gnufied)
- Recognize eu-west-2 region (#38746, @justinsb)
- Recognize ca-central-1 region (#38410, @justinsb)
- Add sequential allocator for device names. (#38818, @jsafrane)
Azure
GCE
- On GCI by default logrotate is disabled for application containers in favor of rotation mechanism provided by docker logging driver. (#40634, @crassirostris)
GKE
vSphere
- Reverts to looking up the current VM in vSphere using the machine's UUID, either obtained via sysfs or via the
vm-uuid parameter in the cloud configuration file. (#40892, @robdaemon)
- Fix for detach volume when node is not present/ powered off (#40118, @BaluDontu)
- Adding vmdk file extension for vmDiskPath in vsphere DeleteVolume (#40538, @divyenpatel)
- Changed default scsi controller type in vSphere Cloud Provider (#38426, @abrarshivani)
- Fixes NotAuthenticated errors that appear in the kubelet and kube-controller-manager due to never logging in to vSphere (#36169, @robdaemon)
- Fix panic in vSphere cloud provider (#38423, @BaluDontu)
- Fix space issue in volumePath with vSphere Cloud Provider (#38338, @BaluDontu)
Federation
kubefed
- Flag cleanup (#41335, @irfanurrehman)
- Create configmap for the cluster kube-dns when cluster joins and remove when it unjoins (#39338, @irfanurrehman)
- Support configuring dns-provider (#40528, @shashidharatd)
- Bug fix relating kubeconfig path in kubefed init (#41410, @irfanurrehman)
- Add override flags options to kubefed init (#40917, @irfanurrehman)
- Add option to expose federation apiserver on nodeport service (#40516, @shashidharatd)
- Add option to disable persistence storage for etcd (#40862, @shashidharatd)
- kubefed init creates a service account for federation controller manager in the federation-system namespace and binds that service account to the federation-system:federation-controller-manager role that has read and list access on secrets in the federation-system namespace. (#40392, @madhusudancs)
- Implement dry run support in kubefed init (#36447, @irfanurrehman)
- Make federation etcd PVC size configurable (#36310, @irfanurrehman)
Other Notable Changes
- Federated Ingress over GCE no longer requires separate firewall rules to be created for each cluster to circumvent flapping firewall health checks. (#41942, @csbell)
- Add support for finalizers in federated configmaps (deletes configmaps from underlying clusters). (#40464, @csbell)
- Add support for DeleteOptions.OrphanDependents for federated services. Setting it to false while deleting a federated service also deletes the corresponding services from all registered clusters. (#36390, @nikhiljindal)
- Add
--controllers flag to federation controller manager for enable/disable federation ingress controller (#36643, @kzwang)
- Stop deleting services from underlying clusters when federated service is deleted. (#37353, @nikhiljindal)
- Expose autoscaling apis through federation api server (#38976, @irfanurrehman)
- Federation: Add
batch/jobs API objects to federation-apiserver (#35943, @jianhuiz)
- Add logging of route53 calls (#39964, @justinsb)
Garbage Collector
- Added foreground garbage collection: the owner object will not be deleted until all its dependents are deleted by the garbage collector. Please checkout the user doc for details. (#38676, @caesarxuchao)
- deleteOptions.orphanDependents is going to be deprecated in 1.7. Please use deleteOptions.propagationPolicy instead.
kubeadm
- A new label and taint is used for marking the master. The label is
node-role.kubernetes.io/master="" and the taint has the effect NoSchedule. Tolerate the node-role.kubernetes.io/master="":NoSchedule taint to schedule a workload on the master (a networking DaemonSet for example).
- The kubelet API is now secured, only cluster admins are allowed to access it.
- Insecure access to the API Server over
localhost:8080 is now disabled.
- The control plane components now talk securely to each other. The API Server talks securely to the kubelets in the cluster.
- kubeadm creates RBAC-enabled clusters. This means that some add-ons which worked previously won't work without having their YAML configuration updated. Please see each vendor's own documentation to check that the add-ons you are using will work with Kubernetes 1.6.
- The kube-discovery Deployment isn't used anymore when creating a kubeadm cluster and shouldn't be used anywhere else either due to its insecure nature.
- The Certificates API has graduated from alpha to beta. This is a backwards-incompatible change since the alpha support is dropped, and therefore kubeadm v1.5 and v1.6 don't work together (for example kubeadm v1.5 can't join a kubeadm v1.6 cluster).
- Supporting only etcd3, with 3.0.14 as the minimum version.
kubeadm reset will no longer drain nodes automatically. This is because the credentials on nodes do not have permission to perform this operation. We have documented an alternate procedure, driven from the API server/master.- Hook up kubeadm against the BootstrapSigner (#41417, @luxas)
- Rename some flags for beta UI and fixup some logic (#42064, @luxas)
- Insecure access to the API Server at localhost:8080 will be turned off in v1.6 when using kubeadm (#42066, @luxas)
- Flag --use-kubernetes-version for kubeadm init renamed to --kubernetes-version (#41820, @kad)
- Remove the --cloud-provider flag for beta init UX (#41710, @luxas)
- Fixed an SELinux issue in kubeadm on Docker 1.12+ by moving etcd SELinux options from container to pod. (#40682, @dgoodwin)
- Add authorization mode to kubeadm (#39846, @andrewrynhard)
- Refactor the certificate and kubeconfig code in the kubeadm binary into two phases (#39280, @luxas)
- Added kubeadm commands to manage bootstrap tokens and the duration they are valid for. (#35805, @dgoodwin)
kubectl
New Commands
apply set-last-applied updates the applied-applied-configuration annotation (#41694, @shiywang)kubectl apply view-last-applied viewing the last configuration file applied (#41146, @shiywang)
Create subcommands
Updates to existing commands
- Printing and output
- Import a numeric ordering sorting library and use it in the sorting printer. (#40746, @matthyx)
- DaemonSet get and describe show status fields. (#42843, @janetkuo)
- Pods describe shows tolerationSeconds (#42162, @kevin-wangzefeng)
- Node describe contains closing paren (#39217, @luksa)
- Display pod node selectors with kubectl describe. (#36396, @aveshagarwal)
- Add Version to the resource printer for 'get nodes' (#37943, @ailusazh)
- Added support for printing in all supported
--output formats to kubectl create ... and kubectl apply ... (#38112, @juanvallejo)
- Add three more columns to
kubectl get deploy -o wide output. (#39240, @xingzhou)
- Fix kubectl get -f -o so it prints all items in the file (#39038, @ncdc)
- kubectl describe no longer prints the last-applied-configuration annotation for secrets. (#34664, @ymqytw)
- Completed pods should not be hidden when requested by name via
kubectl get. (#42216, @smarterclayton)
- Improve formatting of EventSource in kubectl get and kubectl describe (#40073, @matthyx)
- Attach now supports multiple types (#40365, @shiywang)
- Create now accepts the label selector flag for filtering objects to create (#40057, @MrHohn)
- Top now accepts short forms for "node" and "pod" ("no", "po") (#39218, @luksa)
- Begin paths for internationalization in kubectl (#36802, @brendandburns)
Updates to apply
- New command
apply set-last-applied updates the applied-applied-configuration annotation (#41694, @shiywang)
- New command
apply view-last-applied command for viewing the last configuration file applied (#41146, @shiywang)
apply now supports explicitly clearing values by setting them to null (#40630, @liggitt)- Warn user when using
apply on a resource lacking the LastAppliedConfig annotation (#36672, @ymqytw)
- PATCH (i.e. apply and edit) now supports merging lists of primitives (#38665, @ymqytw)
apply falls back to generic 3-way JSON merge patch for Third Party Resource or unregistered types (#40666, @ymqytw)
Updates to edit
edit now supports Third party resources and extension API servers. (#41304, @liggitt)
- Now to edit a particular API version, provide the fully-qualify the resource, version, and group used to fetch the object (for example,
job.v1.batch/myjob)
- Client-side conversion is no longer done, and the
--output-version option is deprecated for kubectl edit.
edit works as intended with apply and will not change the annotation
- No longer updates the last-applied-configuration annotation when --save-config is unspecified or false. (#41924, @ymqytw)
- Fixes issue that caused apply to revert changes made by edit
Bug fixes
- Fixed --save-config in create subcommand to save the annotation (#40289, @xilabao)
- Fixed an issue where 'kubectl get --sort-by=' would return an error if the specified fields in sort were not specified in one or more of the returned objects. (#40541, @fabianofranz)
- Previously this would cause the command to fail regardless of whether or not the field was present in the object model
- Now the command will succeed even if the sort-by field is missing from one or more of the objects
- Fixed issue with kubectl proxy so it will now proxy an empty path - e.g. http://localhost:8001 (#39226, @luksa)
- Fixed an issue where commas were not accepted in --from-literal flags for the creation of secrets. (#35191, @SamiHiltunen)
- Passing multiple values separated by a comma in a single --from-literal flag is no longer supported. Please use multiple --from-literal flags to provide multiple values.
- Fixed a bug where the --server, --token, and --certificate-authority flags were not overriding the related in-cluster configs when provided in a
kubectl call inside a cluster. (#39006, @fabianofranz)
Other Notable Changes
- The api server will publish the extensions/Deployments API as preferred over the apps/Deployment (introduced in 1.6). (#43553, @liggitt)
- This will ensure certain commands in 1.5 versions of kubectl continue function when run against a 1.6 server. (e.g.
kubectl edit deployment)
- Taint
- The
taint command will not function in a skewed 1.5 / 1.6 environment - client and server versions must match (See Action required section above)
- Change taints/tolerations to api fields (#38957, @aveshagarwal)
- Make kubectl taint command respect effect NoExecute (#42120, @kevin-wangzefeng)
- Allow drain --force to remove pods whose managing resource is deleted. (#41864, @marun)
--output-version is ignored for all commands except kubectl convert. This is consistent with the generic nature of kubectl CRUD commands and the previous removal of --api-version. Specific versions can be specified in the resource field: resource.version.group, jobs.v1.batch. (#41576, @deads2k)- Allow missing keys in templates by default (#39486, @ncdc)
- Add error message when trying to use clusterrole with namespace in kubectl (#36424, @xilabao)
- When deleting an object with
--grace-period=0, the client will begin a graceful deletion and wait until the resource is fully deleted. To force deletion, use the --force flag. (#37263, @smarterclayton)
Node Components
- Kubelet config should ignore file start with dots.
(#39196, @resouer)
- Bump GCI to gci-stable-56-9000-84-2.
(#41819,
@dchen1107)
- Bump GCE ContainerVM to container-vm-v20170214 to address CVE-2016-9962.
(#41449,
@zmerlynn)
- Kubelet: Remove the PLEG health check from /healthz, Kubelet will now report
- NodeNotReady on failed PLEG health check.
(#41569,
@yujuhong)
- CRI: upgrade protobuf to v3. Protobuf v2 and v3 are not compatible.
(#39158, @feiskyer)
- kubelet exports metrics for cgroup management (#41988, @sjenning)
- Experimental support to reserve a pod's memory request from being utilized by
pods in lower QoS tiers.
(#41149,
@sjenning)
- Port forwarding can forward over websockets or SPDY.
(#33684,
@fraenkel)
- Flag gate faster evictions based on node memory pressure using kernel memcg
notifications -
--experimental-kernel-memcg-notification.
(#38258,
@derekwaynecarr)
- Nodes can now report two additional address types in their status: InternalDNS
and ExternalDNS. The apiserver can use --kubelet-preferred-address-types to
give priority to the type of address it uses to reach nodes.
(#34259, @liggitt)
Bug fixes
kube-controller-manager
kube-dns
-
Adds support for configurable DNS stub domains and upstream nameservers.
The following configuration options have been added to the kube-system:kube-dns ConfigMap:
"stubDomains": {
"acme.local": ["1.2.3.4"]
},
is a map of domain to list of nameservers for the domain. This is used
to inject private DNS domains into the kube-dns namespace. In the above
example, any DNS requests for *.acme.local will be served by the
nameserver 1.2.3.4.
"upstreamNameservers": ["8.8.8.8", "8.8.4.4"]
is a list of upstreamNameservers to use, overriding the configuration
specified in /etc/resolv.conf.
-
An empty kube-system:kube-dns ConfigMap will be created for the cluster if one did not already exist.
kube-proxy
- - Add tcp/udp userspace proxy support for Windows. (#41487, @anhowe)
- Add DNS suffix search list support in Windows kube-proxy. (#41618, @JiangtianLi)
- Add a KUBERNETES_NODE_* section to build kubelet/kube-proxy for windows (#38919, @brendandburns)
- Remove outdated net.experimental.kubernetes.io/proxy-mode and net.beta.kubernetes.io/proxy-mode annotations from kube-proxy. (#40585, @cblecker)
- proxy/iptables: don't sync proxy rules if services map didn't change (#38996, @dcbw)
- Update kube-proxy image to be based off of Debian 8.6 base image. (#39695, @ixdy)
- Update amd64 kube-proxy base image to debian-iptables-amd64:v5 (#39725, @ixdy)
- Clean up the kube-proxy container image by removing unnecessary packages and files. (#42090, @timstclair)
- Better compat with very old iptables (e.g. CentOS 6) (#37594, @thockin)
Scheduler
- Add the support to the scheduler for spreading pods of StatefulSets. (#41708, @bsalamat)
- Added support to minimize sending verbose node information to scheduler extender by sending only node names and expecting extenders to cache the rest of the node information (#41119, @sarat-k)
- Support KUBE_MAX_PD_VOLS on Azure (#41398, @codablock)
- Mark multi-scheduler graduation to beta and then v1. (#38871, @k82cn)
- Scheduler treats StatefulSet pods as belonging to a single equivalence class. (#39718, @foxish)
- Update FitError as a message component into the PodConditionUpdater. (#39491, @jayunit100)
- Fix comment and optimize code (#38084, @tanshanshan)
- Add flag to enable contention profiling in scheduler. (#37357, @gmarek)
- Try self-repair scheduler cache or panic (#37379, @wojtek-t)
Volume Plugins
Azure Disk
GlusterFS
- The glusterfs dynamic volume provisioner will now choose a unique GID for new persistent volumes from a range that can be configured in the storage class with the "gidMin" and "gidMax" parameters. The default range is 2000 - 2147483647 (max int32). (#37886, @obnoxxx)
Photon
- Fix photon controller plugin to construct with correct PdID (#37167, @luomiao)
rbd
- force unlock rbd image if the image is not used (#41597, @rootfs)
vSphere
Other Notable Changes
- Implement bulk polling of volumes (#41306, @gnufied)
- Check if pathExists before performing Unmount (#39311, @rkouj)
- Unmount operation should not fail if volume is already unmounted (#38547, @rkouj)
- Provide kubernetes-controller-manager flags to control volume attach/detach reconciler sync. The duration of the syncs can be controlled, and the syncs can be shut off as well. (#39551, @chrislovecnm)
- Fix unmountDevice issue caused by shared mount in GCI (#38411, @jingxu97)
- Fix permissions when using fsGroup (#37009, @sjenning)
- Fixed issues (#39202), (#41041) and (#40941) that caused the iSCSI connections to
be prematurely closed when deleting a pod with an iSCSI persistent volume attached and that prevented the use of newly created LUNs on targets with preestablished connections. (#41196), @CristianPop)
Changes to Cluster Provisioning Scripts
AWS
Juju
- The kubernetes-master, kubernetes-worker and kubeapi-load-balancer charms have gained an nrpe-external-master relation, allowing the integration of their monitoring in an external Nagios server. (#41923, @Cynerva)
- Disable anonymous auth on kubelet (#41919, @Cynerva)
- Fix shebangs in charm actions to use python3 (#42058, @Cynerva)
- K8s master charm now properly keeps distributed master files in sync for an HA control plane. (#41351, @chuckbutler)
- Improve status messages (#40691, @Cynerva)
- Splits Juju Charm layers into master/worker roles (#40324, @chuckbutler)
- Adds support for 1.5.x series of Kubernetes
- Introduces a tactic for keeping templates in sync with upstream eliminating template drift
- Adds CNI support to the Juju Charms
- Adds durable storage support to the Juju Charms
- Introduces an e2e Charm layer for repeatable testing efforts and validation of clusters
libvirt CoreOS
GCE
- the
gce provider enables both RBAC authorization and the permissive legacy ABAC policy that makes all service accounts superusers. To opt out of the permissive ABAC policy, export the environment variable ENABLE_LEGACY_ABAC=false before running cluster/kube-up.sh. (#43544, @liggitt)
- the
gce provider ensures the bootstrap admin token user is included in the super-user group (#39537, @liggitt)
- Remove support for debian masters in GCE kube-up. (#41666, @mikedanese)
- Remove support for trusty in GCE kube-up. (#41670, @mikedanese)
- Don't fail if the grep fails to match any resources (#41933, @ixdy)
- Fix the output of health-mointor.sh (#41525, @yujuhong)
- Added configurable etcd initial-cluster-state to kube-up script. (#41332, @jszczepkowski)
- The kube-apiserver basic audit log can be enabled in GCE by exporting the environment variable
ENABLE_APISERVER_BASIC_AUDIT=true before running cluster/kube-up.sh. This will log to /var/log/kube-apiserver-audit.log and use the same logrotate settings as /var/log/kube-apiserver.log. (#41211, @enisoc)
- On kube-up.sh clusters on GCE, kube-scheduler now contacts the API on the secured port. (#41285, @liggitt)
- Use existing ABAC policy file when upgrading GCE cluster (#40172, @liggitt)
- Ensure the GCI metadata files do not have newline at the end (#38727, @Amey-D)
- Fixed detection of master during creation of multizone nodes cluster by kube-up. (#38617, @jszczepkowski)
- Fixed validation of multizone cluster for GCE (#38695, @jszczepkowski)
- Fix GCI mounter issue (#38124, @jingxu97)
- Exit with error if is not the final parameter. (#37723, @mtaufen)
- GCI: Remove /var/lib/docker/network (#37593, @yujuhong)
- Fix the equality checks for numeric values in cluster/gce/util.sh. (#37638, @roberthbailey)
- Modify GCI mounter to enable NFSv3 (#37582, @jingxu97)
- Use gsed on the Mac (#37562, @roberthbailey)
- Bump GCI
- to gci-beta-56-9000-80-0 (#41027, @dchen1107)
- to gci-stable-56-9000-84-2 (#41819, @dchen1107)
- Bump GCE ContainerVM
OpenStack
- Do not daemonize
salt-minion for the openstack-heat provider. (#40722, @micmro)
- OpenStack-Heat will now look for an image named "CentOS-7-x86_64-GenericCloud-1604". To restore the previous behavior set OPENSTACK_IMAGE_NAME="CentOS7" (#40368, @sc68cal)
- Fixes a bug in the OpenStack-Heat kubernetes provider, in the handling of differences between the Identity v2 and Identity v3 APIs (#40105, @sc68cal)
Container Images
- Update gcr.io/google-containers/rescheduler to v0.2.2, which uses busybox as a base image instead of ubuntu. (#41911, @ixdy)
- Remove unnecessary metrics (http/process/go) from being exposed by etcd-version-monitor (#41807, @shyamjvs)
- Align the hyperkube image to support running binaries at /usr/local/bin/ like the other server images (#41017, @luxas)
- Bump up GLBC version from 0.9.0-beta to 0.9.1 (#41037, @bprashanth)
Other Notable Changes
- The default client certificate generated by kube-up now contains the superuser
system:masters group (#39966, @liggitt)
- Added support for creating HA clusters for centos using kube-up.sh. (#39462, @Shawyeok)
- Enable lazy inode table and journal initialization for ext3 and ext4 (#38865, @codablock)
- Since
kubernetes.tar.gz no longer includes client or server binaries, cluster/kube-{up,down,push}.sh now automatically download released binaries if they are missing. (#38730, @ixdy)
- Fix broken cluster/centos and enhance the style (#34002, @xiaoping378)
- Set kernel.softlockup_panic =1 based on the flag. (#38001, @dchen1107)
- Configure local-up-cluster.sh to handle auth proxies (#36838, @deads2k)
kube-up.sh/kube-down.sh no longer force update gcloud for provider=gce|gke. (#36292, @jlowdermilk)- Collect logs for dead kubelets too (#37671, @mtaufen)
Changes to Addons
Dashboard
DNS
-
Updates the dnsmasq cache/mux layer to be managed by dnsmasq-nanny. (#41826, @bowei)
dnsmasq-nanny manages dnsmasq based on values from the
kube-system:kube-dns configmap:
"stubDomains": {
"acme.local": ["1.2.3.4"]
},
is a map of domain to list of nameservers for the domain. This is used
to inject private DNS domains into the kube-dns namespace. In the above
example, any DNS requests for *.acme.local will be served by the
nameserver 1.2.3.4.
upstreamNameservers": ["8.8.8.8", "8.8.4.4"]
is a list of upstreamNameservers to use, overriding the configuration
specified in /etc/resolv.conf.
-
kube-dns now runs using a separate system:serviceaccount:kube-system:kube-dns service account which is automatically bound to the correct RBAC permissions. (#38816, @deads2k)
-
Use kube-dns:1.11.0 (#39925, @sadlil)
DNS Autoscaler
- Patch CVE-2016-8859 in gcr.io/google-containers/cluster-proportional-autoscaler-amd64 (#42933, @timstclair)
Cluster Autoscaler
- Allow the Horizontal Pod Autoscaler controller to talk to the metrics API and custom metrics API as standard APIs. (#41824, @DirectXMan12)
Cluster Load Balancing
etcd Empty Dir Cleanup
- Base etcd-empty-dir-cleanup on busybox, run as nobody, and update to etcdctl 3.0.14 (#41674, @ixdy)
Fluentd
- Migrated fluentd addon to daemon set (#32088, @piosz)
- Fluentd was migrated to Daemon Set, which targets nodes with beta.kubernetes.io/fluentd-ds-ready=true label. If you use fluentd in your cluster please make sure that the nodes with version 1.6+ contains this label. (#42931, @piosz)
- Fluentd-gcp containers spawned by DaemonSet are now configured using ConfigMap (#42126, @crassirostris)
- Cleanup fluentd-gcp image: rebase on debian-base, switch to upstream packages, remove fluent-ui & rails (#41998, @timstclair)
- On GCE, the apiserver audit log (
/var/log/kube-apiserver-audit.log) will be sent through fluentd if enabled. It will go to the same place as kube-apiserver.log, but tagged as its own stream. (#41360, @enisoc)
- If
experimentalCriticalPodAnnotation feature gate is set to true, fluentd pods will not be evicted by the kubelet. (#41035, @vishh)
- fluentd config for GKE clusters updated: detect exceptions in container log streams and forward them as one log entry. (#39656, @thomasschickinger)
- Make fluentd pods critical (#39146, @crassirostris)
- Fluentd/Elastisearch add-on: correctly parse and index kubernetes labels (#36857, @Shrugs)
Heapster
Registry
External Dependency Version Information
Continuous integration builds have used the following versions of external dependencies, however, this is not a strong recommendation and users should consult an appropriate installation or upgrade guide before deciding what versions of etcd, docker or rkt to use.
- Docker versions 1.10.3, 1.11.2, 1.12.6 have been validated
- Docker version 1.12.6 known issues
- overlay2 driver not fully supported
- live-restore not fully supported
- no shared pid namespace support
- Docker version 1.11.2 known issues
- Kernel crash with Aufs storage driver on Debian Jessie (#27885)
which can be identified by the node problem detector
- Leaked File descriptors (#275)
- Additional memory overhead per container (#21737)
- Docker 1.10.3 contains backports provided by RedHat for known issues
- Support for Docker version 1.9.x has been removed
- rkt version 1.23.0+
- etcd version 3.0.17
Changelog since v1.6.0-rc.1
Previous Releases Included in v1.6.0
v1.6.0-rc.1
Documentation & Examples
Downloads for v1.6.0-rc.1
| filename |
sha256 hash |
| kubernetes.tar.gz |
b92be4b71184888ba4a2781d4068ea369442b6030dfb38e23029192a554651e5 |
| kubernetes-src.tar.gz |
e45e993edfdba176a6750c6d3c2207d54d60b5b1fc80a0fe47274d4d9b233d66 |
Client Binaries
Server Binaries
Changelog since v1.6.0-beta.4
Other notable changes
kube-up.sh using the gce provider enables both RBAC authorization and the permissive legacy ABAC policy that makes all service accounts superusers. To opt out of the permissive ABAC policy, export the environment variable ENABLE_LEGACY_ABAC=false before running cluster/kube-up.sh. (#43544, @liggitt)- Bump CNI consumers to v0.5.1 (#43546, @calebamiles)
- The API server discovery document now prioritizes the
extensions API group over the apps API group. This ensures certain commands in 1.5 versions of kubectl (such as kubectl edit deployment) continue to function against a 1.6 API. (#43553, @liggitt)
- Fix adding disks to more than one scsi adapter. (#42422, @kerneltime)
- PodSecurityPolicy authorization is correctly enforced by the PodSecurityPolicy admission plugin. (#43489, @liggitt)
- API fields that previously serialized null arrays as
null and empty arrays as [] no longer distinguish between those values and always output [] when serializing to JSON. (#43422, @liggitt)
- Apply taint tolerations for NoExecute for all static pods. (#43116, @dchen1107)
- Bumped Heapster to v1.3.0. (#43298, @piosz)
v1.6.0-beta.4
Documentation & Examples
Downloads for v1.6.0-beta.4
| filename |
sha256 hash |
| kubernetes.tar.gz |
8f308a87bcc367656c777f74270a82ad6986517c28a08c7158c77b1d7954e243 |
| kubernetes-src.tar.gz |
3ba73cf27a05f78026d1cfb3a0e47c6e5e33932aefc630a0a5aa3619561bc4dc |
Client Binaries
Server Binaries
Changelog since v1.6.0-beta.3
Other notable changes
- Update dashboard version to v1.6.0 (#43210, @floreks)
- Update photon controller go SDK in vendor code. (#43108, @luomiao)
- Fluentd was migrated to Daemon Set, which targets nodes with beta.kubernetes.io/fluentd-ds-ready=true label. If you use fluentd in your cluster please make sure that the nodes with version 1.6+ contains this label. (#42931, @piosz)
- if kube-apiserver is started with
--storage-backend=etcd2, the media type application/json is used. (#43122, @liggitt)
- Add -p to mkdirs in gci-mounter function of gce configure.sh script (#43134, @shyamjvs)
- Rescheduler uses taints in v1beta1 and will remove old ones (in version v1alpha1) right after its start. (#43106, @piosz)
- kubeadm:
kubeadm reset won't drain and remove the current node anymore (#42713, @luxas)
- hack/godep-restore.sh: use godep v79 which works (#42965, @sttts)
- Patch CVE-2016-8859 in gcr.io/google-containers/cluster-proportional-autoscaler-amd64 (#42933, @timstclair)
- Disable devicemapper thin_ls due to excessive iops (#42899, @dashpole)
v1.6.0-beta.3
Documentation & Examples
Downloads for v1.6.0-beta.3
| filename |
sha256 hash |
| kubernetes.tar.gz |
3903f0a49945abe26f5775c20deb25ba65a8607a2944da8802255bd50d20aca7 |
| kubernetes-src.tar.gz |
62f5f9459c14163e319f878494d10296052d75345da7906e8b38a2d6d0d2a25c |
Client Binaries
Server Binaries
Changelog since v1.6.0-beta.2
Other notable changes
- Introduce new generator for apps/v1beta1 deployments (#42362, @soltysh)
- Use Prometheus instrumentation conventions (#36704, @fabxc)
- Add new DaemonSet status fields to kubectl printer and describer. (#42843, @janetkuo)
- Dropped the support for docker 1.9.x and the belows. (#42694, @dchen1107)
v1.6.0-beta.2
Documentation & Examples
Downloads for v1.6.0-beta.2
| filename |
sha256 hash |
| kubernetes.tar.gz |
8199c781f8c98ed7048e939a590ea10a6c39f6a74bd35ed526b459fa18e20f50 |
| kubernetes-src.tar.gz |
f8de26ab6493d4547f9068f5ef396650c169702863535ba63feaf815464a6702 |
Client Binaries
Server Binaries
Changelog since v1.6.0-beta.1
Action Required
- Deployment now fully respects ControllerRef to avoid fighting over Pods and ReplicaSets. At the time of upgrade, you must not have Deployments with selectors that overlap, or else ownership of ReplicaSets may change. (#42175, @enisoc)
- StatefulSet now respects ControllerRef to avoid fighting over Pods. At the time of upgrade, you must not have StatefulSets with selectors that overlap with any other controllers (such as ReplicaSets), or else ownership of Pods may change. (#42080, @enisoc)
Other notable changes
- DaemonSet now respects ControllerRef to avoid fighting over Pods. (#42173, @enisoc)
- restored normalization of custom
--etcd-prefix when --storage-backend is set to etcd3 (#42506, @liggitt)
- kubelet created cgroups follow lowercase naming conventions (#42497, @derekwaynecarr)
- Support whitespace in command path for gcp auth plugin (#41653, @jlowdermilk)
- Updates the dnsmasq cache/mux layer to be managed by dnsmasq-nanny. (#41826, @bowei)
- dnsmasq-nanny manages dnsmasq based on values from the
- kube-system:kube-dns configmap:
- "stubDomains": {
- "acme.local": ["1.2.3.4"]
- },
- is a map of domain to list of nameservers for the domain. This is used
- to inject private DNS domains into the kube-dns namespace. In the above
- example, any DNS requests for *.acme.local will be served by the
- nameserver 1.2.3.4.
- "upstreamNameservers": ["8.8.8.8", "8.8.4.4"]
- is a list of upstreamNameservers to use, overriding the configuration
- specified in /etc/resolv.conf.
- kubelet exports metrics for cgroup management (#41988, @sjenning)
- kubectl: respect deployment strategy parameters for rollout status (#41809, @kargakis)
- Remove cmd/kube-discovery from the tree since it's not necessary anymore (#42070, @luxas)
- kubeadm: Hook up kubeadm against the BootstrapSigner (#41417, @luxas)
- Federated Ingress over GCE no longer requires separate firewall rules to be created for each cluster to circumvent flapping firewall health checks. (#41942, @csbell)
- ScaleIO Kubernetes Volume Plugin added enabling pods to seamlessly access and use data stored on ScaleIO volumes. (#38924, @vladimirvivien)
- Pods are launched in a separate cgroup hierarchy than system services. (#42350, @vishh)
- Experimental support to reserve a pod's memory request from being utilized by pods in lower QoS tiers. (#41149, @sjenning)
- Juju: Disable anonymous auth on kubelet (#41919, @Cynerva)
- Remove support for debian masters in GCE kube-up. (#41666, @mikedanese)
- Implement bulk polling of volumes (#41306, @gnufied)
- stop kubectl edit from updating the last-applied-configuration annotation when --save-config is unspecified or false. (#41924, @ymqytw)
v1.6.0-beta.1
Documentation & Examples
Downloads for v1.6.0-beta.1
| filename |
sha256 hash |
| kubernetes.tar.gz |
ca17c4f1ebdd4bbbd0e570bf3a29d52be1e962742155bc5e20765434f3141f2d |
| kubernetes-src.tar.gz |
4aefc25b42594f0aab48e43608c8ef6eca8c115022fcc76a9a0d34430e33be0f |
Client Binaries
Server Binaries
Changelog since v1.6.0-alpha.3
Action Required
- The --dns-provider argument of 'kubefed init' is now mandatory and does not default to
google-clouddns. To initialize a Federation control plane with Google Cloud DNS, use the following invocation: 'kubefed init --dns-provider=google-clouddns' (#42092, @marun)
- Change taints/tolerations to api fields (#38957, @aveshagarwal)
Other notable changes
- kubeadm: Rename some flags for beta UI and fixup some logic (#42064, @luxas)
- StorageClassName attribute has been added to PersistentVolume and PersistentVolumeClaim objects and should be used instead of annotation
volume.beta.kubernetes.io/storage-class. The beta annotation is still working in this release, however it will be removed in a future release. (#42128, @jsafrane)
- Remove Azure kube-up as the Azure community has focused efforts elsewhere. (#41672, @mikedanese)
- Fluentd-gcp containers spawned by DaemonSet are now configured using ConfigMap (#42126, @Crassirostris)
- Modified kubemark startup scripts to restore master on reboot (#41980, @shyamjvs)
- Added new Api
PodPreset to enable defining cross-cutting injection of Volumes and Environment into Pods. (#41931, @jessfraz)
- AWS cloud provider: allow to run the master with a different AWS account or even on a different cloud provider than the nodes. (#39996, @scheeles)
- Update defaultbackend image to 1.3 (#42212, @timstclair)
- Allow the Horizontal Pod Autoscaler controller to talk to the metrics API and custom metrics API as standard APIs. (#41824, @DirectXMan12)
- Implement support for mount options in PVs (#41906, @gnufied)
- Introduce apps/v1beta1.Deployments resource with modified defaults compared to extensions/v1beta1.Deployments. (#39683, @soltysh)
- Add DNS suffix search list support in Windows kube-proxy. (#41618, @JiangtianLi)
--experimental-nvidia-gpus flag is replaced by Accelerators alpha feature gate along with support for multiple Nvidia GPUs. (#42116, @vishh)
- To use GPUs, pass
Accelerators=true as part of --feature-gates flag.
- Works only with Docker runtime.
- Clean up the kube-proxy container image by removing unnecessary packages and files. (#42090, @timstclair)
- AWS: Support shared tag
kubernetes.io/cluster/<clusterid> (#41695, @justinsb)
- Insecure access to the API Server at localhost:8080 will be turned off in v1.6 when using kubeadm (#42066, @luxas)
- AWS: Do not consider master instance zones for dynamic volume creation (#41702, @justinsb)
- Added foreground garbage collection: the owner object will not be deleted until all its dependents are deleted by the garbage collector. Please checkout the user doc for details. (#38676, @caesarxuchao)
- deleteOptions.orphanDependents is going to be deprecated in 1.7. Please use deleteOptions.propagationPolicy instead.
- force unlock rbd image if the image is not used (#41597, @rootfs)
- The kubernetes-master, kubernetes-worker and kubeapi-load-balancer charms have gained an nrpe-external-master relation, allowing the integration of their monitoring in an external Nagios server. (#41923, @Cynerva)
- make kubectl describe pod show tolerationSeconds (#42162, @kevin-wangzefeng)
- Completed pods should not be hidden when requested by name via
kubectl get. (#42216, @smarterclayton)
- [Federation][Kubefed] Flag cleanup (#41335, @irfanurrehman)
- Add the support to the scheduler for spreading pods of StatefulSets. (#41708, @bsalamat)
- Portworx Volume Plugin added enabling Portworx to be used as a storage provider for Kubernetes clusters. Portworx pools your servers capacity and turns your servers or cloud instances into converged, highly available compute and storage nodes. (#39535, @adityadani)
- Remove support for trusty in GCE kube-up. (#41670, @mikedanese)
- Import a natural sorting library and use it in the sorting printer. (#40746, @matthyx)
- Parameter keys in a StorageClass
parameters map may not use the kubernetes.io or k8s.io namespaces. (#41837, @liggitt)
- Make DaemonSet respect critical pods annotation when scheduling. (#42028, @janetkuo)
- New Kubelet flag
--enforce-node-allocatable with a default value of pods is added which will make kubelet create a top level cgroup for all pods to enforce Node Allocatable. Optionally, system-reserved & kube-reserved values can also be specified separated by comma to enforce node allocatable on cgroups specified via --system-reserved-cgroup & --kube-reserved-cgroup respectively. Note the default value of the latter flags are "". (#41234, @vishh)
- This feature requires a Node Drain prior to upgrade failing which pods will be restarted if possible or terminated if they have a
RestartNever policy.
- Deployment of AWS Kubernetes clusters using the in-tree bash deployment (i.e. cluster/kube-up.sh or get-kube.sh) is obsolete. v1.5.x will be the last release to support cluster/kube-up.sh with AWS. For a list of viable alternatives, see: http://kubernetes.io/docs/getting-started-guides/aws/ (#42196, @zmerlynn)
- kubectl logs allows getting logs directly from deployment, job and statefulset (#40927, @soltysh)
- make kubectl taint command respect effect NoExecute (#42120, @kevin-wangzefeng)
- Flex volume plugin is updated to support attach/detach interfaces. It broke backward compatibility. Please update your drivers and implement the new callouts. (#41804, @chakri-nelluri)
- Implement the update feature for DaemonSet. (#41116, @lukaszo)
- [Federation] Create configmap for the cluster kube-dns when cluster joins and remove when it unjoins (#39338, @irfanurrehman)
- New GKE certificates controller. (#41160, @pipejakob)
- Juju: Fix shebangs in charm actions to use python3 (#42058, @Cynerva)
- Support kubectl apply set-last-applied command to update the applied-applied-configuration annotation (#41694, @shiywang)
- On GCI by default logrotate is disabled for application containers in favor of rotation mechanism provided by docker logging driver. (#40634, @Crassirostris)
- Cleanup fluentd-gcp image: rebase on debian-base, switch to upstream packages, remove fluent-ui & rails (#41998, @timstclair)
- Updating apiserver to return http status code 202 for a delete request when the resource is not immediately deleted because of user requesting cascading deletion using DeleteOptions.OrphanDependents=false. (#41165, @nikhiljindal)
- [Federation][kubefed] Support configuring dns-provider (#40528, @shashidharatd)
- Added support to minimize sending verbose node information to scheduler extender by sending only node names and expecting extenders to cache the rest of the node information (#41119, @sarat-k)
- Guaranteed admission for Critical Pods (#40952, @dashpole)
- Switch to the
node-role.kubernetes.io/master label for marking and tainting the master node in kubeadm (#41835, @luxas)
- Allow drain --force to remove pods whose managing resource is deleted. (#41864, @marun)
- add kubectl can-i to see if you can perform an action (#41077, @deads2k)
- enable DefaultTolerationSeconds admission controller by default (#41815, @kevin-wangzefeng)
- Make DaemonSets survive taint-based evictions when nodes turn unreachable/notReady. (#41896, @kevin-wangzefeng)
- Add configurable limits to CronJob resource to specify how many successful and failed jobs are preserved. (#40932, @peay)
- Deprecate outofdisk-transition-frequency and low-diskspace-threshold-mb flags (#41941, @dashpole)
- Add OWNERS for sample-apiserver in staging (#42094, @sttts)
- Update gcr.io/google-containers/rescheduler to v0.2.2, which uses busybox as a base image instead of ubuntu. (#41911, @ixdy)
- Add storage.k8s.io/v1 API (#40088, @jsafrane)
- Juju - K8s master charm now properly keeps distributed master files in sync for an HA control plane. (#41351, @chuckbutler)
- Fix zsh completion: unknown file attribute error (#38104, @elipapa)
- kubelet config should ignore file start with dots (#39196, @resouer)
- Add an alpha feature that makes NodeController set Taints instead of deleting Pods from not Ready Nodes. (#41133, @gmarek)
- Base etcd-empty-dir-cleanup on busybox, run as nobody, and update to etcdctl 3.0.14 (#41674, @ixdy)
- Fix zone placement heuristics so that multiple mounts in a StatefulSet pod are created in the same zone (#40910, @justinsb)
- Flag --use-kubernetes-version for kubeadm init renamed to --kubernetes-version (#41820, @kad)
kube-dns now runs using a separate system:serviceaccount:kube-system:kube-dns service account which is automatically bound to the correct RBAC permissions. (#38816, @deads2k)- [Kubemark] Fixed hollow-npd container command to log to file (#41858, @shyamjvs)
- kubeadm: Remove the --cloud-provider flag for beta init UX (#41710, @luxas)
- The CertificateSigningRequest API added the
extra field to persist all information about the requesting user. This mirrors the fields in the SubjectAccessReview API used to check authorization. (#41755, @liggitt)
- Upgrade golang versions to 1.7.5 (#41771, @cblecker)
- Added a new secret type "bootstrap.kubernetes.io/token" for dynamically creating TLS bootstrapping bearer tokens. (#41281, @ericchiang)
- Remove unnecessary metrics (http/process/go) from being exposed by etcd-version-monitor (#41807, @shyamjvs)
- Added
kubectl create clusterrole command. (#41538, @xingzhou)
- Support new kubectl apply view-last-applied command for viewing the last configuration file applied (#41146, @shiywang)
- Bump GCI to gci-stable-56-9000-84-2 (#41819, @dchen1107)
- list-resources: don't fail if the grep fails to match any resources (#41933, @ixdy)
- client-go no longer imports GCP OAuth2 and OpenID Connect packages by default. (#41532, @ericchiang)
- Each pod has its own associated cgroup by default. (#41349, @derekwaynecarr)
- Whitelist kubemark in node_ssh_supported_providers for log dump (#41800, @shyamjvs)
- Support KUBE_MAX_PD_VOLS on Azure (#41398, @codablock)
- Projected volume plugin (#37237, @jpeeler)
--output-version is ignored for all commands except kubectl convert. This is consistent with the generic nature of kubectl CRUD commands and the previous removal of --api-version. Specific versions can be specified in the resource field: resource.version.group, jobs.v1.batch. (#41576, @deads2k)- Added bool type support for jsonpath. (#39063, @xingzhou)
- Nodes can now report two additional address types in their status: InternalDNS and ExternalDNS. The apiserver can use
--kubelet-preferred-address-types to give priority to the type of address it uses to reach nodes. (#34259, @liggitt)
- Clients now use the
?watch=true parameter to make watch API calls, instead of the /watch/ path prefix (#41722, @liggitt)
- ResourceQuota ability to support default limited resources (#36765, @derekwaynecarr)
- Fix kubemark default e2e test suite's name (#41751, @shyamjvs)
- federation aws: add logging of route53 calls (#39964, @justinsb)
- Fix ConfigMap for Windows Containers. (#39373, @jbhurat)
- add defaultTolerationSeconds admission controller (#41414, @kevin-wangzefeng)
- Node Problem Detector is beta now. New features added: journald support, standalone mode and arbitrary system log monitoring. (#40206, @Random-Liu)
- Fix the output of health-mointor.sh (#41525, @yujuhong)
- kubectl describe no longer prints the last-applied-configuration annotation for secrets. (#34664, @ymqytw)
- Report node not ready on failed PLEG health check (#41569, @yujuhong)
- Delay Deletion of a Pod until volumes are cleaned up (#41456, @dashpole)
- Alpha version of dynamic volume provisioning is removed in this release. Annotation (#40000, @jsafrane)
- An
automountServiceAccountToken *bool field was added to ServiceAccount and PodSpec objects. If set to false on a pod spec, no service account token is automounted in the pod. If set to false on a service account, no service account token is automounted for that service account unless explicitly overridden in the pod spec. (#37953, @liggitt)
- Bump addon-manager version to v6.4-alpha.1 in kubemark (#41506, @shyamjvs)
- Do not daemonize
salt-minion for the openstack-heat provider. (#40722, @micmro)
- Move private key parsing from serviceaccount/jwt.go to client-go/util/cert (#40907, @cblecker)
- Added configurable etcd initial-cluster-state to kube-up script. (#41332, @jszczepkowski)
v1.6.0-alpha.3
Documentation & Examples
Downloads for v1.6.0-alpha.3
| filename |
sha256 hash |
| kubernetes.tar.gz |
41b5e9edd973cbb8e68eb5d8d758c4f0afa11dfbd65df49e1c361206706a974c |
| kubernetes-src.tar.gz |
ec13e22322c85752918c23b0b498ba02087a1227b8fdc169f19acdf128f907c4 |
Client Binaries
Server Binaries
Changelog since v1.6.0-alpha.2
Other notable changes
- Fix AWS device allocator to only use valid device names (#41455, @gnufied)
- [Federation][Kubefed] Bug fix relating kubeconfig path in kubefed init (#41410, @irfanurrehman)
- The apiserver audit log (
/var/log/kube-apiserver-audit.log) will be sent through fluentd if enabled. (#41360, @enisoc)
- Bump GCE ContainerVM to container-vm-v20170214 to address CVE-2016-9962. (#41449, @zmerlynn)
- Fixed issues #39202, #41041 and #40941 that caused the iSCSI connections to be prematurely closed when deleting a pod with an iSCSI persistent volume attached and that prevented the use of newly created LUNs on targets with preestablished connections. (#41196, @CristianPop)
- The kube-apiserver basic audit log can be enabled in GCE by exporting the environment variable
ENABLE_APISERVER_BASIC_AUDIT=true before running cluster/kube-up.sh. This will log to /var/log/kube-apiserver-audit.log and use the same logrotate settings as /var/log/kube-apiserver.log. (#41211, @enisoc)
- On kube-up.sh clusters on GCE, kube-scheduler now contacts the API on the secured port. (#41285, @liggitt)
- Default RBAC ClusterRole and ClusterRoleBinding objects are automatically updated at server start to add missing permissions and subjects (extra permissions and subjects are left in place). To prevent autoupdating a particular role or rolebinding, annotate it with
rbac.authorization.kubernetes.io/autoupdate=false. (#41155, @liggitt)
- Make EnableCRI default to true (#41378, @yujuhong)
kubectl edit now edits objects exactly as they were retrieved from the API. This allows using kubectl edit with third-party resources and extension API servers. Because client-side conversion is no longer done, the --output-version option is deprecated for kubectl edit. To edit using a particular API version, fully-qualify the resource, version, and group used to fetch the object (for example, job.v1.batch/myjob) (#41304, @liggitt)- We change the default attach_detach_controller sync period to 1 minute to reduce the query frequency through cloud provider to check whether volumes are attached or not. (#41363, @jingxu97)
- RBAC
v1beta1 RoleBinding/ClusterRoleBinding subjects changed apiVersion to apiGroup to fully-qualify a subject. ServiceAccount subjects default to an apiGroup of "", User and Group subjects default to an apiGroup of "rbac.authorization.k8s.io". (#41184, @liggitt)
- Add support for finalizers in federated configmaps (deletes configmaps from underlying clusters). (#40464, @csbell)
- Make DaemonSet controller respect node taints and pod tolerations. (#41172, @janetkuo)
- Added kubectl create role command (#39852, @xingzhou)
- If
experimentalCriticalPodAnnotation feature gate is set to true, fluentd pods will not be evicted by the kubelet. (#41035, @vishh)
v1.6.0-alpha.2
Documentation & Examples
Downloads for v1.6.0-alpha.2
| filename |
sha256 hash |
| kubernetes.tar.gz |
d1a5c7bc435c0f58dca9eab54e8155b6e4797d4b5d6b0cb8feab968dd3132165 |
| kubernetes-src.tar.gz |
8d09b973f3debfe3d10b0ad392e56141446bc0d04aac60df6d761189997d97ed |
Client Binaries
Server Binaries
Changelog since v1.6.0-alpha.1
Other notable changes
- Align the hyperkube image to support running binaries at /usr/local/bin/ like the other server images (#41017, @luxas)
- Native support for token based bootstrap flow. This includes signing a well known ConfigMap in the
kube-public namespace and cleaning out expired tokens. (#36101, @jbeda)
- Reverts to looking up the current VM in vSphere using the machine's UUID, either obtained via sysfs or via the
vm-uuid parameter in the cloud configuration file. (#40892, @robdaemon)
- This PR adds a manager to NodeController that is responsible for removing Pods from Nodes tainted with NoExecute Taints. This feature is beta (as the rest of taints) and enabled by default. It's gated by controller-manager enable-taint-manager flag. (#40355, @gmarek)
- The authentication.k8s.io API group was promoted to v1 (#41058, @liggitt)
- Fixes issue #38418 which, under circumstance, could cause StatefulSet to deadlock. (#40838, @kow3ns)
- Mediates issue #36859. StatefulSet only acts on Pods whose identity matches the StatefulSet, providing a partial mediation for overlapping controllers.
- Introduces an new alpha version of the Horizontal Pod Autoscaler including expanded support for specifying metrics. (#36033, @DirectXMan12)
- Set all node conditions to Unknown when node is unreachable (#36592, @andrewsykim)
- [Federation] Add override flags options to kubefed init (#40917, @irfanurrehman)
- Fix for detach volume when node is not present/ powered off (#40118, @BaluDontu)
- Bump up GLBC version from 0.9.0-beta to 0.9.1 (#41037, @bprashanth)
- The deprecated flags --config, --auth-path, --resource-container, and --system-container were removed. (#40048, @mtaufen)
- Add
kubectl attach support for multiple types (#40365, @shiywang)
- [Kubelet] Delay deletion of pod from the API server until volumes are deleted (#41095, @dashpole)
- remove the create-external-load-balancer flag in cmd/expose.go (#38183, @tianshapjq)
- The authorization.k8s.io API group was promoted to v1 (#40709, @liggitt)
- Bump GCI to gci-beta-56-9000-80-0 (#41027, @dchen1107)
- [Federation][kubefed] Add option to expose federation apiserver on nodeport service (#40516, @shashidharatd)
- Rename --experiemental-cgroups-per-qos to --cgroups-per-qos (#39972, @derekwaynecarr)
- PV E2E: provide each spec with a fresh nfs host (#40879, @copejon)
- Remove the temporary fix for pre-1.0 mirror pods (#40877, @yujuhong)
- fix --save-config in create subcommand (#40289, @xilabao)
- We should mention the caveats of in-place upgrade in release note. (#40727, @Random-Liu)
- The SubjectAccessReview API passes subresource and resource name information to the authorizer to answer authorization queries. (#40935, @liggitt)
- When feature gate "ExperimentalCriticalPodAnnotation" is set, Kubelet will avoid evicting pods in "kube-system" namespace that contains a special annotation -
scheduler.alpha.kubernetes.io/critical-pod (#40655, @vishh)
- make tolerations respect wildcard key (#39914, @kevin-wangzefeng)
- [Federation][kubefed] Add option to disable persistence storage for etcd (#40862, @shashidharatd)
- Init containers have graduated to GA and now appear as a field. The beta annotation value will still be respected and overrides the field value. (#38382, @hodovska)
- apply falls back to generic 3-way JSON merge patch if no go struct is registered for the target GVK (#40666, @ymqytw)
- HorizontalPodAutoscaler is no longer supported in extensions/v1beta1 version. Use autoscaling/v1 instead. (#35782, @piosz)
- Port forwarding can forward over websockets or SPDY. (#33684, @fraenkel)
- Improve kubectl describe node output by adding closing paren (#39217, @luksa)
- Bump GCE ContainerVM to container-vm-v20170201 to address CVE-2016-9962. (#40828, @zmerlynn)
- Use full package path for definition name in OpenAPI spec (#40124, @mbohlool)
- kubectl apply now supports explicitly clearing values not present in the config by setting them to null (#40630, @liggitt)
- Fixed an issue where 'kubectl get --sort-by=' would return an error when the specified field were not present in at least one of the returned objects, even that being a valid field in the object model. (#40541, @fabianofranz)
- Add initial french translations for kubectl (#40645, @brendandburns)
- OpenStack-Heat will now look for an image named "CentOS-7-x86_64-GenericCloud-1604". To restore the previous behavior set OPENSTACK_IMAGE_NAME="CentOS7" (#40368, @sc68cal)
- Preventing nil pointer reference in client_config (#40508, @vjsamuel)
- The bash AWS deployment via kube-up.sh has been deprecated. See http://kubernetes.io/docs/getting-started-guides/aws/ for alternatives. (#38772, @zmerlynn)
- Fix failing load balancers in Azure (#40405, @codablock)
- kubefed init creates a service account for federation controller manager in the federation-system namespace and binds that service account to the federation-system:federation-controller-manager role that has read and list access on secrets in the federation-system namespace. (#40392, @madhusudancs)
- Fixed an SELinux issue in kubeadm on Docker 1.12+ by moving etcd SELinux options from container to pod. (#40682, @dgoodwin)
- Juju kubernetes-master charm: improve status messages (#40691, @Cynerva)
v1.6.0-alpha.1
Documentation & Examples
Downloads for v1.6.0-alpha.1
| filename |
sha256 hash |
| kubernetes.tar.gz |
abda73bc2a27ae16c66a5aea9e96cd59486ed8cf994afc55da35a3cea2edc1db |
| kubernetes-src.tar.gz |
b429579ba83f9a3fa80e72ceb65b046659b17f16a0b7f70105e7096a441f32b9 |
Client Binaries
Server Binaries
Changelog since v1.5.0
Action Required
- Promote certificates.k8s.io to beta and enable it by default. Users using the alpha certificates API should delete v1alpha1 CSRs from the API before upgrading and recreate them as v1beta1 CSR after upgrading. (#39772, @mikedanese)
- Switch default etcd version to 3.0.14. (#36229, @wojtek-t)
- Switch default storage backend flag in apiserver to
etcd3 mode.
- RBAC's special handling of the user
* in RoleBinding and ClusterRoleBinding objects is deprecated and will be removed in v1beta1. To match all users, explicitly bind to the group system:authenticated and/or system:unauthenticated. Existing v1alpha1 bindings to the user * will be automatically converted to the group system:authenticated. (#38981, @liggitt)
- The 'endpoints.beta.kubernetes.io/hostnames-map' annotation is no longer supported. Users can use the 'Endpoints.subsets[].addresses[].hostname' field instead. (#39284, @bowei)
federation/deploy/deploy.sh was an interim solution introduced in Kubernetes v1.4 to simplify the federation control plane deployment experience. Now that we have kubefed, we are deprecating deploy.sh scripts. (#38902, @madhusudancs)- Cluster federation servers have changed the location in etcd where federated services are stored, so existing federated services must be deleted and recreated. Before upgrading, export all federated services from the federation server and delete the services. After upgrading the cluster, recreate the federated services from the exported data. (#37770, @enj)
- etcd2: watching from 0 returns all initial states as ADDED events (#38079, @hongchaodeng)
Other notable changes
- kube-up.sh on GCE now includes the bootstrap admin in the super-user group, and ensures the auth token file is correct on upgrades (#39537, @liggitt)
- genericapiserver: cut off more dependencies – episode 3 (#40426, @sttts)
- Adding vmdk file extension for vmDiskPath in vsphere DeleteVolume (#40538, @divyenpatel)
- Remove outdated net.experimental.kubernetes.io/proxy-mode and net.beta.kubernetes.io/proxy-mode annotations from kube-proxy. (#40585, @cblecker)
- Improve the ARM builds and make hyperkube on ARM working again by upgrading the Go version for ARM to go1.8beta2 (#38926, @luxas)
- Prevent hotloops on error conditions, which could fill up the disk faster than log rotation can free space. (#40497, @lavalamp)
- DaemonSet controller actively kills failed pods (to recreate them) (#40330, @janetkuo)
- forgiveness alpha version api definition (#39469, @kevin-wangzefeng)
- Bump up glbc version to 0.9.0-beta.1 (#40565, @bprashanth)
- Improve formatting of EventSource in kubectl get and kubectl describe (#40073, @matthyx)
- CRI: use more gogoprotobuf plugins (#40397, @yujuhong)
- Adds
shortNames to the APIResource from discovery which is a list of recommended shortNames for clients like kubectl. (#40312, @p0lyn0mial)
- Use existing ABAC policy file when upgrading GCE cluster (#40172, @liggitt)
- Added support for creating HA clusters for centos using kube-up.sh. (#39462, @Shawyeok)
- azure: fix Azure Container Registry integration (#40142, @colemickens)
-
- Splits Juju Charm layers into master/worker roles (#40324, @chuckbutler)
-
- Adds support for 1.5.x series of Kubernetes
-
- Introduces a tactic for keeping templates in sync with upstream eliminating template drift
-
- Adds CNI support to the Juju Charms
-
- Adds durable storage support to the Juju Charms
-
- Introduces an e2e Charm layer for repeatable testing efforts and validation of clusters
- genericapiserver: more dependency cutoffs (#40216, @sttts)
- AWS: trust region if found from AWS metadata (#38880, @justinsb)
- Volumes and environment variables populated from ConfigMap and Secret objects can now tolerate the named source object or specific keys being missing, by adding
optional: true to the volume or environment variable source specifications. (#39981, @fraenkel)
- kubectl create now accepts the label selector flag for filtering objects to create (#40057, @MrHohn)
- A new field
terminationMessagePolicy has been added to containers that allows a user to request FallbackToLogsOnError, which will read from the container's logs to populate the termination message if the user does not write to the termination message log file. The termination message file is now properly readable for end users and has a maximum size (4k bytes) to prevent abuse. Each pod may have up to 12k bytes of termination messages before the contents of each will be truncated. (#39341, @smarterclayton)
- Add a special purpose tool for editing individual fields in a ConfigMap with kubectl (#38445, @brendandburns)
- [Federation] Expose autoscaling apis through federation api server (#38976, @irfanurrehman)
- Powershell script to start kubelet and kube-proxy (#36250, @jbhurat)
- Reduce time needed to attach Azure disks (#40066, @codablock)
- fixing Cassandra shutdown example to avoid data corruption (#39199, @deimosfr)
- kubeadm: add optional self-hosted deployment for apiserver, controller-manager and scheduler. (#40075, @pires)
- kubelet tears down pod volumes on pod termination rather than pod deletion (#37228, @sjenning)
- The default client certificate generated by kube-up now contains the superuser
system:masters group (#39966, @liggitt)
- CRI: upgrade protobuf to v3 (#39158, @feiskyer)
- Add SIGCHLD handler to pause container (#36853, @verb)
- Populate environment variables from a secrets. (#39446, @fraenkel)
- fluentd config for GKE clusters updated: detect exceptions in container log streams and forward them as one log entry. (#39656, @thomasschickinger)
- Made multi-scheduler graduated to Beta and then v1. (#38871, @k82cn)
- Fixed forming resolver search line for pods: exclude duplicates, obey libc limitations, logging and eventing appropriately. (#29666, @vefimova)
- Add authorization mode to kubeadm (#39846, @andrewrynhard)
- Update dependencies: aws-sdk-go to 1.6.10; also cadvisor (#40095, @dashpole)
- Fixes a bug in the OpenStack-Heat kubernetes provider, in the handling of differences between the Identity v2 and Identity v3 APIs (#40105, @sc68cal)
- Update GCE ContainerVM deployment to container-vm-v20170117 to pick up CVE fixes in base image. (#40094, @zmerlynn)
- AWS: Remove duplicate calls to DescribeInstance during volume operations (#39842, @gnufied)
- The
attributeRestrictions field has been removed from the PolicyRule type in the rbac.authorization.k8s.io/v1alpha1 API. The field was not used by the RBAC authorizer. (#39625, @deads2k)
- Enable lazy inode table and journal initialization for ext3 and ext4 (#38865, @codablock)
- azure disk: restrict name length for Azure specifications (#40030, @colemickens)
- Follow redirects for streaming requests (exec/attach/port-forward) in the apiserver by default (alpha -> beta). (#40039, @timstclair)
- Use kube-dns:1.11.0 (#39925, @sadlil)
- Anonymous authentication is now automatically disabled if the API server is started with the AlwaysAllow authorizer. (#38706, @deads2k)
- genericapiserver: cut off kube pkg/version dependency (#39943, @sttts)
- genericapiserver: cut off pkg/serviceaccount dependency (#39945, @sttts)
- Move pkg/api/rest into genericapiserver (#39948, @sttts)
- genericapiserver: cut off pkg/apis/extensions and pkg/storage dependencies (#39946, @sttts)
- genericapiserver: cut off certificates api dependency (#39947, @sttts)
- Admission control support for versioned configuration files (#39109, @derekwaynecarr)
- Fix issue around merging lists of primitives when using PATCH or kubectl apply. (#38665, @ymqytw)
- Fixes API compatibility issue with empty lists incorrectly returning a null
items field instead of an empty array. (#39834, @liggitt)
- [scheduling] Moved pod affinity and anti-affinity from annotations to api fields #25319 (#39478, @rrati)
- PodSecurityPolicy resource is now enabled by default in the extensions API group. (#39743, @pweil-)
- add --controllers to controller manager (#39740, @deads2k)
- proxy/iptables: don't sync proxy rules if services map didn't change (#38996, @dcbw)
- Update amd64 kube-proxy base image to debian-iptables-amd64:v5 (#39725, @ixdy)
- Update dashboard version to v1.5.1 (#39662, @rf232)
- Fix kubectl get -f -o so it prints all items in the file (#39038, @ncdc)
- Scheduler treats StatefulSet pods as belonging to a single equivalence class. (#39718, @foxish)
- --basic-auth-file supports optionally specifying groups in the fourth column of the file (#39651, @liggitt)
- To create or update an RBAC RoleBinding or ClusterRoleBinding object, a user must: (#39383, @liggitt)
- Be authorized to make the create or update API request
- Be allowed to bind the referenced role, either by already having all of the permissions contained in the referenced role, or by having the
bind permission on the referenced role.
- Fixes an HPA-related panic due to division-by-zero. (#39694, @DirectXMan12)
- federation: Adding support for DeleteOptions.OrphanDependents for federated services. Setting it to false while deleting a federated service also deletes the corresponding services from all registered clusters. (#36390, @nikhiljindal)
- Update kube-proxy image to be based off of Debian 8.6 base image. (#39695, @ixdy)
- Update FitError as a message component into the PodConditionUpdater. (#39491, @jayunit100)
- rename kubernetes-discovery to kube-aggregator (#39619, @deads2k)
- Allow missing keys in templates by default (#39486, @ncdc)
- Caching added to the OIDC client auth plugin to fix races and reduce the time kubectl commands using this plugin take by several seconds. (#38167, @ericchiang)
- AWS: recognize eu-west-2 region (#38746, @justinsb)
- Provide kubernetes-controller-manager flags to control volume attach/detach reconciler sync. The duration of the syncs can be controlled, and the syncs can be shut off as well. (#39551, @chrislovecnm)
- Generate OpenAPI definition for inlined types (#39466, @mbohlool)
- ShortcutExpander has been extended in a way that it will examine a ha… (#38835, @p0lyn0mial)
- fixes nil dereference when doing a volume type check on persistent volumes (#39493, @sjenning)
- Fix issue with PodDisruptionBudgets in which
minAvailable specified as a percentage did not work with StatefulSet Pods. (#39454, @foxish)
- fix issue with kubectl proxy so that it will proxy an empty path - e.g. http://localhost:8001 (#39226, @luksa)
- Check if pathExists before performing Unmount (#39311, @rkouj)
- Adding kubectl tests for federation (#38844, @nikhiljindal)
- Fix comment and optimize code (#38084, @tanshanshan)
- When using OIDC authentication and specifying --oidc-username-claim=email, an
"email_verified":true claim must be returned from the identity provider. (#36087, @ericchiang)
- Allow pods to define multiple environment variables from a whole ConfigMap (#36245, @fraenkel)
- Refactor the certificate and kubeconfig code in the kubeadm binary into two phases (#39280, @luxas)
- Added support for printing in all supported
--output formats to kubectl create ... and kubectl apply ... (#38112, @juanvallejo)
- genericapiserver: extract CA cert from server cert and SNI cert chains (#39022, @sttts)
- Endpoints, that tolerate unready Pods, are now listing Pods in state Terminating as well (#37093, @simonswine)
- DaemonSet ObservedGeneration (#39157, @lukaszo)
- The
--reconcile-cidr kubelet flag was removed since it had been deprecated since v1.5 (#39322, @luxas)
- Add ready replicas in Deployments (#37959, @kargakis)
- Remove all MAINTAINER statements in Dockerfiles in the codebase as they are deprecated by docker (#38927, @luxas)
- Remove the deprecated vsphere kube-up. (#39140, @kerneltime)
- Kubectl top now also accepts short forms for "node" and "pod" ("no", "po") (#39218, @luksa)
- Remove 'exec' network plugin - use CNI instead (#39254, @freehan)
- Add three more columns to
kubectl get deploy -o wide output. (#39240, @xingzhou)
- Add path exist check in getPodVolumePathListFromDisk (#38909, @jingxu97)
- Begin paths for internationalization in kubectl (#36802, @brendandburns)
- Fixes an issue where commas were not accepted in --from-literal flags when creating secrets. Passing multiple values separated by a comma in a single --from-literal flag is no longer supported. Please use multiple --from-literal flags to provide multiple values. (#35191, @SamiHiltunen)
- Support loading UTF16 files if a byte-order-mark is present (#39008, @brendandburns)
- Fix fsGroup to vSphere (#38655, @abrarshivani)
- ReplicaSet has onwer ref of the Deployment that created it (#35676, @krmayankk)
- Don't evict static pods (#39059, @bprashanth)
- Fixed a bug where the --server, --token, and --certificate-authority flags were not overriding the related in-cluster configs when provided in a
kubectl call inside a cluster. (#39006, @fabianofranz)
- Make fluentd pods critical (#39146, @Crassirostris)
- assign -998 as the oom_score_adj for critical pods (e.g. kube-proxy) (#39114, @dchen1107)
- delete continue in monitorNodeStatus (#38798, @NickrenREN)
- add create rolebinding (#38991, @deads2k)
- Add new command "kubectl set selector" (#38966, @kargakis)
- Federation: Add
batch/jobs API objects to federation-apiserver (#35943, @jianhuiz)
- ABAC policies using
"user":"*" or "group":"*" to match all users or groups will only match authenticated requests. To match unauthenticated requests, ABAC policies must explicitly specify "group":"system:unauthenticated" (#38968, @liggitt)
- To add local registry to libvirt_coreos (#36751, @sdminonne)
- Add a KUBERNETES_NODE_* section to build kubelet/kube-proxy for windows (#38919, @brendandburns)
- Added kubeadm commands to manage bootstrap tokens and the duration they are valid for. (#35805, @dgoodwin)
- AWS: Recognize ca-central-1 region (#38410, @justinsb)
- Unmount operation should not fail if volume is already unmounted (#38547, @rkouj)
- Changed default scsi controller type in vSphere Cloud Provider (#38426, @abrarshivani)
- Move non-generic apiserver code out of the generic packages (#38191, @sttts)
- Remove extensions/v1beta1 Jobs resource, and job/v1beta1 generator. (#38614, @soltysh)
- Admit critical pods in the kubelet (#38836, @bprashanth)
- Use daemonset in docker registry add on (#35582, @surajssd)
- Node affinity has moved from annotations to api fields in the pod spec. Node affinity that is defined in the annotations will be ignored. (#37299, @rrati)
- Since
kubernetes.tar.gz no longer includes client or server binaries, cluster/kube-{up,down,push}.sh now automatically download released binaries if they are missing. (#38730, @ixdy)
- genericapiserver: turn APIContainer.SecretRoutes into a real ServeMux (#38826, @sttts)
- Migrated fluentd addon to daemon set (#32088, @piosz)
- AWS: Add sequential allocator for device names. (#38818, @jsafrane)
- Add 'X-Content-Type-Options: nosniff" to some error messages (#37190, @brendandburns)
- Display pod node selectors with kubectl describe. (#36396, @aveshagarwal)
- The main repository does not keep multiple releases of clientsets anymore. Please find previous releases at https://github.com/kubernetes/client-go (#38154, @caesarxuchao)
- Remove a release-note on multiple OpenAPI specs (#38732, @mbohlool)
- genericapiserver: unify swagger and openapi in config (#38690, @sttts)
- fix connection upgrades through kuberentes-discovery (#38724, @deads2k)
- Fixes bug in resolving client-requested API versions (#38533, @DirectXMan12)
- apiserver(s): Replace glog.Fatals with fmt.Errorfs (#38175, @sttts)
- Remove Azure Subnet RouteTable check (#38334, @mogthesprog)
- Ensure the GCI metadata files do not have newline at the end (#38727, @Amey-D)
- Significantly speed-up make (#38700, @sttts)
- add QoS pod status field (#37968, @sjenning)
- Fixed validation of multizone cluster for GCE (#38695, @jszczepkowski)
- Fixed detection of master during creation of multizone nodes cluster by kube-up. (#38617, @jszczepkowski)
- Update CHANGELOG.md to warn about anon auth flag (#38675, @erictune)
- Fixes NotAuthenticated errors that appear in the kubelet and kube-controller-manager due to never logging in to vSphere (#36169, @robdaemon)
- Fix an issue where AWS tear-down leaks an DHCP Option Set. (#38645, @zmerlynn)
- Issue a warning when using
kubectl apply on a resource lacking the LastAppliedConfig annotation (#36672, @ymqytw)
- Re-add /healthz/ping handler in genericapiserver (#38603, @sttts)
- Fail kubelet if runtime is unresponsive for 30 seconds (#38527, @derekwaynecarr)
- Add support for Azure Container Registry, update Azure dependencies (#37783, @brendandburns)
- Fix panic in vSphere cloud provider (#38423, @BaluDontu)
- fix broken cluster/centos and enhance the style (#34002, @xiaoping378)
- Ability to quota storage by storage class (#34554, @derekwaynecarr)
- [Part 2] Adding s390x cross-compilation support for gcr.io images in this repo (#36050, @gajju26)
- kubectl run --rm no longer prints "pod xxx deleted" (#38429, @duglin)
- Bump GCE debian image to container-vm-v20161208 (release notes) (#38432, @timstclair)
- Kubelet: Add image cache. (#38375, @Random-Liu)
- Allow a selector when retrieving logs (#32752, @fraenkel)
- Fix unmountDevice issue caused by shared mount in GCI (#38411, @jingxu97)
- [Federation] Implement dry run support in kubefed init (#36447, @irfanurrehman)
- Fix space issue in volumePath with vSphere Cloud Provider (#38338, @BaluDontu)
- [Federation] Make federation etcd PVC size configurable (#36310, @irfanurrehman)
- Allow no ports when exposing headless service (#32811, @fraenkel)
- Wait for the port to be ready before starting (#38260, @fraenkel)
- Add Version to the resource printer for 'get nodes' (#37943, @ailusazh)
- contribute deis/registry-proxy as a replacement for kube-registry-proxy (#35797, @bacongobbler)
- [Part 1] Add support for cross-compiling s390x binaries (#37092, @gajju26)
- kernel memcg notification enabled via experimental flag (#38258, @derekwaynecarr)
- fix permissions when using fsGroup (#37009, @sjenning)
- Pipe get options to storage (#37693, @wojtek-t)
- add a configuration for kubelet to register as a node with taints (#31647, @mikedanese)
- Remove genericapiserver.Options.MasterServiceNamespace (#38186, @sttts)
- The --long-running-request-regexp flag to kube-apiserver is deprecated and will be removed in a future release. Long-running requests are now detected based on specific verbs (watch, proxy) or subresources (proxy, portforward, log, exec, attach). (#38119, @liggitt)
- Better compat with very old iptables (e.g. CentOS 6) (#37594, @thockin)
- Fix GCI mounter issue (#38124, @jingxu97)
- Kubelet will no longer set hairpin mode on every interface on the machine when an error occurs in setting up hairpin for a specific interface. (#36990, @bboreham)
- fix mesos unit tests (#38196, @deads2k)
- Add
--controllers flag to federation controller manager for enable/disable federation ingress controller (#36643, @kzwang)
- Allow backendpools in Azure Load Balancers which are not owned by cloud provider (#36882, @codablock)
- remove rbac super user (#38121, @deads2k)
- API server have two separate limits for read-only and mutating inflight requests. (#36064, @gmarek)
- check the value of min and max in kubectl (#37789, @yarntime)
- The glusterfs dynamic volume provisioner will now choose a unique GID for new persistent volumes from a range that can be configured in the storage class with the "gidMin" and "gidMax" parameters. The default range is 2000 - 2147483647 (max int32). (#37886, @obnoxxx)
- Add kubectl create poddisruptionbudget command (#36646, @kargakis)
- Set kernel.softlockup_panic =1 based on the flag. (#38001, @dchen1107)
- portfordwardtester: avoid data loss during send+close+exit (#37103, @sttts)
- Enable containerized mounter only for nfs and glusterfs types (#37990, @jingxu97)
- Add flag to enable contention profiling in scheduler. (#37357, @gmarek)
- Add kubernetes-anywhere as a new e2e deployment option (#37019, @pipejakob)
- add create clusterrolebinding command (#37098, @deads2k)
- kubectl create service externalname (#34789, @AdoHe)
- Fix logic error in graceful deletion (#37721, @derekwaynecarr)
- Exit with error if is not the final parameter. (#37723, @mtaufen)
- Fix Service Update on LoadBalancerSourceRanges Field (#37720, @freehan)
- Bug fix. Incoming UDP packets not reach newly deployed services (#32561, @zreigz)
- GCI: Remove /var/lib/docker/network (#37593, @yujuhong)
- configure local-up-cluster.sh to handle auth proxies (#36838, @deads2k)
- Add
clusterid, an optional parameter to storageclass. (#36437, @humblec)
- local-up-cluster: avoid sudo for control plane (#37443, @sttts)
- Add error message when trying to use clusterrole with namespace in kubectl (#36424, @xilabao)
kube-up.sh/kube-down.sh no longer force update gcloud for provider=gce|gke. (#36292, @jlowdermilk)- Fix issue when attempting to unmount a wrong vSphere volume (#37413, @BaluDontu)
- Fix the equality checks for numeric values in cluster/gce/util.sh. (#37638, @roberthbailey)
- kubelet: don't reject pods without adding them to the pod manager (#37661, @yujuhong)
- Modify GCI mounter to enable NFSv3 (#37582, @jingxu97)
- Fix photon controller plugin to construct with correct PdID (#37167, @luomiao)
- Collect logs for dead kubelets too (#37671, @mtaufen)
- Set Dashboard UI version to v1.5.0 (#37684, @rf232)
- When deleting an object with
--grace-period=0, the client will begin a graceful deletion and wait until the resource is fully deleted. To force deletion, use the --force flag. (#37263, @smarterclayton)
- federation service controller: stop deleting services from underlying clusters when federated service is deleted. (#37353, @nikhiljindal)
- Fix nil pointer dereference in test framework (#37583, @mtaufen)
- Kubernetes now automatically installs a StorageClass object when deployed on (#31617, @jsafrane)
- AWS, Google Compute Engine, Google Container Engine, and OpenStack using
- the default kube-up.sh scripts. This StorageClass is marked as default so that
- a PersistentVolumeClaim without a StorageClass annotation now results in
- automatic provisioning of storage (GCE PersistentDisk on Google Cloud, AWS
- EBS on AWS, and Cinder on OpenStack). In previous versions of Kubernetes
- a PersistentVolumeClaim without a StorageClass annotation on these cloud
- platforms would be satisfied by manually-created PersistentVolume objects.
- Administrators can choose to disable this behavior by deleting the automatically
- installed StorageClass API object. Alternatively, administrators may choose to
- keep the automatically installed StorageClass and only disable the defaulting
- behavior by removing the "is-default-class" annotation from the StorageClass
- API object.
- Fix TestServiceAlloc flakes (#37487, @wojtek-t)
- Use gsed on the Mac (#37562, @roberthbailey)
- Try self-repair scheduler cache or panic (#37379, @wojtek-t)
- Fluentd/Elastisearch add-on: correctly parse and index kubernetes labels (#36857, @Shrugs)
- Mention overflows when mistakenly call function FromInt (#36487, @xialonglee)
- Update doc for kubectl apply (#37397, @ymqytw)
- Removes shorthand flag -w from kubectl apply (#37345, @MrHohn)
- Curating Owners: pkg/api (#36525, @apelisse)
- Reduce verbosity of volume reconciler when attaching volumes (#36900, @codablock)
- Federated Ingress Proposal (#29793, @quinton-hoole)
Please see the Releases Page for older releases.
Release notes of older releases can be found in:
