Automatic merge from submit-queue (batch tested with PRs 51921, 51829, 51968, 51988, 51986)
COS/GCE: bump the max pids for the docker service
**What this PR does / why we need it**:
TasksMax limits how many threads/processes docker can create. Insufficient limit affects container starts.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*:
fixes#51977
**Special notes for your reviewer**:
**Release note**:
```release-note
Ensure TasksMax is sufficient for docker
```
Automatic merge from submit-queue (batch tested with PRs 51921, 51829, 51968, 51988, 51986)
Fix unbound variable in configure-helper.sh
This isn't plumbed yet on GKE, so results in an unbound variable.
```release-note
NONE
```
During NFS/GlusterFS mount, it requires to have DNS server to be able to
resolve service name. This PR gets the DNS server ip from kubelet and
add it to the containerized mounter path. So if containerized mounter is
used, service name could be resolved during mount
Automatic merge from submit-queue (batch tested with PRs 49727, 51792)
Introducing metrics-server
ref https://github.com/kubernetes/features/issues/271
There is still some work blocked on problems with repo synchronization:
- migrate to `v1beta1` introduced in #51653
- bump deps to HEAD
Will do it in a follow up PRs once the issue is resolved.
```release-note
Introduced Metrics Server
```
Automatic merge from submit-queue
Add RBAC, healthchecks, autoscalers and update Calico to v2.5.1
**What this PR does / why we need it**:
- Updates Calico to `v2.5`
- Calico/node to `v2.5.1`
- Calico CNI to `v1.10.0`
- Typha to `v0.4.1`
- Enable health check endpoints
- Add Readiness probe for calico-node and Typha
- Add Liveness probe for calico-node and Typha
- Add RBAC manifest
- With calico ClusterRole, ServiceAccount and ClusterRoleBinding
- Add Calico CRDs in the Calico manifest (only works for k8s v1.7+)
- Add vertical autoscaler for calico-node and Typha
- Add horizontal autoscaler for Typha
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 51553, 51538, 51663, 51069, 51737)
Allow enable pod priority feature gate for GCE and configure priority for kube-proxy
**What this PR does / why we need it**:
From #23225, this PR adds an option for user to enable pod priority feature gate using GCE startup scripts, and configure pod priority for kube-proxy when enabled.
The setup `priorityClassName: system` derives from: ce1485c626/staging/src/k8s.io/api/core/v1/types.go (L2536-L2542)
The plan is to configure pod priority for kube-proxy daemonset (https://github.com/kubernetes/kubernetes/pull/50705) in the same way.
**Special notes for your reviewer**:
cc @bsalamat @davidopp @thockin
**Release note**:
```release-note
When using kube-up.sh on GCE, user could set env `ENABLE_POD_PRIORITY=true` to enable pod priority feature gate.
```
Automatic merge from submit-queue (batch tested with PRs 51590, 48217, 51209, 51575, 48627)
FlexVolume setup script for COS instance using mounting utility image in GCR.
**What this PR does / why we need it**: This scripts automates FlexVolume installation for a single COS instance. Users need to pre-pack their drivers and mount utilities in a Docker image and upload it to GCR.
For each FlexVolume plugin, the script places a driver wrapper in a writable and executable location. The wrapper calls commands from the actual driver but in a chroot environment, so that mount utilities from the image can be used.
I'm working on a script that automatically executes this on all instances. Will be in a separate PR.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#48626
```release-note
NONE
```
/cc @gmarek @chakri-nelluri
/assign @saad-ali @msau42
/sig storage
/release-note-none
Automatic merge from submit-queue
Adding Flexvolume plugin dir piping for controller manager on COS
**What this PR does / why we need it**: Sets the default Flexvolume plugin directory correctly for controller manager running on COS images.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#51563
```release-note
NONE
```
/release-note-none
/sig storage
/assign @msau42
/cc @wongma7
Automatic merge from submit-queue
Retry master instance creation in case of retriable error (with sleep)
To help with our 5k-node CI tests failing to startup the cluster.
And also towards the greater goal - https://github.com/kubernetes/kubernetes/issues/43140
cc @kubernetes/sig-scalability-misc @kubernetes/sig-cluster-lifecycle-misc
Automatic merge from submit-queue
Add Google cloud KMS service for envelope encryption transformer
This adds the required pieces which will allow addition of KMS based encryption providers (envelope transformer).
For now, we will be implementing it using Google Cloud KMS, but the code should make it easy to add support for any other such provider which can expose Decrypt and Encrypt calls.
Writing tests for Google Cloud KMS Service may cause a significant overhead to the testing framework. It has been tested locally and on GKE though.
Upcoming after this PR:
* Complete implementation of the envelope transformer, which uses LRU cache to maintain decrypted DEKs in memory.
* Track key version to assist in data re-encryption after a KEK rotation.
Development branch containing the changes described above: https://github.com/sakshamsharma/kubernetes/pull/4
Envelope transformer used by this PR was merged in #49350
Concerns #48522
Planned configuration:
```
kind: EncryptionConfig
apiVersion: v1
resources:
- resources:
- secrets
providers:
- kms:
cachesize: 100
configfile: gcp-cloudkms.conf
name: gcp-cloudkms
- identity: {}
```
gcp-cloudkms.conf:
```
[GoogleCloudKMS]
kms-location: global
kms-keyring: google-container-engine
kms-cryptokey: example-key
```
Automatic merge from submit-queue (batch tested with PRs 50932, 49610, 51312, 51415, 50705)
Allow running kube-proxy as a DaemonSet when using kube-up.sh on GCE
**What this PR does / why we need it**:
From #23225, this PR adds an option for user to run kube-proxy as a DaemonSet instead of static pods using GCE startup scripts. By default, kube-proxy will run as static pods.
This is the first step for moving kube-proxy into a DaemonSet in GCE, remaining tasks will be tracked on #23225.
**Special notes for your reviewer**:
The last commit are purely for testing out kube-proxy as daemonset via CIs.
cc @kubernetes/sig-network-misc @kubernetes/sig-cluster-lifecycle-misc
**Release note**:
```release-note
When using kube-up.sh on GCE, user could set env `KUBE_PROXY_DAEMONSET=true` to run kube-proxy as a DaemonSet. kube-proxy is run as static pods by default.
```
Automatic merge from submit-queue (batch tested with PRs 51038, 50063, 51257, 47171, 51143)
update related manifest files to use hostpath type
**What this PR does / why we need it**:
Per [discussion in #46597](https://github.com/kubernetes/kubernetes/pull/46597#pullrequestreview-53568947)
Dependes on #46597
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
Fixes: https://github.com/kubernetes/kubeadm/issues/298
**Special notes for your reviewer**:
/cc @euank @thockin @tallclair @Random-Liu
**Release note**:
```release-note
None
```
Automatic merge from submit-queue (batch tested with PRs 51108, 51035, 50539, 51160, 50947)
Set GCE_ALPHA_FEATURES environment variable in gce.conf
This allows us to gate alpha features in the pkg/cloudprovider/providers/gce.
Automatic merge from submit-queue (batch tested with PRs 50386, 50374, 50444, 50382)
Add explicit API kind and version to the audit policy file on GCE
Adds an explicit API version and kind to the audit policy file in GCE configuration scripts. It's a prerequisite for https://github.com/kubernetes/kubernetes/pull/49115
/cc @tallclair @piosz
Automatic merge from submit-queue (batch tested with PRs 50300, 50328, 50368, 50370, 50372)
Bugfix: set resources only for fluentd-gcp container.
There is more than one container in fluentd-gcp deployment. Previous
implementation was setting resources for all containers, not just
the fluent-gcp one.
**What this PR does / why we need it**:
Bugfix; https://github.com/kubernetes/kubernetes/pull/49009 without this is eating more resources.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#50366
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
There is more than one container in fluentd-gcp deployment. Previous
implementation was setting resources for multiple containers, not just
the fluent-gcp one.
Automatic merge from submit-queue (batch tested with PRs 48487, 49009, 49862, 49843, 49700)
Enable overriding fluentd resources in GCP
**What this PR does / why we need it**: This enables overriding fluentd resources in GCP, when there is a need for custom ones.
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 49989, 49806, 49649, 49412, 49512)
Use existing k8s binaries and images on disk when they are preloaded to gce cos image.
**What this PR does / why we need it**:
This change is to accelerate K8S startup time on gce when k8s tarballs and images are already preloaded in VM image, by skipping the downloading, extracting and file transfer steps.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
**Special notes for your reviewer**:
**Release note**:
```release-note
NONE
```
Automatic merge from submit-queue (batch tested with PRs 49898, 49897, 49919, 48860, 49491)
gce: make append_or_replace.. atomic
Before this change,
* the final echo is not atomically written to the target file
* two concurrent callers will use the same tempfile
Helps with https://github.com/kubernetes/kubernetes/issues/49895
cc @miekg
Nick Sardo