Automatic merge from submit-queue (batch tested with PRs 34763, 38706, 39939, 40020)
Use Statefulset instead in e2e and controller
Quick fix ref: #35534
We should finish the issue to meet v1.6 milestone.
Automatic merge from submit-queue
Move PatchType to apimachinery/pkg/types
Fixes https://github.com/kubernetes/kubernetes/issues/39970
`PatchType` is shared by the client and server, they have to agree, and its critical for our API to function.
@smarterclayton @kubernetes/sig-api-machinery-misc
Automatic merge from submit-queue (batch tested with PRs 39911, 40002, 39969, 40012, 40009)
Fix RBAC role for kube-proxy in Kubemark
Ref #39959
This should ensure that kube-proxy (in Kubemark) has the required role and RBAC binding.
@deads2k PTAL
cc @kubernetes/sig-scalability-misc @wojtek-t @gmarek
Automatic merge from submit-queue (batch tested with PRs 38592, 39949, 39946, 39882)
move api/errors to apimachinery
`pkg/api/errors` is a set of helpers around `meta/v1.Status` that help to create and interpret various apiserver errors. Things like `.NewNotFound` and `IsNotFound` pairings. This pull moves it into apimachinery for use by the clients and servers.
@smarterclayton @lavalamp First commit is the move plus minor fitting. Second commit is straight replace and generation.
Automatic merge from submit-queue (batch tested with PRs 38592, 39949, 39946, 39882)
Add optional per-request context to restclient
**What this PR does / why we need it**: It adds per-request contexts to restclient's API, and uses them to add timeouts to all proxy calls in the e2e tests. An entire e2e shouldn't hang for hours on a single API call.
**Which issue this PR fixes**: #38305
**Special notes for your reviewer**:
This adds a feature to the low-level rest client request feature that is entirely optional. It doesn't affect any requests that don't use it. The api of the generated clients does not change, and they currently don't take advantage of this.
I intend to patch this in to 1.5 as a mostly test only change since it's not going to affect any controller, generated client, or user of the generated client.
cc @kubernetes/sig-api-machinery
cc @saad-ali
Automatic merge from submit-queue
Fix examples e2e permission check
Ref #39382
Follow-up from #39896
Permission check should be done within the e2e test namespace, not cluster-wide
Also improved RBAC audit logging to make the scope of the permission check clearer
Automatic merge from submit-queue
Remove sleep from DynamicProvisioner test.
The comment says that the sleep is there because of 10 minute PV controller
sync. The controller sync is now 15 seconds and it should be quick enough
to hide this in subsequent `WaitForPersistentVolumeDeleted(.. , 20*time.Minute)`
Automatic merge from submit-queue (batch tested with PRs 38427, 39896, 39889, 39871, 39895)
Grant permissions to e2e examples test service account
ref #39382
Automatic merge from submit-queue
Updated unit tests
@janetkuo updated the flaky unit test to have the same structure with regard to uncasting as the rest of the tests. ptal
Automatic merge from submit-queue (batch tested with PRs 39807, 37505, 39844, 39525, 39109)
Made cache.Controller to be interface.
**What this PR does / why we need it**:
#37504
Automatic merge from submit-queue
run staging client-go update
Chasing to see what real problems we have in staging-client-go.
@sttts you get similar results?
Automatic merge from submit-queue
replace global registry in apimachinery with global registry in k8s.io/kubernetes
We'd like to remove all globals, but our immediate problem is that a shared registry between k8s.io/kubernetes and k8s.io/client-go doesn't work. Since client-go makes a copy, we can actually keep a global registry with other globals in pkg/api for now.
@kubernetes/sig-api-machinery-misc @lavalamp @smarterclayton @sttts
Automatic merge from submit-queue
Update images that use ubuntu-slim base image to :0.6
**What this PR does / why we need it**: `ubuntu-slim:0.4` is somewhat old, being based on Ubuntu 16.04, whereas `ubuntu-slim:0.6` is based on Ubuntu 16.04.1.
**Special notes for your reviewer**: I haven't pushed any of these images yet, so I expect all of the e2e builds to fail. If we're happy with the changes, I can push the images and then re-trigger tests.
**Release note**:
```release-note
NONE
```
cc @aledbf as FYI
Automatic merge from submit-queue (batch tested with PRs 39803, 39698, 39537, 39478)
[scheduling] Moved pod affinity and anti-affinity from annotations to api fields #25319
Converted pod affinity and anti-affinity from annotations to api fields
Related: #25319
Related: #34508
**Release note**:
```Pod affinity and anti-affinity has moved from annotations to api fields in the pod spec. Pod affinity or anti-affinity that is defined in the annotations will be ignored.```
Automatic merge from submit-queue
Refactor registry etcd to storage
Fixes#17546
Simple shuffle on naming so any sane new person entering the code base can understand what the actual etcd dependencies are.
Automatic merge from submit-queue (batch tested with PRs 39768, 39463)
Check if path exists before performing unmount
This is part 3 of an effort to check if path exists before performing an unmount operation.
[Part 1](https://github.com/kubernetes/kubernetes/pull/38547) and [part 2](https://github.com/kubernetes/kubernetes/pull/39311) involved auditing the different volume plugins and refactoring their `TearDownAt()s` to use the common util function/or create one if absent.
The ideal way to do this change would involve refactoring of the `TearDownAt()s` of these plugins and make a common util function that checks path. (The plugins involved in this PR use someway of unmounting a bind mount and unmounting a global path, there is also refactoring needed to consolidate disk_manager of fc, rbd and iscsi). A non-goal part of this effort can also involve refactoring all the `SetupAt()s`
In the interest of time and considering other higher priority issues that I am caught up with, I am unable to give the time the refactoring needs. Hence I've made the minimum change that would give the desired output.
I am tracking the work pending in this issue: https://github.com/kubernetes/kubernetes/issues/39251
```release-note
NONE
```
Automatic merge from submit-queue
Add [Volume] tag to all the volume-related E2E tests.
**What this PR does / why we need it**:
Tags all the volume/storage related e2e tests to make it easier to run a volume test suite.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #
fixes#35542
**Special notes for your reviewer**:
Please let me know if there are tests that should/should not be included.
**Release note**:
NONE
```release-note
```
Automatic merge from submit-queue
[CRI] Don't include user data in CRI streaming redirect URLs
Fixes: https://github.com/kubernetes/kubernetes/issues/36187
Avoid userdata in the redirect URLs by caching the {Exec,Attach,PortForward}Requests with a unique token. When the redirect URL is created, the token is substituted for the request params. When the streaming server receives the token request, the token is used to fetch the actual request parameters out of the cache.
For additional security, the token is generated using the secure random function, is single use (i.e. the first request with the token consumes it), and has a short expiration time.
/cc @kubernetes/sig-node
Automatic merge from submit-queue (batch tested with PRs 39475, 38666, 39327, 38396, 39613)
e2e tests: new portforwardertester with another three tests for case …
PR include:
- add new e2e test cases for BIND_ADDRESS='0.0.0.0'
- add to portforwardertester.go os.Getenv("BIND_ADDRESS") and if not set, it should be localhost for backward compability with existing tests
- for existing tests pass explicity BIND_ADDRESS='localhost'
- rename existing tests
It was mention in the issue: #32128
cc @mzylowski @pskrzyns
Automatic merge from submit-queue (batch tested with PRs 39495, 39547)
Tag persistent volume PersistentVolume E2E [Volume][Serial][Flaky]
**What this PR does / why we need it**:
When run parallel with other tests that use PV(C)s, cross-test binding causes flakes. Add `[Serial]` tag.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: f
Partly addresses #39119
**Special notes for your reviewer**:
cc @saad-ali @jsafrane @jeffvance
Automatic merge from submit-queue (batch tested with PRs 39684, 39577, 38989, 39534, 39702)
Set PodStatus QOSClass field
This PR continues the work for https://github.com/kubernetes/kubernetes/pull/37968
It converts all local usage of the `qos` package class types to the new API level types (first commit) and sets the pod status QOSClass field in the at pod creation time on the API server in `PrepareForCreate` and in the kubelet in the pod status update path (second commit). This way the pod QOS class is set even if the pod isn't scheduled yet.
Fixes#33255
@ConnorDoyle @derekwaynecarr @vishh
Automatic merge from submit-queue (batch tested with PRs 39694, 39383, 39651, 39691, 39497)
Fix evictions test
**What this PR does / why we need it**:
Fixes bugs in evictions test. Make vet happy.
**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes#39412
Ref: #39452
cc: @calebamiles
Automatic merge from submit-queue (batch tested with PRs 39694, 39383, 39651, 39691, 39497)
Allow rolebinding/clusterrolebinding with explicit bind permission check
Fixes https://github.com/kubernetes/kubernetes/issues/39176
Fixes https://github.com/kubernetes/kubernetes/issues/39258
Allows creating/updating a rolebinding/clusterrolebinding if the user has explicitly been granted permission to perform the "bind" verb against the referenced role/clusterrole (previously, they could only bind if they already had all the permissions in the referenced role via an RBAC role themselves)
```release-note
To create or update an RBAC RoleBinding or ClusterRoleBinding object, a user must:
1. Be authorized to make the create or update API request
2. Be allowed to bind the referenced role, either by already having all of the permissions contained in the referenced role, or by having the "bind" permission on the referenced role.
```
Automatic merge from submit-queue (batch tested with PRs 38212, 38792, 39641, 36390, 39005)
Updating federated service controller to support cascading deletion
Ref https://github.com/kubernetes/kubernetes/issues/33612
Service controller is special than other federation controllers because it does not use federatedinformer and updater to sync services (it was written before we had those frameworks).
Updating service controller code to instantiate these frameworks and then use deletion helper to perform cascading deletion.
Note that, I havent changed the queuing logic in this PR so we still dont use federated informer to manage the queue. Will do that in the next PR.
cc @kubernetes/sig-federation-misc @mwielgus @quinton-hoole
```release-note
federation: Adding support for DeleteOptions.OrphanDependents for federated services. Setting it to false while deleting a federated service also deletes the corresponding services from all registered clusters.
```
Automatic merge from submit-queue (batch tested with PRs 39695, 37054, 39627, 39546, 39615)
Add configs that run more advanced density and load tests
Wojtek is on vacation this week - @timothysc can you please take a look? It's rather terrible, but I don't have a better idea on how to make parametric tests.
cc @wojtek-t
Automatic merge from submit-queue (batch tested with PRs 39695, 37054, 39627, 39546, 39615)
Use Dynamic Config in e2e_node inode eviction test
Alternative solution to #39249. Similar to solution proposed by @vishh in #36828.
@Random-Liu @mtaufen
Automatic merge from submit-queue (batch tested with PRs 39648, 38167, 39591, 39415, 39612)
Add verbs to thirdparty resources in discovery
The namespace controller ignores thirdparty resources right now because verbs are not set. This PR sets a static list of verbs.
Moreover, integration tests are added for the discovery info of thirdparty resources.
/cc @zhouhaibing089
Automatic merge from submit-queue
Deleting federation-util-14.go that is not being used anywhere
We have the same code in federation-util.go
cc @mwielgus @madhusudancs
Automatic merge from submit-queue (batch tested with PRs 39628, 39551, 38746, 38352, 39607)
Increasing times on reconciling volumes fixing impact to AWS.
#**What this PR does / why we need it**:
We are currently blocked by API timeouts with PV volumes. See https://github.com/kubernetes/kubernetes/issues/39526. This is a workaround, not a fix.
**Special notes for your reviewer**:
A second PR will be dropped with CLI cobra options in it, but we are starting with increasing the reconciliation periods. I am dropping this without major testing and will test on our AWS account. Will be marked WIP until I run smoke tests.
**Release note**:
```release-note
Provide kubernetes-controller-manager flags to control volume attach/detach reconciler sync. The duration of the syncs can be controlled, and the syncs can be shut off as well.
```
Added [Volume] tag per issue #35542; added [Flaky] to GCE tests until confirmed fixed. Added [Serial] to NFS to address possible cross test contamination.
Automatic merge from submit-queue (batch tested with PRs 39394, 38270, 39473, 39516, 36243)
Modified run-gcloud-compute-with-retries and used it wherever possible in kubemark
This PR fixes#39335
Simple changes fixing flaky issues within kubemark.
@kubernetes/sig-scalability-misc @wojtek-t @gmarek
Automatic merge from submit-queue (batch tested with PRs 39394, 38270, 39473, 39516, 36243)
Fix wrong skipf parameter
**How to reproduce**
When run e2e test, it reports `%!!(MISSING)d(MISSING)`:
```
STEP: Checking for multi-zone cluster. Zone count = 1
Dec 6 14:16:43.272: INFO: Zone count is %!!(MISSING)d(MISSING), only run for multi-zone clusters, skipping test
[AfterEach] [k8s.io] Multi-AZ Clusters
```
We need to pass a string parameter to `SkipUnlessAtLeast`
The comment says that the sleep is there because of 10 minute PV controller
sync. The controller sync is now 15 seconds and it should be quick enough
to hide this in subsequent WaitForPersistentVolumeDeleted(.. , 20*time.Minute)
Automatic merge from submit-queue
Updated kubemark with RBAC for controllers, proxy and kubelet
Fixes issue #39244
@kubernetes/sig-scalability-misc @wojtek-t @gmarek
Automatic merge from submit-queue (batch tested with PRs 39493, 39496)
Use privileged containers for host path e2e tests
Test containers need to run as spc_t in order to interact with the host
filesystem under /tmp, as the tests for HostPath are doing. Docker will
transition the container into this domain when running the container as
privileged.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Currently, this test fails with AVC denials like:
```
time->Thu Jan 5 10:17:51 2017
type=SYSCALL msg=audit(1483629471.846:6623): arch=c000003e syscall=257 success=no exit=-13 a0=ffffffffffffff9c a1=c820010120 a2=80241 a3=1a4 items=0 ppid=4112 pid=4130 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="mt" exe="/mt" subj=system_u:system_r:svirt_lxc_net_t:s0:c123,c328 key=(null)
type=AVC msg=audit(1483629471.846:6623): avc: denied { write } for pid=4130 comm="mt" name="sub-path" dev="xvda2" ino=118491348 scontext=system_u:system_r:svirt_lxc_net_t:s0:c123,c328 tcontext=system_u:object_r:container_runtime_tmp_t:s0 tclass=dir
```
```release-note
NONE
```
/cc @ncdc @pmorie
Test containers need to run as spc_t in order to interact with the host
filesystem under /tmp, as the tests for HostPath are doing. Docker will
transition the container into this domain when running the container as
privileged.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
Automatic merge from submit-queue (batch tested with PRs 36229, 39450)
Bump etcd to 3.0.14 and switch to v3 API in etcd.
Ref #20504
**Release note**:
```release-note
Switch default etcd version to 3.0.14.
Switch default storage backend flag in apiserver to `etcd3` mode.
```
Automatic merge from submit-queue (batch tested with PRs 39408, 38981)
Remove RBAC UserAll
* Removes special handling of User * subjects in rolebinding matching evaluation
* Converts v1alpha1 rolebindings to `User *` subjects to `Group system:authenticated` subjects for backwards compatibility
```release-note
RBAC's special handling of the User subject named "*" in RoleBinding and ClusterRoleBinding objects is being deprecated and will be removed in v1beta1. Existing v1alpha1 role bindings to User "*" will be converted to the group "system:authenticated". To match unauthenticated requests, RBAC role bindings must explicitly bind to the group "system:unauthenticated".
```
Automatic merge from submit-queue
Ensure invalid token returns 401 error, not 403
fixes#39267
If a user attempts to use a bearer token, and the token is rejected, the authenticator should return an error. This distinguishes requests that did not provide a bearer token (and are unauthenticated without error) from ones that attempted to, and failed.
Automatic merge from submit-queue
Remove jobs that do not exist from active list of CronJob
**What this PR does / why we need it**: This PR modifies the controller for CronJob to remove from the active job list any job that does not exist anymore, to avoid staying blocked in active state forever. See #37957.
**Which issue this PR fixes**: fixes#37957
**Special notes for your reviewer**:
**Release note**:
```
```
Automatic merge from submit-queue (batch tested with PRs 38433, 36245)
Allow pods to define multiple environment variables from a whole ConfigMap
Allow environment variables to be populated from ConfigMaps
- ConfigMaps represent an entire set of EnvVars
- EnvVars can override ConfigMaps
fixes#26299
Automatic merge from submit-queue (batch tested with PRs 39280, 37350, 39389, 39390, 39313)
Moves e2e service util functions into service_util.go and cleans up
Basically moves codes into a central place for service util functions.
Some other codes are touched mostly only due to this migration. Also put a bunch of network reachability utils functions into network_utils.go. They seem somehow redundant, may consider combine they later.
@bowei @freehan
Automatic merge from submit-queue (batch tested with PRs 39001, 39104, 35978, 39361, 39273)
refactored admission to avoid internal client references
Refactored admission to avoid internal client references. This required switching to plugin initializers for them. And that required some rewiring of the plugin initializers.
Technically I can decouple from the other two commits, but I'm optimistic that those will go through easy. This is slightly move invasive, but I'd like to shoot for pre-christmas to avoid new admission plugins coming through and breaking bits.
@sttts @derekwaynecarr
Automatic merge from submit-queue (batch tested with PRs 39092, 39126, 37380, 37093, 39237)
Endpoints with TolerateUnready annotation, should list Pods in state terminating
**What this PR does / why we need it**:
We are using preStop lifecycle hooks to gracefully remove a node from a cluster. This hook is potentially long running and after the preStop hook is fired, the DNS resolution of the soon to be stopped Pod is failing, which causes a failure there.
**Special notes for your reviewer**:
Would be great to backport that to 1.4, 1.3
**Release note**:
```release-note
Endpoints, that tolerate unready Pods, are now listing Pods in state Terminating as well
```
@bprashanth
Automatic merge from submit-queue (batch tested with PRs 39150, 38615)
Add work queues to PV controller
PV controller should not use Controller.Requeue, as as it is not available in
shared informers. We need to implement our own work queues instead, where we
can enqueue volumes/claims as we want.
Automatic merge from submit-queue
Add Persistent Volume E2E in the context of a disrupted kubelet
This PR adds a test suite for persistent volumes affected by a disrupted kubelet. Two cases are presented:
1. A volume mounted via PVC remains accessible after a kubelet restart.
2. When a pod is deleted while the kubelet is down, the mounted volume is unmounted successfully.
PV controller should not use Controller.Requeue, as as it is not available in
shared informers. We need to implement our own work queues instead where we
can enqueue volumes/claims as we want.
Automatic merge from submit-queue
Remove all MAINTAINER statements in the codebase as they are deprecated
**What this PR does / why we need it**:
ref: https://github.com/docker/docker/pull/25466
**Release note**:
```release-note
Remove all MAINTAINER statements in Dockerfiles in the codebase as they are deprecated by docker
```
@ixdy @thockin (who else should be notified?)
Automatic merge from submit-queue
Remove system:anonymous check from kubectl test
This verbiage doesn't appear when the cluster is `AlwaysAllow` (and just makes the check more brittle).
Follow-on to #39263, this is the last (consistent) failure on [kops-aws](https://k8s-testgrid.appspot.com/google-aws#kops-aws&sort-by-failures=)
Automatic merge from submit-queue
Avoid unnecessary memory allocations
Low-hanging fruits in saving memory allocations. During our 5000-node kubemark runs I've see this:
ControllerManager:
- 40.17% k8s.io/kubernetes/pkg/util/system.IsMasterNode
- 19.04% k8s.io/kubernetes/pkg/controller.(*PodControllerRefManager).Classify
Scheduler:
- 42.74% k8s.io/kubernetes/plugin/pkg/scheduler/algrorithm/predicates.(*MaxPDVolumeCountChecker).filterVolumes
This PR is eliminating all of those.
Clayton Coleman