Automatic merge from submit-queue
Kubelet: Refactor container related functions in DockerInterface
For #23563.
Based on #23506, will rebase after #23506 is merged.
The last 4 commits of this PR are new.
This PR refactors all container lifecycle related functions in DockerInterface, including:
* ListContainers
* InspectContainer
* CreateContainer
* StartContainer
* StopContainer
* RemoveContainer
@kubernetes/sig-node
Automatic merge from submit-queue
Add watch.Until, a conditional watch mechanism
A more powerful tool than wait.Poll, allows a watch interface to drive conditionals to react to changes on a resource or resources. Provide a set of standard conditions that are in common use in the code, and updates e2e to use a few of these.
Extracted from #23567
Automatic merge from submit-queue
the component status health check should check whether the scheme of backend storage url is https or not
fix https://github.com/kubernetes/kubernetes/issues/23897, when querying the component status of etcd (backend storage), the scheme of url is not checked and use `http` always, this commit aims to fix this.
Automatic merge from submit-queue
Flexvolume: Add support for multiple secrets
This PR adds support to pass multiple secrets for flexvolume plugins.
To allow multiple secrets, secrets are now passed as:
"kubernetes.io/secret/id-rsa":"value-2\r\n\r\n","kubernetes.io/secret/id-rsa.pub":"value-1\r\n"
Automatic merge from submit-queue
Fix expired event logic to use 404 instead of 500
It seems this logic was never updated once apiserver started returning 404s for expired (missing) events.
This change corrects it to use a 404 so events will get resent correctly if they were expired in etcd.
Fixes#23637.
Automatic merge from submit-queue
Make kubectl edit not convert GV on edits
Previously, kubectl edit was using a decoder to load in edits that
converted to the internal version. It would then re-encode this
decoded value to produce a patch. However, if you were editing
in the object in a GroupVersion that was not the internal version,
this would cause the kubectl edit command to attempt to produce
a patch which changed the GroupVersion, which would fail.
Now, we use a plain deserializer instead, so no conversion or
defaulting occurs when loading in the edited file.
Ref #23378
Automatic merge from submit-queue
phase 2 of cassandra example overhaul
Here's the next iteration in overhauling this example, towards https://github.com/kubernetes/kubernetes/issues/20961. This removes the pod adoption part, but doesn't (yet) otherwise change any of the resources used.
It also includes some README cleanup, and removes some explicit specification of labels in the rc yaml.
This PR doesn't yet add any commentary on how we're using the seed provider (re: https://github.com/kubernetes/kubernetes/issues/20961#issuecomment-190405959 etc.). Maybe we should add that.
Also: LMK if this PR should include any changes to the links out to the docs.
cc @bgrant0607 @johndmulhausen
Automatic merge from submit-queue
rkt: Fix hostnetwork.
Mount hosts' /etc/hosts, /etc/resolv.conf, set host's hostname
when running the pod in the host's network.
Fix#24235
cc @kubernetes/sig-node
Automatic merge from submit-queue
rkt: Use rkt pod's uuid as the systemd service file's name.
Previously, the service file's name is 'k8s_${POD_UID}.service',
which means we need to `systemctl daemon-reload` if the we replace
the content of the service file (e.g. pod is restarted).
However this makes the journal in the previous pod get disconnected.
This PR solves the issue by using the unique rkt uuid as the service
file's name. After the change, the service file's name will be:
'k8s_${rkt_uuid}.service'.
Fix#23691
Also add helpers for collecting the events that happen during a watch
and a helper that makes it easy to start a watch from any object with
ObjectMeta.
Reduces the surface area of the API server slightly and allows
downstream components to have deleteable metrics. After this change
genericapiserver will *not* have metrics unless the caller defines it
(allows different apiserver implementations to make that choice on their
own).
Automatic merge from submit-queue
rkt: Update the directory path for saving auth config.
Since #23308 is merged, now we have more stable way to determine where to store the auth configs.
cc @yujuhong @sjpotter
Automatic merge from submit-queue
Allow lazy binding in credential providers; don't use it in AWS yet
This is step one for cross-region ECR support and has no visible effects yet.
I'm not crazy about the name LazyProvide. Perhaps the interface method could
remain like that and the package method of the same name could become
LateBind(). I still don't understand why the credential provider has a
DockerConfigEntry that has the same fields but is distinct from
docker.AuthConfiguration. I had to write a converter now that we do that in
more than one place.
In step two, I'll add another intermediate, lazy provider for each AWS region,
whose empty LazyAuthConfiguration will have a refresh time of months or years.
Behind the scenes, it'll use an actual ecrProvider with the usual ~12 hour
credentials, that will get created (and later refreshed) only when kubelet is
attempting to pull an image. If we simply turned ecrProvider directly into a
lazy provider, we would bypass all the caching and get new credentials for
each image pulled.
Automatic merge from submit-queue
Kubelet: Better-defined Container Waiting state
For issue #20478 and #21125.
This PR corrected logic and add unit test for `ShouldContainerBeRestarted()`, cleaned up `Waiting` state related code and added unit test for `generateAPIPodStatus()`.
Fixes#20478Fixes#17971
@yujuhong
Automatic merge from submit-queue
Do not throw creation errors for containers that fail immediately after being started
Fixes (hopefully) #23607
cc @dchen1107
Mount hosts' /etc/hosts, /etc/resolv.conf, set host's hostname
when running the pod in the host's network.
Besides, do not set the DNS flags when running in host's network.
Previously, the service file's name is 'k8s_${POD_UID}.service',
which means we need to `systemctl daemon-reload` if the we replace
the content of the service file (e.g. pod is restarted).
However this makes the journal in the previous pod get disconnected.
This PR solves the issue by using the unique rkt uuid as the service
file's name. After the change, the service file's name will be:
'k8s_${rkt_uuid}.service'.
Automatic merge from submit-queue
rkt: Add pre-stop lifecycle hooks for rkt.
When a pod is being terminated, the pre-stop hooks of all the containers
will be run before the containers are stopped.
cc @yujuhong @Random-Liu @sjpotter
Automatic merge from submit-queue
add a delayed queueing option to the workqueue
Adds delayed requeuing to the workqueue so that I requeue an item to be retried at some later time in my controller with a series of backoff rules. It lets me have the best of the retryManager and the work queue de-duping. Tracking failures and backoffs is on the caller
@smarterclayton @pweil- this would help us move to using the informer everywhere and de-duping at that level.
Automatic merge from submit-queue
Make kubectl bash-completion namespace and resource alias aware
- filter resource listing by `--namespace` flag given before in the command line
```bash
$ kubectl get pod --namespace=kube-system <tab><tab>
kube-dns-v9-2wuzj kube-dns-v9-llqxa
```
- add completion of `--namespace`
```bash
$ kubectl get pod --namespace=<tab><tab>
[*] default ingress kube-system
```
- add support for plural nouns and aliases like `rc`
Automatic merge from submit-queue
Use OomScoreAdj in kubelet for newer docker api
fixes: #20121
Related: client side PR [pull 454](https://github.com/fsouza/go-dockerclient/pull/454)
Godeps has already been updated to `0099401a7342ad77e71ca9f9a57c5e72fb80f6b2`, which included client side's modification. But it seems too aggressive to upgrade the docker api version of kubelet.
Automatic merge from submit-queue
Clientset release 1.3
This PR creates the release 1.3 client set. We'll keep updating this client set until we cut release 1.3. In the meantime, the release 1.2 client set will be locked.
@lavalamp
Automatic merge from submit-queue
Fixed mounting with containerized kubelet
`NsenterMounter.IsLikelyNotMountPoint()` should return `ErrNotExist` when the
checked directory does not exists - the regular mounted does this and
some volume plugins depend on this behavior.
See for example: https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/aws_ebs/aws_util.go#L72
Automatic merge from submit-queue
Implement a streaming serializer for watch
Changeover watch to use streaming serialization. Properly version the
watch objects. Implement simple framing for JSON and Protobuf (but not
YAML).
@wojtek-t @lavalamp
Automatic merge from submit-queue
Add godoc to kubelet/volumes.go
Noticed that `mountExternalVolumes`, of all things, was missing Godoc while working w/ @screeley44. Decided to add some tonight since I have been making noise about grokkability of the kubelet lately.
@kubernetes/sig-storage
Automatic merge from submit-queue
Make watch cache treat resourceVersion consistent with uncached watch
Fixes#24004
This makes the watch cache handle resourceVersion consistent with an uncached watch API call, and the documented behavior. Watching from resourceVersion=X delivers watch events *after* version X (X is not included):
> // When specified with a watch call, shows changes that occur after that particular version of a resource.
> // Defaults to changes from the beginning of history.
> ResourceVersion string
Automatic merge from submit-queue
Move predicates into library
This PR tries to implement #12744
Any suggestions/ideas are welcome. @davidopp
current state: integration test fails if including podCount check in Kubelet.
DONE:
1. refactor all predicates: predicates return fitOrNot(bool) and error(Error) in which the latter is of type PredicateFailureError or InsufficientResourceError
2. GeneralPredicates() is a predicate function, which includes serveral other predicate functions (PodFitsResource, PodFitsHost, PodFitsHostPort). It is registered as one of the predicates in DefaultAlgorithmProvider, and is also called in canAdmitPod() in Kubelet and should be called by other components (like rescheduler, etc if necessary. See discussion in issue #12744
TODO:
1. determine which predicates should be included in GeneralPredicates()
2. separate GeneralPredicates() into: a.) GeneralPredicatesEvictPod() and b.) GeneralPredicatesNotEvictPod()
3. DaemonSet should use GeneralPredicates()
Automatic merge from submit-queue
Preserve int data when unmarshaling
There are several places we use `json.Unmarshal` into an unstructured map (StrategicMergePatch, UnstructuredJSONScheme, many others).
In this scenario, the json package converts all numbers to float64. This exposes many of the int64 fields in our API types to corruption when the unstructured map is marshalled back to json.
A simple example is a pod with an `"activeDeadlineSeconds": 1000000`. Trying to use `kubectl label`, `annotate`, `patch`, etc results in that int64 being converted to a float64, submitted to the server, and the server rejecting it with an error about "cannot unmarshal number 1e+6 into Go value of type int64"
The json package provides a way to defer conversion of numbers (`json.Decoder#UseNumber`), but does not actually do conversions to int or float. This PR makes use of that feature, and post-processes the unmarshalled map to convert json.Number objects to either int64 or float64 values
Automatic merge from submit-queue
Additional go vet fixes
Mostly:
- pass lock by value
- bad syntax for struct tag value
- example functions not formatted properly
DONE:
1. refactor all predicates: predicates return fitOrNot(bool) and error(Error) in which the latter is of type
PredicateFailureError or InsufficientResourceError. (For violation of either MaxEBSVolumeCount or
MaxGCEPDVolumeCount, returns one same error type as ErrMaxVolumeCountExceeded)
2. GeneralPredicates() is a predicate function, which includes serveral other predicate functions (PodFitsResource,
PodFitsHost, PodFitsHostPort). It is registered as one of the predicates in DefaultAlgorithmProvider, and
is also called in canAdmitPod() in Kubelet and should be called by other components (like rescheduler, etc)
if necessary. See discussion in issue #12744
3. remove podNumber check from GeneralPredicates
4. HostName is now verified in Kubelet's canAdminPod(). add TestHostNameConflicts in kubelet_test.go
5. add getNodeAnyWay() method in Kubelet to get node information in standaloneMode
TODO:
1. determine which predicates should be included in GeneralPredicates()
2. separate GeneralPredicates() into:
a. GeneralPredicatesEvictPod() and
b. GeneralPredicatesNotEvictPod()
3. DaemonSet should use GeneralPredicates()
Automatic merge from submit-queue
Allowing type object in kubectl swagger validation
Allowing `"type": "object"` in kubectl swagger validation.
We will cherrypick this in 1.2
This allows 1.2 kubectl to work with 1.3 api server which will use `"type":"object"` instead of `"type":"any"`.
Ref https://github.com/kubernetes/kubernetes/issues/4700#issuecomment-194719759
cc @bgrant0607
Automatic merge from submit-queue
Client-gen: show the command used to generate the package in doc.go
#22928 adds a comment in every generated file to show the arguments supplied to client-gen. I received comment (https://github.com/kubernetes/kubernetes/pull/22928#issuecomment-201078363) that it generates too many one-line changes every time the command line argument is changed. To address this problem, this PR only generates that line in doc.go.
@jianhuiz @krousey
Automatic merge from submit-queue
Add a timeout to the sshDialer to prevent indefinite hangs.
Prevents the SSH Dialer from hanging forever. Fixes a problem where SSH Tunnels get stuck trying to open.
Addresses #23835.
Automatic merge from submit-queue
Ensure object returned by volume getCloudProvider incorporates cloud config
This PR addresses https://github.com/kubernetes/kubernetes/issues/23517.
**Problem**
The existing GCE PD and AWS EBS volume plugin code were fetching cloud provider without specifying a cloud config: `cloudprovider.GetCloudProvider("gce", nil)`
This caused the cloud provider to use default auth mechanism, which is not acceptable for the provisioning controller running on GKE master.
**Fix**
This PR does the following:
* Modifies the GCE PD and AWS EBS volume plugin code to use the cloud provider object pre-constructed by the binary with a cloud config.
* Enable provisioning E2E test for GKE (to catch future issues).
Thanks to @cjcullen for debugging and finding the root cause! 👍
This should be cherry-picked into the v1.2 branch for the next release.
Automatic merge from submit-queue
Kubelet: Remove nsinit related code and bump up minimum docker apiversion
Docker has native exec support after 1.3.x. We never need this code now.
As for the apiversion, because Kubernetes supports 1.8.x - 1.10.x now, we should bump up the minimum docker apiversion.
@yujuhong I checked the [changes](https://github.com/docker/engine-api/blob/master/types/versions/v1p20/types.go), we are not relying on any of those changes. So #23506 should work with docker 1.8.x+
Automatic merge from submit-queue
Move typed clients into clientset folder
Move typed clients from `pkg/client/typed/` to `pkg/client/clientset_generated/${clientset_name}/typed`.
The first commit changes the client-gen, the last commit updates the doc, other commits are just moving things around.
@lavalamp @krousey
Automatic merge from submit-queue
relax restmapper resource matching
We were matching case insensitive on Kinds, not Resources, thus driving me insane.
@liggitt @caesarxuchao
Automatic merge from submit-queue
genericapiserver: Moving InstallSwaggerAPI to Run
Ref https://github.com/kubernetes/kubernetes/pull/21190#discussion_r57494673
Moving InstallSwaggerAPI() from InstallAPIGroups() to Run(). This allows the use of InstallAPIGroups() multiple times or using InstallAPIGroup() directly.
cc @jianhuiz @kubernetes/sig-api-machinery
Automatic merge from submit-queue
add missing attributes to authorization interface
Plumb through missing attributes for authorization decisions. The request parser picked them up a while back, but they were never wired up in kube.
This simply allows authorizers to use them, it doesn't require them to.
@kubernetes/kube-iam
Automatic merge from submit-queue
Check claimRef UID when processing a recycled PV, take 2
Reorder code a bit so it doesn't allow a case where you get some error other than "not found"
combined with a non-nil Claim.
Add test case.
cc @kubernetes/rh-cluster-infra @kubernetes/rh-storage @liggitt
Automatic merge from submit-queue
Kubelet: Start using the official docker engine-api
For #23563.
This is the **first step** in the roadmap of switching to docker [engine-api](https://github.com/docker/engine-api).
In this PR, I keep the old `DockerInterface` and implement it with the new engine-api.
With this approach, we could switch to engine-api with minimum change, so that we could:
* Test the engine-api without huge refactoring.
* Send following PRs to refactor functions in `DockerInterface` separately so as to avoid a huge change in one PR.
I've tested this PR locally, it passed all the node conformance test:
```
make test_e2e_node
Ran 19 of 19 Specs in 823.395 seconds
SUCCESS! -- 19 Passed | 0 Failed | 0 Pending | 0 Skipped PASS
Ginkgo ran 1 suite in 13m49.429979585s
Test Suite Passed
```
And it also passed the jenkins gce e2e test:
```
go run hack/e2e.go -test -v --test_args="--ginkgo.skip=\[Slow\]|\[Serial\]|\[Disruptive\]|\[Flaky\]|\[Feature:.+\]"
Ran 161 of 268 Specs in 4570.214 seconds
SUCCESS! -- 161 Passed | 0 Failed | 0 Pending | 107 Skipped PASS
Ginkgo ran 1 suite in 1h16m16.325934558s
Test Suite Passed
2016/03/25 15:12:42 e2e.go:196: Step 'Ginkgo tests' finished in 1h16m18.918754301s
```
I'm writing the design document, and will post the switching roadmap in an umbrella issue soon.
@kubernetes/sig-node
Automatic merge from submit-queue
Use correct home directory on Windows
As of now the `kubectl` config mechanism that saves/loads `~/.kube/config` is broken on Windows, saving the config file in the *current* directory instead of the user's *home* dir. This happens because most Windows don't respect the `HOME` environment variable.
This PR changes the config file loading mechanism to use the recommended way to detect the user home on Windows (`HOMEDRIVE`+`HOMEPATH` or `USERPROFILE`), and adds a migration for users that might be currently relying on existing config files.
Automatic merge from submit-queue
Refactor streaming code to support interop testing
Refactor exec/attach/port forward client and server code to better
support interop testing of different client and server subprotocol
versions.
Fixes#16119
Automatic merge from submit-queue
A pod never terminated if a container image registry was unavailable
Fixes https://github.com/kubernetes/kubernetes/issues/23742
Fixes https://github.com/kubernetes/kubernetes/issues/22045
Pods will now show proper status when this happens as well:
```
$ cluster/kubectl.sh get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
test foo-4072956304-1g8qs 0/1 RegistryUnavailable 0 7s
test foo-4072956304-i045g 0/1 RegistryUnavailable 0 7s
test foo-4072956304-qem2n 0/1 RegistryUnavailable 0 7s
```
Where as previously they would never report a reason.
I also removed the "temporary" part of the message because we have no idea if its temporary or permanent.
/cc @kubernetes/sig-node @kubernetes/rh-cluster-infra
It seems this logic was never updated once apiserver started returning
404s for expired (missing) events.
This change corrects it to use a 404 so events will get resent correctly
if they were expired in etcd.
Automatic merge from submit-queue
Add support for 3rd party objects to kubectl
@deads2k @jlowdermilk
Instructions for playing around with this:
Run an apiserver with third party resources turned on (`--runtime-config=extensions/v1beta1=true,extensions/v1beta1/thirdpartyresources=true`)
Then you should be able to:
```
kubectl create -f rsrc.json
```
```json
{
"metadata": {
"name": "foo.company.com"
},
"apiVersion": "extensions/v1beta1",
"kind": "ThirdPartyResource",
"versions": [
{
"apiGroup": "group",
"name": "v1"
},
{
"apiGroup": "group",
"name": "v2"
}
]
}
```
Once that is done, you should be able to:
```
curl http://<server>/apis/company.com/v1/foos
```
```
curl -X POST -d @${HOME}/foo.json http://localhost:8080/apis/company.com/v1/namespaces/default/foos
```
```json
{
"kind": "Foo",
"apiVersion": "company.com/v1",
"metadata": {
"name": "baz"
},
"someField": "hello world",
"otherField": 1
}
```
After this PR, you can do:
```
kubectl create -f foo.json
```
```
kubectl get foos
```
etc.
Automatic merge from submit-queue
Migrate to the new conversion generator - part1
This PR contains two commits:
- few more fixes to the generator
- migration of the pkg/api/v1 to use the new generator
The second commit is big, but I reviewed the changes and they contain:
- conversions between types that we didn't even generating conversion between
- changes in how we handle maps/pointers/slices - previously we were explicitly referencing fields, now we are using "shadowing in, out" to make the code more generic
- lack of auto-generated method for ReplicationControllerSpec (because these types are different (*int vs int for Replicas) and a preexisting conversion already exists
Most of issues in the first commit (e.g. adding references to "in" and "out" for slices/maps/points) were discovered by our tests. So I'm pretty confident that this change is correct now.
Automatic merge from submit-queue
storage.Interface KV impl. of etcd v3
This is the initial implementation of #22448.
The PR consists of two parts:
- add godep of "clientv3" and "integration" (for testing)
- create new package "etcd3" under "pkg/storage/"
- implement KV methods of storage.Interface using etcd v3 APIs
- Create, Set, Get, Delete, GetToList, List, GuaranteedUpdate
os/signal.Stop() does not actually close the channel, which means
the channel is blocked. Not serious because we don't use interrupt
protection on the server, just the CLI (which can afford 1 leaked
goroutine).
Automatic merge from submit-queue
Implement network plugin capabilities hook and shaping capability
Allow network plugins to declare that they handle shaping and that
Kuberenetes should not. I've got an OpenShift PR that handles shaping in OVS but the kubelet code sends nasty pod events because it doesn't think shaping is used since --reconcile-cbr0 is not used with most network plugins.
See https://github.com/openshift/openshift-sdn/pull/266 for the OpenShift implementation.
k8s-merge-robot
