Commit Graph

6820 Commits (f0b207df2dd297b0b54920c46978083dd8a081e9)

Author SHA1 Message Date
yankaiz b32cc1be59 e2e test forwarding externalname dns lookup to upstream nameservers. 2018-04-19 16:53:41 -07:00
Kubernetes Submit Queue 2142bff940
Merge pull request #62606 from smarterclayton/defer_delete
Automatic merge from submit-queue (batch tested with PRs 61962, 58972, 62509, 62606). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Allow a test suite reusing framework to register namespaces to delete

If the suite bypasses CreateNamespace (because it wants to create more
specialized namespaces) it has no way to register deletes.

@liggitt this was something that came up in an older rebase and was missed (for things creating projects)
2018-04-19 16:47:26 -07:00
Yecheng Fu 27e2e32b00 Use shorter timeout if possible.
- Add PodStartShortTimeout and ClaimProvisionShortTimeout constants.
- Change framework.PodStartTimeout to framework.PodStartShortTimeout in
  persistent_volumes-local.go. Busybox image is very small, no need to
  wait for a long time.
2018-04-19 10:07:59 +08:00
Kubernetes Submit Queue efadf7b9e7
Merge pull request #61877 from mikedanese/depeid
Automatic merge from submit-queue (batch tested with PRs 62481, 62643, 61877, 62515). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

remove deprecated ExternalID

This field has been deprecated since 1.1. After we remove it we can remove "self delete" from the node's permission set.

@kubernetes/api-reviewers 
@kubernetes/sig-auth-pr-reviews 

fixes https://github.com/kubernetes/kubernetes/issues/61966
part of https://github.com/kubernetes/community/pull/911

```release-note
Kubelets will no longer set `externalID` in their node spec.
```
2018-04-18 17:53:16 -07:00
Kubernetes Submit Queue 17fec00b89
Merge pull request #62433 from davidz627/feature/csiGCETest
Automatic merge from submit-queue (batch tested with PRs 62694, 62569, 62646, 61633, 62433). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Add GCE-PD CSI Driver test to E2E test suite

Fixes: #60462

/sig storage
/kind technical-debt
/assign @saad-ali @msau42 

**What this PR does / why we need it**:
This PR adds an E2E test for the GCE-PD CSI driver that deploys the driver in a production-like setting and tests whether dynamic provisioning with the driver is possible.

```release-note
NONE
```
2018-04-18 14:44:24 -07:00
Kubernetes Submit Queue b758341a04
Merge pull request #62646 from cofyc/reporteventtoapiserver
Automatic merge from submit-queue (batch tested with PRs 62694, 62569, 62646, 61633, 62433). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Report events to apiserver in local volume plugin.

**What this PR does / why we need it**:

See https://github.com/kubernetes/kubernetes/pull/62102#discussion_r179238429.

Report events to apiserver in local volume plugin.

- Add VolumeHost.GetEventRecorder() method
- Add related e2e tests

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #62248

**Special notes for your reviewer**:

Example output of `kubectl describe pods`:

```
Events:
  Type     Reason                 Age   From                Message
  ----     ------                 ----  ----                -------
  Normal   Scheduled              7s    default-scheduler   Successfully assigned e2e-tests-persistent-local-volumes-test-x4h5x/security-context-670da435-4174-11e8-9098-000c29bb0377 to 127.0.0.1
  Warning  AlreadyMountedVolume   7s    kubelet, 127.0.0.1  The requested fsGroup is 4321, but the volume local-pvfbb76 has GID 1234. The volume may not be shareable.
  Normal   SuccessfulMountVolume  7s    kubelet, 127.0.0.1  MountVolume.SetUp succeeded for volume "default-token-996xr"
  Normal   SuccessfulMountVolume  7s    kubelet, 127.0.0.1  MountVolume.SetUp succeeded for volume "local-pvfbb76"
  Normal   Pulled                 6s    kubelet, 127.0.0.1  Container image "k8s.gcr.io/busybox:1.24" already present on machine
  Normal   Created                6s    kubelet, 127.0.0.1  Created container
  Normal   Started                6s    kubelet, 127.0.0.1  Started container
```

**Release note**:

```release-note
NONE
```
2018-04-18 14:44:17 -07:00
David Zhu 4d11dab272 CSI test refactor to be more easily extensible for more plugins when there are more tests 2018-04-18 11:24:36 -07:00
Mike Danese f427531179 boring 2018-04-18 09:55:57 -07:00
Kubernetes Submit Queue bb8f58b6e6
Merge pull request #62195 from serathius/prometheus
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Add prometheus cluster monitoring addon.

This PR adds new cluster monitoring addon based on prometheus.
It adds prometheus deployment with e2e tests.
Additional components will be added iterativly in future.
Manifests based on current Helm chart.
At current state it's not intended for production use.

cc @piosz @kawych @miekg
```release-note
Add prometheus cluster monitoring addon to kube-up
```
/sig instrumentation
/kind feature
/priority important-soon
2018-04-18 02:17:48 -07:00
Yecheng Fu 55ef18ad42 Report events to apiserver in local volume plugin.
- Add VolumeHost.GetEventRecorder() method
- Add related e2e tests
2018-04-18 11:19:36 +08:00
Kubernetes Submit Queue 26135c5f4b
Merge pull request #62303 from msau42/fix-subpath-reconstruct-test
Automatic merge from submit-queue (batch tested with PRs 62650, 62303, 62545, 62375). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Refactor subpath reconstruction tests to use util test

**What this PR does / why we need it**:
Use common util method for reconstruction tests so that test updates will be reflected in all e2es.  Also increase termination grace period to allow more time for volume manager to cleanup before the tests check for leftover mounts

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #62293

**Special notes for your reviewer**:

**Release note**:

```release-note
NONE
```
2018-04-16 13:56:07 -07:00
Clayton Coleman 565f97bf68
Allow a test suite reusing framework to register namespaces to delete
If the suite bypasses CreateNamespace (because it wants to create more
specialized namespaces) it has no way to register deletes.
2018-04-15 20:15:38 -04:00
David Zhu 01d916167b Add GCE-PD CSI Driver test to E2E test suite 2018-04-13 15:31:33 -07:00
Kubernetes Submit Queue 92065407b7
Merge pull request #62462 from jsafrane/private-mount-propagation
Automatic merge from submit-queue (batch tested with PRs 60476, 62462, 61391, 62535, 62394). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Private mount propagation

This PR changes the default mount propagation from "rslave" (newly added in 1.10) to "private" (default in 1.9 and before). "rslave" as default causes regressions, see below.

Value `"None"` has to be added to `MountPropagationMode` enum in API ("I don't want any propagation at all"), which translates to "private" on Linux. [We did not have use cases for it](https://github.com/kubernetes/community/pull/659#discussion_r131454319), but we have them now.

**Which issue(s) this PR fixes**
Fixes #62397, fixes #62396

**Special notes for your reviewer**:
CRI already has an option for private mount propagation in volumes, however it's called "PRIVATE", while Kubernetes API value is "None". I did not change PRIVATE to NONE to keep the interface stable. See `kubelet_pods.go`.

**Release note**:
```release-note
Default mount propagation has changed from "HostToContainer" ("rslave" in Linux terminology) to "None" ("private") to match the behavior in 1.9 and earlier releases. "HostToContainer" as a default caused regressions in some pods.
```


/sig storage
/sig node
2018-04-13 11:20:13 -07:00
Kubernetes Submit Queue ce1cab3bfa
Merge pull request #62350 from jennybuckley/crd-tests-flaky
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

 Fix flaky crd e2e tests

Fixes #62345

/sig api-machinery
/kind flake
/priority failing-test

From the issue:
> All the tests that rely on creating a CustomResourceDefinition are flaky. This is because the test "AdmissionWebhook Should be able to deny custom resource creation" creates a validating webhook, called deny-crd.k8s.io, which denies all crd creations. This is an issue for any other test which will create a crd that gets run in parallel with that webhook test, causing them to fail with the message failed to create CustomResourceDefinition: admission webhook "deny-crd.k8s.io" denied the request: this webhook denies all requests

This PR changes the test webhook image to support handling a new "/crd" path which would accept and deny crd creations based on their labels, only denying if if the object has the label ```"webhook-e2e-test":"webhook-disallow"```, and updates the "AdmissionWebhook Should be able to deny custom resource creation" test to use that label.

**Release note**:
```release-note
NONE
```
2018-04-13 09:28:44 -07:00
Marek Siarkowicz 9544222e91 Test e2e prometheus addon 2018-04-13 11:12:10 +02:00
Kubernetes Submit Queue 47a12e3ab0
Merge pull request #62502 from nicksardo/update-ingress-util
Automatic merge from submit-queue (batch tested with PRs 62503, 62502). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Fix ingress util for setting TLS spec

**What this PR does / why we need it**:
Fixes an issue with ingress TLS testing.  For most tests, we want to only provide one certificate and verify it's being served. In some tests (including a conformance test), a new certificate was added; but only the first certificate was asserted. 

This is because the multi-TLS test uses `WaitForIngressWithCert()` while the other tests use `WaitForGivenIngressWithTimeout`.

**Special notes for your reviewer**:
/assign rramkumar1
/cc rramkumar1

**Release note**:
```release-note
NONE
```
2018-04-12 19:48:10 -07:00
Nick Sardo 97798c4f3d Fix ingress util handling of TLS 2018-04-12 17:50:50 -07:00
Kubernetes Submit Queue daca510a48
Merge pull request #62494 from grayluck/fixe2e
Automatic merge from submit-queue (batch tested with PRs 62466, 62494). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Fix failed e2e tests for dns configmap

**What this PR does / why we need it**:
Fix DNS configMap e2e tests.

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #62490 

**Release note**:

```release-note
NONE
```
2018-04-12 16:47:20 -07:00
Kubernetes Submit Queue b96a9cc876
Merge pull request #62466 from aleksandra-malinowska/fix-parsing-timestamp
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Fix parsing timestamp in test

Go 1.9 changed the default string format, see https://github.com/golang/go/issues/20876 

As discussed there, default format is intended to be readable for humans and can change without warning, so we should probably set the format explicitly when writing status configmap.

```release-note
NONE
```
2018-04-12 15:58:14 -07:00
yankaiz 84a635e21a Fix failed e2e tests for dns configmap. 2018-04-12 15:00:14 -07:00
Kubernetes Submit Queue f3dbc1e3e6
Merge pull request #62476 from rramkumar1/patch-9
Automatic merge from submit-queue (batch tested with PRs 62324, 61459, 62475, 62476, 61914). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Remove unnecessary code in ingress upgrade logic

Now that 1.1 is released, fix the TODO.

/assign @nicksardo 

**Release note**:

```release-note
None
```
2018-04-12 13:18:14 -07:00
Kubernetes Submit Queue 2d10c212b3
Merge pull request #62475 from rramkumar1/patch-8
Automatic merge from submit-queue (batch tested with PRs 62324, 61459, 62475, 62476, 61914). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Bump image in ingress downgrade test

**Release note**:
```release-note
None
```

/assign @nicksardo
2018-04-12 13:18:11 -07:00
Kubernetes Submit Queue 9816b43188
Merge pull request #61198 from jpbetz/etcd-3.2-upgrade-reattempt
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Update default etcd server to 3.2 for kubernetes 1.11

Repply #59836 but with latest etcd 3.2 patch version (3.2.18 which includes mvcc fix and leader election timeout fix) and default `--snapshot-count` to 10k to resolve performance regression in previous etcd 3.2 server upgrade attempt (https://github.com/kubernetes/kubernetes/issues/60589#issuecomment-372458578).

See https://github.com/kubernetes/kubernetes/issues/60589#issuecomment-372458578 for details on the root cause of the performance regression and scalability test results of setting `--snapshot-count` to 10k.

```release-note
Upgrade the default etcd server version to 3.2.18
```
@gyuho @shyamjvs @jdumars @timothysc
2018-04-12 10:46:42 -07:00
Rohit Ramkumar 8331cd5bbf
Remove unnecessary code in ingress upgrade logic 2018-04-12 09:05:44 -07:00
Rohit Ramkumar aec7a794bd
Bump image in ingress downgrade test 2018-04-12 09:00:48 -07:00
Kubernetes Submit Queue dacb9ad692
Merge pull request #62411 from msau42/local-e2e-mountprop
Automatic merge from submit-queue (batch tested with PRs 62455, 62465, 62427, 62416, 62411). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Set slave mount propagation for local provisioner

**What this PR does / why we need it**:
Explicitly set slave mount propagation instead of assuming it's the default mode.

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #

**Special notes for your reviewer**:

**Release note**:

```release-note
NONE
```
2018-04-12 08:54:25 -07:00
Kubernetes Submit Queue 66b5978113
Merge pull request #62427 from nicksardo/bump-glbc
Automatic merge from submit-queue (batch tested with PRs 62455, 62465, 62427, 62416, 62411). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Bump GLBC version and remove Unreleased tag from tests

/assign rramkumar1
/cc mrhohn

**Release note**:
```release-note
GCE: Bump GLBC version to 1.1.0 - supporting multiple certificates and HTTP2
```
2018-04-12 08:54:17 -07:00
Jan Safranek a943fdda8a Update e2e test with private mount propagation 2018-04-12 15:47:30 +02:00
wojtekt 4cb99318d2 Fix PodStore to wait for being initialized 2018-04-12 15:38:11 +02:00
Aleksandra Malinowska 3eb420b773 Fix parsing timestamp in test 2018-04-12 14:22:28 +02:00
Joe Betz a795cb61bc Bump etcd default server version to 3.2.18 2018-04-11 21:12:44 -07:00
Kubernetes Submit Queue dcc6d78ac0
Merge pull request #62042 from grayluck/e2e-dnsfwd
Automatic merge from submit-queue (batch tested with PRs 60585, 62398, 62258, 62042). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Add e2e test for kubedns to forward PTR records to upstream server.

**What this PR does / why we need it**:
Add e2e test for kubedns to forward PTR records to upstream server.

**Release note**:

```release-note
NONE
```
2018-04-11 17:31:28 -07:00
Kubernetes Submit Queue 5427f4fa9d
Merge pull request #62384 from shyamjvs/disable-lb-tests-for-large-scale
Automatic merge from submit-queue (batch tested with PRs 62425, 62212, 60824, 62383, 62384). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Disable some newly added loadbalancer tests for large clusters

We're seeing quite some flakes due to LB tests on large-scale - https://k8s-testgrid.appspot.com/sig-scalability-gce#gce-scale-correctness
Earlier we disabled LB tests, but these ones were newly added few days ago.

/cc @wojtek-t 
fyi - @MrHohn 

```release-note
NONE
```
2018-04-11 16:36:25 -07:00
Nick Sardo fc60d36a8e Bump GLBC version and remove Unreleased tag from tests 2018-04-11 14:39:43 -07:00
Michelle Au 7aca917ea4 Refactor subpath reconstruction tests to use util test 2018-04-11 14:20:29 -07:00
jennybuckley 7fee22b7be Fix flaky crd e2e tests 2018-04-11 14:08:05 -07:00
yankaiz 6a072b9085 Add e2e test for forwarding PTR records to upstream nameserver.
createDNSServerWithPtrRecord in dns_common.go is added to generate dns
server pod with PTR record.
2018-04-11 11:24:31 -07:00
Michelle Au bf70c0af4e Set slave mount propagation for local provisioner 2018-04-11 10:28:06 -07:00
Shyam Jeedigunta adf03e2411 Disable some newly added loadbalancer tests for large clusters 2018-04-11 13:28:56 +02:00
Kubernetes Submit Queue a8899b314b
Merge pull request #61241 from jpbetz/go-migrate-if-needed
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Reimplement migrate-if-needed.sh in go

The `migrate-if-needed.sh` script was already partially implemented in go (see the attachlease and rollback sub-dirs), but was still unnecessarily difficult to understand and test.  This closely reimplements the original logic but with improved code structure, error handling and testing.

Where possible, go code that was previously executed as separate binaries is now statically linked into a single 'migrate' go cobra CLI app, which is then thinly wrapped by`migrate-if-needed.sh`.

There are numerous additional improvements that need to be made, but will be submitted in future PRs.  This PR is focused on achieving parity with the pre-existing functionality and introducing some much needed test coverage, in particular HA cluster upgrade test coverage.

It appears that the `attachlease` and `rollback` go binaries are no longer needed as standalones and so I have consolidated them into the new `migrate` go binary. Other than that, this change aims to be 100% backward compatible.

```release-note
NONE
```
2018-04-11 02:37:09 -07:00
Kubernetes Submit Queue 39847f5340
Merge pull request #62317 from xiangpengzhao/update-sca-test
Automatic merge from submit-queue (batch tested with PRs 46903, 61721, 62317). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Remove the use of storage class beta annotations in e2e tests.

**What this PR does / why we need it**:
Since we are going to remove storage class beta annotations, we'd like to remove the use of it in e2e tests to avoid upgrade test issues when it's finally removed.

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
ref: https://github.com/kubernetes/kubernetes/pull/51440#discussion_r180290591

**Special notes for your reviewer**:
This should be cherry-picked to 1.10 as well. https://github.com/kubernetes/kubernetes/pull/51440#discussion_r178919464

/cc @msau42 

**Release note**:

```release-note
NONE
```
2018-04-11 00:51:14 -07:00
Kubernetes Submit Queue 72a44f95f5
Merge pull request #61721 from liggitt/container-probe-timeout
Automatic merge from submit-queue (batch tested with PRs 46903, 61721, 62317). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Double container probe timeout

in some environments, we see a combination of start latency
and the corresponding effect on sync pod latency causing status
manager to fail to report within the 2 minute window.

```release-note
NONE
```
2018-04-11 00:51:11 -07:00
Kubernetes Submit Queue 1dc6e87f57
Merge pull request #62206 from yujuhong/rm-rkt-refs
Automatic merge from submit-queue (batch tested with PRs 62192, 61866, 62206, 62360). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Remove rkt references in the codebase

```release-note
None
```
2018-04-10 23:52:21 -07:00
Kubernetes Submit Queue 56d6f05d23
Merge pull request #61866 from davidz627/fix/CSIe2e
Automatic merge from submit-queue (batch tested with PRs 62192, 61866, 62206, 62360). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Add CSI External Components ClusterRole to bootstrapped roles 

Added CSI External Components ClusterRole to bootstrapped roles and removed creation from failing e2e test

Fixes: #61781

/sig storage
/kind bug
/assign @liggitt @saad-ali 

```release-note
NONE
```
2018-04-10 23:52:17 -07:00
Joe Betz dc4d92e154 Reimplement migrate-if-needed.sh in go 2018-04-10 09:19:30 -07:00
Kubernetes Submit Queue cc826360ef
Merge pull request #61614 from vmware/vpxd-restart-test
Automatic merge from submit-queue (batch tested with PRs 60197, 61614, 62074, 62071, 62301). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Adds e2e test for vpxd restart scenario in vSphere Cloud Provider

This PR adds a test to verify Volume access in a situation where VMware vCenter's `vpxd` service is down. The test mainly verifies Volume/file access before, during, and after restarting the `vpxd` service on the vCenter host.

See vmware/kubernetes#373 for more details.

**Reviewers note:** This PR was internally reviewed at vmware/kubernetes#468.

/cc @kubernetes/vmware
2018-04-10 08:56:19 -07:00
Kubernetes Submit Queue 6bf1871e73
Merge pull request #62291 from rramkumar1/patch-7
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Patch ingress upgrade test logic to take note of SNI support in next glbc release

**What this PR does / why we need it**:
/assign @MrHohn 

**Release note**:
```release-note
None
```
2018-04-10 06:19:50 -07:00
xiangpengzhao 291b1437a5 Auto generated BUILD files. 2018-04-10 15:30:15 +08:00
xiangpengzhao 1f80008f41 Remove the use of storage class beta annotations in e2e tests. 2018-04-10 15:29:08 +08:00