diff --git a/hack/jenkins/README.md b/hack/jenkins/README.md index 39e01cbecc..2f1d462da1 100644 --- a/hack/jenkins/README.md +++ b/hack/jenkins/README.md @@ -1,13 +1,16 @@ # Jenkins [Jenkins](http://jenkins-ci.org/) is a pluggable continuous -integration system. The Google team is running a Jenkins server on a -private GCE instance for the Kubernetes project in order to run longer -integration tests, continuously, on different providers. Currently, we -are running tests on GCE, GKE, and AWS. +integration system. The Google team is running two Jenkins servers in GCE for +the Kubernetes project. The post-commit instance runs continuous builds, unit +tests, integration tests, code verification tests, and end-to-end tests on +multiple providers using the latest commits to the Kubernetes repo from the +master and release branches. The PR Jenkins instance runs these tests on each +PR by a trusted contributor, it but only runs a subset of the end-to-end tests +and only on GCE. ## General flow -The flow of the Google Jenkins server: +The flow of the post-commit Jenkins instance: * Under the `kubernetes-build` job: Every 2 minutes, Jenkins polls for a batch of new commits, after which it runs the `build.sh` script (in this directory) on the latest tip. This results in build assets getting pushed to GCS and the @@ -18,9 +21,7 @@ The flow of the Google Jenkins server: * `kubernetes-e2e-gce`: Standard GCE e2e. * `kubernetes-e2e-gke`: GKE provider e2e, with head k8s client and GKE creating clusters at its default version. - * `kubernetes-e2e-gke-ci`: GKE provider e2e, with head k8s client and GKE - creating clusters at the head k8s version. - * `kubernetes-e2e-aws`: AWS provider e2e. + * `kubernetes-e2e-aws`: AWS provider e2e. This only runs once a day. * Each job will not run concurrently with itself, so, for instance, Jenkins executor will only ever run one `kubernetes-build` job. However, it may run the jobs in parallel, @@ -41,6 +42,42 @@ outside this repository, it's tricky to keep documentation for it up to date quickly. However, the scripts themselves attempt to provide color for the configuration(s) that each script runs in. +## GCS Log Format + +Our `upload-to-gcs.sh` script runs at the start and end of every job. Logs on +post-commit Jenkins go under `gs://kubernetes-jenkins/logs/`. Logs on PR +Jenkins go under `gs://kubernetes-jenkins-pull/pr-logs/pull/PULL_NUMBER/`. +Individual run logs go into the `JOB_NAME/BUILD_NUMBER` folder. + +At the start of the job, it uploads `started.json` containing the version of +Kubernetes under test and the timestamp. + +At the end, it uploads `finished.json` containing the result and timestamp, as +well as the build log into `build-log.txt`. Under `artifacts/` we put our +test results in `junit_XY.xml`, along with gcp resource lists and cluster logs. + +It also updates `latest-build.txt` at the end to point to this build number. +In the end, the directory structure looks like this: + +``` +gs://kubernetes-jenkins/logs/kubernetes-e2e-gce/ + latest-build.txt + 12345/ + build-log.txt + started.json + finished.json + artifacts/ + gcp-resources-{before, after}.txt + junit_{00, 01, ...}.xml + jenkins-e2e-master/{kube-apiserver.log, ...} + jenkins-e2e-minion-abcd/{kubelet.log, ...} + 12344/ + ... +``` + +The munger uses `latest-build.txt` and the JUnit reports to figure out whether +or not the job is healthy. + ## Job Builder New jobs should be specified as YAML files to be processed by [Jenkins Job @@ -51,4 +88,11 @@ in a Docker container defined in `job-builder-image`, and triggers it using [jenkins_jobs.ini](http://docs.openstack.org/infra/jenkins-job-builder/execution.html) which contains the location and credentials of the Jenkins server. +E2E Job definitions are templated to avoid code duplication. To add a new job, +add a new entry to the appropriate `project`. +[This](https://github.com/kubernetes/kubernetes/commit/eb273e5a4bdd3905f881563ada4e6543c7eb96b5) +is an example of a commit which does this. If necessary, create a new project, as in +[this](https://github.com/kubernetes/kubernetes/commit/09c27cdabc300e0420a2914100bedb565c23ed73) +commit. + [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/hack/jenkins/README.md?pixel)]()