Currently, we run the e2e tests in whatever order readdir happens to
return, which is random on some filesystems, name sorted on others,
create order on others, etc. Eventually, we may want to be
automatically hermetic between e2e tests (especially as we introduce
more resource destructive tests), but until then, it would be useful
if we permute the test order randomly between runs to ensure that
developers don't accidentally rely on a particular order. This
introduces a form of forced hermeticism, since improper state cleanup
from one test may not perturb a given test, but there's probably *a*
test in the suite that the order will perturb, so the RNG will find
that order eventually.
Adds logging of the generated seed, and an --orderseed argument that
can be used to re-run in the same order. Also sorts the pass/fail list
now for easier human reading.
Minor usability nuisance: If you run:
go run hack/e2e.go -v -test
.. and you don't happen to have an up e2e cluster, it should fail
fast, rather than chugging through every test and having them fall
over.
* Rewrite a bunch of the hack/ directory with modular reusable bash libraries.
* Have 'build/*' build on 'hack/*'. The stuff in build now just runs hack/* in a docker container.
* Use a docker data container to enable faster incremental builds.
* Standardize output to _output/{local,dockerized}/bin/OS/ARCH/*. This regularized placement makes cross compilation work.
* Move travis specific scripts under hack/travis
With new dockerized incremental builds, I can do a no-op `make quick-release` in ~30s. This is a significant improvement.
Zach Loafman