When etcd is down today we don't specifically handle the error involved,
which means clients get a generic 500 error. This commit adds a formal
error type internally for both WatchExpired and EtcdUnreachable, and
then converts them to api/errors before returning to the client. It also
upgrades the client to retry on any 429 or 5xx error that has a
Retry-After header, instead of just 429.
In combination, this allows the apiserver to exert backpressure on
controllers that are hotlooping. Picked 2 seconds by default, but we
could potentially ramp that up even further in a future iteration.
In order to support graceful deletion, the resource object will
need access to the TTL value in etcd. Also, in the future we
may want to get the creation index (distinct from modifiedindex)
and expose it to clients. Change EtcdResourceVersioner to be
more type specific (objects vs lists) and provide a default
implementation that relies on the internal API convention.
Also, rename etcd_tools.go to etcd_helper.go and split a few
things up.
Clayton Coleman