Automatic merge from submit-queue
Collect logs for dead kubelets too
Collect logs via journalctl if journalctl is installed, rather than only if
kubelet.service is running. The old way resulted in us losing logs any
time the kubelet was failing. This, of course, breaks on a node if
someone decided to install journalctl but not use it. But that is not
the case on any of the images used by cluster-level tests at present.
^^^^FYI @Random-Liu not sure if `which journalctl` implies that journalctl is actually used on all of the nodes we test in the node-e2e suites. This may be of consequence if we move to using `cluster/log-dump.sh` to scrape logs for node-e2e.
P0 because this is somewhat in the way of debugging https://github.com/kubernetes/kubernetes/issues/33882
@jessfraz @saad-ali This should be cherry-picked to 1.4 and 1.5 as well.
Collect logs via journalctl if journalctl is installed, rather than only if
kubelet.service is running. The old way resulted in us losing logs any
time the kubelet was failing. This, of course, breaks on a node if
someone decided to install journalctl but not use it. But that is not
the case on any of the images used by cluster-level tests at present.
The LOG_DUMP_USE_KUBECTL path is fine, once the cluster is up.
However, we've had a continuous low-grade Up flake in the kops builds,
so I'd like to grab logs using the aws CLI.
This makes log-dump.sh extensible, so you can do:
function log_dump_custom_get_instances() { ... }
export -f log_dump_custom_get_instances
go run hack/e2e.go ...
This refactors log-dump.sh so that it can optionally just use "kubectl
get node" output, for dumping logs off a cluster that was brought up
using sometihng other than kube-up.sh (e.g. kops or kubeadm).
Note that for obvious reasons, if the cluster is fried, it'll probably
fail to hit the apiserver and all bets are off, but for the 95% case
where you're debugging tests, this is good. To cover the remaining 5%,
you could actually squirrel away the relevant external IPs after the
e2e.go:Up stage (because we better be able to hit the apiserver then)
and fetch them from cache later.
Shyam Jeedigunta