github
/
k3s
mirror of https://github.com/k3s-io/k3s
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

StatefulSet: Deflake e2e RunHostCmd more. 

It turns out that at some points while the Node is recovering from a
reboot, we get a different kind of error ("unable to upgrade
connection"). Since we can't distinguish these transient errors from an
error encountered after successfully executing the remote command,
let's just retry all errors for 5min. If this doesn't work, I'm gonna
blame it on sig-node.
pull/6/head
Anthony Yeh 2017-09-12 10:12:46 -07:00
parent 6b6b1e5779
commit bff5f7e6b0
No known key found for this signature in database
GPG Key ID: 339F46A383E6ED08
1 changed files with 1 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
test/e2e/framework/util.go
View File

@ -3311,7 +3311,7 @@ func RunHostCmdOrDie(ns, name, cmd string) string {
return stdout
}
// RunHostCmdWithRetries calls RunHostCmd and retries errors it thinks may be transient
// RunHostCmdWithRetries calls RunHostCmd and retries all errors
// until it succeeds or the specified timeout expires.
// This can be used with idempotent commands to deflake transient Node issues.
func RunHostCmdWithRetries(ns, name, cmd string, interval, timeout time.Duration) (string, error) {
@ -3324,9 +3324,6 @@ func RunHostCmdWithRetries(ns, name, cmd string, interval, timeout time.Duration
if elapsed := time.Since(start); elapsed > timeout {
return out, fmt.Errorf("RunHostCmd still failed after %v: %v", elapsed, err)
}
if !strings.Contains(err.Error(), "Error from server") {
return out, fmt.Errorf("Non-retryable RunHostCmd error: %v", err)
}
Logf("Waiting %v to retry failed RunHostCmd: %v", interval, err)
time.Sleep(interval)
}