Add automatic retries to the workflow package. Most of our workflow
actions are idempotent and at least potentially flaky, and there's no
point making the user hit retry over and over.

Since we do have some non-idempotent tasks, allow the task function to
disable retries by calling (*TaskContext).DisableRetries. This means we
don't know if retries are allowed until after the first one, which makes
for a slightly clumsy implementation but a nice easy API.

Retries are currently off by default while I finish implementing up the
stack, but enabled in all the tests.

For golang/go#53886.

Change-Id: I15c57cf7b23785560238bfd6a45bd7744ba29b32
Reviewed-on: https://go-review.googlesource.com/c/build/+/417589
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Jenny Rakoczy <jenny@golang.org>
Run-TryBot: Heschi Kreinick <heschi@google.com>

Store the retry count in the database and test that it all works.

I removed the definition argument from start; we can read it just as
well from the holder, and it makes it harder to forget to register it.

For golang/go#53886.

Change-Id: I4c3d48a96bae87c31629f76067e75ccd524381b0
Reviewed-on: https://go-review.googlesource.com/c/build/+/417590
Reviewed-by: Jenny Rakoczy <jenny@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Heschi Kreinick <heschi@google.com>

Mark tasks that should not be retried and enable automatic retries by
default.

For golang/go#53886.

Change-Id: I6c2b48cff83477c7c3ed7cf8b753b1646831e233
Reviewed-on: https://go-review.googlesource.com/c/build/+/417591
Auto-Submit: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
Reviewed-by: Jenny Rakoczy <jenny@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>