Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/playground: deploy may take over 10 minute and timeout #28046

Closed
dmitshur opened this issue Oct 5, 2018 · 9 comments
Closed

x/playground: deploy may take over 10 minute and timeout #28046

dmitshur opened this issue Oct 5, 2018 · 9 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unreleased

Comments

@dmitshur
Copy link
Contributor

dmitshur commented Oct 5, 2018 •
edited

I've ran into this when deploying the Go Playground with Go 1.11.1 recently (see CL 140097), where the cloud build failed because it took more than the default 10 minute timeout:

ERROR: (gcloud.app.deploy) Cloud build failed. Check logs at https://console.cloud.google.com Failure status: UNKNOWN: Error Response: [4] DEADLINE_EXCEEDED

In the recent past, some builds have taken around 9~ minutes. They were faster earlier on.

People have run into the 10 minute timeout in other contexts too. E.g., see https://stackoverflow.com/questions/50046545/google-app-engine-build-timed-out-during-deployment.

For now, this issue is to gather data and see if this keeps happening. We can then decide on how to best deal with it. If you run into this, please post data here. /cc @bradfitz @andybons

Workaround

The easiest temporary workaround I'm currently aware of is to increase the timeout before the gcloud app deploy step. For example:

gcloud config set app/cloud_build_timeout 900 # 15 mins

Note that it's a global config value.
@dmitshur dmitshur added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Oct 5, 2018
@gopherbot gopherbot added this to the Unreleased milestone Oct 5, 2018
@ysmolski
Copy link
Member

ysmolski commented Oct 6, 2018

I wonder if this build command includes building the docker container? I have noticed that building playground docker container became very slow. Specifically fetching packages is slow. Since I was not sure if it was my network, I did not create ticket for that.

@dmitshur
Copy link
Contributor Author

dmitshur commented Oct 7, 2018

I wonder if this build command includes building the docker container?

Yes, it does. That's the part that takes up to 8-9 minutes.

Specifically fetching packages is slow.

I didn't notice that to be the slow part, but thanks for pointing out. I'll watch it more closely next time.

@dmitshur
Copy link
Contributor Author

dmitshur commented Oct 8, 2018

A data point from today's deploy (for #25462): the cloud build step took 10 min 58 sec. (It was within my 15 minute timeout.)

@dmitshur
Copy link
Contributor Author

dmitshur commented Oct 24, 2018 •
edited

Another data point from today's deploy: the cloud build step took 10 min 43 sec.

Edit: Another data point from a deploy on 2018-10-26: 10 min 40 sec.

@dmitshur
Copy link
Contributor Author

Two data points for the cloud build step from today's deploy (in #30397), using Go version 1.12:

  • 12 min 4 sec
  • 11 min 27 sec

By now, it's clear we are reliably beyond the 10 minute mark.

@dmitshur dmitshur added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Builders x/build issues (builders, bots, dashboards) and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Feb 26, 2019
@dmitshur
Copy link
Contributor Author

dmitshur commented Feb 28, 2019 •
edited

I'm going to send a fix for this issue by documenting the command to set app/cloud_build_timeout to a value higher than the default 10 minutes. /cc @bradfitz @andybons

If we discover a better solution in the future, we can improve it then.

@gopherbot
Copy link

Change https://golang.org/cl/164577 mentions this issue: README: document the need to increase cloud build timeout

@go101
Copy link

go101 commented Mar 5, 2019

Is it possible it is a problem of app engine? https://groups.google.com/forum/#!topic/google-appengine-go/fBdzFHZ1D5U

@dmitshur
Copy link
Contributor Author

dmitshur commented Mar 6, 2019

This issue was related to the time it takes to build the Docker image using Cloud Build, which grew from 8~ minutes to 12~ minutes over time, and how it intersected with the default timeout of 10 minutes. The linked thread doesn't seem to be directly related.

@dmitshur dmitshur mentioned this issue Oct 16, 2019
Closed
@golang golang locked and limited conversation to collaborators Mar 5, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
4 participants
@ysmolski @dmitshur @gopherbot @go101