Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/app/appengine: deploying with dirty working tree affects the deploy #34936

Closed
bradfitz opened this issue Oct 16, 2019 · 11 comments
Closed

x/build/app/appengine: deploying with dirty working tree affects the deploy #34936

bradfitz opened this issue Oct 16, 2019 · 11 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Unreleased

Comments

@bradfitz
Copy link
Contributor

Following the instructions at https://github.com/golang/build/tree/master/app/appengine#deploying I can no longer deploy:

bradfitz@go:~/src/golang.org/x/build/app/appengine$ GO111MODULE=on gcloud app --account=bradfitz@google.com --project=golang-org deploy app.yaml
Services to deploy:
        
descriptor:      [/home/bradfitz/src/golang.org/x/build/app/appengine/app.yaml]
source:          [/home/bradfitz/src/golang.org/x/build/app/appengine]
target project:  [golang-org]
target service:  [build]
target version:  [20191016t151730]
target url:      [https://build-dot-golang-org.appspot.com]


Do you want to continue (Y/n)?

Beginning deployment of service [build]...
╔════════════════════════════════════════════════════════════╗
╠═ Uploading 263 files to Google Cloud Storage              ═╣
╚════════════════════════════════════════════════════════════╝
File upload done.
ERROR: (gcloud.app.deploy) INVALID_ARGUMENT: Filename cannot contain '.', '..', '\r', start with '-', '_ah/', or '\n': cmd/coordinator/coordinator.

What changed?

Docs old?

/cc @toothrot @dmitshur @bcmills @andybons
@gopherbot gopherbot added this to the Unreleased milestone Oct 16, 2019
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Oct 16, 2019
@dmitshur
Copy link
Contributor

dmitshur commented Oct 16, 2019 •
edited

Docs worked when I deployed last, on Oct 8. I use --no-promote flag explicitly and then promote after making sure the new version looks okay, but I don't expect that to cause the difference above.

Is your git status clean and are you on latest master commit of x/build?

I'm trying a --no-promote deploy now and not seeing any issues so far:


appengine $ GO111MODULE=on gcloud --project=golang-org app deploy --no-promote app.yaml
Services to deploy:

descriptor:      [/Users/dmitshur/go/src/golang.org/x/build/app/appengine/app.yaml]
source:          [/Users/dmitshur/go/src/golang.org/x/build/app/appengine]
target project:  [golang-org]
target service:  [build]
target version:  [20191016t122939]
target url:      [https://20191016t122939-dot-build-dot-golang-org.appspot.com]


     (add --promote if you also want to make this service available from
     [https://build-dot-golang-org.appspot.com])

Do you want to continue (Y/n)?  Y

Beginning deployment of service [build]...
╔════════════════════════════════════════════════════════════╗
╠═ Uploading 1 file to Google Cloud Storage                 ═╣
╚════════════════════════════════════════════════════════════╝
File upload done.
Updating service [build]...done.                                               
Deployed service [build] to [https://20191016t122939-dot-build-dot-golang-org.appspot.com]

You can stream logs from the command line by running:
  $ gcloud app logs tail -s build

To view your application in the web browser run:
  $ gcloud app browse -s build --project=golang-org


Updates are available for some Cloud SDK components.  To install them,
please run:
  $ gcloud components update

@dmitshur dmitshur added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Oct 16, 2019
@gopherbot
Copy link

Change https://golang.org/cl/201477 mentions this issue: app/appengine: update deploy instructions

@bradfitz
Copy link
Contributor Author

I did have an empty file cmd/coordinator/coordinator. (final period). Now that that's deleted, it makes it further but fails with Cloud build did not succeed within 10m:

^Cbradfitz@go:~/src/golang.org/x/build/app/appengine$ GO111MODULE=on gcloud app --account=bradfitz@google.com --project=golang-org deploy app.yaml
Services to deploy:

descriptor:      [/home/bradfitz/src/golang.org/x/build/app/appengine/app.yaml]
source:          [/home/bradfitz/src/golang.org/x/build/app/appengine]
target project:  [golang-org]
target service:  [build]
target version:  [20191016t165030]
target url:      [https://build-dot-golang-org.appspot.com]


Do you want to continue (Y/n)?

Beginning deployment of service [build]...
╔════════════════════════════════════════════════════════════╗
╠═ Uploading 1 file to Google Cloud Storage                 ═╣
╚════════════════════════════════════════════════════════════╝
File upload done.
Updating service [build]...failed.                                                                                                                                                      
ERROR: (gcloud.app.deploy) Error Response: [4] Cloud build did not succeed within 10m.
Build error details: Build error details not available..
Check the build log for errors: https://console.cloud.google.com/gcr/builds/4d8f8f8a-bf65-4e0d-9cc6-ebc8053c8422?project=397748307997

Versions:

$ gcloud version
Google Cloud SDK 246.0.0
alpha 2019.05.10
app-engine-go 
app-engine-python 1.9.85
beta 2019.05.10
bq 2.0.43
cloud-datastore-emulator 2.1.0
core 2019.05.10
gsutil 4.38
kubectl 2019.05.10

@dmitshur
Copy link
Contributor

dmitshur commented Oct 16, 2019 •
edited

That's very strange. The deploy took 2~ minutes for me when I tried it just now.

The error is similar to an issue we had when deploying the playground was taking too long, and hitting that time limit. See #28046. You can try increasing the cloud build timeout with gcloud config set app/cloud_build_timeout 900, but I think we need to fix the build taking more than 10 minutes in the first place.

Looking at the cloud build log mentioned in your output, it just says:

[...]
Finished Step #1 - "builder"
TIMEOUT
ERROR: context deadline exceeded

Compare with my earlier successful build (https://console.cloud.google.com/cloud-build/builds/65043963-4de2-4a3d-bf06-9862eace8b0b?project=golang-org):

[...]
Finished Step #1 - "builder"
PUSH
DONE

Not a lot of relevant detail here. Maybe it was a flake, so try again?

@bradfitz
Copy link
Contributor Author

I can't wait for #34744 to delete all this.

@bradfitz
Copy link
Contributor Author

Ah, I did have some extra files in git status. It was an unknown directory I thought was harmless but it had some huge iso/disk images in it.

That didn't matter prior to the GO111MODULE=on addition. Now I guess we upload everything to The Cloud, rather than just the relevant files.

Maybe I'll add a sanity check step in there to look for big unknown files.

@dmitshur
Copy link
Contributor

dmitshur commented Oct 16, 2019 •
edited

Yeah, GO111MODULE=on makes gcloud app deploy use the module root as the root, rather than the directory containing app.yaml. It was added to make deploys consistent regardless of who was deploying and their local environment’s GO111MODULE value.

I think there are ways to ignore certain files or directories from being included, for example via .gcloudignore file. See https://cloud.google.com/sdk/gcloud/reference/topic/gcloudignore.

@dmitshur dmitshur changed the title x/build/app/appengine: can't deploy anymore x/build/app/appengine: deploying with dirty working tree affects the deploy Oct 31, 2019
@dmitshur
Copy link
Contributor

dmitshur commented Nov 6, 2019 •
edited

@bradfitz Is there more to do here, or did CL 205597 fix this by adding the .cloudignore .gcloudignore file?

Edit: Fixed typo.

@bradfitz
Copy link
Contributor Author

bradfitz commented Nov 6, 2019

@dmitshur, the .gcloudignore (with a g) file fixes the worst of it. That could be enough probably.

@bradfitz bradfitz closed this as completed Nov 6, 2019
@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Nov 6, 2019
@dmitshur dmitshur mentioned this issue Jan 14, 2020
Closed
@dmitshur
Copy link
Contributor

dmitshur commented Jan 14, 2020 •
edited

I ran into this today too. I couldn't deploy x/build/app/appengine because it was failing to build due to a "MemoryError":


dmitshur@dev-dmitshur:~/go/src/golang.org/x/build$ cd app/appengine/
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$ git status
On branch deploy-cl-214433
nothing to commit, working tree clean
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$ go test 
PASS
ok  	golang.org/x/build/app/appengine	0.108s
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$ make deploy-test
go install golang.org/x/build/cmd/xb
GO111MODULE=on gcloud app --account=$(xb google-email) --project=golang-org deploy --no-promote app.yaml
Services to deploy:

descriptor:      [/home/dmitshur/go/src/golang.org/x/build/app/appengine/app.yaml]
source:          [/home/dmitshur/go/src/golang.org/x/build/app/appengine]
target project:  [golang-org]
target service:  [build]
target version:  [20200114t160755]
target url:      [https://20200114t160755-dot-build-dot-golang-org.appspot.com]


     (add --promote if you also want to make this service available from
     [https://build-dot-golang-org.appspot.com])

Do you want to continue (Y/n)?  Y

Beginning deployment of service [build]...
╔════════════════════════════════════════════════════════════╗
╠═ Uploading 560 files to Google Cloud Storage              ═╣
╚════════════════════════════════════════════════════════════╝
File upload done.
Updating service [build]...failed.                                             
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build 9610bd65-62a1-4ba6-bd2e-8f60981f0d64 status: FAILURE.
Error ID: E479B424.
Error type: BuildError.
Error message: 2020/01/14 16:10:38 Building /tmp/staging/srv, with main package at ./app/appengine, saving to /tmp/staging/usr/local/bin/start
2020/01/14 16:10:38 Running /usr/local/go/bin/go build -o /tmp/staging/usr/local/bin/start ./app/appengine
2020/01/14 16:11:13 Wrote build output to /builder/outputs/output
2020/01/14 16:11:13 Wrote output to /tmp/staging/usr/local/bin/start
2020/01/14 16:11:13 Making tarball /tmp/tar-out666215249/app.tar.gz
2020/01/14 16:12:21 Creating app image on GCR: us.gcr.io/golang-org/app-engine-tmp/app/ttl-2h:232fb67d-0b88-4b64-80ac-8bc887b981e5
2020/01/14 16:13:06 Failed to create app image on GCR: [/bin/appender.par --src-image us.gcr.io/gae-runtimes/go113:go113_20191215_1_13_5_RC00 --tarball /tmp/tar-out666215249/app.tar.gz --dst-image us.gcr.io/golang-org/app-engine-tmp/app/ttl-2h:232fb67d-0b88-4b64-80ac-8bc887b981e5]: Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/bin/appender.par/__main__.py", line 90, in <module>
  File "/bin/appender.par/__main__.py", line 76, in main
  File "/bin/appender.par/containerregistry/client/v2_2/append_.py", line 74, in __init__
  File "/bin/appender.par/containerregistry/client/v2_2/docker_image_.py", line 110, in uncompressed_blob
  File "/usr/lib/python2.7/gzip.py", line 260, in read
    self._read(readsize)
  File "/usr/lib/python2.7/gzip.py", line 319, in _read
    self._add_read_data( uncompress )
  File "/usr/lib/python2.7/gzip.py", line 337, in _add_read_data
    self.extrabuf = self.extrabuf[offset:] + data
MemoryError.
Makefile:10: recipe for target 'deploy-test' failed
make: *** [deploy-test] Error 1
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$

I had a "clean working tree" as reported by git status, but the problem was that there are 16 .gitignore files in the x/build repository, and they mask some files from being easily visible while the current .gcloudignore doesn't exclude them from being uploaded:


dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$ git status
On branch deploy-cl-214433
nothing to commit, working tree clean
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$ git-branches 
| Branch               | Base   | Behind | Ahead |
|----------------------|--------|-------:|:------|
| **deploy-cl-214433** | master |      0 | 1     |
| master               | master |      0 | 0     |

| Branch               | Remote        | Behind | Ahead |
|----------------------|---------------|-------:|:------|
| **deploy-cl-214433** |               |        |       |
| master               | origin/master |      0 | 0     |
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$ find ../.. -name '.gitignore'
../../cmd/buildlet/.gitignore
../../cmd/coordinator/.gitignore
../../cmd/pubsubhelper/.gitignore
../../cmd/docker2boot/.gitignore
../../cmd/scaleway/.gitignore
../../cmd/gitmirror/.gitignore
../../env/darwin/macstadium/10_8/.gitignore
../../env/windows/.gitignore
../../env/openbsd-386/.gitignore
../../env/plan9/.gitignore
../../env/netbsd-amd64/.gitignore
../../env/freebsd-amd64/.gitignore
../../env/linux-ppc64le/osuosl/.gitignore
../../env/netbsd-386/.gitignore
../../env/openbsd-amd64/.gitignore
../../maintner/maintnerd/.gitignore
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$ git status
On branch deploy-cl-214433
nothing to commit, working tree clean
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$ rm ../../cmd/buildlet/.gitignore ../../cmd/coordinator/.gitignore ../../cmd/pubsubhelper/.gitignore ../../cmd/docker2boot/.gitignore ../../cmd/scaleway/.gitignore ../../cmd/gitmirror/.gitignore ../../env/darwin/macstadium/10_8/.gitignore ../../env/windows/.gitignore ../../env/openbsd-386/.gitignore ../../env/plan9/.gitignore ../../env/netbsd-amd64/.gitignore ../../env/freebsd-amd64/.gitignore ../../env/linux-ppc64le/osuosl/.gitignore ../../env/netbsd-386/.gitignore ../../env/openbsd-amd64/.gitignore ../../maintner/maintnerd/.gitignore
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$ find ../.. -name '.gitignore'
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$ git status
On branch deploy-cl-214433
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	deleted:    ../../cmd/buildlet/.gitignore
	deleted:    ../../cmd/coordinator/.gitignore
	deleted:    ../../cmd/docker2boot/.gitignore
	deleted:    ../../cmd/gitmirror/.gitignore
	deleted:    ../../cmd/pubsubhelper/.gitignore
	deleted:    ../../cmd/scaleway/.gitignore
	deleted:    ../../env/darwin/macstadium/10_8/.gitignore
	deleted:    ../../env/freebsd-amd64/.gitignore
	deleted:    ../../env/linux-ppc64le/osuosl/.gitignore
	deleted:    ../../env/netbsd-386/.gitignore
	deleted:    ../../env/netbsd-amd64/.gitignore
	deleted:    ../../env/openbsd-386/.gitignore
	deleted:    ../../env/openbsd-amd64/.gitignore
	deleted:    ../../env/plan9/.gitignore
	deleted:    ../../env/windows/.gitignore
	deleted:    ../../maintner/maintnerd/.gitignore

Untracked files:
  (use "git add <file>..." to include in what will be committed)

	../../cmd/buildlet/buildlet.openbsd-386-64
	../../cmd/buildlet/buildlet.openbsd-amd64-64
	../../cmd/coordinator/coordinator
	../../cmd/gitmirror/gitmirror
	../../env/openbsd-386/install64-i386.iso
	../../env/openbsd-386/openbsd-i386-gce.tar.gz
	../../env/openbsd-amd64/install64-amd64.iso
	../../env/openbsd-amd64/openbsd-amd64-gce.tar.gz
	../../maintner/maintnerd/ca-certificates.crt
	../../maintner/maintnerd/maintnerd

no changes added to commit (use "git add" and/or "git commit -a")
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$ ls -lah  ../../cmd/buildlet/buildlet.openbsd-386-64 ../../cmd/buildlet/buildlet.openbsd-amd64-64 ../../cmd/coordinator/coordinator ../../cmd/gitmirror/gitmirror ../../env/openbsd-386/install64-i386.iso ../../env/openbsd-386/openbsd-i386-gce.tar.gz ../../env/openbsd-amd64/install64-amd64.iso ../../env/openbsd-amd64/openbsd-amd64-gce.tar.gz ../../maintner/maintnerd/ca-certificates.crt ../../maintner/maintnerd/maintnerd
-rwxr-xr-x 1 dmitshur dmitshur 6.9M Oct  9  2018 ../../cmd/buildlet/buildlet.openbsd-386-64
-rwxr-xr-x 1 dmitshur dmitshur 7.9M Oct  9  2018 ../../cmd/buildlet/buildlet.openbsd-amd64-64
-rwxr-xr-x 1 dmitshur dmitshur  33M Feb 27  2019 ../../cmd/coordinator/coordinator
-rwxr-xr-x 1 dmitshur dmitshur  13M Jan  4  2019 ../../cmd/gitmirror/gitmirror
-rw-r--r-- 1 dmitshur dmitshur 324M Oct 19  2018 ../../env/openbsd-386/install64-i386.iso
-rw-r--r-- 1 dmitshur dmitshur 326M Oct 26  2018 ../../env/openbsd-386/openbsd-i386-gce.tar.gz
-rw-r--r-- 1 dmitshur dmitshur 344M Oct 19  2018 ../../env/openbsd-amd64/install64-amd64.iso
-rw-r--r-- 1 dmitshur dmitshur 372M Oct 26  2018 ../../env/openbsd-amd64/openbsd-amd64-gce.tar.gz
-rw-r--r-- 1 dmitshur dmitshur 230K Jul 17  2018 ../../maintner/maintnerd/ca-certificates.crt
-rwxr-xr-x 1 dmitshur dmitshur  19M Jan 25  2019 ../../maintner/maintnerd/maintnerd
dmitshur@dev-dmitshur:~/go/src/golang.org/x/build/app/appengine$

In my case, it was trying to deploy more than 1 GB of unwanted binary data.

I've opened #36553 to either remove .gitignore files that we can get by without (so that files aren't invisible when you do git status), or include them into .gcloudignore.

@xyzdevops
Copy link

having a shared go.mod file causes unpredictable deploys. best to have a separate .mod file in each app engine app.yaml folder

@golang golang locked and limited conversation to collaborators Jan 16, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
4 participants
@bradfitz @dmitshur @gopherbot @xyzdevops