Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: Dragonfly builder is failing to remove workdir #30853

Closed
timdarbydotnet opened this issue Mar 14, 2019 · 5 comments
Closed

x/build: Dragonfly builder is failing to remove workdir #30853

timdarbydotnet opened this issue Mar 14, 2019 · 5 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. OS-Dragonfly
Milestone

Comments

@timdarbydotnet
Copy link

What version of Go are you using (go version)?

go version go1.11.4 dragonfly/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
amd64

When the builder tries to remove the existing /tmp workdir, this happens:

2019/03/14 20:05:58 buildlet starting.
2019/03/14 20:05:58 remove /tmp/workdir-host-dragonfly-amd64-tdfbsd/tmp/TestSimpleCases_import_grouping_not_path_dependent_no_groups_Modules_GoPackages625130825/modcache/pkg/mod/github.com/local@v1.0.0/go.mod: permission denied
^C

% ls -l /tmp/workdir-host-dragonfly-amd64-tdfbsd/tmp/TestSimpleCases_import_grouping_not_path_dependent_no_groups_Modules_GoPackages625130825/modcache/pkg/mod/github.com/local@v1.0.0/
total 16
dr-xr-xr-x 2 tim wheel 88 Mar 14 16:04 bar
-r--r--r-- 1 tim wheel 24 Mar 14 16:04 go.mod

So, the direct cause is that go.mod is being created with read-only permissions. I have no idea when this started happening though, or why.

@bradfitz bradfitz changed the title builder: Dragonfly builder is failing to remove workdir x/build: Dragonfly builder is failing to remove workdir Mar 14, 2019
@gopherbot gopherbot added this to the Unreleased milestone Mar 14, 2019
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Mar 14, 2019
@bradfitz bradfitz added OS-Dragonfly NeedsFix The path to resolution is known, but the work has not been done. and removed Builders x/build issues (builders, bots, dashboards) labels Mar 14, 2019
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Mar 14, 2019
@bcmills
Copy link
Contributor

bcmills commented Mar 14, 2019

That probably comes from here:

~/src/golang.org/x$ grep -r TestSimpleCases
tools/imports/fix_test.go:func TestSimpleCases(t *testing.T) {

So, two questions:

  1. Why is TestSimpleCases not cleaning up the directories in creates in /tmp? (@ianthehat, who owns golang.org/x/tools/imports at this point? Possibly @bradfitz or @heschik?)
  2. Why is the builder using a “remove” operation that can fail to remove things? (If it is using rm -rf, it should probably chmod -R u+w before attempting that.)

@timdarbydotnet
Copy link
Author

When this happens, it goes into a loop of continually trying the remove and prevents the builder from running. This just started sometime in the past week.

@heschi
Copy link
Contributor

heschi commented Mar 14, 2019

I don't know why https://golang.org/cl/156078 wouldn't take care of this. I just double-checked and on my workstation it doesn't leave directories behind, even in cases of test failure. I guess we can change Cleanup() to return an error and maybe that'll shed light on the problem?

@timdarbydotnet
Copy link
Author

I can try a debug patch, if you have one. For now, I'm running the builder as root to work around this.

@timdarbydotnet
Copy link
Author

This is working now (running as not root), but I didn't really do anything other than rebuild the buildlet, which hadn't been refreshed in some time.

@golang golang locked and limited conversation to collaborators Mar 23, 2020
@rsc rsc unassigned dmitshur and bcmills Jun 23, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. OS-Dragonfly
Projects
None yet
Development

No branches or pull requests

6 participants