buildet: read-only directory in work area leads to cleanup failure when not root #34980

jclulow · 2019-10-18T05:48:58Z

When a buildlet is running as a non-root user, read-only directories like those managed by the Go module system can cause the work directory cleanup step to fail. The buildlet ends up exiting, and the stage0 program goes into a loop like this:

stage0: 2019/10/18 05:41:18 bootstrap binary running
stage0: 2019/10/18 05:41:18 waiting for network.
stage0: 2019/10/18 05:41:18 network up after 400ms
stage0: 2019/10/18 05:41:18 downloading https://storage.googleapis.com/go-builder-data/buildlet.illumos-amd64 to ./buildlet.exe ...
stage0: 2019/10/18 05:41:18 downloaded ./buildlet.exe (14194957 bytes)
stage0: 2019/10/18 05:41:18 downloaded buildlet in 100ms
2019/10/18 05:41:19 buildlet starting.
2019/10/18 05:41:19 unlinkat /var/tmp/workdir-host-illumos-amd64-jclulow/gopath/pkg/mod/github.com/hashicorp/golang-lru@v0.5.1/LICENSE: permission denied
stage0: 2019/10/18 05:41:19 Error running buildlet: exit status 1
...

The file we're trying to remove is read-only:

[root@gobuild1 ~]# stat /var/tmp/workdir-host-illumos-amd64-jclulow/gopath/pkg/mod/github.com/hashicorp/golang-lru@v0.5.1/LICENSE
  File: /var/tmp/workdir-host-illumos-amd64-jclulow/gopath/pkg/mod/github.com/hashicorp/golang-lru@v0.5.1/LICENSE
  Size: 15921           Blocks: 18         IO Block: 16384  regular file
Device: 5a000109c5h/386547124677d       Inode: 125098      Links: 1
Access: (0444/-r--r--r--)  Uid: (84001/ gobuild)   Gid: (84001/ gobuild)
Access: 2019-10-18 04:37:10.739769295 +0000
Modify: 2019-10-18 04:37:10.740269133 +0000
Change: 2019-10-18 04:37:10.740269133 +0000
 Birth: 2019-10-18 04:37:10.739769295 +0000

But the real issue is that the directory in which the file appears is also read-only:

[root@gobuild1 ~]# stat /var/tmp/workdir-host-illumos-amd64-jclulow/gopath/pkg/mod/github.com/hashicorp/golang-lru@v0.5.1/
  File: /var/tmp/workdir-host-illumos-amd64-jclulow/gopath/pkg/mod/github.com/hashicorp/golang-lru@v0.5.1/
  Size: 14              Blocks: 3          IO Block: 131072 directory
Device: 5a000109c5h/386547124677d       Inode: 125094      Links: 3
Access: (0500/dr-x------)  Uid: (84001/ gobuild)   Gid: (84001/ gobuild)
Access: 2019-10-18 04:37:10.738867957 +0000
Modify: 2019-10-18 04:37:10.741754528 +0000
Change: 2019-10-18 04:37:10.742657507 +0000
 Birth: 2019-10-18 04:37:10.738867957 +0000

According to unlinkat(2):

       The unlink() and unlinkat() functions will fail if:

       EACCES
                       Search permission is denied for a component of the path
                       prefix, or write permission is denied on the directory
                       containing the link to be removed.

You can see this with a relatively simple test case:

#!/bin/bash
  
set -o errexit
set -o pipefail

ROOT=/var/tmp/blahblah

chmod -R u+rwx "$ROOT" || true
rm -rf "$ROOT"

mkdir "$ROOT"
mkdir "$ROOT/first"
mkdir "$ROOT/first/second"
date > "$ROOT/first/second/file0"
date > "$ROOT/first/second/file1"
mkdir "$ROOT/first/third"
date > "$ROOT/first/third/file2"
date > "$ROOT/first/third/file3"
chmod 0444 "$ROOT/first/second/file0"
chmod 0444 "$ROOT/first/third/file2"
chmod 0500 "$ROOT/first/third"

find "$ROOT" -ls

package main

import (
        "os"
        "log"
)

func main() {
        rootdir := "/var/tmp/blahblah"

        if err := os.RemoveAll(rootdir); err != nil {
                log.Fatalf("removing \"%s\": %v", rootdir, err)
        }

        log.Printf("ok, removed \"%s\"", rootdir)
}

$ go build trial.go 
$ ./testcase.sh 
91262    1 drwxr-xr-x   3 jclulow  nfsusers        3 Oct 18 06:54 /var/tmp/blahblah
91263    1 drwxr-xr-x   4 jclulow  nfsusers        4 Oct 18 06:54 /var/tmp/blahblah/first
92547    1 dr-x------   2 jclulow  nfsusers        4 Oct 18 06:54 /var/tmp/blahblah/first/third
92548    1 -r--r--r--   1 jclulow  nfsusers       36 Oct 18 06:54 /var/tmp/blahblah/first/third/file2
92549    1 -rw-r--r--   1 jclulow  nfsusers       36 Oct 18 06:54 /var/tmp/blahblah/first/third/file3
92544    1 drwxr-xr-x   2 jclulow  nfsusers        4 Oct 18 06:54 /var/tmp/blahblah/first/second
92545    1 -r--r--r--   1 jclulow  nfsusers       36 Oct 18 06:54 /var/tmp/blahblah/first/second/file0
92546    1 -rw-r--r--   1 jclulow  nfsusers       36 Oct 18 06:54 /var/tmp/blahblah/first/second/file1
$ ./trial 
2019/10/18 06:54:20 removing "/var/tmp/blahblah": unlinkat /var/tmp/blahblah/first/third/file2: permission denied

As root, though, this succeeds:

$ pfexec ./trial
2019/10/18 06:55:15 ok, removed "/var/tmp/blahblah"

Prior to calling os.RemoveAll(), the buildlet should make an attempt to ensure all directories under the work area are writeable.

The text was updated successfully, but these errors were encountered:

gopherbot · 2019-10-18T06:14:16Z

Change https://golang.org/cl/201917 mentions this issue: cmd/buildlet: handle read-only directories in work area

Fixes golang/go#34980 Change-Id: I9d87e2267529212efc370ca6a858452eb829b5bb Reviewed-on: https://go-review.googlesource.com/c/build/+/201917 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>

gopherbot closed this as completed in golang/build@f377c61 Oct 18, 2019

golang locked and limited conversation to collaborators Oct 17, 2020

gopherbot added the FrozenDueToAge label Oct 17, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

buildet: read-only directory in work area leads to cleanup failure when not root #34980

buildet: read-only directory in work area leads to cleanup failure when not root #34980

jclulow commented Oct 18, 2019

gopherbot commented Oct 18, 2019

buildet: read-only directory in work area leads to cleanup failure when not root #34980

buildet: read-only directory in work area leads to cleanup failure when not root #34980

Comments

jclulow commented Oct 18, 2019

gopherbot commented Oct 18, 2019