Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/compile: internal compiler error: lost track of variable in liveness while building kubernetes #19632

Closed
mkumatag opened this issue Mar 21, 2017 · 16 comments
Closed

cmd/compile: internal compiler error: lost track of variable in liveness while building kubernetes #19632

mkumatag opened this issue Mar 21, 2017 · 16 comments
Labels
FrozenDueToAge
Milestone
Go1.9

Comments

@mkumatag
Copy link

mkumatag commented Mar 21, 2017 •
edited

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

Upstream golang - master branch

$ git rev-parse HEAD
d972dc2
$

What operating system and processor architecture are you using (go env)?

GOARCH="ppc64le"
GOBIN=""
GOEXE=""
GOHOSTARCH="ppc64le"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root/upstream_k8s"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_ppc64le"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build937400080=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?

  1. Downloaded and installed https://storage.googleapis.com/golang/go1.8.linux-ppc64le.tar.gz
  2. Cloned upstream golang and compiled(Master branch)
  3. Cloned kubernetes project. https://github.com/kubernetes/kubernetes.git
  4. make all

What did you expect to see?

Expect to build all the binaries without any errors

What did you see instead?

federation/pkg/federation-controller/service/dns.go:287:6: internal compiler error: lost track of variable in liveness: .autotmp_538 (0xc42a2efb00, 0xc42a2efb00)

complete build log:

root@ubuntu:~/upstream_k8s/src/k8s.io/kubernetes# pwd
/root/upstream_k8s/src/k8s.io/kubernetes
root@ubuntu:~/upstream_k8s/src/k8s.io/kubernetes# export GOPATH=/root/upstream_k8s
root@ubuntu:~/upstream_k8s/src/k8s.io/kubernetes# make all
+++ [0321 02:27:12] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0321 02:27:13] Generating bindata:
    test/e2e/generated/gobindata_util.go
~/upstream_k8s/src/k8s.io/kubernetes ~/upstream_k8s/src/k8s.io/kubernetes/test/e2e/generated
~/upstream_k8s/src/k8s.io/kubernetes/test/e2e/generated
+++ [0321 02:27:14] Building go targets for linux/ppc64le:
    cmd/libs/go2idl/deepcopy-gen
+++ [0321 02:27:20] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0321 02:27:20] Generating bindata:
    test/e2e/generated/gobindata_util.go
~/upstream_k8s/src/k8s.io/kubernetes ~/upstream_k8s/src/k8s.io/kubernetes/test/e2e/generated
~/upstream_k8s/src/k8s.io/kubernetes/test/e2e/generated
+++ [0321 02:27:21] Building go targets for linux/ppc64le:
    cmd/libs/go2idl/defaulter-gen
+++ [0321 02:27:27] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0321 02:27:27] Generating bindata:
    test/e2e/generated/gobindata_util.go
~/upstream_k8s/src/k8s.io/kubernetes ~/upstream_k8s/src/k8s.io/kubernetes/test/e2e/generated
~/upstream_k8s/src/k8s.io/kubernetes/test/e2e/generated
+++ [0321 02:27:28] Building go targets for linux/ppc64le:
    cmd/libs/go2idl/conversion-gen
+++ [0321 02:27:35] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0321 02:27:35] Generating bindata:
    test/e2e/generated/gobindata_util.go
~/upstream_k8s/src/k8s.io/kubernetes ~/upstream_k8s/src/k8s.io/kubernetes/test/e2e/generated
~/upstream_k8s/src/k8s.io/kubernetes/test/e2e/generated
+++ [0321 02:27:36] Building go targets for linux/ppc64le:
    cmd/libs/go2idl/openapi-gen
+++ [0321 02:27:45] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0321 02:27:45] Generating bindata:
    test/e2e/generated/gobindata_util.go
~/upstream_k8s/src/k8s.io/kubernetes ~/upstream_k8s/src/k8s.io/kubernetes/test/e2e/generated
~/upstream_k8s/src/k8s.io/kubernetes/test/e2e/generated
+++ [0321 02:27:46] Building go targets for linux/ppc64le:
    cmd/kube-proxy
    cmd/kube-apiserver
    cmd/kube-controller-manager
    cmd/cloud-controller-manager
    cmd/kubelet
    cmd/kubeadm
    cmd/hyperkube
    vendor/k8s.io/kube-aggregator
    plugin/cmd/kube-scheduler
    cmd/kubectl
    federation/cmd/kubefed
    cmd/gendocs
    cmd/genkubedocs
    cmd/genman
    cmd/genyaml
    cmd/mungedocs
    cmd/genswaggertypedocs
    cmd/linkcheck
    examples/k8petstore/web-server/src
    federation/cmd/genfeddocs
    vendor/github.com/onsi/ginkgo/ginkgo
    test/e2e/e2e.test
    cmd/kubemark
    vendor/github.com/onsi/ginkgo/ginkgo
    test/e2e_node/e2e_node.test
    cmd/gke-certificates-controller
# k8s.io/kubernetes/federation/pkg/federation-controller/service
federation/pkg/federation-controller/service/dns.go:287:6: internal compiler error: lost track of variable in liveness: .autotmp_538 (0xc42a2efb00, 0xc42a2efb00)

goroutine 1 [running]:
runtime/debug.Stack(0x0, 0x0, 0x0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x68
cmd/compile/internal/gc.Fatalf(0x63817c, 0x2f, 0xc4297f9568, 0x3, 0x3)
	/usr/local/go/src/cmd/compile/internal/gc/subr.go:175 +0x1b8
cmd/compile/internal/gc.liveIndex(0xc42a2efb00, 0xc42a340a00, 0x18, 0x20, 0xc400000006)
	/usr/local/go/src/cmd/compile/internal/gc/plive.go:270 +0x1d0
cmd/compile/internal/gc.(*Liveness).valueEffects(0xc42a3501c0, 0xc429a26140, 0x1, 0x85)
	/usr/local/go/src/cmd/compile/internal/gc/plive.go:189 +0x6c
cmd/compile/internal/gc.livenessprologue(0xc42a3501c0)
	/usr/local/go/src/cmd/compile/internal/gc/plive.go:451 +0xc4
cmd/compile/internal/gc.liveness(0xc42a30a960, 0xc42a290780, 0xc42a330af0, 0xc42a330bd0, 0xc42a288ff8)
	/usr/local/go/src/cmd/compile/internal/gc/plive.go:1115 +0xc0
cmd/compile/internal/gc.genssa(0xc42a290780, 0xc42014e210)
	/usr/local/go/src/cmd/compile/internal/gc/ssa.go:4260 +0xcc
cmd/compile/internal/gc.compile(0xc420103cc0)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:358 +0x43c
cmd/compile/internal/gc.funccompile(0xc420103cc0)
	/usr/local/go/src/cmd/compile/internal/gc/dcl.go:1172 +0xbc
cmd/compile/internal/gc.Main(0x63d8a0)
	/usr/local/go/src/cmd/compile/internal/gc/main.go:511 +0x2234
main.main()
	/usr/local/go/src/cmd/compile/main.go:49 +0x94

!!! [0321 02:29:01] Call tree:
!!! [0321 02:29:01]  1: /root/upstream_k8s/src/k8s.io/kubernetes/hack/lib/golang.sh:713 kube::golang::build_binaries_for_platform(...)
!!! [0321 02:29:01]  2: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0321 02:29:01] Call tree:
!!! [0321 02:29:01]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0321 02:29:01] Call tree:
!!! [0321 02:29:01]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
Makefile:89: recipe for target 'all' failed
make: *** [all] Error 1
root@ubuntu:~/upstream_k8s/src/k8s.io/kubernetes#
@mkumatag
Copy link
Author

/cc @laboger @mdempsky @bradfitz

@mkumatag
Copy link
Author

Its happening in x86 architecture as well:

root@localhost:~/upstream_k8s/src/k8s.io/kubernetes# make all
+++ [0321 05:09:24] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0321 05:09:25] Generating bindata:
    test/e2e/generated/gobindata_util.go
~/upstream_k8s/src/k8s.io/kubernetes ~/upstream_k8s/src/k8s.io/kubernetes/test/e2e/generated
~/upstream_k8s/src/k8s.io/kubernetes/test/e2e/generated
+++ [0321 05:09:25] Building go targets for linux/amd64:
    cmd/libs/go2idl/deepcopy-gen
+++ [0321 05:09:31] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0321 05:09:31] Generating bindata:
    test/e2e/generated/gobindata_util.go
~/upstream_k8s/src/k8s.io/kubernetes ~/upstream_k8s/src/k8s.io/kubernetes/test/e2e/generated
~/upstream_k8s/src/k8s.io/kubernetes/test/e2e/generated
+++ [0321 05:09:31] Building go targets for linux/amd64:
    cmd/libs/go2idl/defaulter-gen
+++ [0321 05:09:36] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0321 05:09:36] Generating bindata:
    test/e2e/generated/gobindata_util.go
~/upstream_k8s/src/k8s.io/kubernetes ~/upstream_k8s/src/k8s.io/kubernetes/test/e2e/generated
~/upstream_k8s/src/k8s.io/kubernetes/test/e2e/generated
+++ [0321 05:09:37] Building go targets for linux/amd64:
    cmd/libs/go2idl/conversion-gen
+++ [0321 05:09:42] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0321 05:09:42] Generating bindata:
    test/e2e/generated/gobindata_util.go
~/upstream_k8s/src/k8s.io/kubernetes ~/upstream_k8s/src/k8s.io/kubernetes/test/e2e/generated
~/upstream_k8s/src/k8s.io/kubernetes/test/e2e/generated
+++ [0321 05:09:43] Building go targets for linux/amd64:
    cmd/libs/go2idl/openapi-gen
+++ [0321 05:09:50] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0321 05:09:51] Generating bindata:
    test/e2e/generated/gobindata_util.go
~/upstream_k8s/src/k8s.io/kubernetes ~/upstream_k8s/src/k8s.io/kubernetes/test/e2e/generated
~/upstream_k8s/src/k8s.io/kubernetes/test/e2e/generated
+++ [0321 05:09:51] Building go targets for linux/amd64:
    cmd/kube-proxy
    cmd/kube-apiserver
    cmd/kube-controller-manager
    cmd/cloud-controller-manager
    cmd/kubelet
    cmd/kubeadm
    cmd/hyperkube
    vendor/k8s.io/kube-aggregator
    plugin/cmd/kube-scheduler
    cmd/kubectl
    federation/cmd/kubefed
    cmd/gendocs
    cmd/genkubedocs
    cmd/genman
    cmd/genyaml
    cmd/mungedocs
    cmd/genswaggertypedocs
    cmd/linkcheck
    examples/k8petstore/web-server/src
    federation/cmd/genfeddocs
    vendor/github.com/onsi/ginkgo/ginkgo
    test/e2e/e2e.test
    cmd/kubemark
    vendor/github.com/onsi/ginkgo/ginkgo
    test/e2e_node/e2e_node.test
    cmd/gke-certificates-controller
# k8s.io/kubernetes/federation/pkg/federation-controller/service
federation/pkg/federation-controller/service/dns.go:287:6: internal compiler error: lost track of variable in liveness: .autotmp_539 (0xc42a386c00, 0xc42a386c00)

goroutine 1 [running]:
runtime/debug.Stack(0x0, 0x0, 0x0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x79
cmd/compile/internal/gc.Fatalf(0xad1dc6, 0x2f, 0xc4299e3620, 0x3, 0x3)
	/usr/local/go/src/cmd/compile/internal/gc/subr.go:175 +0x230
cmd/compile/internal/gc.liveIndex(0xc42a386c00, 0xc42a3bfb00, 0x18, 0x20, 0xc400000006)
	/usr/local/go/src/cmd/compile/internal/gc/plive.go:270 +0x207
cmd/compile/internal/gc.(*Liveness).valueEffects(0xc42a23ad20, 0xc429af8140, 0x1, 0x85)
	/usr/local/go/src/cmd/compile/internal/gc/plive.go:189 +0x73
cmd/compile/internal/gc.livenessprologue(0xc42a23ad20)
	/usr/local/go/src/cmd/compile/internal/gc/plive.go:451 +0xd3
cmd/compile/internal/gc.liveness(0xc42a394e10, 0xc42a207900, 0xc42a3da1c0, 0xc42a3da2a0, 0xc4299e3968)
	/usr/local/go/src/cmd/compile/internal/gc/plive.go:1115 +0xce
cmd/compile/internal/gc.genssa(0xc42a207900, 0xc4200ec210)
	/usr/local/go/src/cmd/compile/internal/gc/ssa.go:4260 +0x115
cmd/compile/internal/gc.compile(0xc420031e00)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:358 +0x451
cmd/compile/internal/gc.funccompile(0xc420031e00)
	/usr/local/go/src/cmd/compile/internal/gc/dcl.go:1172 +0xbb
cmd/compile/internal/gc.Main(0xad71d0)
	/usr/local/go/src/cmd/compile/internal/gc/main.go:511 +0x22f4
main.main()
	/usr/local/go/src/cmd/compile/main.go:49 +0x95

!!! [0321 05:14:31] Call tree:
!!! [0321 05:14:32]  1: /root/upstream_k8s/src/k8s.io/kubernetes/hack/lib/golang.sh:713 kube::golang::build_binaries_for_platform(...)
!!! [0321 05:14:32]  2: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0321 05:14:32] Call tree:
!!! [0321 05:14:32]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0321 05:14:32] Call tree:
!!! [0321 05:14:32]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
Makefile:89: recipe for target 'all' failed
make: *** [all] Error 1
root@localhost:~/upstream_k8s/src/k8s.io/kubernetes# go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root/upstream_k8s"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build097721640=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
root@localhost:~/upstream_k8s/src/k8s.io/kubernetes#

@karpovicham
Copy link

karpovicham commented Mar 21, 2017 •
edited

I got similar issue as well:

...
my_application.go:263:6: internal compiler error: lost track of variable in liveness: .autotmp_179 (0xc42101b800, 0xc42101b800)
...

It appears on travis server for go tip version, but looks like everything works fine with go 1.8

thanks

@dgryski
Copy link
Contributor

dgryski commented Mar 21, 2017

Should be easy enough to bisect.

@mvdan mvdan added this to the Go1.9 milestone Mar 21, 2017
@mvdan mvdan changed the title internal compiler error: lost track of variable in liveness: .autotmp_538 while building upstream kubernetes cmd/compile: internal compiler error: lost track of variable in liveness: .autotmp_538 while building upstream kubernetes Mar 21, 2017
@mvdan mvdan changed the title cmd/compile: internal compiler error: lost track of variable in liveness: .autotmp_538 while building upstream kubernetes cmd/compile: internal compiler error: lost track of variable in liveness while building kubernetes Mar 21, 2017
@mvdan
Copy link
Member

mvdan commented Mar 21, 2017

Just hit the same in an unrelated package:

 $ go test
# github.com/mvdan/sh/syntax
./parser_test.go:209:6: internal compiler error: lost track of variable in liveness: .autotmp_2723 (0xc422016700, 0xc422016700)

goroutine 1 [running]:
runtime/debug.Stack(0x0, 0x0, 0x0)
        /home/mvdan/tip/src/runtime/debug/stack.go:24 +0x79
cmd/compile/internal/gc.Fatalf(0xad1d63, 0x2f, 0xc421375620, 0x3, 0x3)
        /home/mvdan/tip/src/cmd/compile/internal/gc/subr.go:175 +0x230
cmd/compile/internal/gc.liveIndex(0xc422016700, 0xc421db7e00, 0x10, 0x10, 0xc4ffffffff)
        /home/mvdan/tip/src/cmd/compile/internal/gc/plive.go:270 +0x207
cmd/compile/internal/gc.(*Liveness).valueEffects(0xc4200e5b20, 0xc4213b4560, 0xffffffff, 0x0)
        /home/mvdan/tip/src/cmd/compile/internal/gc/plive.go:189 +0x73
cmd/compile/internal/gc.livenessprologue(0xc4200e5b20)
        /home/mvdan/tip/src/cmd/compile/internal/gc/plive.go:451 +0xd3
cmd/compile/internal/gc.liveness(0xc4205cda10, 0xc420ed1cc0, 0xc420788310, 0xc4207883f0, 0xc421375968)
        /home/mvdan/tip/src/cmd/compile/internal/gc/plive.go:1115 +0xce
cmd/compile/internal/gc.genssa(0xc420ed1cc0, 0xc4200e6210)
        /home/mvdan/tip/src/cmd/compile/internal/gc/ssa.go:4260 +0x115
cmd/compile/internal/gc.compile(0xc420a5be00)
        /home/mvdan/tip/src/cmd/compile/internal/gc/pgen.go:358 +0x451
cmd/compile/internal/gc.funccompile(0xc420a5be00)
        /home/mvdan/tip/src/cmd/compile/internal/gc/dcl.go:1172 +0xbb
cmd/compile/internal/gc.Main(0xad71a0)
        /home/mvdan/tip/src/cmd/compile/internal/gc/main.go:511 +0x22f4
main.main()
        /home/mvdan/tip/src/cmd/compile/main.go:49 +0x95

FAIL    github.com/mvdan/sh/syntax [build failed]

Will try to do some bisecting.

@mdempsky
Copy link
Member

Probably 325904f, since that caused the noopt builders to fail similarly.

@mdempsky mdempsky self-assigned this Mar 21, 2017
@mvdan
Copy link
Member

mvdan commented Mar 21, 2017

Thanks - will be able to confirm in a few minutes...

@mvdan
Copy link
Member

mvdan commented Mar 21, 2017

@mdempsky confirming that commit is indeed to blame.

@mdempsky
Copy link
Member

@mvdan I can't repro your failure with linux/amd64. What build settings are you use? (go env)

@mdempsky
Copy link
Member

I can repro with go build -a -gcflags='-N -l' std though.

@mvdan
Copy link
Member

mvdan commented Mar 21, 2017 

 $ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/mvdan/go/land:/home/mvdan/go"
GORACE=""
GOROOT="/home/mvdan/tip"
GOTOOLDIR="/home/mvdan/tip/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build800668376=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
 $ go version
go version devel +d972dc2de9 Tue Mar 21 06:36:56 2017 +0000 linux/amd64

Reproducible with go test github.com/mvdan/sh/syntax.

@mdempsky
Copy link
Member

Found the issue. Uploading a CL.

@gopherbot
Copy link

CL https://golang.org/cl/38400 mentions this issue.

@gopherbot
Copy link

CL https://golang.org/cl/38401 mentions this issue.

gopherbot pushed a commit that referenced this issue Mar 21, 2017
@mdempsky
cmd/compile: add regress test for issue 19632
051cbf3 
Updates #19632.

Change-Id: I1411dd997c8c6a789d17d0dcc0bfbd2281447b16
Reviewed-on: https://go-review.googlesource.com/38401
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@mdempsky
Copy link
Member

@mkumatag At your leisure, a confirmation that this fixed the Kubernetes build would be appreciated.

@mkumatag
Copy link
Author

@mdempsky with latest upstream golang I'm able to build kubernetes properly. Thanks for the fix.

@golang golang locked and limited conversation to collaborators Mar 22, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
Go1.9
Development

No branches or pull requests
6 participants
@mdempsky @dgryski @mvdan @gopherbot @mkumatag @karpovicham