x/sys/windows/svc: the service terminated unexpectedly (id 7034) #40074

AlexSiLive · 2020-07-06T15:59:45Z

What version of Go are you using (`go version`)?

$ go version
go version go1.14.4 windows/amd64

Does this issue reproduce with the latest release?

Yes.
Go version: go1.14.4
x/sys version: v0.0.0-20200625212154-ddb9806d33ae

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\alex\AppData\Local\go-build
set GOENV=C:\Users\alex\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GONOPROXY=
set GONOSUMDB=*
set GOOS=windows
set GOPATH=C:\Users\alex\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=c:\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=E:\WorkGo\src\golang.org\x\sys\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\A6473~1.POG\AppData\Local\Temp\go-build406331230=/tmp/go-build -gno-record-gcc-switches
GOROOT/bin/go version: go version go1.14.4 windows/amd64
GOROOT/bin/go tool compile -V: compile version go1.14.4
gdb --version: GNU gdb (GDB) 7.9.1

What did you do?

Problem:
Building a windows service using x/sys/windows/svc on go1.14+ will cause a service to terminate unexpectedly on system shutdown (SERVICE_CONTROL_SHUTDOWN command).

How to reproduce:
Build a default example for x/sys/windows/svc on go1.13 and go1.14. The latter will be terminated unexpectedly and produce a corresponding event in the event log (7034), while the former one will stop just fine.

Remarks:

A service can be stopped and started via service manager without an issue, the problem occurs on system shutdown or reboot.
Reverting x/sys/windows/svc to initial commits does nothing, i.e. the problem, probably, somewhere in-between go1.13 and go1.14.
go_svc_7034.zip

The text was updated successfully, but these errors were encountered:

networkimprov · 2020-07-07T02:18:01Z

cc @alexbrainman @bcmills

@gopherbot add OS-Windows

alexbrainman · 2020-07-12T09:14:15Z

@AlexSiLive

Thank you for reporting the issue and the steps to reproduce the problem. Yes, I can see shutdown event sent for service compiled with go1.13.12, but not with current tip.

We need to try to use git bisect command to find commit that broken the code (it is, probably, https://go-review.googlesource.com/c/go/+/187739/ ). Unfortunately I don't have time to run bisect now, but I will do it when I have time. Unless someone beats me to it.

Alex

AlexSiLive · 2020-07-13T09:38:29Z

@alexbrainman

Just a quick update.
Indeed, commenting out this clause resolves the issue:
9e05d63#diff-4fcd95ba1348be2a6286fdfdd9e8393dR897

AlexSiLive · 2020-07-13T14:26:28Z

I was able to make a dirty fix for this without modifying runtime by adding signal.Notify handling to a wrapping package.
Something similar can be done to x/sys/windows/svc itself:
`

...
func Run(name string, handler Handler) error {
	runtime.LockOSThread()

	tid := windows.GetCurrentThreadId()

	s, err := newService(name, handler)
	if err != nil {
		return err
	}

	signals := make(chan os.Signal, 1)
	signal.Notify(signals)

	go func() {
		for sig := range signals {
			if sig == windows.SIGTERM {
				signal.Stop(signals)
				s.c <- ctlEvent{cmd: Shutdown} // todo: some checks here, i.e. for SERVICE_ACCEPT_SHUTDOWN

				return
			}
		}
	}()
...

`

Please note, that following didn't work for me (unless I'm doing something wrong):
`

signals := make(chan os.Signal, 1)
signal.Notify(signals, windows.SIGTERM) // <- specifying SIGTERM doesn't seem to work.

go func() {
	<- signals // <- this never triggers if SIGTERM is specified (mock program with SIGINT works though).
	signal.Stop(signals)
	s.c <- ctlEvent{cmd: Shutdown} // todo: some checks here, i.e. for SERVICE_ACCEPT_SHUTDOWN
}()

`

JohanKnutzen · 2020-07-13T21:44:52Z

@AlekSi

I believe that our issues are related. I've got another workaround for this that overrides handling in go of that particular event in Windows:
40167

gopherbot · 2020-07-13T23:46:15Z

Change https://golang.org/cl/242378 mentions this issue: windows/svc: restore default console shutdown notifier when starting service

alexbrainman · 2020-07-14T06:25:00Z

Indeed, commenting out this clause resolves the issue:

Cool. Thanks for checking that.

We just need to come up with a solution here.

Alex

AlekSi · 2020-07-14T06:34:35Z

@AlekSi

I'm not @AlexSiLive :)

gopherbot · 2020-07-14T07:46:12Z

Change https://golang.org/cl/242280 mentions this issue: runtime: detect services in signal handler

alexbrainman · 2020-07-17T08:18:18Z

@aclements

This issue is with golang.org/x/sys/windows/svc package.

The problem was introduced in

https://go-review.googlesource.com/c/go/+/187739/

After the CL 187739, every Windows service would not receive "shutdown" event. When computer is shutdown, Go service just exits, instead of handling "shutdown" event.

@zx2c4 sent a fix for this issue

https://golang.org/cl/242280

And the fix is in runtime package.

Both Jason and I think the fix should be back-ported to Go 1.14 (Go 1.13 is good). Should we submit CL 242280 now? Should we wait for tree to be opened first? Do you agree CL 242280 should be back-ported?

There is some risk involved. Here, in particular, we don't even have a test for the fix - you need to reboot computer to test our fix.

What do you think?

Thank you.

Alex

networkimprov · 2020-07-17T09:25:32Z

@aclements, most of the discussion about this is in #40167

gopherbot · 2020-07-20T08:39:48Z

Change https://golang.org/cl/243597 mentions this issue: runtime: do not explicitly exit on ctrl handler

gopherbot · 2020-07-27T08:45:25Z

Change https://golang.org/cl/244957 mentions this issue: [release-branch.go1.14] runtime: detect services in signal handler

gopherbot · 2020-07-27T08:51:12Z

Change https://golang.org/cl/244958 mentions this issue: [release-branch.go1.14] runtime: detect services in signal handler

gopherbot · 2020-07-27T08:54:20Z

Change https://golang.org/cl/244959 mentions this issue: [release-branch.go1.15] runtime: detect services in signal handler

zx2c4 · 2020-07-27T08:55:38Z

Sorry for the churn. The two backports are https://golang.org/cl/244958 and https://golang.org/cl/244959.

The service handler needs to handle CTRL+C-like events -- including those sent by the service manager itself -- using the default Windows implementation if no signal handler from Go is already listening to those events. Ordinarily, the signal handler would call exit(2), but we actually need to allow this to be passed onward to the service handler. So, we detect if we're in a service and skip calling exit(2) in that case, just like we do for shared libraries. Updates #40167. Updates #40074. Fixes #40412. Change-Id: Ia77871737a80e1e94f85b02d26af1fd2f646af96 Reviewed-on: https://go-review.googlesource.com/c/go/+/244959 Run-TryBot: Jason A. Donenfeld <Jason@zx2c4.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>

The service handler needs to handle CTRL+C-like events -- including those sent by the service manager itself -- using the default Windows implementation if no signal handler from Go is already listening to those events. Ordinarily, the signal handler would call exit(2), but we actually need to allow this to be passed onward to the service handler. So, we detect if we're in a service and skip calling exit(2) in that case, just like we do for shared libraries. Updates #40167. Updates #40074. Fixes #40411. Change-Id: Ia77871737a80e1e94f85b02d26af1fd2f646af96 Reviewed-on: https://go-review.googlesource.com/c/go/+/244958 Run-TryBot: Jason A. Donenfeld <Jason@zx2c4.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alex Brainman <alex.brainman@gmail.com>

gopherbot added this to the Unreleased milestone Jul 6, 2020

gopherbot added the OS-Windows label Jul 7, 2020

dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jul 11, 2020

This was referenced Jul 12, 2020

runtime: Windows service lifecycle events behave incorrectly when called within a golang environment #40167

Closed

x/sys/windows/svc/example: does not start #40160

Closed

gopherbot closed this as completed in c0dded0 Aug 15, 2020

networkimprov mentioned this issue Oct 11, 2020

runtime: handling of CTRL_CLOSE_EVENT seems broken #41884

Closed

golang locked and limited conversation to collaborators Aug 15, 2021

gopherbot added the FrozenDueToAge label Aug 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/sys/windows/svc: the service terminated unexpectedly (id 7034) #40074

x/sys/windows/svc: the service terminated unexpectedly (id 7034) #40074

AlexSiLive commented Jul 6, 2020

networkimprov commented Jul 7, 2020

alexbrainman commented Jul 12, 2020

AlexSiLive commented Jul 13, 2020 •

edited

AlexSiLive commented Jul 13, 2020 •

edited

JohanKnutzen commented Jul 13, 2020

gopherbot commented Jul 13, 2020

alexbrainman commented Jul 14, 2020

AlekSi commented Jul 14, 2020

gopherbot commented Jul 14, 2020

alexbrainman commented Jul 17, 2020

networkimprov commented Jul 17, 2020

gopherbot commented Jul 20, 2020

gopherbot commented Jul 27, 2020

gopherbot commented Jul 27, 2020

gopherbot commented Jul 27, 2020

zx2c4 commented Jul 27, 2020

x/sys/windows/svc: the service terminated unexpectedly (id 7034) #40074

x/sys/windows/svc: the service terminated unexpectedly (id 7034) #40074

Comments

AlexSiLive commented Jul 6, 2020

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

networkimprov commented Jul 7, 2020

alexbrainman commented Jul 12, 2020

AlexSiLive commented Jul 13, 2020 • edited

AlexSiLive commented Jul 13, 2020 • edited

JohanKnutzen commented Jul 13, 2020

gopherbot commented Jul 13, 2020

alexbrainman commented Jul 14, 2020

AlekSi commented Jul 14, 2020

gopherbot commented Jul 14, 2020

alexbrainman commented Jul 17, 2020

networkimprov commented Jul 17, 2020

gopherbot commented Jul 20, 2020

gopherbot commented Jul 27, 2020

gopherbot commented Jul 27, 2020

gopherbot commented Jul 27, 2020

zx2c4 commented Jul 27, 2020

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

AlexSiLive commented Jul 13, 2020 •

edited

AlexSiLive commented Jul 13, 2020 •

edited