Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: Fatal Error in heapBitsForObject during GC on Windows #24768

Closed
billziss-gh opened this issue Apr 9, 2018 · 5 comments
Closed

runtime: Fatal Error in heapBitsForObject during GC on Windows #24768

billziss-gh opened this issue Apr 9, 2018 · 5 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows
Milestone
Go1.11

Comments

@billziss-gh
Copy link
Contributor

What version of Go are you using (go version)?

go version go1.10 windows/amd64

Does this issue reproduce with the latest release?

???

What operating system and processor architecture are you using (go env)?

windows/amd64

What did you do?

I have a program that uses cgo. The program implements a user mode file system and is available here. The program runs on Windows, macOS and Linux.

I am experiencing this problem during testing of the Readdir operation on Windows. The problem is rather easy to reproduce with the proper setup (happens 1 every 3-4 test runs). Unfortunately it is not trivial to recreate this setup. (I can provide instructions if desired.)

What did you expect to see?

I expected the program to continue running without problems.

What did you see instead?

Every few test runs the program crashes within heapBitsForObject. I provide stack traces from two different crashes:

fatal error: unexpected signal during runtime execution
fatal error: unexpected signal during runtime execution
[signal 0xc0000005 code=0x0 addr=0xffffffffffffffff pc=0x41382e]

runtime stack:
runtime.throw(0x7164f3, 0x2a)
        C:/Program Files/Go/src/runtime/panic.go:619 +0x88
runtime.sigpanic()
        C:/Program Files/Go/src/runtime/signal_windows.go:161 +0x170
runtime.(*mspan).base(...)
        C:/Program Files/Go/src/runtime/mheap.go:310
runtime.heapBitsForObject(0xc042018ac0, 0x8a2920, 0x61e0, 0xc041ff06f2, 0xc000000002, 0x29b7240, 0xc04201b270, 0x9)
        C:/Program Files/Go/src/runtime/mbitmap.go:393 +0xae
runtime.scanblock(0x8a2920, 0x9410, 0x293d800, 0xc04201b270)
        C:/Program Files/Go/src/runtime/mgcmark.go:1103 +0xde
runtime.markrootBlock(0x8a2920, 0x9410, 0x293d800, 0xc04201b270, 0x0)
        C:/Program Files/Go/src/runtime/mgcmark.go:294 +0x83
runtime.markroot(0xc04201b270, 0xc000000002)
        C:/Program Files/Go/src/runtime/mgcmark.go:195 +0xee
runtime.gcDrain(0xc04201b270, 0x15)
        C:/Program Files/Go/src/runtime/mgcmark.go:934 +0x11f
runtime.gcBgMarkWorker.func2()
        C:/Program Files/Go/src/runtime/mgc.go:1863 +0x1a8
runtime.systemstack(0x0)
        C:/Program Files/Go/src/runtime/asm_amd64.s:409 +0x7e
runtime.mstart()
        C:/Program Files/Go/src/runtime/proc.go:1170

unexpected fault address 0xffffffffffffffff
fatal error: fault
[signal 0xc0000005 code=0x0 addr=0xffffffffffffffff pc=0x41382e]

goroutine 17 [running, locked to thread]:
runtime.throw(0x7097ae, 0x5)
        C:/Program Files/Go/src/runtime/panic.go:619 +0x88 fp=0xc042460878 sp=0xc042460858 pc=0x42c9c8

runtime.sigpanic()
        C:/Program Files/Go/src/runtime/signal_windows.go:170 +0x13a fp=0xc0424608a8 sp=0xc042460878 pc=0x43f5ea
runtime.(*mspan).base(...)
        C:/Program Files/Go/src/runtime/mheap.go:310
runtime.heapBitsForObject(0xc042010d80, 0x0, 0x0, 0x3, 0x1, 0x4182d8, 0x8cdee0, 0xc04211e150)
        C:/Program Files/Go/src/runtime/mbitmap.go:393 +0xae fp=0xc042460900 sp=0xc0424608a8 pc=0x41382e
runtime.gcMarkTinyAllocs()
        C:/Program Files/Go/src/runtime/mgcmark.go:1389 +0x86 fp=0xc042460980 sp=0xc042460900 pc=0x41faf6
runtime.gcStart(0x0, 0x1, 0x0, 0x0)
        C:/Program Files/Go/src/runtime/mgc.go:1357 +0x314 fp=0xc0424609b0 sp=0xc042460980 pc=0x4196c4

runtime.mallocgc(0x40, 0x0, 0x0, 0xc0423a2200)
        C:/Program Files/Go/src/runtime/malloc.go:803 +0x448 fp=0xc042460a50 sp=0xc0424609b0 pc=0x411a18
runtime.growslice(0x6979c0, 0xc0423a2200, 0x20, 0x20, 0x21, 0xc0423a2200, 0x10, 0x20)
        C:/Program Files/Go/src/runtime/slice.go:172 +0x224 fp=0xc042460ab8 sp=0xc042460a50 pc=0x43fe74
strings.(*Builder).WriteRune(0xc042460b60, 0xc00000006c, 0x1, 0x0, 0x0)
        C:/Program Files/Go/src/strings/builder.go:101 +0xbb fp=0xc042460b30 sp=0xc042460ab8 pc=0x4d4d5b
github.com/billziss-gh/objfs/vendor/github.com/billziss-gh/objfs.pkg/objio/onedrive.decodeName(0xc04239b3b0, 0x24, 0x86b5, 0x68eb80)
        C:/Users/billziss/Projects/go/src/github.com/billziss-gh/objfs/vendor/github.com/billziss-gh/objfs.pkg/objio/onedrive/onedrive.go:78 +0x4b fp=0xc042460b90 sp=0xc042460b30 pc=0x658ceb
github.com/billziss-gh/objfs/vendor/github.com/billziss-gh/objfs.pkg/objio/onedrive.(*codedName).UnmarshalJSON(0xc0423a0770, 0xc04242774b, 0x26, 0x86b5, 0x0, 0x4513ae8)
        C:/Users/billziss/Projects/go/src/github.com/billziss-gh/objfs/vendor/github.com/billziss-gh/objfs.pkg/objio/onedrive/onedrive.go:99 +0xac fp=0xc042460be0 sp=0xc042460b90 pc=0x658f0c
encoding/json.(*decodeState).literalStore(0xc04211e3e8, 0xc04242774b, 0x26, 0x86b5, 0x6a6e60, 0xc0423a0770, 0x198, 0x67f800)
        C:/Program Files/Go/src/encoding/json/decode.go:853 +0x2512 fp=0xc042460f50 sp=0xc042460be0 pc=0x641822
encoding/json.(*decodeState).literal(0xc04211e3e8, 0x6a6e60, 0xc0423a0770, 0x198)
        C:/Program Files/Go/src/encoding/json/decode.go:820 +0xdc fp=0xc042460fa8 sp=0xc042460f50 pc=0x63f06c
encoding/json.(*decodeState).value(0xc04211e3e8, 0x6a6e60, 0xc0423a0770, 0x198)
        C:/Program Files/Go/src/encoding/json/decode.go:411 +0x314 fp=0xc042461028 sp=0xc042460fa8 pc=0x63c044
encoding/json.(*decodeState).object(0xc04211e3e8, 0x6cca00, 0xc0421a8c18, 0x196)
        C:/Program Files/Go/src/encoding/json/decode.go:754 +0x1334 fp=0xc0424612a8 sp=0xc042461028 pc=0x63e5b4
encoding/json.(*decodeState).value(0xc04211e3e8, 0x6cca00, 0xc0421a8c18, 0x196)
        C:/Program Files/Go/src/encoding/json/decode.go:408 +0x2da fp=0xc042461328 sp=0xc0424612a8 pc=0x63c00a
encoding/json.(*decodeState).array(0xc04211e3e8, 0x691100, 0xc042256720, 0x197)
        C:/Program Files/Go/src/encoding/json/decode.go:561 +0x1d7 fp=0xc042461428 sp=0xc042461328 pc=0x63c8f7
encoding/json.(*decodeState).value(0xc04211e3e8, 0x691100, 0xc042256720, 0x197)
        C:/Program Files/Go/src/encoding/json/decode.go:405 +0x26d fp=0xc0424614a8 sp=0xc042461428 pc=0x63bf9d
encoding/json.(*decodeState).object(0xc04211e3e8, 0x68f140, 0xc042256720, 0x16)
        C:/Program Files/Go/src/encoding/json/decode.go:754 +0x1334 fp=0xc042461728 sp=0xc0424614a8 pc=0x63e5b4
encoding/json.(*decodeState).value(0xc04211e3e8, 0x68f140, 0xc042256720, 0x16)
        C:/Program Files/Go/src/encoding/json/decode.go:408 +0x2da fp=0xc0424617a8 sp=0xc042461728 pc=0x63c00a
encoding/json.(*decodeState).unmarshal(0xc04211e3e8, 0x68f140, 0xc042256720, 0x0, 0x0)
        C:/Program Files/Go/src/encoding/json/decode.go:189 +0x1ee fp=0xc042461818 sp=0xc0424617a8 pc=0x63b5be
encoding/json.(*Decoder).Decode(0xc04211e3c0, 0x68f140, 0xc042256720, 0x4513a20, 0xc04225a180)
        C:/Program Files/Go/src/encoding/json/stream.go:73 +0x1ae fp=0xc042461858 sp=0xc042461818 pc=0x64d5fe
github.com/billziss-gh/objfs/vendor/github.com/billziss-gh/objfs.pkg/objio/onedrive.(*onedrive).List.func1(0xc04222a5a0, 0x7439d8, 0xc04225a180)
        C:/Users/billziss/Projects/go/src/github.com/billziss-gh/objfs/vendor/github.com/billziss-gh/objfs.pkg/objio/onedrive/onedrive.go:640 +0xef fp=0xc0424618c8 sp=0xc042461858 pc=0x65e8af
github.com/billziss-gh/objfs/vendor/github.com/billziss-gh/objfs.pkg/objio/onedrive.(*onedrive).sendrecv(0xc0420f8d20, 0xc042461b38, 0xc042461ad8, 0x0, 0x0)
        C:/Users/billziss/Projects/go/src/github.com/billziss-gh/objfs/vendor/github.com/billziss-gh/objfs.pkg/objio/onedrive/onedrive.go:576 +0x31d fp=0xc042461a70 sp=0xc0424618c8 pc=0x65ad8d
github.com/billziss-gh/objfs/vendor/github.com/billziss-gh/objfs.pkg/objio/onedrive.(*onedrive).List(0xc0420f8d20, 0xc042056858, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        C:/Users/billziss/Projects/go/src/github.com/billziss-gh/objfs/vendor/github.com/billziss-gh/objfs.pkg/objio/onedrive/onedrive.go:635 +0x21a fp=0xc042461b80 sp=0xc042461a70 pc=0x65baca
github.com/billziss-gh/objfs/cache.(*Cache).readdirNode(0xc042128000, 0xc042394dc0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        C:/Users/billziss/Projects/go/src/github.com/billziss-gh/objfs/cache/cache.go:762 +0x171 fp=0xc042461ca8 sp=0xc042461b80 pc=0x52d451
github.com/billziss-gh/objfs/cache.(*Cache).Readdir(0xc042128000, 0x2e56, 0x0, 0x1, 0x1, 0x18, 0x6ca100, 0x1)
        C:/Users/billziss/Projects/go/src/github.com/billziss-gh/objfs/cache/cache.go:372 +0xa1 fp=0xc042461cf8 sp=0xc042461ca8 pc=0x52b5d1
github.com/billziss-gh/objfs/fs/objfs.(*objfs).Readdir(0xc0420765d0, 0xc0421a5c32, 0x1, 0xc042250300, 0x0, 0x2e56, 0xc042042058)
        C:/Users/billziss/Projects/go/src/github.com/billziss-gh/objfs/fs/objfs/objfs.go:272 +0x1dd fp=0xc042461de0 sp=0xc042461cf8 pc=0x63a56d
github.com/billziss-gh/objfs/vendor/github.com/billziss-gh/cgofuse/fuse.hostReaddir(0x103b00, 0x513fe20, 0x7ff8bace5980, 0x0, 0x513fe40, 0x0)
        C:/Users/billziss/Projects/go/src/github.com/billziss-gh/objfs/vendor/github.com/billziss-gh/cgofuse/fuse/host.go:931 +0x13f fp=0xc042461e48 sp=0xc042461de0 pc=0x53bd4f
github.com/billziss-gh/objfs/vendor/github.com/billziss-gh/cgofuse/fuse._cgoexpwrap_02f8cc1aff87_hostReaddir(0x103b00, 0x513fe20, 0x7ff8bace5980, 0x0, 0x513fe40, 0x7ff8ca82a686)
        _cgo_gotypes.go:776 +0x5a fp=0xc042461e88 sp=0xc042461e48 pc=0x538d8a
runtime.call64(0x0, 0x513fc20, 0x513fd90, 0x30)
        C:/Program Files/Go/src/runtime/asm_amd64.s:574 +0x42 fp=0xc042461ed8 sp=0xc042461e88 pc=0x454602
runtime.cgocallbackg1(0x0)
        C:/Program Files/Go/src/runtime/cgocall.go:316 +0x1aa fp=0xc042461f58 sp=0xc042461ed8 pc=0x402d6a
runtime.cgocallbackg(0x0)
        C:/Program Files/Go/src/runtime/cgocall.go:194 +0xef fp=0xc042461fc0 sp=0xc042461f58 pc=0x402b1f
runtime.cgocallback_gofunc(0x0, 0x0, 0x0, 0x0)
        C:/Program Files/Go/src/runtime/asm_amd64.s:826 +0xb2 fp=0xc042461fe0 sp=0xc042461fc0 pc=0x455bd2
runtime.goexit()
        C:/Program Files/Go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc042461fe8 sp=0xc042461fe0 pc=0x456de1
@mvdan
Copy link
Member

mvdan commented Apr 9, 2018

Have you ran your tests or code with -race, to verify that there isn't any memory corruption? Sometimes, these weird internal panics are because of race conditions.

I assume that the code runs fine on some non-Windows platforms?

@billziss-gh
Copy link
Contributor Author

Have you ran your tests or code with -race, to verify that there isn't any memory corruption? Sometimes, these weird internal panics are because of race conditions.

Thanks for the pointer. I will rerun the tests with -race and report back.

I assume that the code runs fine on some non-Windows platforms?

I have only observed this problem on Windows. However the test in question is a Windows-only test at this point (it is from a Windows-only file system test suite). It is unclear if the same set of conditions could result in a similar problem on other platforms.

@billziss-gh
Copy link
Contributor Author

I built the program with -race. (The build command line was go build -race -ldflags "-s -w" as I usually build it with go build -ldflags "-s -w".)

After a few runs the program appears to have crashed completely (i.e. no panic or printout). The returned ERRORLEVEL is -1073741819==0xc0000005, i.e. STATUS_ACCESS_VIOLATION. It appears that the program has crashed as before, but this time it was unable to print anything on the console.

@odeke-em odeke-em changed the title Fatal Error in heapBitsForObject during GC on Windows runtime: Fatal Error in heapBitsForObject during GC on Windows Apr 9, 2018
@odeke-em odeke-em added OS-Windows NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Apr 9, 2018
@odeke-em
Copy link
Member

odeke-em commented Apr 9, 2018

/cc @aclements @randall77 @cherrymui @alexbrainman

@ianlancetaylor ianlancetaylor added this to the Go1.11 milestone Apr 18, 2018
@billziss-gh
Copy link
Contributor Author

I am closing this as I believe this is related to a problem with one of the cgo packages used (which has now been fixed). I will reopen if it reappears.

@golang golang locked and limited conversation to collaborators Jun 6, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows
Projects
None yet
Milestone
Go1.11
Development

No branches or pull requests
5 participants
@ianlancetaylor @mvdan @odeke-em @gopherbot @billziss-gh