Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: name offset out of range in go1.7.3 #20047

Closed
ianrose14 opened this issue Apr 20, 2017 · 6 comments
Closed

runtime: name offset out of range in go1.7.3 #20047

ianrose14 opened this issue Apr 20, 2017 · 6 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Unplanned

Comments

@ianrose14
Copy link

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.7.3 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/src/mn/projects/fullstory/go"
GORACE=""
GOROOT="/tools/go"
GOTOOLDIR="/tools/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build323058043=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

What did you do?

Unfortunately this occurred during one of our production services and does not seem very common so I have no idea how to repro this :(

What did you expect to see?

Process does not crash.

What did you see instead?

runtime: nameOff 0x50545448 base 0xc43b290117 not in ranges:
        types 0xafa000 etypes 0xdf810c
fatal error: runtime: name offset base pointer out of range

goroutine 50 [running]:
runtime.throw(0xd51dda, 0x2e)
        /tools/go/src/runtime/panic.go:566 +0x95 fp=0xc45f457818 sp=0xc45f4577f8
runtime.resolveNameOff(0xc43b290117, 0x50545448, 0x3)
        /tools/go/src/runtime/type.go:193 +0x221 fp=0xc45f457880 sp=0xc45f457818
runtime.(*_type).nameOff(0xc43b290117, 0x50545448, 0x3)
        /tools/go/src/runtime/type.go:199 +0x33 fp=0xc45f4578a8 sp=0xc45f457880
runtime.additab(0x7f959e1da198, 0x101)
        /tools/go/src/runtime/iface.go:111 +0x1ce fp=0xc45f457998 sp=0xc45f4578a8
runtime.getitab(0xc15960, 0xc43b290117, 0x1, 0xc442d1c3c0)
        /tools/go/src/runtime/iface.go:79 +0x1af fp=0xc45f457a30 sp=0xc45f457998
runtime.assertE2I2(0xc15960, 0xc43b290117, 0x5, 0xc45f457af0, 0xc45b29f5c0)
        /tools/go/src/runtime/iface.go:383 +0x7a fp=0xc45f457a60 sp=0xc45f457a30
fmt.(*pp).handleMethods(0xc4570eb080, 0x73, 0x6485872000)
        /tools/go/src/fmt/print.go:554 +0xb4 fp=0xc45f457b10 sp=0xc45f457a60
fmt.(*pp).printArg(0xc4570eb080, 0xc43b290117, 0x5, 0x73)
        /tools/go/src/fmt/print.go:665 +0x17b fp=0xc45f457c08 sp=0xc45f457b10
fmt.(*pp).doPrintf(0xc4570eb080, 0xd56de5, 0x3a, 0xc45b29f590, 0x4, 0x4)
        /tools/go/src/fmt/print.go:985 +0x123d fp=0xc45f457cf0 sp=0xc45f457c08
fmt.Sprintf(0xd56de5, 0x3a, 0xc45b29f590, 0x4, 0x4, 0xc433f8db90, 0xc456ff0cb0)
        /tools/go/src/fmt/print.go:196 +0x6a fp=0xc45f457d48 sp=0xc45f457cf0
fs/fslog.(*LogEntry).Message(0xc4d0954000, 0xc420016720, 0xc45f457ed0)
        /src/mn/projects/fullstory/go/src/fs/fslog/fslog.go:125 +0x68 fp=0xc45f457de8 sp=0xc45f457d48
(more stack trace here, but probably not helpful since it is all in our code)
@bradfitz
Copy link
Contributor

Usual questions:

  • have you tried Go 1.7.5?
  • have you tried Go 1.8.1?
  • have you run your program under the race detector?

Also, what are your concrete types there? Are you generating any types at runtime?

/cc @ianlancetaylor @crawshaw @aclements

@bradfitz bradfitz added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Apr 20, 2017
@ianlancetaylor
Copy link
Contributor

The nameOff value of 0x50545448 happens to be the ASCII characters HTTP. I suspect some sort of memory corruption.

@ianrose14
Copy link
Author

ianrose14 commented Apr 20, 2017 •
edited

No to the other Go versions, although I don't see any reason why we shouldn't be able to upgrade to at least 1.7.5, if not 1.8.1, in the near future.

Here is the entire stack trace:

goroutine 50 [running]:
runtime.throw(0xd51dda, 0x2e)
	/tools/go/src/runtime/panic.go:566 +0x95 fp=0xc45f457818 sp=0xc45f4577f8
runtime.resolveNameOff(0xc43b290117, 0x50545448, 0x3)
	/tools/go/src/runtime/type.go:193 +0x221 fp=0xc45f457880 sp=0xc45f457818
runtime.(*_type).nameOff(0xc43b290117, 0x50545448, 0x3)
	/tools/go/src/runtime/type.go:199 +0x33 fp=0xc45f4578a8 sp=0xc45f457880
runtime.additab(0x7f959e1da198, 0x101)
	/tools/go/src/runtime/iface.go:111 +0x1ce fp=0xc45f457998 sp=0xc45f4578a8
runtime.getitab(0xc15960, 0xc43b290117, 0x1, 0xc442d1c3c0)
	/tools/go/src/runtime/iface.go:79 +0x1af fp=0xc45f457a30 sp=0xc45f457998
runtime.assertE2I2(0xc15960, 0xc43b290117, 0x5, 0xc45f457af0, 0xc45b29f5c0)
	/tools/go/src/runtime/iface.go:383 +0x7a fp=0xc45f457a60 sp=0xc45f457a30
fmt.(*pp).handleMethods(0xc4570eb080, 0x73, 0x6485872000)
	/tools/go/src/fmt/print.go:554 +0xb4 fp=0xc45f457b10 sp=0xc45f457a60
fmt.(*pp).printArg(0xc4570eb080, 0xc43b290117, 0x5, 0x73)
	/tools/go/src/fmt/print.go:665 +0x17b fp=0xc45f457c08 sp=0xc45f457b10
fmt.(*pp).doPrintf(0xc4570eb080, 0xd56de5, 0x3a, 0xc45b29f590, 0x4, 0x4)
	/tools/go/src/fmt/print.go:985 +0x123d fp=0xc45f457cf0 sp=0xc45f457c08
fmt.Sprintf(0xd56de5, 0x3a, 0xc45b29f590, 0x4, 0x4, 0xc433f8db90, 0xc456ff0cb0)
	/tools/go/src/fmt/print.go:196 +0x6a fp=0xc45f457d48 sp=0xc45f457cf0
fs/fslog.(*LogEntry).Message(0xc4d0954000, 0xc420016720, 0xc45f457ed0)
	/src/mn/projects/fullstory/go/src/fs/fslog/fslog.go:125 +0x68 fp=0xc45f457de8 sp=0xc45f457d48
fs/fslog.LogpipeSenderLoop(0xc42000f9e0, 0xa, 0xd92260, 0x1144720, 0xc420698e80)
	/src/mn/projects/fullstory/go/src/fs/fslog/logpipe.go:132 +0x4cd fp=0xc45f457f78 sp=0xc45f457de8
runtime.goexit()
	/tools/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc45f457f80 sp=0xc45f457f78
created by fs/skeleton.Startup
	/src/mn/projects/fullstory/go/src/fs/skeleton/skeleton.go:406 +0x765

The source code of fs/fslog.(*LogEntry).Message is

type LogEntry struct {
	Time      time.Time
	Level     Level
	Filename  string
	Lineno    int
	OrgId     string // used by logpipe
	RequestId string // used by logpipe
	Format    string
	Args      []interface{}
	refCount  int32
}

[124] func (e *LogEntry) Message() string {
[125]	base := fmt.Sprintf(e.Format, e.Args...)
	if len(e.Args) > 0 {
		// special handling: if the last arg is (or wraps) a PanicError then we also want to log the stack trace
		if err, ok := e.Args[len(e.Args)-1].(error); ok {
			err = stderrors.Root(err)
			if p, ok := err.(*stderrors.PanicError); ok {
				return fmt.Sprintf("%s\n%s", base, p.Trace)
			}
		}
	}

	return base
}

Unfortunately, this is generic log handler code, so there are many possible values for e.Format and e.Args (we use a sprintf-like log function to create these LogEntry instances).

We haven't run the actual binary with race detector on, but we do have some unit tests that call LogEntry.Message and our CI runs those with -race. Of course the unit tests only test certain input values for Format and Args so they are only covering a tiny slice of the possible state space.

@aclements
Copy link
Member

We haven't run the actual binary with race detector on, but we do have some unit tests that call LogEntry.Message and our CI runs those with -race. Of course the unit tests only test certain input values for Format and Args so they are only covering a tiny slice of the possible state space.

Unfortunately, the memory corruption could have happened anywhere (and arbitrarily long ago). LogEntry.Message just happened to trip over it. It's not likely it's the culprit.

@ianlancetaylor
Copy link
Contributor

Is this still a problem?

@ianlancetaylor ianlancetaylor added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Apr 13, 2018
@ianlancetaylor ianlancetaylor added this to the Unplanned milestone Apr 13, 2018
@ALTree
Copy link
Member

ALTree commented Sep 26, 2019

No news for over a year, affected go1.7 which is now unsupported, likely a memory corruption issue, happened only once in production, no way to reproduce the issue... I think we can close here.

@ALTree ALTree closed this as completed Sep 26, 2019
@golang golang locked and limited conversation to collaborators Sep 25, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
6 participants
@bradfitz @ianrose14 @aclements @ianlancetaylor @ALTree @gopherbot