Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: change in Caller prevents detecting program name #23128

Closed
rhysh opened this issue Dec 14, 2017 · 12 comments
Closed

runtime: change in Caller prevents detecting program name #23128

rhysh opened this issue Dec 14, 2017 · 12 comments
Labels
FrozenDueToAge NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Milestone
Go1.10

Comments

@rhysh
Copy link
Contributor

rhysh commented Dec 14, 2017

I used to have a way for packages to determine the name of the program that included them. This is very helpful for providing easy-to-use metrics and logging packages: the package determines the name of the program, and everyone knows how to map between the name of the metrics/logs and the location of the relevant source code. The implementation details I was using to do that have changed during the Go 1.10 cycle, and it's not clear how I can move forward.

What version of Go are you using (go version)?

go1.10beta1:

$ go version
go version devel +9ce6b5c2ed Thu Dec 7 17:38:51 2017 +0000 darwin/amd64

Does this issue reproduce with the latest release?

This is a regression since Go 1.9, present in go1.10beta1.

What operating system and processor architecture are you using (go env)?

darwin/amd64, linux/amd64

What did you do?

Since Go 1.4, I've maintained a (closed-source) package that allows programs to determine their name: the import path of the "main" package. This was based on using runtime.Caller in an init function until encountering main.init. The filename defining that function would allow my package to determine the import path of the program (assuming $GOPATH didn't include a bonus "/src/" element).

This worked well until Go 1.9, when the filenames of all init functions changed to be <autogenerated>. I updated my package to rely on even more arcane implementation details, using runtime.FuncForPC to do a binary search of the executable between the PC for main.init and the PC for its own init function to find other code within package main that included a real filename, and returning the import path it calculated from that.

Then, I upgraded to go1.10beta1 and found that the package was once again broken, with git bisect pointing to the removal of <autogenerated> methods from stack traces (e972095). /cc @aclements

What did you expect to see?

I expected that the caller of an init function would continue to be the code that caused it to run—that is, the package that caused it to be part of the program by importing it.

I expect there to be a way for programs to determine their identity. This package is deployed in a large number of (closed-source) applications, where it is used when naming logs and metric streams for a clear bidirectional mapping between metrics and the programs that generated them.

I expected to be able to work around the change, with some other way of getting ahold of a program counter for something in package main, without having to dig even deeper into unsupported behavior.

From #22231 (comment), "Using Caller(1) in an init function is definitely depending on an implementation detail that could easily change for any number of reasons." makes it clear that this behavior should not be trusted .. but what is the alternative?

What did you see instead?

Today https://play.golang.org/p/-RRrePXZMo prints three frames, and would print more if it were in a non-main package:

goroutine 1 [running, locked to thread]:
main.printStack()
	/tmp/sandbox953472673/main.go:10 +0x80
main.init.0()
	/tmp/sandbox953472673/main.go:15 +0x20
main.init()
	<autogenerated>:1 +0xa0

go1.9

But with go1.10beta1, it prints only two:

$ go run /tmp/main.go 
goroutine 1 [running, locked to thread]:
main.printStack()
	/tmp/main.go:10 +0x77
main.init.0()
	/tmp/main.go:15 +0x20

devel +9ce6b5c2ed Thu Dec 7 17:38:51 2017 +0000

Critically, the final frame is in the function that calls runtime.Stack/runtime.Caller and no longer leads all the way up the import tree to the "main" package.
@ianlancetaylor ianlancetaylor added this to the Go1.10 milestone Dec 14, 2017
@ianlancetaylor ianlancetaylor added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Dec 14, 2017
@ianlancetaylor
Copy link
Member

Can you just look at os.Args[0] instead?

@bradfitz
Copy link
Contributor

Or https://golang.org/pkg/os/#Executable

@rhysh
Copy link
Contributor Author

rhysh commented Dec 14, 2017

I'm looking for the full import path of the program. When adding instrumentation to https://godoc.org/golang.org/x/tools/cmd/present, the path I'd look for is "golang.org/x/tools/cmd/present". os.Args[0] and os.Executable might be able to give me "present". The name of the directory containing the program isn't generally meaningful in my environments.

We'd end up with a bunch of different programs claiming to be "server", "api", or "worker".

I guess it could be done via debug/elf (and os.Executable): the symbol table would give an offset that I could use with runtime.FuncForPC. I'll give that a shot and measure its overhead. Thanks for the suggestions!

@cznic
Copy link
Contributor

cznic commented Dec 14, 2017

I'm looking for the full import path of the program.

Something like this?

@cherrymui
Copy link
Member

Why does it need to be called in init? Calling it from main.main would work, right?

@rhysh
Copy link
Contributor Author

rhysh commented Dec 15, 2017

Calling from main.main would give the correct results, but changing to require that is impractical. The results of the lookup are used in several places in our programs. Some programs may initialize everything in main.main, some may use goroutines. Some packages have their own init functions, which would run before main.main starts. These details may change as service owners refactor their code.

@cznic I'm looking for the import path of the program's main package. The code you linked looks like it gives the import path of the immediate caller. I use a function like that one for a related purpose—for importable packages to give unique names to data they provide in expvar and runtime/pprof—but it doesn't help to differentiate between metrics emitted by different programs using the same stats package.

Reading the symbol table has worked so far on linux and darwin. I haven't tackled windows support yet.

Looking at the call stack relied on implementation details, but it was cross-platform. The runtime package provides access to GOOS/GOARCH, the compiler name, the compiler version, and GOROOT. Maybe it could also give access to the import path ("name") of the program?

@cznic
Copy link
Contributor

cznic commented Dec 15, 2017

I'm looking for the import path of the program's main package. The code you linked looks like it gives the import path of the immediate caller.

Correct, but then why not just call it from within the main package? I'm probably missing something.

@rhysh
Copy link
Contributor Author

rhysh commented Dec 15, 2017

The results are used in several metrics/logging packages. They are initialized in a variety of places across a large number of programs (each with a different main package) which are maintained by different teams. Migrating all of our programs to use a new API which only works when called from the main package is impractical for us.

@aclements
Copy link
Member

Hi @rhysh. Sorry that this broke for you.

If you're willing to try more implementation-dependent tricks, I believe it should still work to call runtime.Callers and then iterate over the returned PCs manually (rather than using CallersFrames). One of those PCs (probably the last one) should still the be generated super-init function in the main package, which should keep your existing code working.

The runtime package provides access to GOOS/GOARCH, the compiler name, the compiler version, and GOROOT. Maybe it could also give access to the import path ("name") of the program?

My concern with doing something like this is that package/import paths aren't actually part of the Go language. They're part of the build system, and there are build systems that define them differently (e.g., Blaze [Bazel used to, but it seems that was fixed, thankfully]). And this may get more complicated even in the go tool with changes for package versioning.

@aclements
Copy link
Member

Reading the symbol table has worked so far on linux and darwin. I haven't tackled windows support yet.

I would argue that this is actually the "right" way to do this, even though I understand it requires a little platform-dependent code (I think only a few lines per platform though?). It doesn't depend on any implementation details and it goes through well-defined interfaces. Even the runtime doesn't really have this information handy; it obviously has a map from PC to function, but there's no map the other way around.

@ianlancetaylor
Copy link
Member

This is unfortunate but it sounds like there are workarounds. Closing. Please comment if you disagree.

@rhysh
Copy link
Contributor Author

rhysh commented Jan 5, 2018

Right @ianlancetaylor , the workarounds are sufficient for me. Thanks @aclements for the advice on runtime.Callers.

@golang golang locked and limited conversation to collaborators Jan 5, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Projects
None yet
Milestone
Go1.10
Development

No branches or pull requests
7 participants
@bradfitz @rhysh @cznic @aclements @ianlancetaylor @gopherbot @cherrymui