Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

os: opening/locating/copying system libraries is broken in macOS Big Sur beta #39832

Closed
DiegoMagdaleno opened this issue Jun 24, 2020 · 14 comments
Closed

os: opening/locating/copying system libraries is broken in macOS Big Sur beta #39832

DiegoMagdaleno opened this issue Jun 24, 2020 · 14 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Go1.16

Comments

@DiegoMagdaleno
Copy link

DiegoMagdaleno commented Jun 24, 2020 •
edited

What version of Go are you using (go version)?

$ go version
go version go1.14.4 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/me/Library/Caches/go-build"
GOENV="/Users/me/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/me/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.14.4/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.14.4/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/77/cbrny7lx71vbbgw8jt0zt6x80000gn/T/go-build140751726=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

This recipe is very clear and I actually have an example with my own software:

  1. Use the default go (or any tbh) library to copy a library that is part of the macOS operating system for example /usr/lib/system.b.dylib.
  2. This will return a "No such file or directory".

There is a reason of why this happens, according to Apple, they changed how default system libraries work on macOS (example the C library).

Quote:

New in macOS Big Sur 11 beta, the system ships with a built-in dynamic linker cache of all system-provided libraries. As part of this change, copies of dynamic libraries are no longer present on the filesystem. Code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail. Instead, check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache. (62986286)

My project can be located here

This code checks otool to get linked libraries and then copies them (The code might be bad).

What did you expect to see?

  • I expect to get no errors and have the library copied.

What did you see instead?

  • I get an error "no such file or directory" due to this change Apple made.

Proposed solution

Perhaps an idea of mine will be having an array of the default libraries of macOS (such as libsystem.b.dylib) and check at runtime if the OS is macOS if it is, and the user wants to copy some of this libraries, the solution is to syscall dlopen() instead of open(), hope this issue can help anyone and if not I guess I will have to fix it on my side.
@randall77
Copy link
Contributor

I'm a bit confused as to what this implies. Does the Go toolchain not work? Do Go binaries not work? What about shared libraries and plugins?
Or is it just that os.Open("/usr/lib/libSystem.dylib") doesn't work?
Does the same problem happen in C? If so, I don't think there's anything for us to fix - this sounds like intentional behavior on Apple's part. (But why? Weird.)

@randall77
Copy link
Contributor

Note that this problem is listed as a "known issue" of the beta, not as an intended feature. Maybe it will be fixed.

@DiegoMagdaleno
Copy link
Author

@randall77 Well what works is basically everything in Golang, however, yeah the os.Open("/usr/lib/libsystem.dylib"), is basically broken, I tested in C and yes the same happens unless I do dlopen(), that is Apple's new behavior, while the issue only affects systems libraries, means that without a workaround opening/copying default libraries on macOS will no longer work (in Go or other stuff that doesn't do a workaround)

@dmitshur dmitshur changed the title Opening/Locating/Copying System libraries is broken in macOS Big Sur. Opening/Locating/Copying System libraries is broken in macOS Big Sur beta Jun 24, 2020
@ianlancetaylor
Copy link
Contributor

If Go programs generally work, but they fail in exactly the ways that C programs fail, then I don't think there is anything we should or even can change in the Go project.

@cagedmantis cagedmantis changed the title Opening/Locating/Copying System libraries is broken in macOS Big Sur beta os: opening/locating/copying system libraries is broken in macOS Big Sur beta Jun 24, 2020
@cagedmantis cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 24, 2020
@cagedmantis cagedmantis modified the milestones: Backlog, Go1.16 Jun 24, 2020
@dentarg dentarg mentioned this issue Jul 9, 2020
Closed
@seiflotfy
Copy link

any updates on this?

@randall77
Copy link
Contributor

@seiflotfy If there are updates, they will be posted here.
I don't think anyone knows if this should be fixed, and if so, what such a fix would look like.

@DiegoMagdaleno
Copy link
Author

There is a solution for this, and it is dumping the libraries, the tool is provided by Apple but not by default on macOS, I have the tool on my brew repository.

is dyld_shared_cache_util you can read more online

@DiegoMagdaleno
Copy link
Author

Still not an ideal solution, and I dont have the knowledge if we can interact with the cache, the cache is located at /System/Library/dyld/dyld_shared_cache_x86_64

@davecheney
Copy link
Contributor

@DiegoMagdaleno could you step back a second and explain what not being able to open a system library is preventing you from doing? That would help @randall77 identify the answer to his question, #39832 (comment)

@DiegoMagdaleno
Copy link
Author

@davecheney I have a tool that makes Chroots, so this means, for example, binaries depend on libs they are linked to. Here is an example.

/bin/ls:
	/usr/lib/libutil.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.0.0)

As you can see this are linked to /usr/lib that are system libraries on macOS, however, when you try to copy them it is gonna fail saying there is no such file or directory.

I am trying to copy them from dir A to dir B

@davecheney
Copy link
Contributor

Is this a limitation of Go or are other utils like cp, rsync, or install also affected?

@DiegoMagdaleno
Copy link
Author

Other utilities also appear to be affected.

When I opened this issue this wasn't very know, so I thought it was Golang.

@davecheney
Copy link
Contributor

Given that this problem is not restricted to go, would you consider closing this. If Apple fixes the problem it will be fixed for all languages and tools simultaneously

@DiegoMagdaleno
Copy link
Author

Sure thing, I was considering closing it a few days ago.

Sorry for the inconvenience

@golang golang locked and limited conversation to collaborators Aug 16, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Go1.16
Development

No branches or pull requests
7 participants
@davecheney @seiflotfy @cagedmantis @ianlancetaylor @randall77 @gopherbot @DiegoMagdaleno