cmd/compile/internal/base: GODEBUG=zerocopy=0 tag doesn't seem to work #64427

cuishuang · 2023-11-28T16:13:55Z

Go version

go version devel go1.22-769469fb5e Fri Nov 24 10:25:33 2023 +0800 darwin/arm64

What operating system and processor architecture are you using (`go env`)?

GO111MODULE='on'
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/fliter/Library/Caches/go-build'
GOENV='/Users/fliter/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/fliter/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=off
GOOS='darwin'
GOPATH='/Users/fliter/go'
GOPRIVATE=''
GOPROXY='https://goproxy.cn,direct'
GOROOT='/Users/fliter/20231119/go'
GOSUMDB='off'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/Users/fliter/20231119/go/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='devel go1.22-769469fb5e Fri Nov 24 10:25:33 2023 +0800'
GCCGO='gccgo'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='1'
GOMOD='/Users/fliter/go/src/shuang/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/9t/839s3jmj73bcgyp5x_xh3gw00000gn/T/go-build504757815=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

CL520599 added optimization for zero memory overhead of string-->[]byte. CL520600 enables zerocopy by default.

I want to test and compare,

package main

import (
"reflect"
"testing"
"unsafe"
)


func BenchmarkString(b *testing.B) {
byteSli := []byte{123, 34, 100, 101, 102, 97, 117, 108, 116, 34, 58, 123, 34, 99, 111, 109, 109, 111, 110, 34, 58, 123 , 34, 112, 101, 116, 34, 58, 123, 34, 102, 105, 118, 101, 34, 58, 34, 230, 150, 145, 230, 150, 145, 34, 44, 34, 102 , 111, 117, 114, 34, 58, 34, 231, 154, 174, 231, 147, 156, 231, 147, 156, 34, 44, 34, 111, 110, 101, 34, 58, 34, 229 , 188, 165, 229, 188, 165, 230, 135, 181, 34, 44, 34, 116, 104, 114, 101, 101, 34, 58, 34, 229, 145, 134, 229, 145, 134 , 34, 44, 34, 116, 119, 111, 34, 58, 34, 233, 187, 132, 230, 169, 153, 230, 169, 153, 34, 125, 44, 34, 114, 101, 108 , 97, 116, 105, 111, 110, 34, 58, 123, 34, 102, 97, 116, 104, 101, 114, 34, 58, 34, 99, 117, 105, 120, 120, 120, 120 , 120, 120, 120, 34, 44, 34, 109, 111, 116, 104, 101, 114, 34, 58, 34, 121, 105, 110, 120, 120, 120, 120, 120, 34, 44 , 34, 119, 105, 102, 101, 34, 58, 34, 112, 101, 110, 103, 120, 120, 34, 125, 125, 125, 125}

_ = string(byteSli)

}

func BenchmarkUnsafe(b *testing.B) {
byteSli := []byte{123, 34, 100, 101, 102, 97, 117, 108, 116, 34, 58, 123, 34, 99, 111, 109, 109, 111, 110, 34, 58, 123 , 34, 112, 101, 116, 34, 58, 123, 34, 102, 105, 118, 101, 34, 58, 34, 230, 150, 145, 230, 150, 145, 34, 44, 34, 102 , 111, 117, 114, 34, 58, 34, 231, 154, 174, 231, 147, 156, 231, 147, 156, 34, 44, 34, 111, 110, 101, 34, 58, 34, 229 , 188, 165, 229, 188, 165, 230, 135, 181, 34, 44, 34, 116, 104, 114, 101, 101, 34, 58, 34, 229, 145, 134, 229, 145, 134 , 34, 44, 34, 116, 119, 111, 34, 58, 34, 233, 187, 132, 230, 169, 153, 230, 169, 153, 34, 125, 44, 34, 114, 101, 108 , 97, 116, 105, 111, 110, 34, 58, 123, 34, 102, 97, 116, 104, 101, 114, 34, 58, 34, 99, 117, 105, 120, 120, 120, 120 , 120, 120, 120, 34, 44, 34, 109, 111, 116, 104, 101, 114, 34, 58, 34, 121, 105, 110, 120, 120, 120, 120, 120, 34, 44 , 34, 119, 105, 102, 101, 34, 58, 34, 112, 101, 110, 103, 120, 120, 34, 125, 125, 125, 125}

_ = *(*string)(unsafe.Pointer(&byteSli))
}

func BenchmarkByteStyle(b *testing.B) {

str := `{"default":{"common":{"pet":{"five":"aa","four":"bb","one":"cc","three":"dd ","two":"ee"},"relation":{"father":"ff","mother":"mm","wife":"ww"}}}}`

_ = []byte(str)

}

func BenchmarkWithUnsafe(b *testing.B) {

str := `{"default":{"common":{"pet":{"five":"aa","four":"bb","one":"cc","three":"dd ","two":"ee"},"relation":{"father":"ff","mother":"mm","wife":"ww"}}}}`

sh := (*reflect.StringHeader)(unsafe.Pointer(&str))
bh := reflect.SliceHeader{
Data: sh.Data,
Len: sh.Len,
Cap: sh.Len,
}
_ = *(*[]byte)(unsafe.Pointer(&bh))

}

package main

import (
"testing"
)


func BenchmarkTest1(b *testing.B) {
for i := 0; i < b.N; i++ {
BenchmarkString(b)
}
}

func BenchmarkTest2(b *testing.B) {
for i := 0; i < b.N; i++ {
BenchmarkUnsafe(b)
}
}

func BenchmarkTest3(b *testing.B) {
for i := 0; i < b.N; i++ {
BenchmarkByteStyle(b)
}
}

func BenchmarkTest4(b *testing.B) {
for i := 0; i < b.N; i++ {
BenchmarkWithUnsafe(b)
}
}

when go version is 1.21

  go version
go version go1.21.0 darwin/arm64

goos: darwin
goarch: arm64
pkg: bc
BenchmarkTest1-8 37078008 32.45 ns/op 192 B/op 1 allocs/op
BenchmarkTest2-8 144106840 9.181 ns/op 0 B/op 0 allocs/op
BenchmarkTest3-8 38973375 28.94 ns/op 192 B/op 1 allocs/op
BenchmarkTest4-8 1000000000 0.3130 ns/op 0 B/op 0 allocs/op
PASS
ok bc 6.038s

when go version is 1.22

  go version
go version devel go1.22-631a6c2abf Fri Nov 17 23:34:11 2023 +0000 darwin/arm64


  go test -test.bench=".*" -benchmem
goos: darwin
goarch: arm64
pkg: bc
BenchmarkTest1-8 35727334 33.51 ns/op 192 B/op 1 allocs/op
BenchmarkTest2-8 147172425 8.157 ns/op 0 B/op 0 allocs/op
BenchmarkTest3-8 1000000000 0.3136 ns/op 0 B/op 0 allocs/op
BenchmarkTest4-8 1000000000 0.3153 ns/op 0 B/op 0 allocs/op
PASS
ok bc 5.095s

Compared with 1.21, it is indeed optimized for sting-->[]byte.

But also in the go 1.22 environment, GODEBUG=zerocopy=0 go test -test.bench="." -benchmem or GODEBUG='zerocopy=0' go test -test.bench="." -benchmem does not work at all, BenchmarkTest3-8 still has no memory overhead

What did you expect to see?

BenchmarkTest3-8 should have memory allocated when the GODEBUG=zerocopy=0 tag is added.

What did you see instead?

BenchmarkTest3-8 has no memory overhead.

When I remove Debug.ZeroCopy = 1 in the source code src/cmd/compile/internal/base/flag.go, recompile and execute, I can see that BenchmarkTest3-8 has memory overhead as expected.

The text was updated successfully, but these errors were encountered:

dmitshur · 2023-11-28T18:37:54Z

CC @golang/compiler.

mknyszek · 2023-11-28T18:55:47Z

I haven't looked closely, but at first glance I think BenchmarkTest3 is getting optimized into an empty loop by the compiler, since it can prove the function's computations aren't used at all. In general, a time/op of <1ns typically suggests that the loop is doing nothing.

If you change your Benchmark* functions to write the result into a global, you should be more likely to get the results you expect. See #61179 for a proposal for a more general solution to this problem.

cherrymui · 2023-11-28T20:09:59Z

This is not a GODEBUG setting, but a compiler command-line debug flag. Try go test -gcflags=all=-d=zerocopy=0. (Also note that the compiler's command-line debug flag is not subject to compatibility guarantee, so it can be removed in the future.)

cuishuang · 2023-11-29T02:36:56Z

This is not a GODEBUG setting, but a compiler command-line debug flag. Try go test -gcflags=all=-d=zerocopy=0. (Also note that the compiler's command-line debug flag is not subject to compatibility guarantee, so it can be removed in the future.)

Indeed, thanks for the guidance.

cuishuang · 2023-11-29T03:01:09Z

func BenchmarkByteStyle(b *testing.B) {

str := `{"default":{"common":{"pet":{"five":"aa","four":"bb","one":"cc","three":"dd ","two":"ee"},"relation":{"father":"ff","mother":"mm","wife":"ww"}}}}`

_ = []byte(str)

}

func BenchmarkWithUnsafe(b *testing.B) {

str := `{"default":{"common":{"pet":{"five":"aa","four":"bb","one":"cc","three":"dd ","two":"ee"},"relation":{"father":"ff","mother":"mm","wife":"ww"}}}}`

sh := (*reflect.StringHeader)(unsafe.Pointer(&str))
bh := reflect.SliceHeader{
Data: sh.Data,
Len: sh.Len,
Cap: sh.Len,
}
_ = *(*[]byte)(unsafe.Pointer(&bh))

}

``

I haven't looked closely, but at first glance I think BenchmarkTest3 is getting optimized into an empty loop by the compiler, since it can prove the function's computations aren't used at all. In general, a time/op of <1ns typically suggests that the loop is doing nothing.

If you change your Benchmark* functions to write the result into a global, you should be more likely to get the results you expect. See #61179 for a proposal for a more general solution to this problem.

I modified use case 3 and use case 4 as per your guidance and it looks like yes. Thanks!

var result []byte

func BenchmarkByteStyle(b *testing.B) {

	str := `{"default":{"common":{"pet":{"five":"aa","four":"bb","one":"cc","three":"dd ","two":"ee"},"relation":{"father":"ff","mother":"mm","wife":"ww"}}}}`

	result = []byte(str)

}

func BenchmarkWithUnsafe(b *testing.B) {

	str := `{"default":{"common":{"pet":{"five":"aa","four":"bb","one":"cc","three":"dd ","two":"ee"},"relation":{"father":"ff","mother":"mm","wife":"ww"}}}}`

	sh := (*reflect.StringHeader)(unsafe.Pointer(&str))
	bh := reflect.SliceHeader{
		Data: sh.Data,
		Len: sh.Len,
		Cap: sh.Len,
	}
	result = *(*[]byte)(unsafe.Pointer(&bh))

}

 go test  -test.bench=".*" -benchmem
goos: darwin
goarch: arm64
pkg: bc
BenchmarkTest1-8        36800676                32.41 ns/op          192 B/op          1 allocs/op
BenchmarkTest2-8        143107815                8.233 ns/op           0 B/op          0 allocs/op
BenchmarkTest3-8        40202856                29.65 ns/op          192 B/op          1 allocs/op
BenchmarkTest4-8        942046788                1.271 ns/op           0 B/op          0 allocs/op
PASS
ok      bc      6.996s

cuishuang · 2023-11-29T03:02:37Z

I have a misunderstanding about the use of GODEBUG, thanks for all the replies. I think this issue can be closed, this is not a bug.

Sorry to bother everyone

cuishuang changed the title ~~import/path: GODEBUG=zerocopy=0 tag doesn't seem to work~~ cmd/compile/internal/base: GODEBUG=zerocopy=0 tag doesn't seem to work Nov 28, 2023

bcmills added the compiler/runtime Issues related to the Go compiler and/or runtime. label Nov 28, 2023

dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 28, 2023

dmitshur added this to the Backlog milestone Nov 28, 2023

cherrymui added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Nov 28, 2023

cuishuang closed this as completed Nov 29, 2023

dmitshur closed this as not planned Won't fix, can't repro, duplicate, stale Nov 29, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/compile/internal/base: GODEBUG=zerocopy=0 tag doesn't seem to work #64427

cmd/compile/internal/base: GODEBUG=zerocopy=0 tag doesn't seem to work #64427

cuishuang commented Nov 28, 2023 •

edited

dmitshur commented Nov 28, 2023

mknyszek commented Nov 28, 2023

cherrymui commented Nov 28, 2023

cuishuang commented Nov 29, 2023

cuishuang commented Nov 29, 2023

cuishuang commented Nov 29, 2023

cmd/compile/internal/base: GODEBUG=zerocopy=0 tag doesn't seem to work #64427

cmd/compile/internal/base: GODEBUG=zerocopy=0 tag doesn't seem to work #64427

Comments

cuishuang commented Nov 28, 2023 • edited

Go version

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

dmitshur commented Nov 28, 2023

mknyszek commented Nov 28, 2023

cherrymui commented Nov 28, 2023

cuishuang commented Nov 29, 2023

cuishuang commented Nov 29, 2023

cuishuang commented Nov 29, 2023

cuishuang commented Nov 28, 2023 •

edited

What operating system and processor architecture are you using (`go env`)?