Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: optimize []byte(string1 + string2) #62407

Open
mdempsky opened this issue Aug 31, 2023 · 13 comments
Open

cmd/compile: optimize []byte(string1 + string2) #62407

mdempsky opened this issue Aug 31, 2023 · 13 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. help wanted NeedsFix The path to resolution is known, but the work has not been done. Suggested Issues that may be good for new contributors looking for work to do.
Milestone

Comments

@mdempsky
Copy link
Member

A somewhat common code pattern is []byte(string1 + string2).

Today this gets compiled as two separate steps: concatenating the two strings into a new string, and then copying that string result into a new []byte slice.

But it shouldn't be too much trouble to have the compiler instead recognize these patterns (e.g., recognize an OSTR2BYTES whose operand is an OADDSTR), and optimize accordingly.

@mdempsky mdempsky added Suggested Issues that may be good for new contributors looking for work to do. help wanted NeedsFix The path to resolution is known, but the work has not been done. labels Aug 31, 2023
@mdempsky mdempsky added this to the Backlog milestone Aug 31, 2023
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Aug 31, 2023
@robpike
Copy link
Contributor

robpike commented Sep 1, 2023

What is your evidence that it's common code? It doesn't sound familiar to me.

@mdempsky
Copy link
Member Author

mdempsky commented Sep 1, 2023

Searching kubernetes, it happens a lot in test files, but not exclusively there.

string concatenations converted to bytes
go/src/bufio/bufio_test.go:1003:33: string concatenation converted to bytes
go/src/bytes/bytes_test.go:1846:9: string concatenation converted to bytes
go/src/crypto/tls/tls_test.go:106:15: string concatenation converted to bytes
go/src/crypto/tls/tls_test.go:110:15: string concatenation converted to bytes
go/src/encoding/base64/base64_test.go:571:52: string concatenation converted to bytes
go/src/encoding/json/decode_test.go:106:15: string concatenation converted to bytes
go/src/encoding/json/encode_test.go:584:15: string concatenation converted to bytes
go/src/encoding/pem/pem.go:235:28: string concatenation converted to bytes
go/src/encoding/pem/pem.go:254:31: string concatenation converted to bytes
go/src/encoding/pem/pem.go:301:28: string concatenation converted to bytes
go/src/mime/multipart/multipart.go:120:13: string concatenation converted to bytes
go/src/net/http/readrequest_test.go:440:15: string concatenation converted to bytes
go/src/net/http/serve_test.go:1212:17: string concatenation converted to bytes
go/src/net/http/serve_test.go:135:15: string concatenation converted to bytes
go/src/net/netip/netip_test.go:296:16: string concatenation converted to bytes
go/src/regexp/all_test.go:591:13: string concatenation converted to bytes
go/src/regexp/all_test.go:634:13: string concatenation converted to bytes
kubernetes/cmd/kubeadm/app/cmd/upgrade/common_test.go:180:25: string concatenation converted to bytes
kubernetes/cmd/kubeadm/app/preflight/checks_test.go:799:52: string concatenation converted to bytes
kubernetes/pkg/api/testing/serialization_test.go:191:70: string concatenation converted to bytes
kubernetes/pkg/api/testing/serialization_test.go:199:35: string concatenation converted to bytes
kubernetes/pkg/api/testing/serialization_test.go:205:36: string concatenation converted to bytes
kubernetes/pkg/proxy/iptables/proxier.go:671:30: string concatenation converted to bytes
kubernetes/pkg/proxy/iptables/proxier.go:715:30: string concatenation converted to bytes
kubernetes/pkg/proxy/ipvs/ipset/ipset_test.go:468:18: string concatenation converted to bytes
kubernetes/pkg/proxy/ipvs/ipset/ipset_test.go:472:18: string concatenation converted to bytes
kubernetes/pkg/proxy/ipvs/ipset/ipset_test.go:516:18: string concatenation converted to bytes
kubernetes/pkg/util/iptables/iptables_test.go:53:50: string concatenation converted to bytes
kubernetes/pkg/util/iptables/iptables_test.go:55:50: string concatenation converted to bytes
kubernetes/pkg/util/iptables/iptables_test.go:778:50: string concatenation converted to bytes
kubernetes/pkg/util/iptables/iptables_test.go:842:50: string concatenation converted to bytes
kubernetes/pkg/util/tail/tail_test.go:34:21: string concatenation converted to bytes
kubernetes/pkg/util/tail/tail_test.go:95:21: string concatenation converted to bytes
kubernetes/test/e2e/apimachinery/apply.go:1022:17: string concatenation converted to bytes
kubernetes/test/e2e/apimachinery/apply.go:274:16: string concatenation converted to bytes
kubernetes/test/e2e/apimachinery/apply.go:319:15: string concatenation converted to bytes
kubernetes/test/e2e/apimachinery/apply.go:379:18: string concatenation converted to bytes
kubernetes/test/e2e/apimachinery/apply.go:426:17: string concatenation converted to bytes
kubernetes/test/e2e/apimachinery/apply.go:486:16: string concatenation converted to bytes
kubernetes/test/e2e/apimachinery/apply.go:972:18: string concatenation converted to bytes
kubernetes/test/e2e/apimachinery/namespace.go:332:10: string concatenation converted to bytes
kubernetes/test/e2e/apimachinery/resource_quota.go:1109:10: string concatenation converted to bytes
kubernetes/test/e2e/apimachinery/resource_quota.go:1161:10: string concatenation converted to bytes
kubernetes/test/e2e/apps/cronjob.go:424:10: string concatenation converted to bytes
kubernetes/test/e2e/apps/job.go:667:10: string concatenation converted to bytes
kubernetes/test/e2e/auth/certificates.go:396:10: string concatenation converted to bytes
kubernetes/test/e2e/common/node/pods.go:1119:10: string concatenation converted to bytes
kubernetes/test/e2e/network/ingress.go:755:10: string concatenation converted to bytes
kubernetes/test/e2e/network/service.go:3405:10: string concatenation converted to bytes
kubernetes/test/integration/apiserver/admissionwebhook/client_auth_test.go:120:59: string concatenation converted to bytes
kubernetes/test/integration/apiserver/admissionwebhook/client_auth_test.go:94:54: string concatenation converted to bytes
kubernetes/test/integration/apiserver/apply/apply_test.go:153:20: string concatenation converted to bytes
kubernetes/test/integration/apiserver/apply/apply_test.go:619:15: string concatenation converted to bytes
kubernetes/test/integration/apiserver/apply/apply_test.go:885:20: string concatenation converted to bytes
kubernetes/test/integration/apiserver/max_json_patch_operations_test.go:49:21: string concatenation converted to bytes
kubernetes/test/integration/apiserver/max_request_body_bytes_test.go:106:22: string concatenation converted to bytes
kubernetes/test/integration/apiserver/max_request_body_bytes_test.go:117:22: string concatenation converted to bytes
kubernetes/test/integration/apiserver/max_request_body_bytes_test.go:128:22: string concatenation converted to bytes
kubernetes/test/integration/apiserver/max_request_body_bytes_test.go:139:22: string concatenation converted to bytes
kubernetes/test/integration/apiserver/max_request_body_bytes_test.go:150:22: string concatenation converted to bytes
kubernetes/test/integration/apiserver/max_request_body_bytes_test.go:161:22: string concatenation converted to bytes
kubernetes/test/integration/apiserver/max_request_body_bytes_test.go:172:22: string concatenation converted to bytes
kubernetes/test/integration/apiserver/max_request_body_bytes_test.go:193:21: string concatenation converted to bytes
kubernetes/test/integration/apiserver/max_request_body_bytes_test.go:216:21: string concatenation converted to bytes
kubernetes/test/integration/apiserver/max_request_body_bytes_test.go:239:21: string concatenation converted to bytes
kubernetes/test/integration/apiserver/max_request_body_bytes_test.go:259:21: string concatenation converted to bytes
kubernetes/test/integration/apiserver/max_request_body_bytes_test.go:283:21: string concatenation converted to bytes
kubernetes/test/integration/apiserver/max_request_body_bytes_test.go:308:21: string concatenation converted to bytes
kubernetes/test/integration/apiserver/max_request_body_bytes_test.go:95:22: string concatenation converted to bytes
kubernetes/vendor/github.com/coreos/go-semver/semver/semver.go:131:15: string concatenation converted to bytes
kubernetes/vendor/github.com/google/go-cmp/cmp/report_reflect.go:337:70: string concatenation converted to bytes
kubernetes/vendor/github.com/google/go-cmp/cmp/report_reflect.go:346:70: string concatenation converted to bytes
kubernetes/vendor/github.com/google/go-cmp/cmp/report_reflect.go:352:18: string concatenation converted to bytes
kubernetes/vendor/github.com/google/go-cmp/cmp/report_reflect.go:354:17: string concatenation converted to bytes
kubernetes/vendor/github.com/gorilla/websocket/proxy.go:45:58: string concatenation converted to bytes
kubernetes/vendor/github.com/opencontainers/runc/libcontainer/intelrdt/intelrdt.go:317:57: string concatenation converted to bytes
kubernetes/vendor/github.com/russross/blackfriday/v2/block.go:493:20: string concatenation converted to bytes
kubernetes/vendor/github.com/vmware/govmomi/toolbox/vix/protocol.go:838:22: string concatenation converted to bytes
kubernetes/vendor/github.com/vmware/govmomi/toolbox/vix/protocol.go:838:45: string concatenation converted to bytes
kubernetes/vendor/go.etcd.io/etcd/server/v3/etcdserver/api/v2error/error.go:163:26: string concatenation converted to bytes
kubernetes/vendor/golang.org/x/oauth2/google/internal/externalaccount/aws.go:240:22: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apiextensions-apiserver/test/integration/limit_test.go:208:22: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apiextensions-apiserver/test/integration/limit_test.go:219:22: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apiextensions-apiserver/test/integration/limit_test.go:230:22: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/embedded_test.go:45:38: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/embedded_test.go:57:37: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json_limit_test.go:102:21: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json_limit_test.go:107:21: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json_limit_test.go:112:21: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json_limit_test.go:117:21: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json_limit_test.go:53:20: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json_limit_test.go:58:20: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json_limit_test.go:63:20: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json_limit_test.go:72:21: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json_limit_test.go:77:21: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json_limit_test.go:82:21: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json_limit_test.go:87:21: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json_limit_test.go:92:21: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json_limit_test.go:97:21: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/yaml/yaml_test.go:280:21: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/yaml/yaml_test.go:286:16: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/yaml/yaml_test.go:299:17: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/yaml/yaml_test.go:313:21: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/yaml/yaml_test.go:318:21: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/yaml/yaml_test.go:323:21: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/yaml/yaml_test.go:328:21: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/yaml/yaml_test.go:333:21: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/yaml/yaml_test.go:338:21: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/yaml/yaml_test.go:343:21: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/yaml/yaml_test.go:348:21: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/yaml/yaml_test.go:353:21: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/yaml/yaml_test.go:358:21: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/util/managedfields/internal/lastappliedupdater_test.go:43:29: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy/upgradeaware_test.go:517:20: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers_test.go:346:32: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apiserver/pkg/server/options/encryptionconfig/config.go:591:24: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apiserver/pkg/server/options/encryptionconfig/config.go:643:24: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apiserver/pkg/server/options/encryptionconfig/config.go:739:23: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apiserver/pkg/server/options/encryptionconfig/config.go:818:22: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apiserver/pkg/storage/testing/store_tests.go:2137:31: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apiserver/pkg/storage/value/encrypt/envelope/envelope_test.go:63:15: string concatenation converted to bytes
kubernetes/vendor/k8s.io/apiserver/plugin/pkg/audit/truncate/truncate_test.go:56:17: string concatenation converted to bytes
kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy_test.go:533:20: string concatenation converted to bytes

The opposite scenario happens about as frequently too, which we could also optimize:

bytes converted to concatenated strings
go/src/archive/tar/reader_test.go:1151:20: bytes converted to concatenated string
go/src/archive/tar/reader_test.go:1286:20: bytes converted to concatenated string
go/src/bytes/bytes_test.go:1194:30: bytes converted to concatenated string
go/src/bytes/example_test.go:506:36: bytes converted to concatenated string
go/src/bytes/example_test.go:507:35: bytes converted to concatenated string
go/src/bytes/example_test.go:602:36: bytes converted to concatenated string
go/src/bytes/example_test.go:603:35: bytes converted to concatenated string
go/src/bytes/example_test.go:617:36: bytes converted to concatenated string
go/src/bytes/example_test.go:618:35: bytes converted to concatenated string
go/src/encoding/binary/varint_test.go:41:36: bytes converted to concatenated string
go/src/encoding/binary/varint_test.go:42:73: bytes converted to concatenated string
go/src/encoding/binary/varint_test.go:67:36: bytes converted to concatenated string
go/src/encoding/binary/varint_test.go:68:74: bytes converted to concatenated string
go/src/encoding/hex/hex_test.go:64:41: bytes converted to concatenated string
go/src/encoding/hex/hex_test.go:65:61: bytes converted to concatenated string
go/src/encoding/json/decode.go:1000:62: bytes converted to concatenated string
go/src/encoding/json/decode.go:1008:62: bytes converted to concatenated string
go/src/encoding/json/decode.go:992:62: bytes converted to concatenated string
go/src/encoding/json/encode.go:1164:38: bytes converted to concatenated string
go/src/encoding/xml/xml.go:1191:54: bytes converted to concatenated string
go/src/go/doc/comment/html.go:105:21: bytes converted to concatenated string
go/src/go/scanner/scanner.go:252:49: bytes converted to concatenated string
go/src/go/scanner/scanner.go:267:53: bytes converted to concatenated string
go/src/go/scanner/scanner.go:277:49: bytes converted to concatenated string
go/src/math/big/ftoa.go:391:40: bytes converted to concatenated string
go/src/math/big/nat.go:44:22: bytes converted to concatenated string
go/src/net/http/clientserver_test.go:663:15: bytes converted to concatenated string
go/src/net/http/clientserver_test.go:663:35: bytes converted to concatenated string
go/src/net/mail/message_test.go:374:14: bytes converted to concatenated string
go/src/net/mail/message_test.go:375:21: bytes converted to concatenated string
go/src/net/mail/message_test.go:376:23: bytes converted to concatenated string
go/src/net/textproto/reader.go:514:74: bytes converted to concatenated string
go/src/net/textproto/reader.go:526:67: bytes converted to concatenated string
go/src/net/textproto/reader.go:530:67: bytes converted to concatenated string
go/src/net/textproto/reader.go:534:68: bytes converted to concatenated string
go/src/os/dir_unix.go:132:44: bytes converted to concatenated string
go/src/os/env.go:44:15: bytes converted to concatenated string
go/src/path/filepath/path.go:65:41: bytes converted to concatenated string
go/src/runtime/export_test.go:1961:46: bytes converted to concatenated string
go/src/runtime/malloc_test.go:145:22: bytes converted to concatenated string
go/src/runtime/string_test.go:230:22: bytes converted to concatenated string
go/src/runtime/string_test.go:231:69: bytes converted to concatenated string
go/src/runtime/string_test.go:64:41: bytes converted to concatenated string
go/src/runtime/string_test.go:90:24: bytes converted to concatenated string
go/src/strings/export_test.go:32:37: bytes converted to concatenated string
go/src/strings/strings_test.go:1113:30: bytes converted to concatenated string
go/src/syscall/exec_linux_test.go:414:46: bytes converted to concatenated string
go/src/time/time.go:335:28: bytes converted to concatenated string
go/src/time/time.go:358:30: bytes converted to concatenated string
kubernetes/pkg/controlplane/controller/clusterauthenticationtrust/cluster_authentication_trust_controller_test.go:195:31: bytes converted to concatenated string
kubernetes/pkg/controlplane/controller/clusterauthenticationtrust/cluster_authentication_trust_controller_test.go:195:57: bytes converted to concatenated string
kubernetes/pkg/controlplane/controller/clusterauthenticationtrust/cluster_authentication_trust_controller_test.go:228:51: bytes converted to concatenated string
kubernetes/pkg/controlplane/controller/clusterauthenticationtrust/cluster_authentication_trust_controller_test.go:228:74: bytes converted to concatenated string
kubernetes/test/e2e/apimachinery/namespace.go:332:107: bytes converted to concatenated string
kubernetes/test/e2e/apimachinery/resource_quota.go:1109:40: bytes converted to concatenated string
kubernetes/test/e2e/apimachinery/resource_quota.go:1161:40: bytes converted to concatenated string
kubernetes/test/e2e/apps/cronjob.go:424:82: bytes converted to concatenated string
kubernetes/test/e2e/apps/job.go:667:82: bytes converted to concatenated string
kubernetes/test/e2e/auth/certificates.go:396:97: bytes converted to concatenated string
kubernetes/test/e2e/common/node/pods.go:1119:82: bytes converted to concatenated string
kubernetes/test/e2e/network/ingress.go:755:98: bytes converted to concatenated string
kubernetes/test/e2e/network/service.go:3405:98: bytes converted to concatenated string
kubernetes/test/e2e/network/util_iperf.go:75:36: bytes converted to concatenated string
kubernetes/vendor/github.com/Azure/go-autorest/autorest/preparer.go:434:39: bytes converted to concatenated string
kubernetes/vendor/github.com/cilium/ebpf/internal/output.go:79:27: bytes converted to concatenated string
kubernetes/vendor/github.com/cpuguy83/go-md2man/v2/md2man/roff.go:113:25: bytes converted to concatenated string
kubernetes/vendor/github.com/godbus/dbus/v5/auth_sha1.go:66:59: bytes converted to concatenated string
kubernetes/vendor/github.com/go-errors/errors/error.go:181:59: bytes converted to concatenated string
kubernetes/vendor/github.com/google/go-cmp/cmp/report_reflect.go:352:61: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_array.go:22:77: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_array.go:50:77: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_array.go:62:68: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter.go:177:85: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_int.go:129:65: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_int.go:251:65: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_object.go:124:84: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_object.go:135:85: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_object.go:153:73: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_object.go:161:69: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_object.go:177:83: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_object.go:189:84: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_object.go:209:70: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_object.go:217:66: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_object.go:234:72: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_object.go:24:84: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_object.go:250:94: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_object.go:31:71: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_object.go:37:83: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_object.go:53:66: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_object.go:71:69: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_object.go:80:69: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_skip.go:28:65: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_str.go:142:75: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_str.go:159:70: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/iter_str.go:31:68: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/reflect_array.go:101:65: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/reflect_array.go:79:70: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/reflect_map.go:163:67: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/reflect_map.go:175:81: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/reflect_map.go:186:82: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/reflect_map.go:194:62: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/reflect_map.go:205:62: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/reflect_map.go:211:62: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/reflect_slice.go:75:70: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/reflect_slice.go:96:65: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/reflect_struct_decoder.go:1085:76: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/reflect_struct_decoder.go:1094:76: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/reflect_struct_decoder.go:514:66: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/reflect_struct_decoder.go:543:83: bytes converted to concatenated string
kubernetes/vendor/github.com/json-iterator/go/reflect_struct_decoder.go:550:82: bytes converted to concatenated string
kubernetes/vendor/github.com/mailru/easyjson/jlexer/lexer.go:421:16: bytes converted to concatenated string
kubernetes/vendor/github.com/mailru/easyjson/jlexer/lexer.go:463:15: bytes converted to concatenated string
kubernetes/vendor/github.com/onsi/ginkgo/v2/internal/suite.go:354:73: bytes converted to concatenated string
kubernetes/vendor/github.com/opencontainers/runc/libcontainer/container_linux.go:592:37: bytes converted to concatenated string
kubernetes/vendor/github.com/robfig/cron/v3/chain.go:50:56: bytes converted to concatenated string
kubernetes/vendor/github.com/russross/blackfriday/v2/html.go:344:28: bytes converted to concatenated string
kubernetes/vendor/github.com/vmware/govmomi/vim25/xml/xml.go:1184:54: bytes converted to concatenated string
kubernetes/vendor/go.etcd.io/etcd/server/v3/etcdserver/api/membership/confstate.go:52:39: bytes converted to concatenated string
kubernetes/vendor/golang.org/x/net/websocket/hybi.go:418:47: bytes converted to concatenated string
kubernetes/vendor/golang.org/x/net/websocket/hybi.go:561:51: bytes converted to concatenated string
kubernetes/vendor/google.golang.org/protobuf/internal/encoding/json/decode_number.go:251:22: bytes converted to concatenated string
kubernetes/vendor/k8s.io/apimachinery/pkg/util/managedfields/internal/pathelement.go:128:28: bytes converted to concatenated string
kubernetes/vendor/k8s.io/apimachinery/pkg/util/managedfields/internal/pathelement.go:134:30: bytes converted to concatenated string
kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy/upgradeaware_test.go:517:38: bytes converted to concatenated string
kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/apiserver_test.go:1640:19: bytes converted to concatenated string
kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/discovery/aggregated/handler_test.go:140:24: bytes converted to concatenated string
kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers_test.go:346:44: bytes converted to concatenated string
kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:362:17: bytes converted to concatenated string
kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:320:33: bytes converted to concatenated string
kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/store.go:787:23: bytes converted to concatenated string
kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/store.go:807:45: bytes converted to concatenated string
kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/watcher.go:264:23: bytes converted to concatenated string
kubernetes/vendor/k8s.io/apiserver/pkg/storage/testing/store_tests.go:2137:38: bytes converted to concatenated string
kubernetes/vendor/k8s.io/client-go/plugin/pkg/client/auth/exec/exec_test.go:1051:42: bytes converted to concatenated string
kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy_test.go:533:38: bytes converted to concatenated string
kubernetes/vendor/k8s.io/kubectl/pkg/cmd/get/get_test.go:2812:42: bytes converted to concatenated string
kubernetes/vendor/k8s.io/kubectl/pkg/cmd/get/get_test.go:2836:37: bytes converted to concatenated string
kubernetes/vendor/k8s.io/kube-openapi/pkg/builder3/openapi.go:355:15: bytes converted to concatenated string
kubernetes/vendor/k8s.io/kube-openapi/pkg/builder/openapi.go:370:15: bytes converted to concatenated string
kubernetes/vendor/k8s.io/kube-openapi/pkg/internal/third_party/go-json-experiment/json/arshal_any.go:161:59: bytes converted to concatenated string
kubernetes/vendor/k8s.io/kube-openapi/pkg/internal/third_party/go-json-experiment/json/arshal_default.go:1032:62: bytes converted to concatenated string
kubernetes/vendor/k8s.io/kube-openapi/pkg/internal/third_party/go-json-experiment/json/arshal_default.go:1052:60: bytes converted to concatenated string
kubernetes/vendor/k8s.io/kube-openapi/pkg/internal/third_party/go-json-experiment/json/arshal_default.go:781:61: bytes converted to concatenated string
kubernetes/vendor/k8s.io/kube-openapi/pkg/internal/third_party/go-json-experiment/json/arshal_inlined.go:109:60: bytes converted to concatenated string
kubernetes/vendor/k8s.io/kube-openapi/pkg/internal/third_party/go-json-experiment/json/arshal_inlined.go:69:60: bytes converted to concatenated string
kubernetes/vendor/k8s.io/kube-openapi/pkg/internal/third_party/go-json-experiment/json/decode.go:495:58: bytes converted to concatenated string
kubernetes/vendor/k8s.io/kube-openapi/pkg/internal/third_party/go-json-experiment/json/decode.go:665:58: bytes converted to concatenated string
kubernetes/vendor/k8s.io/kube-openapi/pkg/internal/third_party/go-json-experiment/json/decode.go:904:67: bytes converted to concatenated string
kubernetes/vendor/k8s.io/kube-openapi/pkg/internal/third_party/go-json-experiment/json/encode.go:451:58: bytes converted to concatenated string
kubernetes/vendor/k8s.io/kube-openapi/pkg/internal/third_party/go-json-experiment/json/encode.go:546:59: bytes converted to concatenated string
kubernetes/vendor/k8s.io/kube-openapi/pkg/internal/third_party/go-json-experiment/json/encode.go:619:58: bytes converted to concatenated string
kubernetes/vendor/k8s.io/kube-openapi/pkg/internal/third_party/go-json-experiment/json/encode.go:776:68: bytes converted to concatenated string
kubernetes/vendor/sigs.k8s.io/kustomize/api/internal/plugins/execplugin/execplugin.go:188:43: bytes converted to concatenated string

@go101
Copy link

go101 commented Sep 1, 2023

This is just another evidence that a builtin concat function is needed.

With the builtin concat supported, this pattern can be rewritten as concat([]byte{}, string1, string2).

@go101
Copy link

go101 commented Sep 1, 2023

BTW, currently, without the builtin concat, this pattern should not be encouraged.
Using bytes.Buffer or make([]byte,...)+copy/append is more efficient (though the
two are still less efficient than the builtin concat way).

@mateusz834
Copy link
Member

But turning the conversions around causes only one allocation.

slices.Concat([]byte(str), []byte(str[:32]))
BenchmarkConcat-4       19893025                59.09 ns/op           96 B/op          1 allocs/op

@go101
Copy link

go101 commented Sep 1, 2023

@mateusz834 Please read this comment. Generally, it makes 3 allocations.

@mateusz834
Copy link
Member

@go101 I belive that is not true on gotip, []byte(str) doesn't cause allocations when the function does not mutate the slice.

#2205

@go101
Copy link

go101 commented Sep 1, 2023

Aha, that is a cool improvement.

In Go programming, there are many string and byte slice manipulation patterns which are needed to be specially optimized.
The long-tail effect of these patterns are hard to neglect. But it is also tedious (and error-prone) to optimize them one by one.
So I think it would be a good idea to find a universal way to rewrite these patterns. The builtin concat is such an attempt.

@tpaschalis
Copy link
Contributor

I'm looking into a CL for this, I might ask for some help around tests in Gerrit. 👀

@mdempsky
Copy link
Member Author

@tpaschalis Great, thanks! Feel free to add me as a reviewer. Happy to give advice

@gopherbot
Copy link

Change https://go.dev/cl/527935 mentions this issue: [draft] cmd/comile: optimize []byte(string1 + string2)

@aimuz
Copy link
Contributor

aimuz commented Nov 29, 2023

Will this improvement be available in version 1.22?

@randall77
Copy link
Contributor

No, it will not make it for 1.22.

However, there is now slices.Concat which seems to work. slices.Concat([]byte(string1), []byte(string2)) only does a single allocation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. help wanted NeedsFix The path to resolution is known, but the work has not been done. Suggested Issues that may be good for new contributors looking for work to do.
Projects
Development

No branches or pull requests

8 participants