Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: runtime: dynamically de-duplicate strings #61063

Closed
namewxt1220 opened this issue Jun 29, 2023 · 10 comments
Closed

proposal: runtime: dynamically de-duplicate strings #61063

namewxt1220 opened this issue Jun 29, 2023 · 10 comments
Labels
FrozenDueToAge Proposal WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone

Comments

@namewxt1220
Copy link

str1 := "hello world"
str2 := "hello world"

At runtime, it is desirable to point str1 and str2 to a single instance in the same string pool, since they contain the same string constants.

This optimization technique reduces the pressure on memory usage and garbage collection, while also improving the performance of the program.

Is it possible to do this in go?

@gopherbot gopherbot added this to the Proposal milestone Jun 29, 2023
@jaloren
Copy link

jaloren commented Jun 29, 2023

I believe you are looking for sync pool

https://pkg.go.dev/sync#Pool

it predates generics so unfortunately you’ll be casting to an interface but you could write generic wrapper around it.

@namewxt1220
Copy link
Author

I believe you are looking for sync pool

https://pkg.go.dev/sync#Pool

it predates generics so unfortunately you’ll be casting to an interface but you could write generic wrapper around it.

@jaloren
Thanks, but I'm afraid the sync pool doesn't meet my requirements.
I want is similar to the java string constant pool.

@namewxt1220 namewxt1220 changed the title proposal: add string pool in go proposal: add string constant pool in go Jun 29, 2023
@atdiar
Copy link

atdiar commented Jun 29, 2023

@namewxt1220 maybe you are looking for string interning?

https://mdlayher.com/blog/unsafe-string-interning-in-go/

@namewxt1220
Copy link
Author

@namewxt1220 maybe you are looking for string interning?

https://mdlayher.com/blog/unsafe-string-interning-in-go/

@atdiar
Thanks, maybe it's better to solve it from go runtime.

@bcmills
Copy link
Contributor

bcmills commented Jun 29, 2023

This proposal needs more detail.

@bcmills bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jun 29, 2023
@ianlancetaylor ianlancetaylor moved this to Incoming in Proposals Jun 30, 2023
@merykitty
Copy link

I think what this proposal is really trying to achieve is string deduplication, not string interning.

Can string deduplication be applied after the creation of the string, a.k.a we change the pointer in str2 to point to the buffer of str1 after both strings have been created?

@namewxt1220
Copy link
Author

This proposal needs more detail.

@bcmills
I think I was expecting the same thing as runtime add string intern #5160, but I'm not sure if runtime add string intern is a good solution.
github.com/josharian/intern may be an alternative, but I'd like to address this from runtime, if that's possible.

@alresvor
Copy link

If it's just your example, then they will be the same address

package main

import "unsafe"

func main() {
	str1 := "hello world"
	str2 := "hello world2"
	str3 := "hello world3"
	str4 := "hello world"
	println(unsafe.StringData(str1))
	println(unsafe.StringData(str2))
	println(unsafe.StringData(str3))
	println(unsafe.StringData(str4))
	// 0xc34e3d	<-----
	// 0xc3518c
	// 0xc35198
	// 0xc34e3d	<-----
}

@namewxt1220
Copy link
Author

namewxt1220 commented Jun 30, 2023

If it's just your example, then they will be the same address

package main

import "unsafe"

func main() {
	str1 := "hello world"
	str2 := "hello world2"
	str3 := "hello world3"
	str4 := "hello world"
	println(unsafe.StringData(str1))
	println(unsafe.StringData(str2))
	println(unsafe.StringData(str3))
	println(unsafe.StringData(str4))
	// 0xc34e3d	<-----
	// 0xc3518c
	// 0xc35198
	// 0xc34e3d	<-----
}

@alresvor
Thanks, I think I didn't express myself clearly, the scenario is similar to the following.

package main

import (
	"reflect"
	"unsafe"
)

func main() {
	var buf1 = []byte("hello world")
	var buf2 = []byte("hello world")
	str1 := decode(buf1)
	str2 := decode(buf2)
	println((*reflect.StringHeader)(unsafe.Pointer(&str1)).Data)
	println((*reflect.StringHeader)(unsafe.Pointer(&str2)).Data)
}

func decode(buf []byte) string {
	return string(buf)
}

buf1 and buf2 are the two array spaces generated at runtime, str1 and str2 is pointing to different instances, and I want str1 and str2 to point to the same instance.

@seankhliao seankhliao added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Jul 9, 2023
@adonovan adonovan changed the title proposal: add string constant pool in go proposal: runtime: dynamically de-duplicate strings Jul 19, 2023
@gopherbot
Copy link
Contributor

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@gopherbot gopherbot closed this as not planned Won't fix, can't repro, duplicate, stale Aug 9, 2023
@golang golang locked and limited conversation to collaborators Aug 8, 2024
@rsc rsc removed this from Proposals Aug 14, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge Proposal WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

8 participants