But it is safe to do so in go, strings are immutable. Right?

I don't see anything to do here. strings consist of more than just the data part, appending to the builder won't change the original string.

As others have said, I'm not sure I see what the problem here is. Strings are safely allowed to alias, and IIRC the only requirement is that their underlying data is immutable. You also can't re-slice a string to be larger like you can with a slice, because it doesn't have a capacity like slices do. Finally, strings.Builder provides no facility for mutating that data, only appending.

Basically, we are able to break a cornerstone Go invariant with some really basic usage of "beginner" level facilities without any warning.

Which invariant do you mean?

As a side note, it is not obvious why Builder is explicitly made non-copyable. Nothing bad will happen if a similar structure is copied here and there, it keeps working just fine, unlike this here issue.

This requirement is necessary to achieve the performance characteristics Builder strives for while maintaining string invariants. Namely, if you copy Builder and write to both copies, the buffer the internal implementation carries may have the same memory written to more than once. Making that safe for copying would require having String return a copy of the internal buffer, which the type is explicitly trying to avoid.

This restriction does make for some weird semantics, but there's precedent for no-copy data structures elsewhere (e.g. sync.Mutex). It doesn't seem unreasonable to me to require the same of another type, even if it is just for performance.