Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

proposal: encoding: add AppendText and AppendBinary #62384

Open
dsnet opened this issue Aug 30, 2023 · 20 comments
Open

proposal: encoding: add AppendText and AppendBinary #62384

dsnet opened this issue Aug 30, 2023 · 20 comments
Labels
Performance Proposal
Milestone
Proposal

Comments

@dsnet
Copy link
Member

dsnet commented Aug 30, 2023 •
edited

The MarshalText and MarshalBinary methods are a trash factory.
They create a short-lived string that is almost always garbage collectable shortly after creation.
Furthermore, the caller almost always copied the result into another buffer, resulting in further waste.

I propose the addition of the following interfaces to the "encoding" package:

// TextAppender is the interface implemented by an object
// that can append the textual representation of itself.
// If a type implements both [TextAppender] and [TextMarshaler],
// then v.MarshalText() must be semantically identical to v.AppendText(nil).
type TextAppender interface {
    // AppendText appends the textual representation of itself to the end of b
    // (allocating a larger slice if necessary) and returns the updated slice.
    //
    // Implementations must not retain b, nor mutate any bytes within b[:len(b)].
    AppendText(b []byte) ([]byte, error)
}

type BinaryAppender interface {
    AppendBinary([]byte) ([]byte, error)
}

In addition to the above, we would update all types in the stdlib that implement MarshalText or MarshalBinary to have the Append equivalent (and switch the Marshal implementation to just call the Append variant). Also, we would teach all stdlib libraries that looks for TextMarshaler or BinaryMarshaler to also look for the append (and unmarshal) variant.

\cc @bradfitz @maisem
@dsnet dsnet added the Proposal label Aug 30, 2023
@gopherbot gopherbot added this to the Proposal milestone Aug 30, 2023
@dsnet
Copy link
Member Author

dsnet commented Aug 30, 2023

This is motivated by netip.Addr.MarshalText being a non-trivial amount of slowdown in a large JSON object.

@dsnet dsnet mentioned this issue Aug 30, 2023
Closed
@dsnet
Copy link
Member Author

dsnet commented Aug 30, 2023

Note that I did not propose json.Appender. There are other possible API designs that allow for streaming JSON encoding and decoding which would obviate the need for it. A separate proposal or discussion will be made for this in the future.

@dsnet
Copy link
Member Author

dsnet commented Aug 30, 2023

Analyzing all code on the module proxy from 2023-07-01:

I do not believe there are any backwards compatibility issues with this change.

barista.run@v0.0.0-20230630192130-1694cee7a398/pango/pango.go:95: func (n *Node) AppendText(texts ...string) *Node {
bitbucket.org/afiszone/gotk3@v0.0.0-20200317205801-68a9308a074c/gtk/combo_box.go:263: func (v *ComboBoxText) AppendText(text string) {
code.afis.me/gotk3@v0.0.0-20201225185300-6238108e03d9/gtk/combo_box.go:354: func (v *ComboBoxText) AppendText(text string) {
code.aliyun.com/newrank-tools/document@v0.0.3/api/markdown/markdown.go:47: func (md *Markdown) AppendText(content string, params ...bool) *Markdown {
codeberg.org/japh/psv@v0.2.0/table.go:149: func (tbl *Table) AppendText(textLines ...string) {
git.gammaspectra.live/P2Pool/p2pool-observer@v0.0.0-20230605214634-bad1e302c8c8/monero/transaction/extra.go:54: func (t *ExtraTags) AppendBinary(preAllocatedBuf []byte) (buf []byte, err error) {
git.gammaspectra.live/P2Pool/p2pool-observer@v0.0.0-20230605214634-bad1e302c8c8/monero/transaction/extra.go:124: func (t *ExtraTag) AppendBinary(preAllocatedBuf []byte) ([]byte, error) {
git.gammaspectra.live/P2Pool/p2pool-observer@v0.0.0-20230605214634-bad1e302c8c8/monero/transaction/output.go:78: func (s *Outputs) AppendBinary(preAllocatedBuf []byte) (data []byte, err error) {
git.gammaspectra.live/P2Pool/p2pool-observer@v0.0.0-20230605214634-bad1e302c8c8/p2pool/sidechain/sidedata.go:52: func (b *SideData) AppendBinary(preAllocatedBuf []byte, version ShareVersion) (buf []byte, err error) {
git.milar.in/milarin/tui@v0.0.21/rune.go:46: func (t *Text) AppendText(txt *Text) {
git.tordarus.net/Tordarus/tui@v0.0.15/rune.go:46: func (t *Text) AppendText(txt *Text) {
git.wit.org/wit/gui@v0.8.6/common.go:91: func (n *Node) AppendText(str string) {
gitee.com/scpro/gotk3@v1.0.1/gtk/combo_box.go:364: func (v *ComboBoxText) AppendText(text string) {
gitee.com/zhouhailin/hutool-go@v0.0.2/hutool/io/io_util.go:27: func (f *IoUtil) AppendText(pathname string, text string) error {
github.com/45638654564654/appoptics-apm-go@v0.0.0-20221005200510-e098d2000040/v1/ao/internal/bson/bson.go:43: func (b *Buffer) AppendBinary(k string, v []byte) {
github.com/474420502/libxml2@v0.0.0-20200807040518-4ef6186ae68c/dom/node_element.go:62: func (n *Element) AppendText(s string) error {
github.com/BlackMocca/mongo-go-driver@v1.0.4/x/bsonx/bsoncore/bson_arraybuilder.go:94: func (a *ArrayBuilder) AppendBinary(subtype byte, b []byte) *ArrayBuilder {
github.com/BlackMocca/mongo-go-driver@v1.0.4/x/bsonx/bsoncore/bson_documentbuilder.go:82: func (db *DocumentBuilder) AppendBinary(key string, subtype byte, b []byte) *DocumentBuilder {
github.com/Existed/libxml2@v0.0.0-20190422014600-3dcec2c44e24/dom/node_element.go:62: func (n *Element) AppendText(s string) error {
github.com/FabianWe/etherpadlite-golang@v1.1.0/etherpadlite.go:308: func (pad *EtherpadLite) AppendText(ctx context.Context, padID, text interface{}) (*Response, error) {
github.com/Gipcomp/winapi@v0.0.0-20211028125119-f5f50e8984c7/textedit.go:245: func (te *TextEdit) AppendText(value string) {
github.com/GoTyro/walk@v0.0.0-20220112065045-02bb5eb37497/textedit.go:241: func (te *TextEdit) AppendText(value string) {
github.com/GoTyro/walk@v0.0.0-20220112065045-02bb5eb37497/examples/logview/logview.go:55: func (lv *LogView) AppendText(value string) {
github.com/KemalovMaulen/libxml2@v1.0.0/dom/node_element.go:62: func (n *Element) AppendText(s string) error {
github.com/Skarm/walk@v0.0.0-20150220080041-98cb48b69104/textedit.go:94: func (te *TextEdit) AppendText(value string) {
github.com/Skarm/walk@v0.0.0-20150220080041-98cb48b69104/examples/logview/logview.go:63: func (lv *LogView) AppendText(value string) {
github.com/StephanVerbeeck/walk@v0.0.0-20230202143337-d63a2e470fda/textedit.go:217: func (te *TextEdit) AppendText(value string) {
github.com/StephanVerbeeck/walk@v0.0.0-20230202143337-d63a2e470fda/examples/logview/logview.go:63: func (lv *LogView) AppendText(value string) {
github.com/adejoux/nmon2influxdb@v2.1.7-0.20190605081557-84e3405360f7+incompatible/nmon/nmon.go:42: func (nmon *Nmon) AppendText(text string) {
github.com/admpub/spider@v0.0.0-20171028061754-3a892f2f9e33/gui/logview.go:59: func (lv *LogView) AppendText(value string) {
github.com/agl/go-gtk@v0.0.0-20151214221645-91c1edb38c24/gtk/gtk.go:5447: func (v *GtkComboBoxText) AppendText(text string) {
github.com/anoshenko/rui@v0.13.0/editView.go:357: func (edit *editViewData) AppendText(text string) {
github.com/appoptics/appoptics-apm-go@v1.16.0/v1/ao/internal/bson/bson.go:43: func (b *Buffer) AppendBinary(k string, v []byte) {
github.com/auroralaboratories/gotk3@v0.0.0-20170323014032-52f9ba079f44/gtk/combo_box.go:229: func (v *ComboBoxText) AppendText(text string) {
github.com/bakharal/mongo-go-driver@v1.10.11/x/bsonx/bsoncore/bson_arraybuilder.go:94: func (a *ArrayBuilder) AppendBinary(subtype byte, b []byte) *ArrayBuilder {
github.com/bakharal/mongo-go-driver@v1.10.11/x/bsonx/bsoncore/bson_documentbuilder.go:82: func (db *DocumentBuilder) AppendBinary(key string, subtype byte, b []byte) *DocumentBuilder {
github.com/bennicholls/burl-E@v0.0.0-20190708032244-24ccafc7d477/burl/textbox.go:29: func (t *Textbox) AppendText(txt string) {
github.com/blorticus/tpcli@v0.3.1/tpcli.go:391: func (panel *outputPanel) AppendText(s string) {
github.com/bontibon/go-pipeglade@v0.0.0-20150212114355-0ac764b2f9e2/manipulation.go:141: func (ui *UI) AppendText(widget, text string) {
github.com/bytbox/gogobject@v0.0.0-20120219142239-4e70a8595dcf/gtk-3.0/gtk.go:14947: func (this0 *ComboBoxText) AppendText(text0 string) {
github.com/chai2010/walk@v0.0.2/textedit.go:240: func (te *TextEdit) AppendText(value string) {
github.com/chai2010/walk@v0.0.2/examples/logview/logview.go:55: func (lv *LogView) AppendText(value string) {
github.com/clockworkpi/LauncherGoDev@v0.0.0-20230126075719-2743d6b9dc1e/sysgo/UI/textarea.go:80: func (self *Textarea) AppendText(alphabet string) {
github.com/clockworksoul/mediawiki@v0.0.5/edit.go:259: func (w *EditClient) AppendText(s string) *EditClient {
github.com/cloudfly/alarmd@v0.0.0-20191209053936-d369a3264630/pkg/events/channel.go:61: func (spec ChannelSpec) AppendText(data map[string]interface{}) string {
github.com/cloudfly/ecenter@v0.0.0-20191220053526-dfd69a52662f/pkg/events/channel.go:61: func (spec ChannelSpec) AppendText(data map[string]interface{}) string {
github.com/cloudradar-monitoring/frontman@v0.0.0-20210924115943-7c61842a19b8/pkg/winui/winui_logview.go:64: func (lv *LogView) AppendText(value string) {
github.com/coyim/gotk3adapter@v0.0.2/gtk_mock/combo_box_text.go:7: func (*MockComboBoxText) AppendText(v1 string) {
github.com/coyim/gotk3adapter@v0.0.2/gtka/combo_box_text.go:31: func (v *comboBoxText) AppendText(v1 string) {
github.com/coyim/gotk3mocks@v0.0.2/gtk/combo_box_text.go:7: func (m *MockComboBoxText) AppendText(v1 string) {
github.com/d2r2/go-gtk@v0.0.0-20131218031741-e35a6dfb5a85/gtk/gtk.go:5223: func (v *ComboBox) AppendText(text string) {
github.com/d2r2/go-gtk@v0.0.0-20131218031741-e35a6dfb5a85/gtk/gtk.go:5309: func (v *ComboBoxText) AppendText(text string) {
github.com/d2r2/gotk3@v0.0.0-20211023153135-0ceb344f1671/gtk/combobox.go:240: func (v *ComboBoxText) AppendText(text string) {
github.com/davidli2010/gobson_exp@v0.0.0-20161004043538-c651a288efad/bson/bsonarraybuilder.go:103: func (a *BsonArrayBuilder) AppendBinary(value Binary) *BsonArrayBuilder {
github.com/davidli2010/gobson_exp@v0.0.0-20161004043538-c651a288efad/bson/bsonbuilder.go:183: func (b *BsonBuilder) AppendBinary(name string, value Binary) *BsonBuilder {
github.com/diamondburned/gotk4/pkg@v0.0.5/gtk/v3/gtkcomboboxtext.go:235: func (comboBox *ComboBoxText) AppendText(text string) {
github.com/diamondburned/gotk4/pkg@v0.0.5/gtk/v4/gtkcomboboxtext.go:213: func (comboBox *ComboBoxText) AppendText(text string) {
github.com/dimansk12/libxml2@v1.0.0/dom/node_element.go:62: func (n *Element) AppendText(s string) error {
github.com/dontpanic92/wxGo@v0.0.0-20181107055819-fbf544604121/wx/wx_darwin.go:84776: func (arg1 SwigClassTextEntry) AppendText(arg2 string) {
github.com/dontpanic92/wxGo@v0.0.0-20181107055819-fbf544604121/wx/wx_darwin.go:245455: func (arg1 SwigClassStyledTextCtrl) AppendText(arg2 string) {
github.com/dontpanic92/wxGo@v0.0.0-20181107055819-fbf544604121/wx/wx_darwin.go:290453: func (arg1 SwigClassRichTextCtrl) AppendText(arg2 string) {
github.com/dontpanic92/wxGo@v0.0.0-20181107055819-fbf544604121/wx/wx_linux.go:84736: func (arg1 SwigClassTextEntry) AppendText(arg2 string) {
github.com/dontpanic92/wxGo@v0.0.0-20181107055819-fbf544604121/wx/wx_linux.go:245543: func (arg1 SwigClassStyledTextCtrl) AppendText(arg2 string) {
github.com/dontpanic92/wxGo@v0.0.0-20181107055819-fbf544604121/wx/wx_linux.go:288912: func (arg1 SwigClassRichTextCtrl) AppendText(arg2 string) {
github.com/dontpanic92/wxGo@v0.0.0-20181107055819-fbf544604121/wx/wx_windows.go:85111: func (arg1 SwigClassTextEntry) AppendText(arg2 string) {
github.com/dontpanic92/wxGo@v0.0.0-20181107055819-fbf544604121/wx/wx_windows.go:246501: func (arg1 SwigClassStyledTextCtrl) AppendText(arg2 string) {
github.com/dontpanic92/wxGo@v0.0.0-20181107055819-fbf544604121/wx/wx_windows.go:289885: func (arg1 SwigClassRichTextCtrl) AppendText(arg2 string) {
github.com/ebitenui/ebitenui@v0.5.4/widget/textarea.go:336: func (l *TextArea) AppendText(value string) {
github.com/electricface/go-gir@v0.0.0-20220119144140-f2fd0857a7da/gtk-3.0/gtk_auto.go:15477: func (v ComboBoxText) AppendText(text string) {
github.com/friendstechday/codeDocToMarkdown@v0.0.0-20150523005008-fc62c7cac2de/converter/java/javaDoc/comment.go:11: func (c *Comment) AppendText(text string) {
github.com/friendstechday/codeDocToMarkdown@v0.0.0-20150523005008-fc62c7cac2de/converter/java/javaDoc/param.go:13: func (p *Param) AppendText(text string) {     
github.com/gangming/walk@v0.0.0-20220825033616-0c0e321abef9/textedit.go:240: func (te *TextEdit) AppendText(value string) {
github.com/gangming/walk@v0.0.0-20220825033616-0c0e321abef9/examples/logview/logview.go:55: func (lv *LogView) AppendText(value string) {
github.com/gimalay/binx@v0.0.0-20200623152039-84df950e8e35/binx.go:78: func (e *page) AppendBinary(data []byte) (bool, error) {
github.com/gimalay/binx@v0.0.0-20200623152039-84df950e8e35/binx.go:94: func (c *Count) AppendBinary([]byte) error {
github.com/gimalay/binx@v0.0.0-20200623152039-84df950e8e35/binx_test.go:37: func (e *indexable) AppendBinary(data []byte) (bool, error)  { return false, json.Unmarshal(data, e) }
github.com/gimalay/binx@v0.0.0-20200623152039-84df950e8e35/binx_test.go:52: func (e *indexableSlice) AppendBinary(data []byte) (bool, error) {
github.com/godot-go/godot-go@v0.0.0-20230312082217-9d5a86402ae8/pkg/gdextension/classes.gen.go:341178: func (cx *RichTextLabelImpl) AppendText(
github.com/gofaith/walk@v0.0.0-20200803081117-681a3effcc77/textedit.go:240: func (te *TextEdit) AppendText(value string) {
github.com/gofaith/walk@v0.0.0-20200803081117-681a3effcc77/examples/logview/logview.go:55: func (lv *LogView) AppendText(value string) {
github.com/goki/gi@v1.3.19/giv/textbuf.go:707: func (tb *TextBuf) AppendText(text []byte, signal bool) *textbuf.Edit {
github.com/gontikr99/bidbot2@v0.0.0-20190924173835-de946e3c6c1e/controller/gui/logview.go:60: func (lv *LogView) AppendText(value string) {
github.com/gotk3/gotk3@v0.6.2/gtk/combo_box.go:364: func (v *ComboBoxText) AppendText(text string) {
github.com/hegeng1212/pholcus@v0.0.0-20230627120450-78db5a490cf2/gui/logview.go:59: func (lv *LogView) AppendText(value string) {
github.com/henrylee2cn/pholcus@v1.3.4/gui/logview.go:59: func (lv *LogView) AppendText(value string) {
github.com/ianatha/walk@v0.0.0-20200225051714-2c8e61d8d5e8/textedit.go:240: func (te *TextEdit) AppendText(value string) {
github.com/ianatha/walk@v0.0.0-20200225051714-2c8e61d8d5e8/examples/logview/logview.go:55: func (lv *LogView) AppendText(value string) {
github.com/icowan/walk@v0.0.0-20201223064526-8326e90d8f36/textedit.go:240: func (te *TextEdit) AppendText(value string) {
github.com/icowan/walk@v0.0.0-20201223064526-8326e90d8f36/examples/logview/logview.go:55: func (lv *LogView) AppendText(value string) {
github.com/ingmardrewing/gomic@v0.0.0-20170621162852-b6f2c68bbe53/fs/html.go:116: func (n *htmlNode) AppendText(txt string) node {
github.com/ionous/sashimi@v0.0.0-20160915000703-9034458fa291/metal/lists.go:36: func (ar arrayValues) AppendText(v string) error {
github.com/ionous/sashimi@v0.0.0-20160915000703-9034458fa291/metal/panicValue.go:31: func (p panicValue) AppendText(string) error {
github.com/ionous/sashimi@v0.0.0-20160915000703-9034458fa291/runtime/internal/gameList.go:126: func (l *gameList) AppendText(v string) {
github.com/ionous/sashimi@v0.0.0-20160915000703-9034458fa291/runtime/internal/nullList.go:13: func (n nullList) AppendText(string)         {}
github.com/iquanxin/walk@v0.0.0-20230418091138-4b9082def936/textedit.go:241: func (te *TextEdit) AppendText(value string) {
github.com/jannson/walk@v0.0.0-20221010063135-328f21216f11/textedit.go:240: func (te *TextEdit) AppendText(value string) {
github.com/jannson/walk@v0.0.0-20221010063135-328f21216f11/examples/logview/logview.go:55: func (lv *LogView) AppendText(value string) {
github.com/jason-wj/pholcus@v0.0.0-20230628053201-2b89c876f2ed/gui/logview.go:59: func (lv *LogView) AppendText(value string) {
github.com/joeirimpan/libxml2@v1.0.4/dom/node_element.go:62: func (n *Element) AppendText(s string) error {
github.com/johnbartholomew/go-gtk@v0.0.0-20130206210405-e86225ead39e/gtk/gtk.go:4955: func (v *ComboBox) AppendText(text string) {
github.com/johnbartholomew/go-gtk@v0.0.0-20130206210405-e86225ead39e/gtk/gtk.go:5041: func (v *ComboBoxText) AppendText(text string) {
github.com/jopbrown/gtk-sugar@v0.0.0-20191124065551-9a86599f5831/lib/gtk/combo.go:42: func (obj *ComboBoxText) AppendText(text string) {
github.com/jopbrown/gtk-sugar@v0.0.0-20191124065551-9a86599f5831/lib/gtk/combo.gtk2.go:10: func (obj *ComboBox) AppendText(text string) {
github.com/justinyaoqi/pholcus@v0.0.0-20221204035437-fbc1b0707612/gui/logview.go:65: func (lv *LogView) AppendText(value string) {
github.com/jzs/walk@v0.0.0-20181112095039-c7d36e6fcacf/textedit.go:147: func (te *TextEdit) AppendText(value string) {
github.com/jzs/walk@v0.0.0-20181112095039-c7d36e6fcacf/examples/logview/logview.go:63: func (lv *LogView) AppendText(value string) {
github.com/karashiiro/kartlobby@v0.0.0-20221019160947-d0596f565030/pkg/colortext/builder.go:11: func (b *ColorTextBuilder) AppendText(text string) *ColorTextBuilder {
github.com/kckrinke/go-gtk@v0.0.0-20180506212049-4cde83b81e6e/gtk/gtk.go:6126: func (v *ComboBox) AppendText(text string) {
github.com/kckrinke/go-gtk@v0.0.0-20180506212049-4cde83b81e6e/gtk/gtk.go:6221: func (v *ComboBoxText) AppendText(text string) {
github.com/ktye/ui@v0.0.0-20210211114936-94158a1502e8/editor/edit.go:47: func (e *Edit) AppendText(s string) {
github.com/kumakichi/walk@v0.0.0-20150122092410-7e83dfcca18a/textedit.go:94: func (te *TextEdit) AppendText(value string) {
github.com/kumakichi/walk@v0.0.0-20150122092410-7e83dfcca18a/examples/logview/logview.go:63: func (lv *LogView) AppendText(value string) {
github.com/kvark128/walk@v0.0.0-20230207150301-925603f55833/textedit.go:240: func (te *TextEdit) AppendText(value string) {
github.com/kvark128/walk@v0.0.0-20230207150301-925603f55833/examples/logview/logview.go:55: func (lv *LogView) AppendText(value string) {
github.com/lazyshot/gotk3@v0.0.0-20150809002019-08a6d4f8db4c/gtk/gtk.go:2629: func (v *ComboBoxText) AppendText(text string) {
github.com/leaanthony/go-gtk@v0.0.0-20211127021733-72e2f9c2edc1/gtk/gtk.go:6189: func (v *ComboBox) AppendText(text string) {
github.com/leaanthony/go-gtk@v0.0.0-20211127021733-72e2f9c2edc1/gtk/gtk.go:6284: func (v *ComboBoxText) AppendText(text string) {
github.com/lestrrat/go-libxml2@v0.0.0-20180221004755-bb78334e2019/dom/node_element.go:62: func (n *Element) AppendText(s string) error {
github.com/lestrrat-go/libxml2@v0.0.0-20210328064523-a6d61dc4e799/dom/node_element.go:64: func (n *Element) AppendText(s string) error {
github.com/liwh011/gonebot@v0.2.1/message.go:44: func (m *Message) AppendText(t string) {
github.com/loyalpartner/go-nigori@v0.0.5/stream.go:31: func (n *nigoriStream) AppendText(str string) {
github.com/lufuhu/gotk3@v0.6.3/gtk/combo_box.go:364: func (v *ComboBoxText) AppendText(text string) {
github.com/lxn/walk@v0.0.0-20210112085537-c389da54e794/textedit.go:240: func (te *TextEdit) AppendText(value string) {
github.com/lxn/walk@v0.0.0-20210112085537-c389da54e794/examples/logview/logview.go:55: func (lv *LogView) AppendText(value string) {
github.com/mad-day/bsonbox@v0.0.0-20211002061631-2f125da2d253/bsoncore/bson_arraybuilder.go:94: func (a *ArrayBuilder) AppendBinary(subtype byte, b []byte) *ArrayBuilder {
github.com/mad-day/bsonbox@v0.0.0-20211002061631-2f125da2d253/bsoncore/bson_documentbuilder.go:82: func (db *DocumentBuilder) AppendBinary(key string, subtype byte, b []byte) *DocumentBuilder {
github.com/marcusolsson/passtray@v0.0.0-20150813080859-7d312850bdc9/Godeps/_workspace/src/github.com/mattn/go-gtk/gtk/gtk.go:5310: func (v *ComboBox) AppendText(text string) {
github.com/marcusolsson/passtray@v0.0.0-20150813080859-7d312850bdc9/Godeps/_workspace/src/github.com/mattn/go-gtk/gtk/gtk.go:5396: func (v *ComboBoxText) AppendText(text string) {
github.com/mardukbp/atk@v1.0.3/tk/text.go:410: func (w *Text) AppendText(text string) error {
github.com/mattn/go-gtk@v0.0.0-20230419095350-e099c1bf7abc/gtk/gtk.go:6188: func (v *ComboBox) AppendText(text string) {
github.com/mattn/go-gtk@v0.0.0-20230419095350-e099c1bf7abc/gtk/gtk.go:6283: func (v *ComboBoxText) AppendText(text string) {
github.com/maurofran/hamcrest4go@v0.2.2/matcher/description.go:70: func (d Description) AppendText(text string) {
github.com/mjm/mpsanity@v0.0.0-20211020031355-e32cecb0bc89/block/builder.go:96: func (b *Builder) AppendText(text string) {
github.com/mrbear258/gotk3@v1.0.0/gtk/combo_box.go:364: func (v *ComboBoxText) AppendText(text string) {
github.com/muumlover/walk@v0.0.0-20181212150522-fb9fbbcc6391/textedit.go:151: func (te *TextEdit) AppendText(value string) {
github.com/muumlover/walk@v0.0.0-20181212150522-fb9fbbcc6391/examples/logview/logview.go:63: func (lv *LogView) AppendText(value string) {
github.com/natemaia/gotk3@v0.6.4/gtk/combo_box.go:364: func (v *ComboBoxText) AppendText(text string) {
github.com/norisatir/go-gtk3@v0.0.0-20120410091437-f6c2ba810f40/gtk3/gtk3.go:11356: func (self *ComboBoxText) AppendText(text string) {
github.com/novln/soba@v0.0.0-20201126173112-c6f1bc371303/encoder_test.go:140: func (encoder *TestEncoder) AppendBinary(value []byte) {}
github.com/novln/soba@v0.0.0-20201126173112-c6f1bc371303/encoder/json/types.go:434: func (encoder *Encoder) AppendBinary(value []byte) {
github.com/oldkingnearby/walk@v1.0.0/textedit.go:240: func (te *TextEdit) AppendText(value string) {
github.com/oldkingnearby/walk@v1.0.0/examples/logview/logview.go:55: func (lv *LogView) AppendText(value string) {
github.com/pauldub/go-gtk3@v0.0.0-20120812130113-4604e9480944/gtk3/gtk3.go:11358: func (self *ComboBoxText) AppendText(text string) {
github.com/pekim/gobbi@v0.1.1/lib/gtk/v-2.24.go:16816: func (recv *ComboBoxText) AppendText(text string) {
github.com/pekim/gobbi@v0.1.1/lib/gtk/v-3.0.go:18683: func (recv *ComboBoxText) AppendText(text string) {
github.com/pekim/gobbi@v0.1.1/lib/gtk/v-3.10.go:19473: func (recv *ComboBoxText) AppendText(text string) {
github.com/pekim/gobbi@v0.1.1/lib/gtk/v-3.12.go:19619: func (recv *ComboBoxText) AppendText(text string) {
github.com/pekim/gobbi@v0.1.1/lib/gtk/v-3.14.go:19877: func (recv *ComboBoxText) AppendText(text string) {
github.com/pekim/gobbi@v0.1.1/lib/gtk/v-3.16.go:19976: func (recv *ComboBoxText) AppendText(text string) {
github.com/pekim/gobbi@v0.1.1/lib/gtk/v-3.18.go:20013: func (recv *ComboBoxText) AppendText(text string) {
github.com/pekim/gobbi@v0.1.1/lib/gtk/v-3.2.go:18901: func (recv *ComboBoxText) AppendText(text string) {
github.com/pekim/gobbi@v0.1.1/lib/gtk/v-3.20.go:20182: func (recv *ComboBoxText) AppendText(text string) {
github.com/pekim/gobbi@v0.1.1/lib/gtk/v-3.22.26.go:20199: func (recv *ComboBoxText) AppendText(text string) {
github.com/pekim/gobbi@v0.1.1/lib/gtk/v-3.22.29.go:20217: func (recv *ComboBoxText) AppendText(text string) {
github.com/pekim/gobbi@v0.1.1/lib/gtk/v-3.22.6.go:20190: func (recv *ComboBoxText) AppendText(text string) {
github.com/pekim/gobbi@v0.1.1/lib/gtk/v-3.22.go:20190: func (recv *ComboBoxText) AppendText(text string) {
github.com/pekim/gobbi@v0.1.1/lib/gtk/v-3.4.go:19167: func (recv *ComboBoxText) AppendText(text string) {
github.com/pekim/gobbi@v0.1.1/lib/gtk/v-3.6.go:19278: func (recv *ComboBoxText) AppendText(text string) {
github.com/pekim/gobbi@v0.1.1/lib/gtk/v-3.8.go:19293: func (recv *ComboBoxText) AppendText(text string) {
github.com/pietroglyph/walk@v0.0.0-20180728074727-d9c93ed8784b/textedit.go:147: func (te *TextEdit) AppendText(value string) {
github.com/pietroglyph/walk@v0.0.0-20180728074727-d9c93ed8784b/examples/logview/logview.go:63: func (lv *LogView) AppendText(value string) {
github.com/pirogom/walk@v0.0.0-20230128105705-492944454fc3/textedit.go:241: func (te *TextEdit) AppendText(value string) {
github.com/planetscale/log@v0.0.0-20230320220358-9d4ed51abd8d/encoder.go:195: func (enc *prettyEncoder) AppendBinary(value []byte) {
github.com/pritunl/mongo-go-driver@v1.11.3/x/bsonx/bsoncore/bson_arraybuilder.go:94: func (a *ArrayBuilder) AppendBinary(subtype byte, b []byte) *ArrayBuilder {
github.com/pritunl/mongo-go-driver@v1.11.3/x/bsonx/bsoncore/bson_documentbuilder.go:82: func (db *DocumentBuilder) AppendBinary(key string, subtype byte, b []byte) *DocumentBuilder {
github.com/pwiecz/atk@v0.0.0-20210227100443-cb0bf65e43e2/tk/text.go:410: func (w *Text) AppendText(text string) error {
github.com/pzartem/libxml2@v0.0.0-20200831082948-c3d9892b3a0f/dom/node_element.go:62: func (n *Element) AppendText(s string) error {
github.com/qsirwyk/walk@v1.1.1/textedit.go:240: func (te *TextEdit) AppendText(value string) {
github.com/qsirwyk/walk@v1.1.1/examples/logview/logview.go:55: func (lv *LogView) AppendText(value string) {
github.com/raceresult/walk@v0.0.4/textedit.go:240: func (te *TextEdit) AppendText(value string) {
github.com/readykit/gd@v0.0.0-20221015225739-25a193a18dd7/api.go:25532: func (gdClass RichTextLabel) AppendText(bbcode string)  { methodCall[struct{}](gdClass.obj.get(), methodRichTextLabel[76], &bbcode) }
github.com/reusee/ggir@v0.0.0-20141013104036-88ce4855c267/gtk/gtk_traits.go:5592: func (self *TraitComboBoxText) AppendText(text string) {
github.com/richardwilkes/cef@v0.5.3/cef/Textfield_gen.go:111: func (d *Textfield) AppendText(text string) {
github.com/romychs/gotk3@v0.0.0-20210328092339-116e09e25f5e/gtk/combobox.go:240: func (v *ComboBoxText) AppendText(text string) {
github.com/rstshardware/walk@v0.0.1/textedit.go:241: func (te *TextEdit) AppendText(value string) {
github.com/rstshardware/walk@v0.0.1/examples/logview/logview.go:53: func (lv *LogView) AppendText(value string) {
github.com/scjtqs2/walk@v0.0.0-20211230061331-3545c43f16e8/textedit.go:240: func (te *TextEdit) AppendText(value string) {
github.com/scjtqs2/walk@v0.0.0-20211230061331-3545c43f16e8/examples/logview/logview.go:55: func (lv *LogView) AppendText(value string) {
github.com/tHinqa/outside-gtk2@v0.0.0-20131129203352-f05e200c0477/gtk/gtk_c.go:1059: func (c *ComboBox) AppendText(text string) { ComboBoxAppendText(c, text) }
github.com/tHinqa/outside-gtk2@v0.0.0-20131129203352-f05e200c0477/gtk/gtk_c.go:1109: func (c *ComboBoxText) AppendText(text string) { ComboBoxTextAppendText(c, text) }
github.com/tHinqa/outside-gtk2@v0.0.0-20131129203352-f05e200c0477/gtk/gtk_e.go:194: func (e *Entry) AppendText(text string)            { EntryAppendText(e, text) }
github.com/tailscale/walk@v0.0.0-20230502163740-ea8725bca6d8/textedit.go:240: func (te *TextEdit) AppendText(value string) {
github.com/tailscale/walk@v0.0.0-20230502163740-ea8725bca6d8/examples/logview/logview.go:55: func (lv *LogView) AppendText(value string) {
github.com/tiborvass/uniline@v0.0.0-20150528164013-b39ee5b15151/utils.go:58: func (t text) AppendText(n text) text {
github.com/timelessnesses/l4d2-fmsa-tk@v0.0.0-20230302131554-ce07a2f78b70/tk/text.go:410: func (w *Text) AppendText(text string) error {
github.com/tmaxmax/go-sse@v0.4.2/message.go:98: func (e *Message) AppendText(chunks ...string) {
github.com/turutcrane/cefingo@v0.5.9/capi/cefingo_gen.go:27863: func (self *CTextfieldT) AppendText(
github.com/vikulin/walk@v0.0.0-20210329133517-4f03df2e225a/textedit.go:240: func (te *TextEdit) AppendText(value string) {
github.com/vikulin/walk@v0.0.0-20210329133517-4f03df2e225a/examples/logview/logview.go:55: func (lv *LogView) AppendText(value string) {
github.com/visualfc/atk@v1.2.3/tk/text.go:410: func (w *Text) AppendText(text string) error {
github.com/wangch/walk@v0.0.0-20150802023425-cdcaffe9ed2d/textedit.go:94: func (te *TextEdit) AppendText(value string) {
github.com/wangch/walk@v0.0.0-20150802023425-cdcaffe9ed2d/examples/logview/logview.go:63: func (lv *LogView) AppendText(value string) {
github.com/wayf-dk/go-libxml2@v0.0.0-20210718205101-95691c57c1ac/dom/node_element.go:65: func (n *Element) AppendText(s string) error {
github.com/weili71/go-filex@v0.0.0-20230321132739-7a825b3f5691/filex.go:200: func (f *Filex) AppendText(file *os.File, text string) (int, error) {
github.com/whtiehack/wingui@v0.0.6/edit.go:50: func (e *Edit) AppendText(value string) {
github.com/winxxp/walk@v0.0.0-20200901104446-3e7ddcbafc7d/textedit.go:240: func (te *TextEdit) AppendText(value string) {
github.com/winxxp/walk@v0.0.0-20200901104446-3e7ddcbafc7d/examples/logview/logview.go:55: func (lv *LogView) AppendText(value string) {
github.com/xlplbo/walk@v0.0.0-20201127063646-ea1cc4883b4d/textedit.go:240: func (te *TextEdit) AppendText(value string) {
github.com/xlplbo/walk@v0.0.0-20201127063646-ea1cc4883b4d/examples/logview/logview.go:38: func (lv *LogView) AppendText(value string) {
github.com/yamnikov-oleg/go-gtk@v0.0.0-20160312170510-bc91c0fbef0c/gtk/gtk.go:5343: func (v *ComboBox) AppendText(text string) {
github.com/yamnikov-oleg/go-gtk@v0.0.0-20160312170510-bc91c0fbef0c/gtk/gtk.go:5430: func (v *ComboBoxText) AppendText(text string) {
github.com/zhangweiii/pholcus@v0.0.0-20200228101341-71bf9bad259b/gui/logview.go:59: func (lv *LogView) AppendText(value string) {
github.com/zhangyiming748/sendEmailAlert@v0.0.14/alert.go:67: func (i *Info) AppendText(s string) {
github.com/zurek87/go-gtk3@v0.0.0-20171203223943-bec742635bb7/gtk3/gtk3.go:5867: func (v *ComboBoxText) AppendText(text string) {
github.com/zxjsdp/specimen-go@v1.9.4/gui/windows/logview.go:63: func (lv *LogView) AppendText(value string) {
gitlab.com/abibino-lab/mobal@v0.0.0-20170907170701-2d91a9572e70/gui/logview.go:59: func (lv *LogView) AppendText(value string) {
gitlab.com/gitlab-org/security-products/analyzers/common/table@v0.0.0-20181219150649-131c639c2176/table.go:30: func (t *Table) AppendText(text string) {
gitlab.com/gitlab-org/security-products/analyzers/common/table/v2@v2.0.0-20201123204353-3743d5d8b4cc/table.go:30: func (t *Table) AppendText(text string) {
go.mongodb.org/mongo-driver@v1.12.0/x/bsonx/bsoncore/bson_arraybuilder.go:94: func (a *ArrayBuilder) AppendBinary(subtype byte, b []byte) *ArrayBuilder {
go.mongodb.org/mongo-driver@v1.12.0/x/bsonx/bsoncore/bson_documentbuilder.go:82: func (db *DocumentBuilder) AppendBinary(key string, subtype byte, b []byte) *DocumentBuilder {
modernc.org/atk@v1.4.3/text.go:410: func (w *Text) AppendText(text string) error {

@mateusz834
Copy link
Member

This is motivated by netip.Addr.MarshalText being a non-trivial amount of slowdown in a large JSON object.

Just so you now there is netip.Addr.AppendTo, that does the same thing as MarshalText, but appends.

@dsnet
Copy link
Member Author

dsnet commented Aug 31, 2023

There are unfortunately two problems with AppendTo:

  • It doesn't return an error, and so can't implement equivalent semantics as MarshalText
  • It's naming is ambiguous with regard to whether it does "binary" or "textual" marshaling

@mateusz834
Copy link
Member

@dsnet Yes I agree, just wanted to point that there will be now two methods that do the same thing on netip.Addr, probably after this proposal gets accepted the AppendTo should be deprecated.

@mvdan
Copy link
Member

mvdan commented Aug 31, 2023

Scratching my head and wondering how this hasn't been proposed before :) sounds great to me.

@flimzy
Copy link
Contributor

flimzy commented Aug 31, 2023 •
edited

What are the possible security implications wrt:

// Implementations must not retain b, nor mutate any bytes within b[:len(b)].

I can imagine a malicious, or sloppy, library not following this rule, and introducing nastiness.

Given the other vectors for "nastiness" that already exist throughout the stdlib when using untrusted code, probably not a meaningful consideration, but it does seem a bit icky to me, to expose some other code's data to arbitrary code this way.

@dsnet
Copy link
Member Author

dsnet commented Aug 31, 2023

Given the other vectors for "nastiness" that already exist throughout the stdlib when using untrusted code, probably not a meaningful consideration, but it does seem a bit icky to me, to expose some other code's data to arbitrary code this way.

Agreed. That's the strength of the MarshalText API. It's much harder to hold wrong.

That said, two of the top-three interfaces in Go by far are io.Reader and io.Writer, both of which expose a buffer to the implementation and document:

Implementations must not retain p.

I don't think we're making the situation any worse since encoding.AppendText will be multiple orders of magnitude less popular than io.Reader or io.Writer.

@seankhliao seankhliao mentioned this issue Sep 1, 2023
Closed
@seebs
Copy link
Contributor

seebs commented Sep 1, 2023

This seems very desireable, the large allocations can be a problem. Although I think in a lot of the cases that are particularly messy, append might not help, because i really need to do an encoder to an io.Writer? But I would still very much like to have this, because iterative loops that are marshaling a bunch of things but only need one at a time are pretty common.

@mateusz834
Copy link
Member

@seebs That is true that something like this would be a nice alternative.

type TextWriter interface {
    WriteText(b io.Writer) error
}

But, this as as side effect would cause another bunch of memory allocations inside each implementation of WriteText, because escape analysis is not great for this use case.

@dsnet
Copy link
Member Author

dsnet commented Sep 1, 2023 •
edited

Any encoder writing eventually to an io.Writer almost certainly have an intermediate buffer on hand that it can pass to AppendText and then pass eventually to io.Writer.Write.

@yerden
Copy link

yerden commented Sep 22, 2023

Upvoting. I cannot imagine any heavily loaded application tinkering with bytes and text that wouldn't implement an interface of similar semantics.

@mitar
Copy link
Contributor

mitar commented Oct 6, 2023

nor mutate any bytes within b[:len(b)].

Why, performance wise, would that not be allowed? Maybe we can open additional use cases if we do allow that?

@dsnet
Copy link
Member Author

dsnet commented Oct 6, 2023

Why, performance wise, would that not be allowed?

For a function named "Append", it just seems highly suspicious for it to mutate data within b[:len(b)]. The verb "append" implies only modifying data within b[len(b):cap(b)].

A performance argument could be made in the reverse. The caller of AppendText could in theory spin off a goroutine to do some form of read-only validation within b[:len(b)]. Mutating that section would result in a race condition.

@mitar
Copy link
Contributor

mitar commented Oct 6, 2023

For a function named "Append", it just seems highly suspicious for it to mutate data within b[:len(b)]. The verb "append" implies only modifying data within b[len(b):cap(b)].

So should we rename it then? Like MutateText(b []byte) ([]byte, error)?

A performance argument could be made in the reverse. The caller of AppendText could in theory spin off a goroutine to do some form of read-only validation within b[:len(b)]. Mutating that section would result in a race condition.

I think this is handled by the "Implementations must not retain b", if you make a goroutine, you are retaining b.

@dsnet
Copy link
Member Author

dsnet commented Oct 6, 2023

So should we rename it then?

I'm arguing that we should keep it named AppendText and also keep the documentation that implementations shouldn't mutate b[:len(b)].

I think this is handled by the "Implementations must not retain b", if you make a goroutine, you are retaining b.

In my example, the asynchronous validation logic is done by the caller of AppendText, not by the implementation of AppendText.

@mitar
Copy link
Contributor

mitar commented Oct 7, 2023

@dsnet Thanks for the explanation!

@cristaloleg cristaloleg mentioned this issue Dec 16, 2023
Merged
@mvdan
Copy link
Member

mvdan commented Apr 2, 2024

@ianlancetaylor could I nudge this proposal forward? It would help with the encoding/json/v2 efforts, and it seems to have reasonable consensus :)

@icholy icholy mentioned this issue Apr 23, 2024
Open
@rsc
Copy link
Contributor

rsc commented Apr 24, 2024

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Performance Proposal
Projects
Proposals
Status: Active
Milestone
Proposal
Development

No branches or pull requests
9 participants
@rsc @seebs @mitar @mvdan @dsnet @yerden @flimzy @gopherbot @mateusz834