proposal: regexp: add CompileSyntax #14078

alexflint · 2016-01-23T17:59:47Z

Currently there is, I believe, no way to turn a syntax.Regex AST into an executable regexp.Regex from outside the regexp package. How about adding

func CompileSyntax(ast *syntax.Regexp) (*Regexp, error)

to the regexp package?

The implementation would be mostly trivial because the first step in regexp.Compile is already to build a syntax.Regexp.

The regexp.Regexp struct does have an expr string field, which could be populated with ast.String() in CompileSyntax.

If people agree that this is a good idea then I'm happy to make the CL.

The text was updated successfully, but these errors were encountered:

ianlancetaylor · 2016-01-23T19:18:05Z

Can you give an example of why somebody might want to do that?

alexflint · 2016-01-24T01:07:28Z

The reason I wanted it is that I'm building regexp ASTs from tagged struct fields. So I look through the fields of a struct and build a syntax.Regexp based on their tags. I then want to execute the AST I've built against some input.

But also it seems that if you're going to expose syntax.Parse, syntax.Compile, and so on in the public API, then isn't it a bit remiss not to provide any way to execute the results?

alexflint · 2016-01-24T17:20:51Z

Another use case would be if you wanted to use a custom syntax.Flag (e.g. if you wanted to use POSIX syntax but allow . to match newlines). Since regexp.Compile does not allow you to specify a syntax.Flag, your only option would be to use syntax.Compile, but then you would end up with a syntax.Regexp and no way to build a regexp.Regexp.

alexflint · 2016-02-04T17:38:58Z

@ianlancetaylor Ping. The immediate use case for this is constructing regexp ASTs from struct tag fields. But I do think this request is more general than my one use case: why have machinery for describing a regexp as an AST, and machinery for executing regular expressions, but no way to bridge the divide?

bradfitz · 2016-02-04T18:23:56Z

why have machinery for describing a regexp as an AST, and machinery for executing regular expressions, but no way to bridge the divide?

Because the regexp/syntax package probably wasn't meant to be public in the first place. Had the internal package mechanism existed at the time Go 1 was released, I think that detail would remain hidden.

As far as I can tell by searching public github code, nobody but @rsc really uses the regexp/syntax package. It's mostly forks of his regmerge code:

alexflint · 2016-02-04T18:43:43Z

I think the reason it makes sense to expose well-defined parse and compile
steps for the regular expression language is the same reason that it makes
sense to expose the scan/parse/compile steps for the go language itself in
the go/ package: although most users just want to "go build ..." their
source, exposing a few key internals allows tools that plug in at lower
levels.

Similarly, most users just want to compile and run their regular expression
from a string, but exposing a few key internals allows tools like this:

https://github.com/alexflint/go-restructure/blob/master/builder.go

On Thu, Feb 4, 2016 at 10:25 AM Brad Fitzpatrick notifications@github.com
wrote:

why have machinery for describing a regexp as an AST, and machinery for
executing regular expressions, but no way to bridge the divide?

Because the regexp/syntax package probably wasn't meant to be public in
the first place. Had the internal package mechanism existed at the time Go
1 was released, I think that detail would remain hidden.

As far as I can tell by searching public github code, nobody but @rsc
https://github.com/rsc really uses the regexp/syntax package. It's
mostly forks of his regmerge code:

https://github.com/0x434D53/gddo/blob/master/gosrc/data.go
https://github.com/0xmohit/hound/blob/master/codesearch/regexp/regexp.go
https://github.com/287008/rsc/blob/master/regexp/regmerge/main.go
https://github.com/287008/rsc/blob/master/regexp/regmerge/utf.go
https://github.com/CarlosGtrz/codesearch/blob/master/regexp/copy.go
https://github.com/CarlosGtrz/codesearch/blob/master/regexp/utf.go

https://github.com/ChanglinZhou/codesearch/blob/master/src/codesearch/regexp/utf.go
https://github.com/Debian/dcs/blob/master/regexp/utf.go

https://github.com/GoChallenge/GCSolutions/blob/master/march15/normal/alex-holden/drum/cmd/drum_machine.go
https://github.com/Gorilla/reverse/blob/master/reverse.go
https://github.com/GreatStuff660/rsc/blob/master/regexp/regmerge/copy.go
https://github.com/GreatStuff660/rsc/blob/master/regexp/regmerge/main.go
https://github.com/GreatStuff660/rsc/blob/master/regexp/regmerge/utf.go
https://github.com/JoergReinhardt/gorilla/blob/master/reverse/reverse.go
https://github.com/JustinHuang917/rsc/blob/master/regexp/regmerge/main.go
https://github.com/LiJiefei/codesearch/blob/master/index/regexp.go
https://github.com/LiJiefei/codesearch/blob/master/regexp/match.go
https://github.com/MARFMS/codesearch/blob/master/regexp/copy.go
https://github.com/MaGrosseGit/rsc/blob/master/regexp/regmerge/copy.go
https://github.com/NgaNguyenDuy/codesearch/blob/master/regexp/utf.go
https://github.com/ReversalFate/rsc/blob/master/regexp/regmerge/copy.go
https://github.com/ReversalFate/rsc/blob/master/regexp/regmerge/utf.go
https://github.com/SDGophers/gddo/blob/master/gosrc/data.go
https://github.com/StephanieMak/codesearch/blob/master/regexp/regexp.go
https://github.com/UlricQin/rsc/blob/master/regexp/regmerge/main.go
https://github.com/achenge07/rsc/blob/master/regexp/regmerge/copy.go
https://github.com/alfeg/rsc/blob/master/regexp/regmerge/main.go
https://github.com/amrhassan/rsc/blob/master/regexp/regmerge/copy.go
https://github.com/amrhassan/rsc/blob/master/regexp/regmerge/utf.go
https://github.com/ancientlore/codesearch/blob/master/regexp/copy.go
https://github.com/ancientlore/codesearch/blob/master/regexp/regexp.go
https://github.com/andaru/codesearch/blob/master/regexp/copy.go
https://github.com/andaru/codesearch/blob/master/regexp/match.go
https://github.com/andrewzeneski/rsc/blob/master/regexp/regmerge/merge.go
https://github.com/arthurdick/rsc/blob/master/regexp/regmerge/merge.go
https://github.com/arthurdick/rsc/blob/master/regexp/regmerge/utf.go
https://github.com/balachandranc/rsc/blob/master/regexp/regmerge/match.go
https://github.com/balachandranc/rsc/blob/master/regexp/regmerge/merge.go
https://github.com/bdarnell/cockroach/blob/master/sql/parser/builtins.go

https://github.com/betfair/opentsp/blob/master/internal/tsdb/filter/filter.go
https://github.com/billzhuang/codesearch/blob/master/index/regexp.go
https://github.com/brtzsnr/brtzsnr-gorilla/blob/master/reverse/reverse.go
https://github.com/bundgaard/rsc/blob/master/regexp/regmerge/main.go
https://github.com/cardamaro/codesearch/blob/master/regexp/regexp.go
https://github.com/chrsmithdemos/codesearch/blob/master/regexp/match.go
https://github.com/chrsmithdemos/codesearch/blob/master/regexp/regexp.go
https://github.com/csaunders/themekit/blob/master/event_filter.go
https://github.com/dleffert/rsc/blob/master/regexp/regmerge/merge.go
https://github.com/dnephin/hound/blob/master/codesearch/regexp/copy.go
https://github.com/dnephin/hound/blob/master/codesearch/regexp/match.go
https://github.com/dnephin/hound/blob/master/codesearch/regexp/utf.go
https://github.com/donlzx/rsc/blob/master/regexp/regmerge/main.go
https://github.com/edusegzy/reflex/blob/master/match.go
https://github.com/erasin/rsc/blob/master/regexp/regmerge/copy.go
https://github.com/etsy/Hound/blob/master/codesearch/regexp/copy.go
https://github.com/etsy/Hound/blob/master/codesearch/regexp/utf.go
https://github.com/f2prateek/hound/blob/master/codesearch/regexp/match.go
https://github.com/flowbywind/hound/blob/master/codesearch/regexp/copy.go
https://github.com/francoishill/rsc/blob/master/regexp/regmerge/copy.go
https://github.com/frankban/juju-testing/blob/master/imports.go
https://github.com/frederich/codesearch/blob/master/index/regexp.go
https://github.com/fsiler/rsc/blob/master/regexp/regmerge/main.go
https://github.com/fuhj/rsc/blob/master/regexp/regmerge/match.go
https://github.com/fuhj/rsc/blob/master/regexp/regmerge/merge.go
https://github.com/guiquanz/goqr/blob/master/regexp/regmerge/match.go
https://github.com/h2so5/goback/blob/master/regexp/syntax/parser.go
https://github.com/h2so5/goback/blob/master/regexp/syntax/regexp.go
https://github.com/honsty/rsc/blob/master/regexp/regmerge/copy.go
https://github.com/ikawaha/regexp/blob/master/internal/factor/analyze.go
https://github.com/jamessan/dcs/blob/master/regexp/utf.go
https://github.com/jeff66ruan/codesearch/blob/master/regexp/regexp.go
https://github.com/jempe/rsc/blob/master/regexp/regmerge/copy.go
https://github.com/jempe/rsc/blob/master/regexp/regmerge/main.go
https://github.com/jempe/rsc/blob/master/regexp/regmerge/utf.go
https://github.com/jmelfi/cockroach/blob/master/sql/parser/builtins.go
https://github.com/jonlai/hound/blob/master/codesearch/regexp/copy.go
https://github.com/jonlai/hound/blob/master/codesearch/regexp/utf.go
https://github.com/junkblocker/codesearch/blob/master/regexp/match.go
https://github.com/junkblocker/codesearch/blob/master/regexp/regexp.go
https://github.com/jwilk/dcs/blob/master/regexp/utf.go
https://github.com/kakaiu/codesearch/blob/master/index/regexp.go
https://github.com/ksmaheshkumar/codesearch/blob/master/regexp/copy.go
https://github.com/ksmaheshkumar/codesearch/blob/master/regexp/match.go
https://github.com/kusora/rsc/blob/master/regexp/regmerge/main.go
https://github.com/kusora/rsc/blob/master/regexp/regmerge/merge.go
https://github.com/lamby/dcs/blob/master/regexp/match.go
https://github.com/magus424/codesearch/blob/master/regexp/copy.go
https://github.com/magus424/codesearch/blob/master/regexp/utf.go
https://github.com/margaery/gddo/blob/master/gosrc/data.go
https://github.com/mars9/kanabe/blob/master/codesearch/regexp/copy.go
https://github.com/mars9/kanabe/blob/master/codesearch/regexp/match.go
https://github.com/mashuai/goji/blob/master/web/regexp_pattern.go

https://github.com/matthewdoerksen-porch/Hound/blob/master/src/hound/codesearch/regexp/copy.go
https://github.com/mattn/codesearch/blob/master/regexp/copy.go
https://github.com/maxlang/cockroach/blob/master/sql/parser/builtins.go
https://github.com/mengzhuo/rsc/blob/master/regexp/regmerge/copy.go
https://github.com/morria/hound/blob/master/codesearch/regexp/copy.go
https://github.com/morria/hound/blob/master/codesearch/regexp/regexp.go
https://github.com/mrKrizz/hound/blob/master/codesearch/regexp/match.go
https://github.com/mrheung/rsc/blob/master/regexp/regmerge/main.go
https://github.com/mukgupta/rsc/blob/master/regexp/regmerge/main.go
https://github.com/mukgupta/rsc/blob/master/regexp/regmerge/utf.go
https://github.com/mvdan/interfacer/blob/master/std.go
https://github.com/nieyy/cockroach/blob/master/sql/parser/builtins.go
https://github.com/nomadli/rsc/blob/master/regexp/regmerge/main.go
https://github.com/nomadli/rsc/blob/master/regexp/regmerge/merge.go
https://github.com/odeke-em/rsc/blob/master/regexp/regmerge/match.go
https://github.com/onealtang/rsc/blob/master/regexp/regmerge/main.go
https://github.com/onealtang/rsc/blob/master/regexp/regmerge/match.go
https://github.com/paulswanson/sumo/blob/master/delta.go
https://github.com/petar/rsc/blob/master/regexp/regmerge/copy.go
https://github.com/petar/rsc/blob/master/regexp/regmerge/merge.go
https://github.com/petar/rsc/blob/master/regexp/regmerge/merge.go
https://github.com/petermattis/codesearch/blob/master/regexp/regexp.go
https://github.com/plumbum/rsc/blob/master/regexp/regmerge/copy.go
https://github.com/plumbum/rsc/blob/master/regexp/regmerge/utf.go
https://github.com/pombredanne/dcs/blob/master/index/regexp.go
https://github.com/rberrelleza/hound/blob/master/codesearch/regexp/utf.go
https://github.com/rjpower/codesearch/blob/master/regexp/regexp.go
https://github.com/rocky/go-fish/blob/master/repl_imports.go
https://github.com/sarathsp06/gddo/blob/master/gosrc/data.go
https://github.com/seveas/codesearch/blob/master/regexp/regexp.go
https://github.com/sevki/rsc/blob/master/regexp/regmerge/utf.go
https://github.com/sirocyl/rsc/blob/master/regexp/regmerge/copy.go
https://github.com/snowsnail/hound/blob/master/codesearch/index/regexp.go
https://github.com/snowsnail/hound/blob/master/codesearch/regexp/match.go
https://github.com/splicers/rsc/blob/master/regexp/regmerge/copy.go
https://github.com/splicers/rsc/blob/master/regexp/regmerge/main.go
https://github.com/splicers/rsc/blob/master/regexp/regmerge/match.go
https://github.com/splicers/rsc/blob/master/regexp/regmerge/merge.go
https://github.com/spuranam/codesearch/blob/master/index/regexp.go
https://github.com/spuranam/codesearch/blob/master/regexp/copy.go
https://github.com/spuranam/rsc/blob/master/regexp/regmerge/main.go
https://github.com/spuranam/rsc/blob/master/regexp/regmerge/merge.go
https://github.com/sriram-srinivasan/gore/blob/master/eval/eval.go
https://github.com/sunsimon/rsc/blob/master/regexp/regmerge/main.go
https://github.com/sunsimon/rsc/blob/master/regexp/regmerge/merge.go
https://github.com/supaplextor/rsc/blob/master/regexp/regmerge/main.go
https://github.com/supaplextor/rsc/blob/master/regexp/regmerge/merge.go
https://github.com/swenson/rsc/blob/master/regexp/regmerge/match.go
https://github.com/taolu0615/rsc/blob/master/regexp/regmerge/main.go
https://github.com/taolu0615/rsc/blob/master/regexp/regmerge/match.go
https://github.com/taolu0615/rsc/blob/master/regexp/regmerge/utf.go
https://github.com/tchssk/goji/blob/master/web/regexp_pattern.go
https://github.com/tka/rsc/blob/master/regexp/regmerge/copy.go
https://github.com/tka/rsc/blob/master/regexp/regmerge/utf.go
https://github.com/toolkits/rsc/blob/master/regexp/regmerge/copy.go
https://github.com/toolkits/rsc/blob/master/regexp/regmerge/match.go
https://github.com/tschottdorf/cockroach/blob/master/sql/parser/builtins.go
https://github.com/tylerl/codesearch/blob/master/regexp/match.go
https://github.com/tylerl/codesearch/blob/master/regexp/regexp.go
https://github.com/vishnuvaradaraj/redwood/blob/master/restring.go
https://github.com/vrajanap/codesearch/blob/master/regexp/copy.go
https://github.com/vsco/goji/blob/master/web/regexp_pattern.go
https://github.com/waigani/testing/blob/master/imports.go
https://github.com/wiless/codesearch/blob/master/regexp/match.go
https://github.com/wiless/codesearch/blob/master/regexp/utf.go
https://github.com/wonktnodi/rsc/blob/master/regexp/regmerge/match.go
https://github.com/wonktnodi/rsc/blob/master/regexp/regmerge/utf.go
https://github.com/yh453926638/rsc/blob/master/regexp/regmerge/copy.go
https://github.com/yh453926638/rsc/blob/master/regexp/regmerge/utf.go
https://github.com/z-ray/rsc/blob/master/regexp/regmerge/copy.go
https://github.com/z-ray/rsc/blob/master/regexp/regmerge/utf.go
https://github.com/zach-klippenstein/goregen/blob/master/regexp_format.go

—
Reply to this email directly or view it on GitHub
#14078 (comment).

rsc · 2016-02-04T20:47:21Z

Sorry, we're not processing proposals right now due to Go 1.6 release work. I will say that this is not the trivial thing it looks like.

robpike · 2016-08-15T23:36:44Z

The regexp/syntax package is really an internal package that appeared before the official internal mechanism launched. Changes for outside users are therefore strongly discouraged.

ianlancetaylor changed the title ~~proposal to add regex.CompileSyntax~~ regexp: proposal to add regex.CompileSyntax Jan 23, 2016

ianlancetaylor added this to the Unplanned milestone Jan 23, 2016

rsc changed the title ~~regexp: proposal to add regex.CompileSyntax~~ proposal: regexp: add CompileSyntax Feb 4, 2016

rsc modified the milestones: Proposal, Unplanned Feb 4, 2016

rsc added the Proposal label Feb 4, 2016

robpike closed this as completed Aug 15, 2016

robpike added the Proposal-Declined label Aug 15, 2016

jimmyfrasche mentioned this issue Feb 2, 2017

regexp/syntax: Enable hand crafted regexp/syntax.Regexp being compiled into a regexp.Regex #18888

Closed

golang locked and limited conversation to collaborators Aug 15, 2017

gopherbot added the FrozenDueToAge label Aug 15, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

proposal: regexp: add CompileSyntax #14078

proposal: regexp: add CompileSyntax #14078

alexflint commented Jan 23, 2016

ianlancetaylor commented Jan 23, 2016

alexflint commented Jan 24, 2016

alexflint commented Jan 24, 2016

alexflint commented Feb 4, 2016

bradfitz commented Feb 4, 2016

alexflint commented Feb 4, 2016

rsc commented Feb 4, 2016

robpike commented Aug 15, 2016

proposal: regexp: add CompileSyntax #14078

proposal: regexp: add CompileSyntax #14078

Comments

alexflint commented Jan 23, 2016

ianlancetaylor commented Jan 23, 2016

alexflint commented Jan 24, 2016

alexflint commented Jan 24, 2016

alexflint commented Feb 4, 2016

bradfitz commented Feb 4, 2016

alexflint commented Feb 4, 2016

rsc commented Feb 4, 2016

robpike commented Aug 15, 2016