Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: x/tools/go/packages: Provide an AST loader for single binary applications #52974

Closed
thomaspeugeot opened this issue May 18, 2022 · 6 comments

Comments

@thomaspeugeot
Copy link

thomaspeugeot commented May 18, 2022

This request describes a need for a new function that provides the same results as the Load() function of the golang.org/x/tools/go/packages package but that allows embedding in a single binary.

This request is based on end-user use cases with UML diagrams, on an implementation of UML diagrams in go that uses the packages.Load() function and on the description of the current limitations of this Load() function.

Uses cases for using UML diagrams at run time

UML diagrams are useful artifacts for documenting code but they can be also useful at runtime as it is shown here with both structural and state UML diagrams.

For UML structural diagram, 2 use cases can be met at run time:

  1. Visualizing the UML structural data model to understand how data is stored and the relationships between data items
  2. Using the UML structural diagram to pinpoint class shapes and access the corresponding table

The following movie illustrates the use case with a toy plane simulation. At time stamp 10'', the end-user opens the UML diagrams describing the simulation model (use case #1) and then he checks the values of the Liner agent with clicking on the Liner class shape (use case #2).

https://vimeo.com/709811836

For UML state diagrams, there is one other case that can be useful when developing simulations of protocols between devices that are statefull agents. In this case, it is interesting to to check the state of each agent during the simulation. The following movie shows such a use case. It is a toy laundromat simulation. The UX provides insight into the state of the washer and the state of the machine.

https://vimeo.com/709811805

Persistence of UML diagrams in go code

In the above use cases, UML diagrams are integral parts of applications developed with the gong framework, a go+angular full stack framework for developing applications for system engineers.

With gong, UML diagrams such a the simple one below are marshaled as go code.

image

var Test uml.Classdiagram = uml.Classdiagram{
	Classshapes: []*uml.Classshape{
		{
			Struct: &(models.Country{}),
			Position: &uml.Position{
				X: 20.000000,
				Y: 20.000000,
			},
			Width:  240.000000,
			Heigth: 78.000000,
			Links: []*uml.Link{
				{
					Field: models.Country{}.Hello,
					Middlevertice: &uml.Vertice{
						X: 380.000000,
						Y: 129.000000,
					},
					TargetMultiplicity: "0..1",
					SourceMultiplicity: "*",
				},
			},
			Fields: []*uml.Field{
				{
					Field: models.Country{}.Name,
				},
			},
		},
[...]

Load of the UML diagram with the packages.Load() go function

In a gong diagram, every shape references a go element such a struct or a field via a a zero value of the struct of the field of the actual go code.

[...]
			Struct: &(models.Country{}),
[...]		
			                Field: models.Country{}.Hello,
[...]				
                                         Field: models.Country{}.Name,

This solution brings two benefits:

  • go compilation provides good confidence that the UML diagram is in sync with the go models
  • it allows the go refactoring tool (gopls) to refactor the UML diagram while refactoring the code (for renaming a struct or a field).

At run time, an Unmarshall function loads the go code by using the packages.Load() function and then parses the Abstract Syntax Tree to find which UML shapes references which go struct or field

Limitations of the current packages.Load function

In system engineering, it is interesting to ship simulation binaries to interested stakeholders. This is where go is interesting because one great advantage of go is the single binary feature. However, the above simulations cannot be considered single binary for two limitations.

packages.Load makes a system call to the go compiler

In order to be executed, uses cases described above cannot be executed on a machine where go is not installed. Indeed, the packages.Load function makes a system call to the go compiler.

package.Load cannot support embed directories

In order to be executed, uses cases described above needs the go files describing the UML diagram as well as the go model both accessible in local directories. This is because, unless mistaken, one cannot use the embed.FS type in the Dir field of the packages.Config struct that is passed as a parameter to the packages.Load function.

Expected feature

A packages.<name to be defined>() function that does not makes a system call to the go compiler and that allows embedded go files and that returns the AST.

@gopherbot gopherbot added this to the Proposal milestone May 18, 2022
@seankhliao seankhliao changed the title proposal: golang.org/x/tools/go/packages : Provide an AST loader for single binary applications proposal: x/tools/go/packages: Provide an AST loader for single binary applications May 18, 2022
@ianlancetaylor
Copy link
Contributor

CC @taking @alandonovan

@ianlancetaylor ianlancetaylor added this to Incoming in Proposals (old) May 18, 2022
@mvdan
Copy link
Member

mvdan commented May 18, 2022

If you mean that packages.Load as it exists today should work without requiring Go to be installed, that generally seems like a bad idea to me. You could potentially do it by embedding the entire Go toolchain and standard library into the binary (hundreds of megabytes!), to then extract it in a temporary directory and use it.

A simpler alternative, if you only need ASTs, is https://pkg.go.dev/go/parser#ParseDir. That does require you to have the Go package already on the filesystem somewhere, but it then means you don't need go/packages at all.

@thomaspeugeot
Copy link
Author

thomaspeugeot commented May 19, 2022

@mvdan , thks for the feedback. I retract the proposal.

You could potentially do it by embedding the entire Go toolchain and standard library into the binary (hundreds of megabytes!), to then extract it in a temporary directory and use it.

That would be doable indeed but complex to implement from my perspective. My idea of the go standard library is to provide functions with most complexity hidden from the programmer. BTW : indicating in the package.Load() documentation that the go tool need to be installed in the execution environnement would be a plus in my opinion.

A simpler alternative, if you only need ASTs, is https://pkg.go.dev/go/parser#ParseDir.

Thank-you for pointing this but the example above needs a bit more than the results of the of parser.ParseDir() function.

Field: models.Country{}.Hello,

In order to construe the link between the Country shape and the Hello shape, the UML diagram unmarshaller needs to know that the type of the Hello field is of type pointer to Hello struct. The unmarshaller get the type of association fields by packages.Load().. Maybe I am mistaken but getting the type would not be possible with parser.ParseDir() (the type is within the package).

PS : a workaround is possible with parser.ParseDir() but this is not the proper forum for this discussion

@adonovan
Copy link
Member

@mvdan , thks for the feedback. I retract the proposal.

Thanks for writing up your interesting use case so clearly.

The root of the problem is that the go list embodies a lot of computation on Go workspaces that it would be infeasible to maintain as a linked library. Also, golang.org/go/packages is intended to work with other build systems than go, such as Bazel and Blaze.

In system engineering, it is interesting to ship simulation binaries to interested stakeholders.

I would have thought that a stakeholder who cares about the code at the level of UML diagrams would be working in an environment where 'go build' works. But if not, perhaps you could package your program as Linux container image that contains a Go toolchain.

@thomaspeugeot
Copy link
Author

The root of the problem is that the go list embodies a lot of computation on Go workspaces that it would be infeasible to maintain as a linked library.

@adonovan , thks for the explanation. I admit that when I used packages.Load() for the first time, I was a bit amazed that a simple go program could perform such a go compilation by itself. I only discovered later that there was a go system call (after sharing the binary on a file server with a stakeholder who had not go installed :-)).

I would have thought that a stakeholder who cares about the code at the level of UML diagrams would be working in an environment where 'go build' works.

For your information, there is a recent trend in system engineering where stakeholders routinely deal with complex custom data model to explore their problem domain (see ref. below). Data model, also named ontology, can be complex (for instance in the model I currently work with, there are 40 classes and 61 associations). For stakeholders, it is natural to navigate the ontology/data thanks to a UML diagram (this is the use case # 1). Most stakeholders do not know that this is a UML diagram, they see (and understand) a diagram of interrelated concepts.

Ernadote, Dominique. "An ontology mindset for system engineering." 2015 IEEE International Symposium on Systems Engineering (ISSE). IEEE, 2015.

@rsc rsc moved this from Incoming to Declined in Proposals (old) May 25, 2022
@rsc
Copy link
Contributor

rsc commented May 25, 2022

This proposal has been declined as retracted.
— rsc for the proposal review group

@golang golang locked and limited conversation to collaborators May 25, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
No open projects
Development

No branches or pull requests

6 participants