Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/sys/windows: use win32metadata? #43838

Open
bradfitz opened this issue Jan 22, 2021 · 31 comments
Open

x/sys/windows: use win32metadata? #43838

bradfitz opened this issue Jan 22, 2021 · 31 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. OS-Windows
Milestone

Comments

@bradfitz
Copy link
Contributor

Microsoft released this win32 metadata project: https://github.com/microsoft/win32metadata

It or its output is basically this 9.5MB machine-readable file describing all the Win32 APIs: https://github.com/microsoft/windows-rs/blob/master/crates/winmd/default/Windows.Win32.winmd

Maybe x/sys/windows could use it to auto-generate bindings more easily/accurately.

Might be fun. Might be tedious. Might be worth it.

/cc @alexbrainman @ianlancetaylor @zx2c4

@gopherbot gopherbot added this to the Unreleased milestone Jan 22, 2021
@dmitshur dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jan 22, 2021
@zx2c4
Copy link
Contributor

zx2c4 commented Jan 22, 2021

That's a great idea. https://docs.microsoft.com/en-us/uwp/winrt-cref/winmd-files is the file format.

I'd be in favor of making x/win32 or x/windows/win32 or similar for this -- something totally separate from the incredibly cluttered and disorganized x/sys/windows. It'll probably take us a few tries to get it right, and I assume that Microsoft will periodically issue corrections to the metadata, which we'll use to regenerate. In that sense, we wouldn't want to strive at an API compatibility guarantee beyond what Microsoft provides. If Microsoft fixes a function signature, then we'd want that to reflect in x/win32 without additional hassle or compatibility burden. This in turn means there might be a little bit more churn than the usual x/ repos, at least in the beginning, but in exchange, we get a complete API picture that's much better than what we have now.

@zx2c4
Copy link
Contributor

zx2c4 commented Jan 22, 2021

CC @jstarks in case you're interested too.

@bcmills
Copy link
Contributor

bcmills commented Jan 22, 2021

@zx2c4

we wouldn't want to strive at an API compatibility guarantee beyond what Microsoft provides. If Microsoft fixes a function signature, then we'd want that to reflect in x/win32 without additional hassle or compatibility burden.

At that point would it make sense to rewrite x/sys/windows in terms of small compatibility shims over x/sys/win32?

@jstarks
Copy link

jstarks commented Jan 22, 2021

Yes, this sounds like a great idea. I agree that starting fresh, rather than trying to replat x/sys/windows (at least initially), is the right approach.

Note that there are fidelity issues with win32metadata today--for example, C unions are not yet supported correctly. But I'm told the team is working on it.

I bet @sotteson1 and @kennykerr would be happy to give advice and take feedback.

@zx2c4
Copy link
Contributor

zx2c4 commented Jan 22, 2021

At that point would it make sense to rewrite x/sys/windows in terms of small compatibility shims over x/sys/win32?

That would potentially be a lot of work but wouldn't be impossible. It'd probably easier, though, to just deprecate x/sys/windows and encourage people to gradually move over to x/windows/win32 (or x/windows/winrt).

@zx2c4
Copy link
Contributor

zx2c4 commented Jan 22, 2021

Note that there are fidelity issues with win32metadata today--for example, C unions are not yet supported correctly. But I'm told the team is working on it.

C unions are the big annoying thing for Go too, and we'd probably want to come up with an overall approach at handling these, whether it's just using unsafe.Pointer when possible, or exploding one struct into multiple, or something else. A lot of fun decisions to make, I guess. C# doesn't quite have unions either, I don't think, which perhaps is where the trouble is coming from on the win32metadata side. I'll be interested to learn what you guys come up with there, as perhaps it's a model we could copy for Go.

@kennykerr
Copy link

Note that there are fidelity issues with win32metadata today--for example, C unions are not yet supported correctly. But I'm told the team is working on it.

Note that the metadata fully supports C-style unions. Some of the early language support don't yet support them, but the metadata is ready for consumption/trial.

@jstarks
Copy link

jstarks commented Jan 22, 2021

Ah, good, I misunderstood the source of the union bug in the Rust crate. Thanks Kenny.

@staticgc
Copy link

Does this mean golang's CGO will link with MSVC compiler in future? or I am missing something.

If I am correct, golang on windows works with MingGW & not MSVC

@zx2c4
Copy link
Contributor

zx2c4 commented Jan 24, 2021

Does this mean golang's CGO will link with MSVC compiler in future? or I am missing something.

If I am correct, golang on windows works with MingGW & not MSVC

CGO and MSVC are unrelated concerns to this one, as far as I can see here.

@zx2c4
Copy link
Contributor

zx2c4 commented Jan 24, 2021

Yes, I agree with most of what you said. We'll try not to repeat the mistakes of x/sys/windows, of which there are many many, when generating from winmd. Fully generating it means we can easily tweak the output format until it looks like something acceptable. And relaxing the compatibility requirements -- mentioned in #43838 (comment) -- means that we'd be able to actually perfect the output, rather than getting stuck with inconsistent signatures, like the one you mentioned.

@deadprogram
Copy link

I cannot begin to tell you how much I want this feature. 💯

@gonutz
Copy link

gonutz commented Apr 8, 2021

Note that the file referenced in Brad's first comment here is now located at https://github.com/microsoft/windows-rs/blob/master/crates/gen/default/Windows.Win32.winmd

@smasher164
Copy link
Member

To follow up, that file can be fetched from this package: https://www.nuget.org/packages/Microsoft.Windows.SDK.Win32Metadata

Download and unzip the file (nuget packages are just zip files) and you should find Windows.Win32.winmd in the root directory.

@smasher164
Copy link
Member

I also just discovered today that WinMD files are valid PE files, so someone could potentially use pkg/debug/pe to implement a parser for WinMD files.

@mugli
Copy link

mugli commented May 28, 2021

Regarding parsing the WinMD files, Jonathan Marler created a project to generate JSON representation from the win32 metadata, which was used to create the win32 binding for Zig. There's also a talk on Youtube explaining the project.

The generated JSON files are here: https://github.com/marlersoft/win32json/tree/main/api

@tdakkota
Copy link

https://docs.microsoft.com/en-us/uwp/winrt-cref/winmd-files is the file format.

There is a full ECMA-335 specification https://www.ecma-international.org/wp-content/uploads/ECMA-335_6th_edition_june_2012.pdf.
See II.22 Metadata logical format: tables.

@tdakkota
Copy link

https://github.com/tdakkota/win32metadata

Wrote metadata parser in Go.

There is a simple tool to generate some Go defintions from metadata:

$ go install github.com/tdakkota/win32metadata/cmd/printsig
$ printsig -file ./md/testdata/.windows/winmd/Windows.Win32.winmd -method ShellExecuteW
func ShellExecuteW(
	p0 HWND,
	hwnd PWSTR,
	lpOperation PWSTR,
	lpFile PWSTR,
	lpParameters PWSTR,
	lpDirectory int32,
) HINSTANCE

type PWSTR *uint16

type HINSTANCE int

type HWND int

@smasher164
Copy link
Member

Awesome work @tdakkota! After looking at the repo, I'm considering what would be the next steps to move this issue forward. I notice that printsig does a straightforward translation of win32 datatypes to Go datatypes (e.g. PWSTR is a *uint16). Does the metadata say anything about which parameters are supposed to be strings? And if Go bindings are to be idiomatic, should we do UTF-8 <-> UTF-16 conversion on our end?

@tdakkota
Copy link

ECMA-335 defines some flags for types (II.23.1.15 Flags for types [TypeAttributes]) and there are AnsiClass and UnicodeClass attributes, but it seems Windows metadata does not use it.

There is also a ELEMENT_TYPE_STRING, but it used only in WinRT.
For example:

printsig -file ./md/testdata/.windows/winmd/Windows.WinRT.winmd -method RenameAsync
func RenameAsync(
	desiredName string,
) IAsyncAction

type IAsyncAction struct { // printsig does not support managed structures and inheritance yet 
}

windows-rs adds PSTR and PWSTR conversions manually

https://github.com/microsoft/windows-rs/blob/61ed91e31ce7c181d570257a793f8dd3a9df92d6/crates/gen/src/types/pstr.rs#L37-L47

https://github.com/microsoft/windows-rs/blob/61ed91e31ce7c181d570257a793f8dd3a9df92d6/crates/gen/src/types/pwstr.rs#L37-L47

zigwin32gen too

https://github.com/marlersoft/zigwin32gen/blob/b63c76ee2f070dbb3cb027484c194f61316b9eb2/src/pass1.zig#L132-L138

I think if you want a pretty Go idiomatic API, you need a well-known types mapping.

@smasher164
Copy link
Member

smasher164 commented Jul 27, 2021

There is also a ELEMENT_TYPE_STRING, but it used only in WinRT.

Interesting. That sounds like something worth bringing up upstream. I agree that in the meantime, having a mapping of well-known types is the right approach.

Edit: I filed an issue upstream at microsoft/win32metadata#589.

@zzl
Copy link

zzl commented Apr 14, 2022

The generated JSON files are here: https://github.com/marlersoft/win32json/tree/main/api

I've created a project go-win32api that provides win32 API bindings in go, which are generated based the the mentioned JSON files. Anyone interested could go and take a look.

@MattBDev
Copy link

It would be great if the WinRT API was added as well but that may be out of scope for this issue. They release those over NuGet as well: https://www.nuget.org/packages/Microsoft.Windows.SDK.Contracts

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 7, 2022
@smasher164 smasher164 added NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jul 10, 2022
@dagood
Copy link
Contributor

dagood commented Dec 3, 2022

I'm (also!) interested in helping with this if possible, but I'm not totally sure what the current plan is after making the new repo. Sorry if it's simply too early in the process for these kinds of questions. 😄

Based on how x/sys/windows works, it seems like the new repo (say, x/win32) would contain the generated bindings as well as the tools that generate the bindings. This would let people use the tools to generate extra bindings if they aren't included in x/win32, like what you can do now with manually written //sys and the mkwinsyscall generator. Is that generally what's in mind?

On the dependency side, it seems to me that parsing the winmd files directly would be preferable to using an intermediate json, just because it would keep the critical update path a little shorter if there's an important (e.g. security) release from microsoft/win32metadata. But, maybe this isn't an important consideration because of the nature of the calls and win32 API?

I work for Microsoft, and a couple of us have done some work on another implementation that we can publish on the microsoft org and commit to maintaining. However, it seems to me to be better for everyone if we end up with an implementation in x/win32, so ideally, I'd like to help get there if I can.

@deadprogram
Copy link

Also, please see https://github.com/saltosystems/winrt-go which has seen some active development recently, and that we are currently using in the https://github.com/tinygo-org/bluetooth project.

Great to see some movement on this!

@glerchundi
Copy link

Thanks for mentioning us @deadprogram and kudos for the fantastic work done by @jagobagascon there!

@dagood
Copy link
Contributor

dagood commented Jan 3, 2023

Now that we're past the Dec holidays, wanted to ping this. I imagine that because creating some kind of x/win32 repository is part of the consensus (it seems to me), it'll take work/investment from the Go team to make that happen, even before the further questions enter the picture.

Is someone driving this proposal? This GitHub issue has been around for about a year now, so I wonder if priorities and perspectives may have shifted and a fresh, perhaps more detailed proposal is needed.

@slimsag
Copy link

slimsag commented Feb 14, 2023

Beware that the data portions of the github.com/microsoft/win32metadata repository are.. not open source, and the derivative versions built from those sources are.. questionably licensed:

microsoft/win32metadata#766

microsoft/win32metadata#761

https://github.com/microsoft/win32metadata/blob/main/licenses/sdk_license.txt

@riverar
Copy link

riverar commented Mar 13, 2023

There is no derivative issue here (Microsoft owns all the goop in there). The license on the bits you care about (the .winmd) is MIT. I'll poke some folks again to get that clarified in the README.

@mugli
Copy link

mugli commented Sep 14, 2023

fyi, there is also https://github.com/microsoft/go-winmd now

@dagood
Copy link
Contributor

dagood commented Sep 14, 2023

Yeah, just to clarify, https://github.com/microsoft/go-winmd is the repo for the project I mentioned here:

I work for Microsoft, and a couple of us have done some work on another implementation that we can publish on the microsoft org and commit to maintaining. However, it seems to me to be better for everyone if we end up with an implementation in x/win32, so ideally, I'd like to help get there if I can.

There are a number of open issues on the repo, https://github.com/microsoft/go-winmd/issues, and we haven't been able to work on this recently to get it to a point where we think it's stable and produces truly usable syscall signatures.

After a little more work, what we're planning to do as our next step is create a new proposal to create a x/win32 repo that contains the code that's currently in the microsoft/go-winmd repo along with syscalls generated by that tooling.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. OS-Windows
Projects
Status: Triage Backlog
Development

No branches or pull requests