Just some more information. I implemented my own Hotkeys without syscall using cgo. The same thing happens.

package main

/*
#include <windows.h>
#include <stdio.h>

void registerHotKey(int id, UINT modifiers, UINT key) {
    if (RegisterHotKey(NULL, id, modifiers, key))    //0x42 is 'b' 0x4000 is MOD_NOREPEAT
    {
        wprintf(L"Hotkey 'alt+b' registered, using MOD_NOREPEAT flag\n");
    } else {
        wprintf(L"It failed\n");
    }
}

MSG msg;
int getMessageHotKey() {
    BOOL rt;

    //rt = PeekMessageA(&msg, NULL, 0, 0, 0);	//this works as well
    rt = GetMessageA(&msg, NULL, 0, 0);
    if (rt == -1 || rt == 0) {
        return rt;
    } else {
        if (msg.message == WM_HOTKEY) {
            return msg.wParam;
        }
    }
}

*/
import "C"

import (
	"fmt"
	"runtime"
)

func main() {
	C.registerHotKey(1, 0x0001, 0x42)
	count := 0
	for {
		count++
		id := C.getMessageHotKey()
		if id == 1 {
			fmt.Println("alt+b pressed")
		}
		fmt.Println("Alloc %v", getAlloc())
		runtime.GC()
	}
}

func getAlloc() uint64 {
	var m runtime.MemStats
	runtime.ReadMemStats(&m)
	return m.Alloc
}

Thank you for filing this bug @Ragnoroct and welcome to the Go project!

I shall kindly ping some Windows, CGO and runtime experts: @alexbrainman @ianlancetaylor @aclements.

Thanks :)

Just to add some more info too. I tried adding

TranslateMessage(&msg);
DispatchMessage(&msg);

right after GetMessage to my cgo example and that also didn't work. I thought that possible https://docs.microsoft.com/en-us/windows/win32/winmsg/about-messages-and-message-queues#message-deadlocks might be a clue as to what is going on but I couldn't really figure out the how

@Ragnoroct May I kindly ask you something? Do you really need to run runtime.GC() every time after the message got from the queue? When I try it in my local environment and inspect it with task manager, I don't think there is any memory leak even I comment the runtime.GC().

No, I don't need to run runtime.GC() every time after GetMessage.
The only reason I was doing it was to try to actually see if there is a memory leak. When I did it I noticed that GetMessage just stopped working after.

It "probably" wouldn't be a problem but what if the GC does need to run. The application will just break without any chance of recovery or restarting.

Where I got a hunch that syscall's GetMessage on windows has a memory leak is from the second comment on this SO answer https://stackoverflow.com/a/38954281/10111548

@Ragnoroct I see your point there, thank you for explaining to me. However, I feel strange for the GetMessage to stop working after the GC run. It seems I still don't have any clue about it.

I don't know what is going on here. Can anybody else recreate the problem?

GetMessage must be called in same thread. So you need to call runtime.LockOSThread() at top of main.

I think this is not a bug of Go.

@mattn That fixes it. When I printf("thread %d", pthread_self()); in my cgo code without the lock the first time is thread 1 and after runtime.GC() it is thread 2.
It seems that runtime.GC() changes the thread/goroutine?

Adding runtime.LockOSThread() to the top of the application fixes it.

I'll mark it as closed now. Thanks for the help.