Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/env/wasip1-wasm-wazero: wasip1 tests are failing #61043

Closed
johanbrandhorst opened this issue Jun 27, 2023 · 14 comments
Closed

x/build/env/wasip1-wasm-wazero: wasip1 tests are failing #61043

johanbrandhorst opened this issue Jun 27, 2023 · 14 comments
Labels
arch-wasm WebAssembly issues Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Unreleased

Comments

@johanbrandhorst
Copy link
Member

johanbrandhorst commented Jun 27, 2023 •
edited by bcmills 

#!watchflakes
post <- pkg == "test" && test == "Test" && date >= "2023-07-12" && builder ~ `wazero` && log ~ `tmp__.go:\d+:\d+: syntax error:`

After enabling nonblocking io on stdin, stdout and stderr in CL 498196, the wazero wasip1 runner started to fail. The errors look like this:

##### ../test
--- FAIL: Test (0.07s)
    --- FAIL: Test/fixedbugs/bug449.go (0.94s)
        testdir_test.go:141: exit status 1
            # command-line-arguments
            ../../tmp/Testfixedbugsbug449.go4132839843/001/tmp__.go:3072:6: syntax error: unexpected package, expected name
            ../../tmp/Testfixedbugsbug449.go4132839843/001/tmp__.go:3866:53: syntax error: unexpected main at end of statement
            ../../tmp/Testfixedbugsbug449.go4132839843/001/tmp__.go:3870:6: syntax error: unexpected call, expected (
            ../../tmp/Testfixedbugsbug449.go4132839843/001/tmp__.go:4660:53: syntax error: unexpected main at end of statement
            ../../tmp/Testfixedbugsbug449.go4132839843/001/tmp__.go:4664:6: syntax error: unexpected call, expected (
            ../../tmp/Testfixedbugsbug449.go4132839843/001/tmp__.go:6214:13: syntax error: unexpected package, expected expression
            ../../tmp/Testfixedbugsbug449.go4132839843/001/tmp__.go:6216:1: syntax error: unexpected var at end of statement
            ../../tmp/Testfixedbugsbug449.go4132839843/001/tmp__.go:6218:6: syntax error: unexpected call, expected (
            ../../tmp/Testfixedbugsbug449.go4132839843/001/tmp__.go:9285:6: syntax error: unexpected package, expected name

Sample error log.

The link to the standard I/O change seems pretty clear, but the cause is likely a runtime bug. An issue has been filed with the wazero runtime project. The fix will likely be an update to the wazero version used in the builders.

The wasmtime runner continues to pass all tests as expected.
@johanbrandhorst johanbrandhorst added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. arch-wasm WebAssembly issues labels Jun 27, 2023
@gopherbot
Copy link

Change https://go.dev/cl/506695 mentions this issue: dashboard: add wazero known issue

gopherbot pushed a commit to golang/build that referenced this issue Jun 28, 2023
@johanbrandhorst @gopherbot
dashboard: add wazero known issue
4d5bc55 
The current version of the wazero runtime has a known issue
surrounding nonblocking I/O. Add a reference to the
tracking issue.

For golang/go#61043

Change-Id: Ic460058736b99bcf4e2a426924228ed1e2fb6c01
Reviewed-on: https://go-review.googlesource.com/c/build/+/506695
Reviewed-by: Cherry Mui <cherryyz@google.com>
Auto-Submit: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
Run-TryBot: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
@johanbrandhorst
Copy link
Member Author

The wazero issue has been closed and will be available in the next release. We'll upgrade the Wazero runner and hopefully close this issue then.

@johanbrandhorst
Copy link
Member Author

johanbrandhorst commented Jun 29, 2023 •
edited

The tests pass locally for me with the new wazero version

@gopherbot
Copy link

Change https://go.dev/cl/508875 mentions this issue: env: upgrade wazero version

gopherbot pushed a commit to golang/build that referenced this issue Jul 11, 2023
@johanbrandhorst @gopherbot
env: upgrade wazero version
60cba8a 
The latest version of wazero contains a proposed fix for an I/O
bug that is causing build failures.

For golang/go#61043

Change-Id: Ifd7d5b0f6d36d52692b22faa85ecfd1a7d385781
Reviewed-on: https://go-review.googlesource.com/c/build/+/508875
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Auto-Submit: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
@johanbrandhorst
Copy link
Member Author

It seems this isn't quite fixed yet in Wazero v1.3.0, I've posted an update to the original issue.

@gopherbot
Copy link

Change https://go.dev/cl/512255 mentions this issue: env: upgrade wazero version

@johanbrandhorst johanbrandhorst mentioned this issue Jul 23, 2023
Closed
@dmitshur dmitshur added this to the Backlog milestone Jul 23, 2023
@dmitshur dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jul 23, 2023
gopherbot pushed a commit to golang/build that referenced this issue Jul 24, 2023
@johanbrandhorst
env: upgrade wazero version
c9aa849 
The latest version of wazero contains a proposed fix for an I/O
bug that is causing build failures.

For golang/go#61043

Change-Id: Ibcd77469969b51c34c1ae94d6a670a6e86025147
Reviewed-on: https://go-review.googlesource.com/c/build/+/512255
Reviewed-by: Heschi Kreinick <heschi@google.com>
Run-TryBot: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Auto-Submit: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "test" && test == "Test" && date >= "2023-07-14" && builder ~ `wazero` && `tmp__.go:\d+:\d+: syntax error:`
2023-07-14 12:49 wasip1-wasm-wazero go@c30faf9c test.Test (log) 
--- FAIL: Test (0.08s)
    --- FAIL: Test/fixedbugs/bug449.go (0.88s)
        testdir_test.go:141: exit status 1
            # command-line-arguments
            ../../tmp/Testfixedbugsbug449.go1458940196/001/tmp__.go:3072:6: syntax error: unexpected package, expected name
            ../../tmp/Testfixedbugsbug449.go1458940196/001/tmp__.go:3866:53: syntax error: unexpected main at end of statement
            ../../tmp/Testfixedbugsbug449.go1458940196/001/tmp__.go:3870:6: syntax error: unexpected call, expected (
            ../../tmp/Testfixedbugsbug449.go1458940196/001/tmp__.go:4660:53: syntax error: unexpected main at end of statement
            ../../tmp/Testfixedbugsbug449.go1458940196/001/tmp__.go:4664:6: syntax error: unexpected call, expected (
            ../../tmp/Testfixedbugsbug449.go1458940196/001/tmp__.go:6214:13: syntax error: unexpected package, expected expression
            ../../tmp/Testfixedbugsbug449.go1458940196/001/tmp__.go:6216:1: syntax error: unexpected var at end of statement
            ../../tmp/Testfixedbugsbug449.go1458940196/001/tmp__.go:6218:6: syntax error: unexpected call, expected (
            ../../tmp/Testfixedbugsbug449.go1458940196/001/tmp__.go:9285:6: syntax error: unexpected package, expected name
2023-07-14 17:30 wasip1-wasm-wazero go@847d40d6 test.Test (log) 
--- FAIL: Test (0.07s)
    --- FAIL: Test/fixedbugs/bug449.go (0.97s)
        testdir_test.go:141: exit status 1
            # command-line-arguments
            ../../tmp/Testfixedbugsbug449.go1168914229/001/tmp__.go:3072:6: syntax error: unexpected package, expected name
            ../../tmp/Testfixedbugsbug449.go1168914229/001/tmp__.go:4626:13: syntax error: unexpected package, expected expression
            ../../tmp/Testfixedbugsbug449.go1168914229/001/tmp__.go:4628:1: syntax error: unexpected var at end of statement
            ../../tmp/Testfixedbugsbug449.go1168914229/001/tmp__.go:4630:6: syntax error: unexpected call, expected (
            ../../tmp/Testfixedbugsbug449.go1168914229/001/tmp__.go:9214:7: syntax error: unexpected package, expected expression
            ../../tmp/Testfixedbugsbug449.go1168914229/001/tmp__.go:9216:1: syntax error: unexpected var at end of statement
            ../../tmp/Testfixedbugsbug449.go1168914229/001/tmp__.go:9218:6: syntax error: unexpected call, expected (
            ../../tmp/Testfixedbugsbug449.go1168914229/001/tmp__.go:12285:6: syntax error: unexpected package, expected name
            ../../tmp/Testfixedbugsbug449.go1168914229/001/tmp__.go:15356:6: syntax error: unexpected package, expected name

— watchflakes

@johanbrandhorst
Copy link
Member Author

@cherrymui are these errors still happening with the new version deployed?

@cherrymui
Copy link
Member

I deployed the new version. So far there is only one run after it is deployed and the test passes. So it looks promising. If it passes consistently after a few more runs, we can call it fixed. Thanks!

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jul 24, 2023
@dmitshur dmitshur modified the milestones: Backlog, Unreleased Jul 24, 2023
@dmitshur dmitshur changed the title build: wasip1 tests are failing on wazero x/build/env/wasip1-wasm-wazero: wasip1 tests are failing Jul 24, 2023
@dmitshur dmitshur added the Builders x/build issues (builders, bots, dashboards) label Jul 24, 2023
@johanbrandhorst
Copy link
Member Author

Looks like it's passed the last 4 builds so I think we can call this fixed for now!
image

@dmitshur
Copy link
Contributor

dmitshur commented Jul 25, 2023 •
edited

Thanks @johanbrandhorst!

This issue is still referenced as a "known issue" in x/build/dashboard (here). Since you've marked this as closed, please also remove it there. Thanks.

@dmitshur dmitshur removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jul 25, 2023
@gopherbot
Copy link

Change https://go.dev/cl/513055 mentions this issue: dashboard: remove wazero issue

gopherbot pushed a commit to golang/build that referenced this issue Jul 25, 2023
@johanbrandhorst @gopherbot
dashboard: remove wazero issue
6412f5a 
The issue has been resolved.

For golang/go#61043

Change-Id: Ia3c4223fec18c7987c30f154346222391a969e4e
Reviewed-on: https://go-review.googlesource.com/c/build/+/513055
Reviewed-by: Cherry Mui <cherryyz@google.com>
Auto-Submit: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@bcmills bcmills mentioned this issue Aug 8, 2023
Closed
@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "test" && test == "Test" && date >= "2023-07-12" && builder ~ `wazero` && log ~ `tmp__.go:\d+:\d+: syntax error:`
2023-07-12 21:23 wasip1-wasm-wazero go@cc0cb302 test.Test (log) 
--- FAIL: Test (0.08s)
    --- FAIL: Test/fixedbugs/bug449.go (0.92s)
        testdir_test.go:141: exit status 1
            # command-line-arguments
            ../../tmp/Testfixedbugsbug449.go3129408765/001/tmp__.go:3072:6: syntax error: unexpected package, expected name
            ../../tmp/Testfixedbugsbug449.go3129408765/001/tmp__.go:4626:13: syntax error: unexpected package, expected expression
            ../../tmp/Testfixedbugsbug449.go3129408765/001/tmp__.go:4628:1: syntax error: unexpected var at end of statement
            ../../tmp/Testfixedbugsbug449.go3129408765/001/tmp__.go:4630:6: syntax error: unexpected call, expected (
            ../../tmp/Testfixedbugsbug449.go3129408765/001/tmp__.go:6180:13: syntax error: unexpected package, expected expression
            ../../tmp/Testfixedbugsbug449.go3129408765/001/tmp__.go:6182:1: syntax error: unexpected var at end of statement
            ../../tmp/Testfixedbugsbug449.go3129408765/001/tmp__.go:6184:6: syntax error: unexpected call, expected (
            ../../tmp/Testfixedbugsbug449.go3129408765/001/tmp__.go:9251:6: syntax error: unexpected package, expected name
2023-07-13 16:46 wasip1-wasm-wazero go@9480b4ad test.Test (log) 
--- FAIL: Test (0.02s)
    --- FAIL: Test/clearfat.go (0.95s)
        testdir_test.go:141: exit status 1
            # command-line-arguments
            ../../tmp/Testclearfat.go3174433507/001/tmp__.go:4229:13: syntax error: unexpected main at end of statement
            ../../tmp/Testclearfat.go3174433507/001/tmp__.go:4234:6: syntax error: unexpected poison1, expected (
            ../../tmp/Testclearfat.go3174433507/001/tmp__.go:4236:2: syntax error: unexpected var at end of statement
            ../../tmp/Testclearfat.go3174433507/001/tmp__.go:4498:14: syntax error: unexpected main, expected comma, : or ]
            ../../tmp/Testclearfat.go3174433507/001/tmp__.go:4500:1: syntax error: unexpected var, expected expression
            ../../tmp/Testclearfat.go3174433507/001/tmp__.go:4503:6: syntax error: unexpected poison1, expected (
            ../../tmp/Testclearfat.go3174433507/001/tmp__.go:4505:2: syntax error: unexpected var at end of statement
            ../../tmp/Testclearfat.go3174433507/001/tmp__.go:4511:6: syntax error: unexpected clearfat1, expected (
            ../../tmp/Testclearfat.go3174433507/001/tmp__.go:4512:2: syntax error: unexpected var at end of statement
            ../../tmp/Testclearfat.go3174433507/001/tmp__.go:4767:14: syntax error: unexpected main, expected comma, : or ]
            ../../tmp/Testclearfat.go3174433507/001/tmp__.go:4767:14: too many errors

— watchflakes

@johanbrandhorst
Copy link
Member Author

These last two flakes are from June, it's likely the same bug that should be fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arch-wasm WebAssembly issues Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Projects
Test Flakes
Status: Done
Milestone
Unreleased
Development

No branches or pull requests
4 participants
@dmitshur @johanbrandhorst @gopherbot @cherrymui