Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/playground: ExampleNoOutput test is flaky #24615

Closed
andybons opened this issue Mar 30, 2018 · 10 comments
Closed

x/playground: ExampleNoOutput test is flaky #24615

andybons opened this issue Mar 30, 2018 · 10 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@andybons
Copy link
Member

When running the playgrounds within the Dockerfile, it sometimes fails with

Step 60/62 : RUN /app/playground test
 ---> Running in 8fbf8f5df18a
App (project ID: "") is NOT caching results
2018/03/30 23:30:42 unexpected output: "PASS\ntesting: warning: no tests to run\n", want "testing: warning: no tests to run\nPASS"

after updating the test code, it will sometimes fail with:

Step 60/62 : RUN /app/playground test
 ---> Running in 411206ccfd4d
App (project ID: "") is NOT caching results
2018/03/30 22:51:30 unexpected output: "testing: warning: no tests to run\nPASS\n", want "PASS\ntesting: warning: no tests to run"

The output seems non-deterministic, which is odd.

@ysmolsky

@andybons andybons added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 30, 2018
@andybons andybons added this to the Unreleased milestone Mar 30, 2018
@ysmolski
Copy link
Member

I am unable to reproduce this. How do you run those tests? When I run make test using macOS native Docker I have different step numbers:

...golang.org/x/playground (master $=) % git a
* 76d1738 - (HEAD -> master, origin/master, origin/HEAD) playground: support examples as tests

... SKIPPED ...
Step 45/48 : WORKDIR /app
Removing intermediate container 7782ffd5187f
 ---> 87a0a8813a76
Step 46/48 : RUN /go/bin/playground test
 ---> Running in 8188688d1019
App (project ID: "") is NOT caching results
OK
Removing intermediate container 8188688d1019
 ---> 226e01540b07
Step 47/48 : EXPOSE 8080
 ---> Running in d7deedd5a872
Removing intermediate container d7deedd5a872
 ---> cebe18a41b3c
Step 48/48 : ENTRYPOINT ["/go/bin/playground"]
 ---> Running in df745592a4fd
Removing intermediate container df745592a4fd
 ---> 91f5b1decaa6
Successfully built 91f5b1decaa6
Successfully tagged playground:latest
go test
PASS
ok  	golang.org/x/playground	0.018s

Does it happen for the commit 76d1738?

@ysmolski
Copy link
Member

ysmolski commented Mar 31, 2018

That particular test is the only one expecting output of two different kinds in Events slice: [{"testing: warning: no tests to run\n" "stderr" "0s"} {"PASS\n" "stdout" "0s"}]

Trying to write to stderr and stdout gives inconsistent output when writing both to stderr and stdout at the same "playground time":

package main

import (
	"fmt"
	"os"
)

func main() {
	fmt.Fprintln(os.Stderr, "Error")
	fmt.Fprintln(os.Stdout, "Hello")
}

https://play.golang.org/p/8du1N7eAxmq gives

Hello
Error

https://play.golang.org/p/yky_PWrcZvu gives

Error1
Hello1

This program produces correct order in my local playground.

@ysmolski
Copy link
Member

And then I got to this function: https://github.com/golang/go/blob/master/src/runtime/sys_nacl_amd64p32.s#L63

TEXT runtime·write(SB),NOSPLIT,$16-20
	// If using fake time and writing to stdout or stderr,
	// emit playback header before actual data.

242050d

ping @minux

@ysmolski
Copy link
Member

ysmolski commented Apr 2, 2018

The problem is not about patched runtime.write function. It's about how data got copied in os/exec.writeDescriptor:

	c.goroutine = append(c.goroutine, func() error {
		_, err := io.Copy(w, pr)
		pr.Close() // in case io.Copy stopped due to write error
		return err
	})

Since we have two parallel writers, for stderr and stdout, io.Copy happens in goroutines competing with each other. Then there is no guarantee that copy operations happen in correct order. Fixing this would mean patching os/exec, but I doubt that it's worth it.

Another crude temporary fix would be to patch sandbox code to capture both kind of output into the same writer Stdout():

	rec := new(Recorder)
	cmd.Stdout = rec.Stdout()
	cmd.Stderr = rec.Stdout()

Since stdout and stderr output looks exactly the same on playground UI, then we could apply this patch until we find a proper fix.

@ysmolski
Copy link
Member

ysmolski commented Apr 2, 2018

Of course we can resort to just fixing the test and closing the issue. Then spawn another issue for fixing stdout/stderr messed up order.

@andybons, what do you think?

@gopherbot
Copy link

Change https://golang.org/cl/104857 mentions this issue: playground: fix the order of interleaved stdout and stderr

@gopherbot
Copy link

Change https://golang.org/cl/105235 mentions this issue: runtime: make fake timestamps in Playground writes strict

@gopherbot
Copy link

Change https://golang.org/cl/105496 mentions this issue: playground: use strict fake timestamps for sorting output

@gopherbot
Copy link

Change https://golang.org/cl/106216 mentions this issue: playground: patch runtime with https://golang.org/cl/105235

gopherbot pushed a commit that referenced this issue Apr 11, 2018
The process reading the output of the binary may read stderr and
stdout separately, and may interleave reads from the two streams
arbitrarily. Because we explicitly serialize writes on the writer
side, we can reuse a timestamp within a single stream without losing
information; however, if we use the same timestamp for write on both
streams, the reader can't tell how to interleave them.

This change ensures that every time we change between the two fds, we
also bump the timestamp. That way, writes within a stream continue to
show the same timestamp, but a sorted merge of the contents of the two
streams always interleaves them in the correct order.

This still requires a corresponding change to the Playground parser to
actually reconstruct the correct interleaving. It currently merges the
two streams without reordering them; it should instead buffer them
separately and perform a sorted merge. (See
https://golang.org/cl/105496.)

Updates #24615.
Updates #24659.

Change-Id: Id789dfcc02eb4247906c9ddad38dac50cf829979
Reviewed-on: https://go-review.googlesource.com/105235
Run-TryBot: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Yury Smolsky <yury@smolsky.by>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@gopherbot
Copy link

Change https://golang.org/cl/106475 mentions this issue: playground: add integration test for https://golang.org/cl/105235

gopherbot pushed a commit to golang/playground that referenced this issue Apr 13, 2018
Add a test case (currently flaky) that checks for deterministic output
ordering.

Updates golang/go#24615.
Updates golang/go#24659.

Change-Id: I7edc5d77a890edcd684ddf2aeee4c7a7dea68af1
Reviewed-on: https://go-review.googlesource.com/106216
Reviewed-by: Andrew Bonventre <andybons@golang.org>
gopherbot pushed a commit to golang/playground that referenced this issue Apr 13, 2018
Add a test case that checks fake timestamps generated by the patched
runtime.write.

Updates golang/go#24615
Updates golang/go#24659

Change-Id: I210cfd4500e7f72567539dd9b3e23da4f907a3e2
Reviewed-on: https://go-review.googlesource.com/106475
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@golang golang locked and limited conversation to collaborators Apr 13, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

3 participants