Thanks for reporting, I'll work on fixing this.

I only tested using latest Go version when I sent that CL. It looks like (still need to confirm confirmed, see below for details) the test failure is simply because I was fairly aggressive about removing unneeded accepted error message strings.

I can bring back the accepted error strings that were there before, but while doing that, I'll think if there's a better way to fix this.

Broken for Go 1.7 and Go 1.6 (but not tip)

@bradfitz, did you mean it's broken for Go 1.8 and Go 1.7 (but not tip)? That would make more sense, given what I'm seeing (I expect it to fail for Go 1.8 too).

I see only Go 1.8 and 1.7 mentioned at https://build.golang.org/.

I have a better understanding (it's more subtle than what I wrote above).

CL 33158 changed behavior of go/build (fixing a bug, making the behavior more consistent). This fix only went into Go 1.9 and newer, so Go 1.6, 1.7, 1.8 still have old go/build behavior.

The go/build change allowed for a better expected error message, which CL 37768 updated for. However, that better error message doesn't apply for Go 1.6, 1.7, 1.8 because they don't have the original go/build fix.

Fix

I see two possible fixes. Either make the test more lax and accept both new and old error messages:

args: []string{"nonexistingpkg"},
matches: []string{
	// For Go 1.9 and newer.
	"cannot find package" +

		// For Go 1.8 and older, because it doesn't have CL 33158 change to go/build.
		// The last pattern (does not e) is for plan9:
		// http://build.golang.org/log/2d8e5e14ed365bfa434b37ec0338cd9e6f8dd9bf
		`|no such file or directory|does not exist|cannot find the file|(?:' does not e)`,
},

Or if we want to be really precise about expected error messages, we'd need to use _test.go files build tags for this:

args:    []string{"nonexistingpkg"},
matches: nonexistingpkgMatches,

// +build go1.9

package main_test

var nonexistingpkgMatches = []string{"cannot find package"}

// +build !go1.9

package main_test

// For Go 1.8 and older, because it doesn't have CL 33158 change to go/build.
var nonexistingpkgMatches = []string{
	// The last pattern (does not e) is for plan9:
	// http://build.golang.org/log/2d8e5e14ed365bfa434b37ec0338cd9e6f8dd9bf
	`no such file or directory|does not exist|cannot find the file|(?:' does not e)`,
}

Are there preferences as to which solution to use for this?

Second one is more precise, but also more verbose (and less readable, probably).

Let's do something in-between those two extremes.

I'd make a new file with just:

// +build go1.9
package main_test
func init() { isGo19 = true }

And then in the main test, have:

var isGo19 bool
....
args: []string{"nonexistingpkg"},
matches: []string{
		"cannot find package" +
                  condStr(!isGo19, 
    		         // For Go 1.8 and older, because it doesn't have CL 33158 change to go/build.
		         // The last pattern (does not e) is for plan9:
		         // http://build.golang.org/log/2d8e5e14ed365bfa434b37ec0338cd9e6f8dd9bf
		        `|no such file or directory|does not exist|cannot find the file|(?:' does not e)`)
},

func condStr(bool, string) string { .... }

What you suggested exactly doesn't work, because package-level variables are initialized before init gets a chance to run (https://golang.org/ref/spec#Package_initialization). So when condStr(!isGo19, ...) gets executed, isGo19 will always be false.

I can fix that by creating two files instead of one:

// +build go1.9
package main_test
const isGo19 = true

// +build !go1.9
package main_test
const isGo19 = false

I assumed the condStr() expression was inside of a test. I didn't go read the code.

I'd prefer the list of test cases to be a func variable instead of a package global anyway.

Looked at the code now. Yes, just move that var godocTests into the test. It's only used those two places:

godoc_test.go:25:var godocTests = []struct {
godoc_test.go:102:      for _, test := range godocTests {

Ok, that'll work.

Will send a CL soon, right after dinner.

CL https://golang.org/cl/41810 mentions this issue.