Probably has something to do with the poller changes, but I don't know what.

CC @alexbrainman

@jakans your program works with Windows type command:

c:\Users\Alex\dev\src\issue\go\22024>type pipehangs.go
package main

import (
        "fmt"
        "io/ioutil"
        "os"
)

func main() {

        str, err := ioutil.ReadAll(os.Stdin)
        fmt.Fprintf(os.Stderr, "n: %d\n", len(str))
        if err != nil {
                fmt.Fprintf(os.Stderr, "err: %s\n", err)
        }
}

c:\Users\Alex\dev\src\issue\go\22024>type pipehangs.go | go run pipehangs.go
n: 215

c:\Users\Alex\dev\src\issue\go\22024>

I do not have curl program installed on my computer to test your scenario. How do I install it?
But even after I install curl program, I am not sure how to debug this - it could be curl program that hangs for all we know.

What are you trying to do? If you just want to download some file from the Internet, Go standard library have plenty of tools to do that.

Alex

MSYS (with separately downloaded cURL)

$ curl.exe --version
curl 7.50.2 (i386-pc-win32) libcurl/7.50.2 OpenSSL/1.0.2e zlib/1.2.8
Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtsp
smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile NTLM SSL libz

$ go.exe version
go version go1.9 windows/386

$ curl.exe -s \
> "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&id=26422376&version=2.0" | \
> go.exe run pipehangs.go
n: 0

$ 

MSYS2

http://nurmi-labs.blogspot.com/2016/11/git.html

$ curl.exe --version
curl 7.50.1 (i686-w64-mingw32) libcurl/7.50.1 OpenSSL/1.0.2h zlib/1.2.8 libidn/1
.33 libssh2/1.7.0 nghttp2/1.13.0 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s
 rtmp rtsp scp sftp smtp smtps telnet tftp
Features: IDN IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL libz TLS-SRP HTTP2 Me
talink

$ go.exe version
go version go1.9 windows/386

$ curl.exe -s \
> "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&id=26422376&version=2.0" | \
> go.exe run pipehangs.go
n: 3910

$

Does it hang also, if you run it from cmd.exe?
It could have something to do with the terminal (cygwin, msys/mintty vs console).
For me on mintty, pipe sometimes work, sometimes hang. On cmd.exe they always work.

the content of #22024 (comment) indicates NO outout from curl 7.50.2

in both examples from that comment the Windows Console had been utilised

for what its worth here is MSYS2's mintty.exe running a C Shell

% curl.exe -s \
? "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&id=26422376&version=2.0" | \
? go.exe run pipehangs.go
n: 3910
%

You don't mention what cURL version you are using, or if you've run your cURL command without the pipe.

@forskning thanks for testing. Looks like "go run pipehangs.go" does not hang in either of your scenarios.

I wonder why curl 7.50.2 does not output any data. What happens if you redirect curl output to a file? Do you get anything written to the file?

Thank you

Alex

@alexbrainman I should have looked into that prior to adding the MSYS content

http://nurmi-labs.blogspot.com/2015/11/bcc55.html

It was Dirk Paehl's cURL 7.50.2 (Download WITH SUPPORT SSL) I used for a Borland/Dmake compile of perl-5.10.1.tar.gz, the CPAN setup, and, as those were http addresses, I never added the SSL support files.

Perl/perl5@378eeda#diff-65539b463d4890c68be9c1e3de589c4d

You can read about the events which led up to the "The Borland Chainsaw Massacre" here:

http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2011-09/msg00034.html

Unable to reproduce with various versions of curl 7.55.1.
n: 0 when curl fails to initialize; but a hang never occurs.

http://www.paehl.com/open_source/?CURL_7.55.1

set URL=https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed^&id=26422376^&version=2.0

curl -s "%URL%" | go run pipehangs.go  & echo curl_755_1
n: 0
curl_755_1

No crash, instant return

curl -s "%URL%" | go run pipehangs.go  & echo curl_755_1_ssh2_ssl_sspi
n: 0
curl_755_1_ssh2_ssl_sspi

curl -s "%URL%" | go run pipehangs.go & echo curl_755_1
n: 3910
curl_755_1

curl -s "%URL%" | go run pipehangs.go  & echo curl_755_1_rtmp_ssh2_ssl_sspi
n: 0
curl_755_1_rtmp_ssh2_ssl_sspi

go get github.com/as/torgo/hget
go run %GOPATH%\src\github.com\as\torgo\hget\hget.go "%URL%" | go run pipehangs.go
n: 3910

@jakans

Does your curl command when run without the pipe output the xml to the console?

@alexbrainman I should have looked into that prior to adding the MSYS content
...

Simply speaking you curl.exe was broken. Cool. Thank you for explaining.

Unable to reproduce with various versions of curl 7.55.1.
n: 0 when curl fails to initialize; but a hang never occurs.

Thank you @as for trying.

Alex

moot point

curl_750_2_ssl (without adding the SSL support files) returns output for http not https

$ curl.exe -s "http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&id=26422376&version=2.0"
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://eutils.ncbi.nlm.nih.gov/entrez/eutils
/esummary.fcgi?db=pubmed&amp;id=26422376&amp;version=2.0">here</a>.</p>
</body></html>

$ curl.exe -s "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&id=26422376&version=2.0"

$ curl.exe --version
curl 7.50.2 (i386-pc-win32) libcurl/7.50.2 OpenSSL/1.0.2e zlib/1.2.8
Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtsp
smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile NTLM SSL libz

$

"That they be not forced to sue the lawe, wrapped with so infinite crickes and moot poyntes."

Laurence Humphrey - 1563

The problem was discovered by users of Entrez Direct (https://www.ncbi.nlm.nih.gov/books/NBK179288/) who had installed the latest release on their PCs. While most of EDirect is implemented as a Perl wrapper to NCBI's URL-based Entrez Utilities (https://www.ncbi.nlm.nih.gov/books/NBK25501/), the xtract component is written in Go, and it failed to read pipes from the upstream Perl steps. I quickly reverted to the Go 1.8 xtract binary for CYGWIN (Mac and Linux versions are fine under Go 1.9), and then isolated the problem and created the minimal pipehangs.go program for debugging.

Entrez Direct provides a simple way for non-programmers to do sophisticated data mining in PubMed and other interconnected NCBI databases. The target audience includes scientists, medical librarians, bibliometric researchers, and academic administrators, as well as computational biologists who are looking for an easy way to get specific information from our databases. Users supply query details in command-line arguments. Separate steps in the query are connected by Unix pipes. Everything else is handled behind the scenes, with no additional coding required.

The sample ad hoc EDirect query shown below searches for journal articles in PubMed and retrieves the records in a defined XML form. The XML data set is piped to xtract, which limits results to journals published in the U.S. and then visits each author, printing one author name per line. This output is piped to a standard script that produces a frequency table of publications per author:

esearch -db pubmed -query "rattlesnake phospholipase" |
efetch -format xml |
xtract -pattern PubmedArticle
-if MedlineJournalInfo/Country -equals "United States"
-block Author -tab "\n" -sep " " -element LastName,Initials |
sort-uniq-count-rank

9 Wells MA
8 Marangoni S
8 Toyama MH
6 Dennis EA
5 Bon C
5 Kézdy FJ
5 Sigler PB
5 Soares AM
4 Carlini CR
4 Faure G
4 Francischetti IM
4 Guimarães JA
4 HANAHAN DJ
4 Heinrikson RL
...

To answer the earlier question, curl.exe --version produces:
curl 7.43.0 (x86_64-unknown-cygwin) libcurl/7.43.0 OpenSSL/1.0.2d zlib/1.2.8 libidn/1.29 libssh2/1.5.0

The latest xtract.go source code now has an undocumented -echo command to help with debugging. xtract -sample will send a small sample XML file to stdout. xtract -echo will read from stdin, using the same method that is hanging when reading from a pipe. To obtain the source code without doing the full EDirect installation you can run:

ftp ftp://ftp.ncbi.nlm.nih.gov/entrez/entrezdirect/edirect.tar.gz
gunzip -c edirect.tar.gz | tar xf -
rm edirect.tar.gz

To build the PC binary using a Go 1.9 compiler on any platform, run:

cd edirect
env GOOS=windows GOARCH=386 go build -o testxtract -v xtract.go
chmod +x testxtract

On a PC under CYGWIN, the following will print a sample XML record:

./testxtract -sample

Redirecting stdout will save it to a file:

./testxtract -sample > sample.xml

Piping it to another instance of testxtract and reading with the -echo command will fail:

./testxtract -sample | ./testxtract -echo

However, redirecting stdin from an existing file will work:

./testxtract -echo < sample.xml

The actual code being tested is excerpted here:

in := os.Stdin

// test reading from input pipe or file (undocumented)
if args[0] == "-echo" {
	const XMLBUFSIZE = 65536 + 16384
	buffr := make([]byte, XMLBUFSIZE)
	for {
		n, err := in.Read(buffr)
		if n == 0 {
			break
		}
		if err != nil {
			fmt.Fprintf(os.Stderr, "err: %s\n", err)
			break
		}
		fmt.Fprintf(os.Stdout, "%s", buffr[:n])
	}
	return
}

It would be nice to see the dataflow through the pipe. Can you run your pipeline through pv? If you're using Windows you can use the implementation here:

go get github.com/as/torgo/pv
hget http://badurl.com | %GOBIN%\pv > nul
pv: t=1   b=0.000 B   Δ=0.000 B/s  a=0.000 B/s
pv: t=2   b=0.000 B   Δ=0.000 B/s  a=0.000 B/s
pv: t=3   b=0.000 B   Δ=0.000 B/s  a=0.000 B/s
pv: t=4   b=0.000 B   Δ=0.000 B/s  a=0.000 B/s
pv: t=5   b=0.000 B   Δ=0.000 B/s  a=0.000 B/s
pv: t=6   b=0.000 B   Δ=0.000 B/s  a=0.000 B/s
pv: t=7   b=0.000 B   Δ=0.000 B/s  a=0.000 B/s
pv: t=8   b=0.000 B   Δ=0.000 B/s  a=0.000 B/s
pv: t=9   b=0.000 B   Δ=0.000 B/s  a=0.000 B/s
pv: t=10   b=0.000 B   Δ=0.000 B/s  a=0.000 B/s
pv: t=11   b=0.000 B   Δ=0.000 B/s  a=0.000 B/s
2017/09/27 14:00:15 Get http://badurl.com: dial tcp: lookup badurl.com: no such host
pv: t=11   b=0.000 B   Δ=0.000 B/s  a=0.000 B/s

./testxtract -sample | pv | ./testxtract -echo

To obtain the source code without doing the full EDirect installation you can run:

ftp ftp://ftp.ncbi.nlm.nih.gov/entrez/entrezdirect/edirect.tar.gz
gunzip -c edirect.tar.gz | tar xf -
rm edirect.tar.gz

I did that.

To build the PC binary using a Go 1.9 compiler on any platform, run:

cd edirect
env GOOS=windows GOARCH=386 go build -o testxtract -v xtract.go
chmod +x testxtract

I did that.

On a PC under CYGWIN,

I don't have CYGWIN, I used cmd.exe program.

the following will print a sample XML record:

./testxtract -sample

I did that. It worked.

Redirecting stdout will save it to a file:

./testxtract -sample > sample.xml

I did that. It worked.

Piping it to another instance of testxtract and reading with the -echo command will fail:

./testxtract -sample | ./testxtract -echo

I did that. It worked.

However, redirecting stdin from an existing file will work:

./testxtract -echo < sample.xml

I did that. It worked.

The actual code being tested is excerpted here:

I think your code has a bug. When in.Read(buffr) returns with n == 0 you break out of the read loop. But there could be more data coming. The Reader documentation says https://golang.org/pkg/io/#Reader about that: "... Callers should treat a return of 0 and nil as indicating that nothing happened; in particular it does not indicate EOF. ...".

Perhaps your CYGWIN shell program (unlike my cmd.exe) inserts 0 length writes into the pipe between 2 programs.

Alex

In the event anyone decides to pursue testing the Cygnus port.

A search of the current curl and perl implementations follows.

https://cygwin.com/cgi-bin2/package-grep.cgi

curl-7.55.1-1
perl-5.24.1-1

Pour faire l'Archiviste il faut être un homme intelligent.

Un homme intelligent ne fait pas l'Archiviste.

I've modified the -echo code to report if "n == 0" without breaking out of the loop, but the test had been added merely to suppress the EOF error message. The actual function that calls Read only looks for "err != nil" to determine end of data.

To further investigate, I added a -read command to specifically test the actual read block function. These are in the xtract.go that is now on the ftp site. The relevant code section is:

// DEBUGGING

// test reading from input pipe or file (undocumented)
if args[0] == "-echo" {
	const XMLBUFSIZE = 65536 + 16384
	buffr := make([]byte, XMLBUFSIZE)
	for {
		n, err := in.Read(buffr)
		if err != nil {
			fmt.Fprintf(os.Stderr, "ERR: %s, N: %d\n", err, n)
			break
		}
		if n == 0 {
			fmt.Fprintf(os.Stderr, "N: zero\n")
			continue
		}
		fmt.Fprintf(os.Stdout, "%s", buffr[:n])
	}
	return
}

// CREATE XML BLOCK READER FROM STDIN OR FILE

rdr := NewXMLReader(in, doCompress, doCleanup, doStrict || doMixed)
if rdr == nil {
	fmt.Fprintf(os.Stderr, "\nERROR: Unable to create XML Block Reader\n")
	os.Exit(1)
}

// DEBUGGING

// test reading blocks from xml reader (undocumented)
if args[0] == "-read" {
	for {
		str := rdr.NextBlock()
		if str == "" {
			fmt.Fprintf(os.Stderr, "\n\nSTR: empty\n")
			break
		}
		fmt.Fprintf(os.Stdout, "%s", str)
	}
	fmt.Fprintf(os.Stdout, "\n")
	return
}

I asked my colleague with a PC to try:

testxtract -sample | testxtract -echo

and:

testxtract -sample | testxtract -read

and he got wildly inconsistent results. Sometimes it would hang, sometimes it wouldn't. Inserting pv in between would either hang, print one "pv:" line and then the expected output, or print "pv:" lines forever.

My expert-of-last-resort colleague got involved, and quickly found that he could force consistent hangs by replacing:

testxtract -sample

in the above commands with:

(sleep 1; testxtract -sample)

That was immediately confirmed by the other colleague.

Do you know of any change between Go 1.8 and Go 1.9 that might explain this sort of timing dependency with pipes on the PC?

		n, err := in.Read(buffr)
		if err != nil {

Err can be io.EOF, and n > 0, but we break out of the loop so that data is never processed

			fmt.Fprintf(os.Stderr, "ERR: %s, N: %d\n", err, n)
			break
		}
		if n == 0 {
			fmt.Fprintf(os.Stderr, "N: zero\n")
			continue
		}
		fmt.Fprintf(os.Stdout, "%s", buffr[:n])

Just because err != nil doesn't mean you're done processing the data, it may be the case that n > 0. The test err == io.EOF tells you that the reader is done, so err != nil it says nothing about the data it copied to the slice for you.

for {
   n, err := fd.Read(p)
   if err != nil && err != io.EOF{
       // hard stop; a real error happens
       break
   }
   datain <- append([]byte{}, p[:n]...) // edited here 
   if err == io.EOF{
     break
  }
}

Above is a cautious way to do this, i.e., don't process the bytes if a real error occurred. The go doc io.Reader recommends p[:n] is handled regardless of what err is though.

I had noticed this in the documentation, but in my early tests err != nil always came with n == 0. To be robust against future implementation changes, I've modified the code, but I still prefer to err on the side of caution and ignore bytes that come with a real error. It will now print any non-EOF error message to Stderr, at least making the user aware that something is amiss. The -echo test code was also changed, with the relevant section shown below:

		n, err := in.Read(buffr)
		if err != nil {
			if err != io.EOF {
				fmt.Fprintf(os.Stderr, "ERR: %s, N: %d\n", err, n)
				break
			}
			if n == 0 {
				// EOF and no more data
				break
			}
		}
		if n == 0 {
			fmt.Fprintf(os.Stderr, "N: zero\n")
			continue
		}
		fmt.Fprintf(os.Stdout, "%s", buffr[:n])

The source code on the ftp site has been updated, but I'll allow some time for in-house testing before making a new release of the precompiled binaries.

Thanks for the advice.

Not a problem, but just FYI my example and yours are semantically similar - they don't handle data if there's a real error. This is because p[:n], when n==0 is a valid zero-length slice. The disadvantage is that it doesn't guard against non-conforming implementations of io.Reader that return -1.

Below I used n >= 0 and not n > 0 because sometimes a zero-length read carries information through the virtue of the read call itself, which I believe @alexbrainman mentioned somewhere earlier in the thread. It's up to you whether you need this information or not.

// Edit: as pointed out below, this is still incorrect as the data should 
// be processed regardless of error's value
for {
   n, err := fd.Read(p)
   if err != nil && err != io.EOF{
       // hard stop; a real error happens
       break
    }
   if n >= 0{
         // handle data in p[:n]
   }
   if err == io.EOF{
     break
  }
}

@jakans and @as, I think, you are still wrong with your code. As per io.Reader documentation: "... When Read encounters an error or end-of-file condition after successfully reading n > 0 bytes, it returns the number of bytes read. ...". So if Read returns n > 0, you must process p[:n] bytes regardless what the err is. And you don't, because you break out of for loop.

@jakans are you still having a problem? If yes, I take it you can reproduce it with a Go program (without curl). Is that correct? If so, can you, show us that program, how you run it, what the program outputs, and why you think the output is wrong. If we need CYGWIN to reproduce, tell us where to get it and how to install it. Thank you.

Alex

I agree it is incorrect, but a mild amount of APIs outside of stdlib return mangled data on a non-EOF error. I would love for this to work everywhere, it a lot easier on the eyes too.

for {
	n, err = r.Read(b)
	if n > 0 {
		// process b[:n]
	}
	if err != nil {
		break
	}
}

presuming from #22024 (comment)

the hardware utilised is 64bit

https://cygwin.com/install.html

Installing and Updating Cygwin for 64-bit versions of Windows

That is the CYGWIN address I was about to pass along.

The current xtract.go source code has the modified -echo command that can handle n == 0 blocks. You can download it by running:

ftp ftp://ftp.ncbi.nlm.nih.gov/entrez/entrezdirect/edirect.tar.gz
gunzip -c edirect.tar.gz | tar xf -
rm edirect.tar.gz

and build the PC executable with:

env GOOS=windows GOARCH=386 go build -o testxtract -v xtract.go
chmod +x testxtract

You should be able to reproduce the hanging problem when built under Go 1.9 by running:

(sleep 1; testxtract -sample) | testxtract -echo

and:

(sleep 1; testxtract -sample) | pv | testxtract -echo

For reference, here is the current -echo source code:

if args[0] == "-echo" {
	const XMLBUFSIZE = 65536 + 16384
	buffr := make([]byte, XMLBUFSIZE)
	for {
		n, err := in.Read(buffr)
		if err != nil {
			if err != io.EOF {
				fmt.Fprintf(os.Stderr, "ERR: %s, N: %d\n", err, n)
				break
			}
			if n == 0 {
				// EOF and no more data
				break
			}
		}
		if n == 0 {
			fmt.Fprintf(os.Stderr, "N: zero\n")
			continue
		}
		fmt.Fprintf(os.Stdout, "%s", buffr[:n])
	}
	return
}

@jakans

If CYGWIN is needed to reproduce, beyond the minimal base packages, what added packages would be relevant?

Not pertinent to this thread, but IIRC historically bi-directional named pipes have been problematic on CYGWIN.

I'm told that the base system should be sufficient.

Thanks for testing it, and of course for all the work to get to this point. Since this fixes the problem, are you and the other developers satisfied with the change, and will be in 1.9.2 and future versions?

No problem. This issue did not affect me personally (yet), I just wanted to help see what the problem was and fix it. I am happy that @alexbrainman got the poller to behave, and time will tell if the change is satisfying.

https://groups.google.com/forum/#!topic/golang-dev/naQlgeWVtvA

@jakans I believe Go 1.9.2 is the subject of the above discussion.

are you and the other developers satisfied with the change,

I satisfy myself that https://go-review.googlesource.com/#/c/go/+/69871 change is not doing bad things. I wouldn't know if it fixes your problem, because I could not reproduce it. But @as checked that. So I will take his word.

and will be in 1.9.2 and future versions?

I suggested that change for go1.9.2 on
https://groups.google.com/forum/#!topic/golang-dev/naQlgeWVtvA

Alex

CL 69871 OK for Go 1.9.2.

It looks like if we are going to fix this for Go 1.9.2, the patch will apply cleanly only if we first also fix #22149, which it looks like we should fix anyway.

Change https://golang.org/cl/70990 mentions this issue: [release-branch.go1.9] internal/poll: only call SetFileCompletionNotificationModes for sockets

It looks like if we are going to fix this for Go 1.9.2, the patch will apply cleanly only if we first also fix #22149, which it looks like we should fix anyway.

SGTM to also fix #22149

Alex

Since I still have the environment to reproduce this issue, I can re-test any intermediate versions if necessary.

CL 53530 OK for Go 1.9.2.
CL 65810 OK for Go 1.9.2.
CL 69871 OK for Go 1.9.2.

go1.9.2 has been packaged and includes:

• CL 53530
• CL 65810
• CL 69871

The release is posted at golang.org/dl.

— golang.org/x/build/cmd/releasebot, Oct 26 21:09:18 UTC

tomnomnom/gron#32 occurs on Go 1.9 but not on 1.10.
The symptom is that the program doesn't read stdin redirect.