Change https://golang.org/cl/101075 mentions this issue: fixes #24422 leading double quote in non-quoted \t separated field

I believe this is a dup of #19019

cc @dsnet

Issue #19019 is more about not having enough detail in an error, whereas I'm saying we should not have an error at all. I have updated https://play.golang.org/p/YFyJ7uoGcXW example. There are two fields in every row, therefore the error is erroneous.

This is working as intended. A leading quote in a field indicates that the field is quoted. In that situation, the parser must look for a terminating quote. In your example, there is no terminating quote before the next delimiter.

If you want to the CSV parser to treat fields as unescaped fields separated by some delimiter, then csv is not the right package. Most of the complexity comes from quoted-string parsing, and you are intentionally avoiding that functionality.

You are better off doing this yourself as this example:

in := `item-name	item-description	listing-id	seller-sku	price	quantity	open-date	image-url	item-is-marketplace	product-id-type	zshop-shipping-fee	item-note	item-condition	zshop-category1	zshop-browse-path	zshop-storefront-feature	asin1	asin2	asin3	will-ship-internationally	expedited-shipping	zshop-boldface	product-id	bid-for-featured-placement	add-delete	pending-quantity	fulfillment-channel	merchant-shipping-group
"Medaka Box" Stray Bushiroad Storage Box Collection Vol.50 [ Japan Import ]		0124O7XD4ZR	crdcase-bushi-50-medaka-box	14.88	2	24/01/2014 10:08:14 GMT		y	1		Official Item Packed and Shipped Carefully and Quickly	11				B008DQZ0DA						B008DQZ0DA			0	DEFAULT	Migrated Template\
`
for _, s := range strings.Split(in, "\n") {
	if i := strings.Index(s, "#"); i >= 0 {
		s = s[:i]
	}
	if s == "" {
		continue
	}
	records := strings.Split(s, "\t")
	fmt.Printf("%d %q\n", len(records), records)
}

Under https://tools.ietf.org/html/rfc4180#section-2.

Each field may or may not be enclosed in double quotes (however
some programs, such as Microsoft Excel, do not use double quotes
at all).  If fields are not enclosed with double quotes, then
double quotes may not appear inside the fields.

Under that definition then a field which starts with a double quote but does not end with a double quote is not a valid CSV field. So under that strict definition, then yes, I would agree that [it works as intended] (for comma separated values).

However, the encoding/csv package already breaks with the strictest interpretation of the spec via LazyQuotes option. LazyQuotes = true allows double quotes in the field without the entire field being quoted, directly counter to this sentence:

 double quotes may not appear inside the fields.

Clearly the intention of the LazyQuote option is to support files from vendors that do not produce valid quoted fields. If the package allows the flexibility for use LazyQuotes and the flexibility to switch the delimiter to '\t', I would argue that developers would and should expect encoding/csv to be THE defacto standard package for handing csv files and for handling other delimited files by changing the r.Comma rune. Other CSV parsing libs in other languages, such as ruby, can handle delimiting '\t' and do not run into this issue.

The error message generated here https://play.golang.org/p/YFyJ7uoGcXW just makes no sense, (clearly the number of fields is uniform between lines) is unintuitive and wastes developer time. It should be handled.

The CSV spec is silent on using other delimiters. But the package here not only allows alternate delimiters, it also allows flexibility for breaking out of strictly enclosed quoted fields so we should handle this case correctly.

The necessity of using quoted fields at all is partial to using commas as a delimiter in the first place.
But when switching to a tab delimiter, many problems are avoided: commas to do need to be enclosed with double quotes, and double quotes do not need to be escaped since we do not require quoted fields.

If the package advertises that it is flexible to break the spec as a pragmatic consideration to parsing quotes in non-quoted fields and flexible to switching to a non-comma delimiter then I think this issue lies in it's domain.

The paragraph you cite says:

Each field may or may not be enclosed in double quotes

This says nothing about what is a quoted field or not.

The ABNF grammar is as such:

file = [header CRLF] record *(CRLF record) [CRLF]
header = name *(COMMA name)
record = field *(COMMA field)
name = field
field = (escaped / non-escaped)
escaped = DQUOTE *(TEXTDATA / COMMA / CR / LF / 2DQUOTE) DQUOTE
non-escaped = *TEXTDATA
COMMA = %x2C
CR = %x0D ;as per section 6.1 of RFC 2234 [2]
DQUOTE = %x22 ;as per section 6.1 of RFC 2234 [2]
LF = %x0A ;as per section 6.1 of RFC 2234 [2]
CRLF = CR LF ;as per section 6.1 of RFC 2234 [2]
TEXTDATA = %x20-21 / %x23-2B / %x2D-7E

Note that a field may either be escaped or non-escaped. The grammar for escaped insists that it starts and ends with a double quote.

Let's look at an example. The problematic field in question in your data is: "Medaka Box" Stray Bushiroad Storage Box Collection Vol.50 [ Japan Import ]. This field starts with a quote, so it is by definition a quoted field, which must terminate with a quote, but it doesn't.

In order to be valid it would have to be:
"""Medaka Box"" Stray Bushiroad Storage Box Collection Vol.50 [ Japan Import ]" or
""Medaka Box" Stray Bushiroad Storage Box Collection Vol.50 [ Japan Import ]" if LazyQuotes is enabled.