Does %q not solve this?

https://go.dev/play/p/i97Z2XmpmcE

It absolutely does. So part of a solution could be making REALLY CLEAR to EVERY user of the package that %v and %s should only ever be used for strings that cannot contain user input.

Also, %q cannot be used safely on structs: https://go.dev/play/p/FLoC2-CJSwk, so %#v seems to be the only safe way to log structs with user input.

But yes, I kinda am open to a documentation-only solution to this issue.

I suppose what you're really after might be a vis(3)-like thing that's built into a special logger function? (log.SafePrintf("%v", attackerControlled)) Or maybe even a modifier to % that does the right thing, without necessarily adding quotes like %q does? (log.Printf("%^v", attackerControlled))

Of these, I like the second idea better.

But there's also an argument to be made that %q is the only safe way to do this anyway, since otherwise the space character becomes an interpretive hazard. For example https://go.dev/play/p/czlaVxlHTwn

So this becomes a kind of hard thing to specify generally, which I guess is why vis(3) has 8 flags for it and why Go just has %q.

I think we have two problems there.

Problem 1: User input can look like e.g. spaces around log messages. A possible solution could be creating something "between" %#v and %+v" that is safe for structs containing user input and virtually anything else (but prints less "noise" - think of a "%+v that uses %q for strings") and till then to recommend that anyone who logs user input shall use %q, %#v or explicit numeric formats, but never %v or %s. I like your idea of a %^v format for this purpose; sadly it is likely not possible to just make %+v quote strings, as that is likely to break existing code.

Problem 2: Multi-line log messages are never "safe". This could be solved as suggested above - or we could explicitly not solve it but explain in documentation that logging of strings containing newlines is strongly discouraged because the output format becomes ambiguous. In addition, we could add as new Logger flag to "harden" the logger against this - basically, a flag Lhardened that makes Logger.Output sanitize what it prints to the writer. Why I believe this is needed even if Problem 1 is solved is that existing code may already be vulnerable, and it is hard to audit code for this issue, so it would be nice to have some defense in depth at runtime at least against the worst of the worst.

As a hack, one could of course instead provide a hardened Writer by assuming that Logger.Output calls Write precisely once. Not sure if such a guarantee should be made, though.

The log package already defines that each logging operation makes exactly one call to the Write method. (This is more or less required to provide good support for simultaneous logging from multiple goroutines.)

This is indistinguishable from "Go away, world" being a real log message. Can be used by an attacker ...

I would argue that the log package is the wrong place to solve this purported security vulnerability. The stderr output of a UNIX process is a stream of bytes, some of which are human-readable lines of text that are vaguely useful to a maintainer. That's the extent of the contract. Any dependency package with which your application logic shares stderr is liable to emit burps of unpredictable data. The real security problem is in a tool that assumes the log is structured. (This includes any log viewer that simply serves raw log files in a form that allows a client to interpret them as HTML or JavaScript.)

This category of security problems is recognized by https://cwe.mitre.org/data/definitions/117.html, but I'm skeptical that changing the expectations around stderr is a practical mitigation. And it is frustrating when overzealous static checkers obstruct the build because of a temporary log.Print(req.Form.whatever) statement in an HTTP handler.

But logs ARE structured. They are an array of lines, each being a timestamp and a string.
And exactly this structure should be encoded clearly without exceptions. This is what I am asking for.
My general expectation of a logging system is that it is its responsibility that log messages are
associated with timestamp and separated cleanly no matter what the input.
Right now Go's log package violates this when newlines are in the string to be logged.

I don't think the maintainers of Go's log package share your expectation. The log package does not provide structured logging: its formatting is nothing more than a convenience. If you want structured logging---and that's a perfectly reasonable thing to want---you'll need to use a different package, one that quotes every argument value, and writes log entries to a different sink than stderr, which is fundamentally unstructured. There are many third-party Go logging packages, structured and unstructured; see https://www.client9.com/logging-packages-in-golang/ for examples.

Another option would be stripping or escaping newlines in output, but it
may not be possible to square that with the compatibility promise for 1.x.

Indeed, we can't change the existing behavior. Many tests rely on the precise formatting of error messages.

Yet another option would be just providing a way to do so (e.g. an
io.Writer wrapper that encapsulates write calls to escape inner newlines ...

By the time of the Write call, all the argument structure has been lost, so it's too late to do quoting then.

If it is the case that the log package is not expected to provide safe
logging provided "bad input", it would make sense to document this fact.

The log package's documentation is quite clear about what it does (timestamp, printf, newline, stderr), and I don't see how a reader could conclude from it that the output is structured or unambiguous. But perhaps you can think of a helpful sentence that makes things clearer, without giving credit to the misconception that stderr is anything other than fundamentally unstructured.

First of all, it is true that stderr can't be very structured, but stderr is only one possible destination of the log package.

By the time of the write call, indeed internal structure of the log line is lost, however the structure of the log itself (array of timestamped lines) is still there.

As for why a reader could conclude that the output is unambiguous as per the log package's own formatting (again, not WITHIN a line of log but ACROSS), simply because other logging systems, e.g. syslog, do ensure that, while also providing no structure guarantees within each log line. If my C program contains syslog(LOG_INFO, "%s just logged in", username), then there is no way to inject a fake timestamp or second log event into the syslog with that - but clearly a malicious user could set their username so this will log "I rule the world, and root just logged in". I would expect "log" to provide the same guarantees as syslog provided the output is not stderr (or provided I ensure that no code outside the log package writes to stderr, which is usually a given - e.g. I am not aware of any Go package that does so without going via the "log" package, which also ensures serialization of calls for this specific reason).

As logs are text files, having escape sequences in them could be similarly dangerous, so maybe a good recommendation would be

Printf calls Output to print to the standard logger. Arguments are handled in the manner of fmt.Printf.

As logging output tends to be written to terminals or text files, it is recommended to only ever log untrusted (including user-provided) textual data with appropriate quoting, as ensured by the %q or %#v format specifiers.

Similarly, log.Print and log.Println should be entirely discouraged on untrusted textual data.

... provided I ensure that no code outside the log package writes to stderr, which is usually a given - e.g. I am not aware of any Go package that does so without going via the "log" package ...

Many Go packages write directly to stderr, sometimes just because debug log wasn't removed.

As for why a reader could conclude that the output is unambiguous as per the log package's own formatting (again, not WITHIN a line of log but ACROSS), simply because other logging systems, e.g. syslog, do ensure that, while also providing no structure guarantees within each log line. If my C program contains syslog(LOG_INFO, "%s just logged in", username), then there is no way to inject a fake timestamp or second log event into the syslog...

If all you need is to ensure that the boundaries of each log entry are unambiguous, the current API affords you the means to do that. Call log.Default().SetOutput, passing an io.Writer that removes interior newlines, perhaps replacing them with the two-byte sequence \n, before passing the result to a call to out.Write, where out is a file other than stderr.

since you bring up syslog, the standard library does have log/syslog

As I see it, the log package makes no claim of safety or that output will all be on one line, only that each call starts on a new line. (I for one would be very annoyed of it started escaping things like newlines automatically). In my experience, if you wanted generic structured output, you'd likely want json or logfmt to make parsing easier, at which point it is out of scope for log.