proposal: go/doc: support explicit hotlinking syntax #45533

dsnet · 2021-04-13T00:49:28Z

Introduction

Even though the hotlinks proposal (#25444) was accepted, one reason it wasn't implemented is because detection of identifiers experienced many false positives. See #44447#related-proposals for details.

I propose modifying the hotlinks proposal to use an explicit syntax to indicate the user's intent that some string is a reference to a Go identifier. An explicit syntax should drastically reduce false positives.

Proposal

(The following proposal uses brackets, but other delimiters are reasonable to consider. See #45533 (comment) for alternatives.)

I propose modifying https://pkg.go.dev/ to support hot-linking using a syntax similar to the Markdown syntax for reference-style links. For example, [Buffer.Write] in source code would rendered as Buffer.Write. The reference comprises of a left bracket (i.e, [), some text in-between (e.g., Buffer.Write), and a right bracket (i.e., ]). When rendering, the brackets are removed and the text in-between is linked to the referent. Functionally, it operates just like a Markdown inlined reference where the referent is implicitly provided by GoDoc.

The reference must match one of the following:

[ExportedIdent] where it references a method or field on the current type.
- E.g., [Write] within the documentation of the bytes.Buffer type refers to the bytes.Buffer.Write method.
[ExportedIdent] where it references a variable, constant, type, or function in the current package.
- E.g., [Buffer] in the bytes package would refer to the bytes.Buffer type.
[ExportedIdent.ExportedIdent] where it references a method or field on a type in the package.
- E.g., [Buffer.Write] in the bytes package would refer to the bytes.Buffer.Write method.
[PackageName.ExportedIdent] where it references variable, constant, type, or function in another package.
- E.g., [io.EOF] would refer to the io.EOF variable.
[PackageName.ExportedIdent.ExportedIdent] where it references a method or field on a type in another package.
- E.g., [io.Reader.Read] would refer to the io.Reader.Read method.
["ImportPath"] where it references a package or module.
- E.g., ["google.golang.org/protobuf/proto"] would refer to the proto package.
["ImportPath".ExportedIdent] where it references a method or field in another package.
- E.g., ["google.golang.org/protobuf/proto".Message] would refer to the proto.Message type.
["ImportPath".ExportedIdent.ExportedIdent] where it references a method or field on a type in another package.
- E.g., ["google.golang.org/protobuf/proto".MarshalOptions.Deterministic] would refer to the proto.MarshalOptions.Deterministic field.

ExportedIdent is a valid Go identifier that is exported.

The set of PackageNames that are allowed is determined by what that Go source file imports (e.g., [io.EOF] is only hot-linked if the io package is imported). For example, if the code imports github.com/other/json, then we will link [json.Marshal] to "github.com/other/json".Marshal and not "encoding/json".Marshal.

ImportPath must be a valid import path and either:

be a standard library package path, or
be a multi segment path where the first segment contains a dot and at least one letter (simple heuristic to detect valid domain names).

In order to avoid false positives, the left bracket must be preceded by whitespace and the right bracket must be succeeded by whitespace or punctuation (i.e., period, comma, or semi-colon). Hot-linking is not performed on pre-formatted code blocks (i.e., indented paragraphs).

Examples

This code snippet:

// FooMethod performs the foo functionality.
// It implements ["github.com/example/project".FooInterface].
// The arguments may be of either the map[Key]Foo or map[Key]Bar types.
// Under certain situation, it returns an error that matches [ErrCondition] according to [errors.Is].
// This method may print [INFO] messages to [os.Stderr].
//
// Example usage:
//    t.FooMethod(map[Key]Value{...})
func (t *MyType) FooMethod(...) error

would render as:

FooMethod performs the foo functionality.
It implements "github.com/example/project".FooInterface.
The arguments may be of either the map[Key]Foo or map[Key]Bar types.
Under certain situation, it returns an error that matches ErrCondition according to errors.Is.
This method may print [INFO] messages to os.Stderr.

Example usage:
t.FooMethod(map[Key]Value{...})

Observations:

The brackets for successfully hot-linked references are removed and replaced with an HTML anchor:
Supposing there is a local declaration for Key, the [Key] is not hot-linked since it is not surrounded by whitespace. We avoid hot-linking this since removal of the brackets would render poorly:
- mapKeyFoo
If the reference does not match anything, then it is left as is:
- [Info]
Indented blocks do not have hot-linking applied:
- t.FooMethod(map[Key]Value{...})

See CL/309430 for possible changes made to the protobuf module to make use of this feature.

Design and Analysis

(The following analysis uses the latest version (as of 2021-03-21) of all public modules.)

This feature can be broken down into two problems:

How to identify references in Go documentation (e.g., io.EOF), and
How to identify what these references refer to (e.g., https://pkg.go.dev/io#EOF).

Identifying references

The occurence of [...] occurs ~229k times in Go documentation, most of which should not be hot-linked.

First, we restrict the text within brackets to valid identifiers and/or valid import paths. This reduces the number of matches to ~19k.

Second, we restrict the grammar to require leading whitespace and trailing whitespace (or punctuation). We do this since some of the matches:

are Go code inlined within a paragraph of normal text:
- E.g., map[Key]bool
- E.g., [gcs.KeySize]byte
- While these are technically correct matches, the removal of the brackets would cause the documentation to render strangely (e.g., mapKeyFoo). We could improve this situation in the future, but it's probably better to not link these initially.
are some pre-existing custom markup (usually Markdown):
- E.g., [ConsensusCreateTopicTransactionBody](#proto.ConsensusCreateTopicTransactionBody)
- E.g., [ListClusterAdminCredentials](https://docs.microsoft.com/en-us/rest/api/aks/managedclusters/listclusteradmincredentials)
- It appears that the most common markup syntax is some flavor of Markdown, but this proposal deliberately does not aim to handle arbitrary Markdown, which can be a future proposal.

This restriction reduces the number of matches to ~6k.

Third, we avoid performing hot-linking within indented paragraphs. We do this because:

Indented paragraphs are printed as code blocks and should be preserved ad-verbatim if possible. The removal of brackets could break intentional character alignment that the user desired to be present.
Indented paragraphs often contain code-like snippets, which more often includes the bracket character, leading to false-positives.
This proposal aims to be subset of Markdown and [...] references are not respected within code blocks.
Not hot-linking indented paragraphs provides a means to opt-out of this feature.

This reduces the number of matches to ~3.7k. The list of results at this point can be found here.

For the remaining results:

Most of them are from the same module with it's own markup syntax:
- ~1.1k from godot-go/godot-go
- ~700 from tiezhong2004/iecp5 and forks of it
- ~200 from modernc.org/sqlite
- ~200 from yandex-cloud/go-genproto
Some are generated packages, which don't tend to aim for readable GoDoc packages anyways.
- Example
Some already contain non-standard markup (e.g., HTML). The Go documentation already renders strangely and I don't think this feature is going to make it any worse.
- Example
- Example
Some matches seem to do what the user intended:
- Example
- Example
- Example
- Example

Identifying referents

Of the ~3.7k results, the type of referents are as follows:

~3k are references to locally defined identifier (e.g., MyType.MyMethod)
~500 are references to locally defined identifier, but scoped within some type (e.g., MyMethod)
~60 are references to an identifier in an imported package by package name (e.g., io.EOF)
~5 are references to another package by import path (e.g., "google.golang.org/protobuf/proto")

Hot-linking local referents is relatively easy since we a can derive the set of explicitly defined identifiers within the package using the *doc.Package we have on hand. These referents will never have false positives, but may have false negatives (since we can't easily know implicit declarations obtained through embedding or type aliases).

Hot-linking remote referents is challenging and may lead to false positives. The GoDoc implementation does not assume that it has type information available for other remote packages and we should maintain this property for the implementation. As such, it cannot verify whether some remote reference truly exists and whether some declaration is defined within it.

For references by import path (e.g., ["google.golang.org/protobuf"]), we require that it be valid, that the first path segment must contain a dot since it is always a domain name, and that there be at least one letter. With this heuristic, there were no false positives in the above results.

For references by package name (e.g., [os.Exit]), we determine the set of supported package names based on what packages are imported by the current package. For example, [os.Exit] would not be hot-linked if the os package was not imported. Unfortunately, the package name cannot always be determined given the import statement alone (see #29036). As such, we use a heuristic where the package name is the last path segment if it is a valid identifier. While this may lead to false positives, it is actually the same heuristic that go/doc uses and the https://pkg.go.dev/ is already subject to this potential false positive. To explicitly avoid incorrectly identifying the correct package name, the user code can always use a named import if the real package name is different from the last import path segment.

Even if we could identify the package name, we don't know what declarations exist in that package. In theory we could fetch the documentation for that package, but that would incur significant complexity that doesn't currently exist. Furthermore, for cases where a go.mod file is missing or incomplete, we won't even know what version of the remote package to load. Instead, we hot-link package-scoped identifiers according to the following heuristics:

we don't hot-link standalone package names (e.g., [os] will not be linked to https://pkg.go.dev/os),
we require that the identifiers must be exported (e.g., [os.Exit] will be linked to https://pkg.go.dev/os#Exit, while [os.exit] will not be linked to https://pkg.go.dev/os#exit).

Using these heuristics, none of the existing ~60 matches had non-existing referrents.

Summary:

There is an unwritten goal that Go documentation remain readable without being passed through some type of renderer. The use of brackets adds two characters and does not seem to obstruct reading the documentation directly in source code.
Compared to the original (x/tools/cmd/godoc: add support for hotlinks #25444) proposal, code will generally need to be modified to opt-into using this feature. However, relatively little code will need to be modified to opt-out of accidental matches due to the drastically reduced amount of false-positives.
Over the years there has been requests for Go documentation to support markdown. This proposal does not prevent that possibility as this is a sub-set of Markdown.

Implementation

I can do the work for this since I implemented the original hot-linking proposal, which is actually already merged into the pkgsite code-base, but currently disabled. I suspect that the modifications to the original hot-linking implementation will be relatively minimal.

Note: I originally considered using back-ticks as the marker. Credit goes to @rsc for suggesting the use of brackets to match the Markdown syntax for reference-style links.

The text was updated successfully, but these errors were encountered:

dsnet · 2021-04-13T00:49:49Z

\cc @julieqiu @jba @dmitshur

mvdan · 2021-04-13T08:15:13Z

I assume that full markdown links, such as [foo](bar) or [foo][3], would be left untouched by this change? That's how I understood the proposal, but just making sure.

that the first path segment must contain a dot since it is always a domain name

You might be interested in #32819, since this "domain" rule is currently unwritten.

Also note #37641, though I don't think having pkgsite support links within module example or module test is hugely important.

we don't hot-link standalone package names (e.g., [os] will not be linked to https://pkg.go.dev/os),

I assume that the advice there would be to link the entire import path instead, which makes sense to me.

I originally considered using back-ticks as the marker.

I've seen lots of code use backticks in ways that fit your supported bracket syntax, modulo the surrounding character. I wonder if we could have some way to automatically convert those that would result in a valid link, such as `MyType.MyMethod` to [MyType.MyMethod]. Maybe via go fix, using the same heuristic code?

peterbourgon · 2021-04-13T08:24:50Z

I strongly object to this proposal.

mvdan · 2021-04-13T08:28:07Z

@peterbourgon care to explain why?

peterbourgon · 2021-04-13T08:46:52Z

Go documentation currently reads fluently as prose, whether in source or lightly rendered form. Semantic markup turns it into something else: source code in the .go file, compiled artifact in the browser.

Why? Why change this lovely, lightweight, refreshing aspect of the language into something that it explicitly chose not to be when designed?

As usual, I understand the value that this proposal would deliver. Any conceivable change delivers value to someone, somehow, somewhere. Should we add everything that has some value to the language? I hope the clear answer of "no" is uncontroversial. But, man, there are just so many proposals being thrown against the wall in the last year, to add more and more stuff to the language, almost never (IMO) with sufficient motivating cause. The language loses its definition, its principles, as it expands in this way, but those things are important to maintain.

mvdan · 2021-04-13T09:18:41Z

For what it's worth, I agree that plaintext documentation is nice. But links are also nice, and proposals to add some form of them were accepted as early as 2017. It's four years later, and automatic hot-linking has not worked, as Joe explained in the detailed proposal.

Your answer then might be "we'll never have links then", but I'd personally find that very unfortunate.

I ultimately defer to Joe, who has spent five years thinking about this problem and attempting different solutions. If we can agree that links are a nice thing to have, then I think any criticism would have far more value in the form of alternative ways to add the feature.

peterbourgon · 2021-04-13T09:26:11Z

If we can agree that links are a nice thing to have,

Insofar as links mean semantic markup, I don't agree. I don't want any kind of semantic markup in Go package docs.

josharian · 2021-04-13T13:38:17Z

there are just so many proposals being thrown against the wall in the last year

This is a natural consequence of Go’s growth. I for one welcome this; it’s noisier, and messier, but I believe it’s also valuable. More voices leads to better outcomes (after more time and more work).

In any case, that hardly seems to apply to this proposal. This is a refinement of a proposal accepted years ago—one which Joe chose not to implement because he didn’t feel at the time that it hit the high bar that Go maintains, despite it being accepted.

Go documentation currently reads fluently as prose

This particular proposal strikes me as not impeding fluent prose but augmenting it. (I would not say so about adding full markdown links, which adds extra non-textual letters.)

seebs · 2021-04-13T14:43:24Z

I'm sort of wary of this because it seems like it's sort of error-prone, but I like the idea sort of? I usually use "name" for references to names, though, which I like better than "[Name]".

I would like to be able to indicate that [os] is a reference to the os built-in package, but I can't make the import path any fuller...

dsnet · 2021-04-13T19:30:30Z

@peterbourgon

The fact that Go documentation remains readable in un-rendered form is a good goal that we should keep aiming for. This proposal only adds two characters of markup, which doesn't seem too obtrusive (obviously a subjective metric). However, readable documentation on a single declaration isn't the only good goal to aim for.

At the present moment, we have a deficiency where the entire documentation for a Go package can be somewhat incomprehensible. The documentation is primarily presented as a list of exported declarations sorted alphabetically (with some grouping based on type). Due to the alphabetical sorting, it can be difficult approaching a new package since the primary functionality may not be near the top, and related functionality are not co-located in the ordering (e.g., the spread of declarations in the http or proto packages).

For the author of a package, there aren't many good options available to present the information in a way that's most accessible to new users. Since they can't control the ordering of declarations†, the best option is to describe high-level organization of the package in the package-level documentation that always appears at the top (e.g., similar to what the http and proto packages do). However, currently there is no way to reference declarations from this documentation to the more detailed documentation (that's what the hotlinks feature aims to resolve: #25444, and what this issue aims to improve on). Alternatively, we could provide the course-grain ability to order and group the declarations in a package (that's what the sections proposal aims to resolve: #44447). I personally prefer the sections proposal, but I'll take either the sections or hotlinks features over nothing.

† Technically, you can by carefully naming declarations so that it sorts as desired, but that's not a fun game to play. For example, I sometimes find myself putting the adjective after the noun just so all declarations with the same noun sort together. Othertimes, there just aren't any good alternative names to order declarations.

dsnet · 2021-04-13T19:35:32Z

I assume that full markdown links, such as foo or [foo][3], would be left untouched by this change? That's how I understood the proposal, but just making sure.

Correct. They will be left as is.

You might be interested in #32819, since this "domain" rule is currently unwritten.

At least for the pkgsite, it is functionally enforced since every non-standard package must be fetchable over the internet.

I would like to be able to indicate that [os] is a reference to the os built-in package, but I can't make the import path any fuller...

You could still use ["os"] where you refer to it by package path. The reason why I decided not to support referencing by package name alone is due to the number of existing matches. Granted, ~60 isn't that many and most of them are already have documentation with custom markup, so it may actually be fine.

ianlancetaylor · 2021-04-13T19:43:33Z

CC @robpike

kortschak · 2021-04-13T22:03:57Z

@dsnet Was there a particular reason to choose the brackets for this markup? It is similar to Markdown, but that doesn't really seem relevant. If you perform your analysis with other less typographically obtrusive markers with the same rules do you get similar results, say ` and ', or ` and `. I think at least part of the problem here is the problem of anecdotal reports of the impact of sigils on reading. Choosing sigils that are less obtrusive or less semiotically laden with other meanings ('[' and ']' have significant meaning in normal text that differs from the intention here, while quote marks are less different) may be less problematic.

dsnet · 2021-04-14T05:26:47Z

It is similar to Markdown, but that doesn't really seem relevant.

There's minor benefit to using a syntax that is similar to some other well-known markup language for something semantically similar (i.e., making links). I'm not tied to using brackets and my original idea was actually to use backticks.

EDIT: The analysis below is wrong. There was a bug in my tool. I'm re-running the tool.

Running my tool to use backticks instead: * There are ~993k occurrences of `` `...` ``. * Requiring that they be syntactically valid identifiers and import paths reduces the results down to ~297k matches. * Further requiring leading whitespace and trailing whitespace (or punctuation) reduces the results down to ~291k matches. * Further restricting hotlinking from applying in indented paragraphs reduces the results down to ~276k matches.
Of those references, the type of referrants are:

~251k are a reference to a local declaration (e.g., Buffer)

~15k are a reference to a local declaration scoped within a type (e.g., Write)

~6.7k are a reference to a local declaration scoped within the local package name (e.g., bytes.Buffer)

461x are a reference to a remote declaration scoped within a locally imported package name (e.g., imported_package.RemoteType)

60x are a reference to a remote package path (e.g., "github.com/example/remote_package")

21x are a reference to a standard package path (e.g., "encoding/json")

1x are a reference to a remote declaration scoped within a standard package path (e.g., "encoding/json".Marshal)

I mentioned in the introduction that remote references can have false positives since we don't know whether those remote packages actually exist or have the given declaration. Of all the remote references, %60.2 of them would be dead links (i.e., they refer to a declaration or path that actually doesn't exist.). In contrast, the use of brackets has almost none of them being dead links.

~~The fact that backticks are already so widely used is a double edged sword. Most of them seem to be actual references to local and remote declarations, but a non-trivial number also are not.~~

kortschak · 2021-04-14T05:30:18Z

Yes, I suspected that paired ` would be higher, but I also included ` and ' which I suspect (no evidence, but based on the observation that the differentiated leading and lagging single quote usage is less common now than it used to be except in users of LaTeX) would be lower. Do you have data for that?

dsnet · 2021-04-14T09:08:38Z

@kortschak

(disregard my earlier comment; there was a bug in my tool that invalidated all results)

Total number of existing matches for each syntax form:

	`...`	`[...]`	`...'
All matches	991268	210049	2267
After filtering for valid identifiers and paths	308972	18481	109
After requiring surrounding whitespace	302230	6397	68
After removing for indented paragraphs	284562	3785	53

The type of each link referent:

Target Type	`...`	`[...]`	`...'	Description
LocalIdentifier	250676	3061	30	reference to local declaration (e.g., `Buffer`)
LocalScopedIdentifier	15522	510	10	reference to local declaration within scope of a type (e.g., `Write`)
LocalPackageIdentifier	6853	0	0	reference to local declaration within scope of package name (e.g., `bytes.Buffer`)
LocalPackage	2398	78	7	reference to local package by package name(e.g., `bytes`)
RemotePackageIdentifier	3650	57	0	reference to remote declaration within scope of imported package name (e.g., `imported_package.MyType`)
RemotePackage	715	54	1	reference to remote package by imported package name (e.g., `imported_package`)
RemotePathIdentifier	0	0	0	reference to remote declaration within scope of package path (e.g., `"github.com/example/remote_package".MyType`)
RemotePath	60	0	0	reference to remote package by package path (e.g., `"github.com/example/remote_package"`)
StandardPackageIdentifier	3628	3	2	reference to declaration package within scope of imported package name (e.g., `json.Marshal`)
StandardPackage	1038	14	3	reference to standard package by imported package name (e.g., `json`)
StandardPathIdentifier	1	0	0	reference to standard declaration within scope of package path (e.g., `"encoding/json".Marshal`)
StandardPath	21	8	0	reference to standard package by package path (e.g., `"encoding/json"`)

As mentioned in the proposal above, there may be false positives when linking to remote references where the target package or declaration does not exist.

Syntax	Accuracy
`...`	93.7%
`[...]`	97.7%
`...'	100%

All the results for the analysis can be found here.

mvdan · 2021-04-14T09:32:10Z

For the false positives involving `...`, could we not detect that they are dead links and leave them untouched? That gives authors time to fix them if they so wish, without breaking any other uses of this syntax that don't relate to linking.

smasher164 · 2021-04-14T16:02:08Z

@peterbourgon
Depending on your definition of "semantic markup," I would argue that Go comments already have that in the form of directives. I hardly think

// TokenType represents the category of lexeme scanned from the input.
//go:stringer -type Token
type TokenType int

meets your idyllic definition of prose. And IIRC, these comments are now stripped by godoc. The only difference here is the tool that primarily ingests these comments is the documentation tool, not the compiler or some code generation tool.

FWIW, I think the bracket syntax is human readable, especially since it kind of converges with the way type parameters and map KeyTypes are presented.

peterbourgon · 2021-04-14T16:14:12Z

@smasher164 Directives aren't part of Go documentation.

smasher164 · 2021-04-14T16:20:28Z

Directives aren't part of Go documentation.

They used to show up in godoc until it specifically recognized and stripped the comment. My point is that like with this proposal, they are an artifact in Go comments that manipulated by some tool. At least with this change, the comment is still (arguably) human readable.

seebs · 2021-04-14T16:32:10Z

So, as a practical reality, right now, we have existing things which are using some amount of markup in doc comments, and existing things which are parsing things in doc comments as markup to varying extents, not always in desireable ways. For instance:

// DoThingToPointer takes a *T and modifies the pointed-to object,
// returning a *T which may be the original object or a new object.

Might come out as documentation saying:

DoThingToPointer takes a T and modifies the pointed-to object, returning a T which may be the original object or a new object.

(Only with the space at the end also italicized.)

So I think the ship has already sailed on whether people might choose to interpret doc comments as markdown.

When I've been intending to distinguish between a type or function that I'm referring to, and just a plain word that might be capitalized, I've usually written it with backticks.

It's obviously much harder to measure the accuracy rate in the other direction, which is to say, checking for things which might be intended as linkable references, but which are not currently marked, or are marked with either brackets or backticks. My intuition is that the false-positive rate of backticks is higher than the false-positive rate of brackets, but also that the true-positive rate would be dramatically higher. So, in the sample corpus, roughly 30% of backticked names appear to be plausibly things that were intended as linkable references, while under 10% of bracketed names did.

Of the ~280k backticked things that seem like they're valid references, how many are "valid" in the sense of "the thing they appear to refer to actually exists"? Establishing the rate of "plausibly-valid" vs "apparently doesn't exist" links that the tool would generate might be also informative. Basically, a higher rate of false positives that also comes with a higher rate of true positives may be better than a lower rate of both.

I think this is a convenience feature. It seems to me that if it has an error rate that can be corrected by updating comments in some cases, that's probably pretty livable, if it provides a significant improvement in quality-of-life. Possibly it should be an optional feature, and then we could do tests comparing the experience of using the docs with and without it.

peterbourgon · 2021-04-14T16:42:09Z

@dsnet

At the present moment, we have a deficiency where the entire documentation for a Go package can be somewhat incomprehensible. The documentation is primarily presented as a list of exported declarations sorted alphabetically (with some grouping based on type). Due to the alphabetical sorting, it can be difficult approaching a new package since the primary functionality may not be near the top, and related functionality are not co-located in the ordering (e.g., the spread of declarations in the http or proto packages).

For the author of a package, there aren't many good options available to present the information in a way that's most accessible to new users. Since they can't control the ordering of declarations†, the best option is to describe high-level organization of the package in the package-level documentation that always appears at the top (e.g., similar to what the http and proto packages do).

Why do you see these things as deficiencies? The doc block at the top of the page serves well as a home for a didactic introduction at the authors' discretion, and the consistent ordering of identifiers makes it easy to find things consistently between packages.

However, currently there is no way to reference declarations from this documentation to the more detailed documentation (that's what the hotlinks feature aims to resolve: #25444, and what this issue aims to improve on).

I agree that the ability to link from the package docs to a package identifier would be valuable. But does it represent so much more value over Cmd+F that it sufficiently motivates this very large change to the language?

seebs · 2021-04-14T16:45:53Z

Being aware of import paths actually creates some interesting qualities.

Consider what happens after

import (
  ...
  "github.example.com/nonstandard/json"
)

...
// SomeFunc uses `json.Unmarshal` to extract values from the provided bytes

In this case, it seems to me that the correct link from json is to the docs for github.example.com/nonstandard/json, not to the docs for stdlib's encoding/json.

This might actually be extremely helpful if you're looking at multiple things which all use different types packages, since there's more than one of those floating around...

dsnet · 2021-04-14T17:34:38Z

For the false positives involving ..., could we not detect that they are dead links and leave them untouched?

@mvdan, yes, in theory. Most (if not all) of the GoDoc-like implementations operate on a standalone ast.Package. It's a fairly large architectural change to various GoDoc tool to support looking at dependencies. There is much value that can be gained from information in dependencies even for current features and proposed features of GoDoc (e.g., #44905), but I think we should consider such a change separately given it's wide impact.

Note that dead links are already possible today due to #29036 and the fact that GoDoc tools use a trivial ast.Importer. Thus, it seems that GoDoc tools have already accepted the fact that some degree of dead links is already acceptable.

dsnet · 2021-04-14T17:39:04Z

Being aware of import paths actually creates some interesting qualities.

@seebs, I'm not sure if you were suggesting that the proposal should do this or that you're pleased that the proposal already suggests this. My current prototype will correctly link to github.example.com/nonstandard/json.Marshal since it notices that the json package name likely came from the local import of github.example.com/nonstandard/json in the source code. Of course, whether it gets this right is subject to package name ambiguities due to #29036.

dsnet · 2021-04-14T17:52:16Z

Why do you see these things as deficiencies? The doc block at the top of the page serves well as a home for a didactic introduction at the authors' discretion, and the consistent ordering of identifiers makes it easy to find things consistently between packages.

@peterbourgon I don't believe consistent ordering implies ease of finding things. Many Go packages have groups of declarations that are related to each other in some way. I want some way to express this grouping to the user (either with sections or hotlinks).

But does it represent so much more value over Cmd+F that it sufficiently motivates this very large change to the language?

This isn't a change to the Go language. It's a change to how tools interpret Go comments in source code. Unlike a language change, failure of the tool to parse the markup doesn't result in a build failure or buggy program. Furthermore, tools that don't support this syntax still presents comments that are readable.

And yes, I do believe it brings sufficient benefit and that the benefit outweighs the readability cost of two characters. Imagine trying to read an Wikipedia article where there were no links (either to other targets within the same article or to other articles). You could argue that the user has the ability to use the find feature to get to what they need, but life is easier being able to just click on the link.

seebs · 2021-04-14T18:46:27Z

To clarify, I had noticed that this seemed to be part of the proposal, and it surprised me at first, but actually it seems really useful and makes me more in favor of the thing, because if I'm reading the docs, I'm more likely to miss distinctions between, say, [stdlib]/errors and github.com/pkg/errors, but this fixes that.

peterbourgon · 2021-04-14T19:05:46Z

@dsnet

And yes, I do believe it brings sufficient benefit and that the benefit outweighs the readability cost of two characters.

Two characters now. And in six months, two more characters, permitting links to arbitrary scopes. Then, an alternative form to allow authors provide custom link text. Then, since we have links, couldn't we have bold and italic modifiers? Tables? And so on.

But, I've made my position clear, and I appear to be in the minority, so I'll leave it at that.

dsnet · 2021-04-14T19:12:02Z

@peterbourgon, the slippery slope argument is valid and reasonable concern, but it alone can't be the justification for why nothing should be added. Nearly every feature ever added has to sit somewhere on the slippery slope.

josharian · 2021-04-14T19:20:38Z

@dsnet I believe you just countered a slippery slope argument with a slippery slope argument. :P

jba · 2021-04-14T20:10:05Z

It is a slippery slope, but the magnitude of the slope is 1 feature/decade.

ChrisHines · 2021-04-16T17:40:27Z

I can see the value of the proposal, but like others am leary of adding noise to the experience of reading the docs in their native form. I am not strongly against the proposal, but I believe its implementation should be as lightweight as possible if it is accepted.

When I've been intending to distinguish between a type or function that I'm referring to, and just a plain word that might be capitalized, I've usually written it with backticks.

I have seen other developers do this as well, and it has always been back-ticks. I've seen back-ticks used to mark function parameter names used in the docs as well.

Thinking ahead, I would not be in favor of a proposal for full markdown support. That's a bridge too far in my opinion. With that position staked out I don't see any value in choosing square brackets as the markup for this proposal. Back-ticks are lighter on the page and I think there is some precedent for using them to mark identifiers in documentation-ish things in Go. Consider that flag.PrintDefaults interprets back-ticks in a somewhat similar fashion in flag usage messages.

My current position on this proposal is 👎 in it's current form due to the square brackets being too heavy and no reason to be compatible with markdown. Switching to back-ticks changes my opinion to a more neutral stance.

@seebs asked:

Of the ~280k backticked things that seem like they're valid references, how many are "valid" in the sense of "the thing they appear to refer to actually exists"? Establishing the rate of "plausibly-valid" vs "apparently doesn't exist" links that the tool would generate might be also informative. Basically, a higher rate of false positives that also comes with a higher rate of true positives may be better than a lower rate of both.

I think this is a good question that hasn't been answered yet.

rsc · 2021-04-21T18:02:43Z

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

rsc · 2021-04-21T18:07:19Z

x/pkgsite is a go/doc renderer. If we make a change here, it would be for all of go/doc, not just x/pkgsite. Retitled.

kylelemons · 2021-08-28T17:40:40Z

I have used backticks in my own documentation before, and I think the data looks like this would result in many more links without users having to do it themselves. I'm in support of this proposal under whichever syntax, but that's the one I'm drawn to. I also tend to think the markdown ship has sailed.

I wonder if some of the code used for the analysis or implementation could be used to adjust whichever syntaxes we don't choose to make a gofix-like tool to swap them, and maybe even have an optional flag to add the syntax to #25444 compatible matches for authors to review.

rsc · 2021-09-10T00:27:55Z

See #48305.

rsc · 2022-02-09T18:42:57Z

Closing as duplicate of #51082.

rsc · 2022-02-09T19:17:16Z

This proposal is a duplicate of a previously discussed proposal, as noted above,
and there is no significant new information to justify reopening the discussion.
The issue has therefore been declined as a duplicate.
— rsc for the proposal review group

gopherbot added this to the Proposal milestone Apr 13, 2021

gopherbot added the Proposal label Apr 13, 2021

dsnet added pkgsite and removed Proposal labels Apr 13, 2021

This was referenced Apr 13, 2021

x/tools/cmd/godoc: add support for hotlinks #25444

Closed

proposal: go/doc: add support for sections #44447

Open

This comment has been minimized.

Sign in to view

ianlancetaylor added the Proposal label Apr 13, 2021

ianlancetaylor added this to Incoming in Proposals (old) Apr 13, 2021

rsc moved this from Incoming to Active in Proposals (old) Apr 21, 2021

rsc changed the title ~~proposal: x/pkgsite: support explicit hotlinking syntax~~ proposal: go/doc: support explicit hotlinking syntax Apr 21, 2021

rsc mentioned this issue Apr 21, 2021

proposal: review meeting minutes #33502

Open

dsnet mentioned this issue Jun 4, 2021

x/tools/cmd/godoc: generate static docs #2381

Open

rsc moved this from Active to Declined in Proposals (old) Feb 9, 2022

rsc closed this as completed Feb 9, 2022

golang locked and limited conversation to collaborators Feb 9, 2023

gopherbot added the FrozenDueToAge label Feb 9, 2023

proposal: go/doc: support explicit hotlinking syntax #45533

proposal: go/doc: support explicit hotlinking syntax #45533

Comments

dsnet commented Apr 13, 2021 • edited

Introduction

Proposal

Examples

Design and Analysis

Identifying references

Identifying referents

Summary:

Implementation

dsnet commented Apr 13, 2021

This comment has been minimized.

This comment has been minimized.

mvdan commented Apr 13, 2021

peterbourgon commented Apr 13, 2021

mvdan commented Apr 13, 2021

peterbourgon commented Apr 13, 2021

mvdan commented Apr 13, 2021

peterbourgon commented Apr 13, 2021 • edited

josharian commented Apr 13, 2021 • edited

seebs commented Apr 13, 2021

dsnet commented Apr 13, 2021

dsnet commented Apr 13, 2021 • edited

ianlancetaylor commented Apr 13, 2021

kortschak commented Apr 13, 2021

dsnet commented Apr 14, 2021 • edited

kortschak commented Apr 14, 2021 • edited

dsnet commented Apr 14, 2021 • edited

mvdan commented Apr 14, 2021

smasher164 commented Apr 14, 2021

peterbourgon commented Apr 14, 2021

smasher164 commented Apr 14, 2021

seebs commented Apr 14, 2021

peterbourgon commented Apr 14, 2021

seebs commented Apr 14, 2021 • edited

dsnet commented Apr 14, 2021 • edited

dsnet commented Apr 14, 2021

dsnet commented Apr 14, 2021 • edited

seebs commented Apr 14, 2021

peterbourgon commented Apr 14, 2021

dsnet commented Apr 14, 2021

josharian commented Apr 14, 2021

jba commented Apr 14, 2021

ChrisHines commented Apr 16, 2021 • edited

rsc commented Apr 21, 2021

rsc commented Apr 21, 2021

kylelemons commented Aug 28, 2021

rsc commented Sep 10, 2021

rsc commented Feb 9, 2022

rsc commented Feb 9, 2022

dsnet commented Apr 13, 2021 •

edited

peterbourgon commented Apr 13, 2021 •

edited

josharian commented Apr 13, 2021 •

edited

dsnet commented Apr 13, 2021 •

edited

dsnet commented Apr 14, 2021 •

edited

kortschak commented Apr 14, 2021 •

edited

dsnet commented Apr 14, 2021 •

edited

seebs commented Apr 14, 2021 •

edited

dsnet commented Apr 14, 2021 •

edited

dsnet commented Apr 14, 2021 •

edited

ChrisHines commented Apr 16, 2021 •

edited