I am proposing a new option enabling us to NOT use internal directories all over the module and instead use a more centralized configuration at the module level.

Unless I'm misunderstanding what you mean, you don't need to do that. internal directories are accessible from the directory that they're in and any subdirectory thereof. You can just have one internal directory in the module root and every package in the module will be able to access the packages in it. For example,

.
├── go.mod
├── internal
│   ├── a
│   └── b
├── one
├── sub
│   └── three
└── two

one, two, and sub/three can all import both internal/a and internal/b.

What concrete problem would this solve? Can you give some specific examples of repos that already exist that could be substantially improved using this?

@bcmills I guess the problem is the opinionated nature of forcing certain directories of certain names at certain places in the repo. This optional feature would free users from that constraint. Plus, it would be more easily discoverable where internal packages are since they are specified in one central place at the root of the module.

the problem is the opinionated nature of forcing certain directories of certain names at certain places in the repo

"opinionated nature" and enforcing a structure sound like desirable properties of a programming language to me.

@DeedleFake The proposal is a feature that frees people from a particular constraint in favor of a solution that is centralized. Specifying the internal (or not) nature of a package/directory need not require internal directories with a solution like this.

it would be more easily discoverable where internal packages are since they are specified in one central place at the root of the module.

It's easy to discover where internal packages are today: go list ./... | grep '/internal/', or go list -deps . | grep '/internal/' (depending on which internal packages you're looking for).

@bcmills Yes but it is not easy to change or specify them as that requires directory structure manipulation. I think the internal directory solution is great I am just proposing something more sublime, centralized, easier to change, etc. Plus, it would introduce the ability to make an entire module internal, which is currently not possible without an internal folder at the module root. I am not sure that is practical for many repos as mostly I have seen the main package outside of this directory still and so it's not really an internal "module".

the problem is the opinionated nature of forcing certain directories of certain names at certain places in the repo

"opinionated nature" and enforcing a structure sound like desirable properties of a programming language to me.

Yes, I agree and love Go's idiomatic style. I think this change is overdue since the internal packages came around Go 1.4 and modules came around Go 1.11 to 1.13. To me, this a "Go modules" approach to internal.

I don't see these as related. internal is about packages and go.mod is about modules. Today, nothing in go.mod works on the package level.

@gophun Modules and packages are definitely related. Modules are made up exclusively of packages with a certain directory structure which is where internal comes in. In fact, we use internal more like a directory and less like a package. There are generally no Go files directly within an internal directory, further pointing to it being a design smell as a Go package. So it's really not used as a package, it's used as a directory. This can be configured away by using go.mod.

what does it mean for a module to be "internal"? who can use it?
note that you can name an entire module internal, eg github.com/org/internal

what are the rules for who can use internal directories specified centrally like this?
if it's the same rule as before (shared prefix), it sounds like it would make for a worse ux as you now don't know if you can use a package until you check a different file.

what does it mean for a module to be "internal"? who can use it? note that you can name an entire module internal, eg github.com/org/internal

I suppose an internal module is just a module where internal is implicitly at the module root. It's similar to your example of naming a module "internal" except you can name the directories whatever you want.

what are the rules for who can use internal directories specified centrally like this? if it's the same rule as before (shared prefix), it sounds like it would make for a worse ux as you now don't know if you can use a package until you check a different file.

It would be the same rules, a prefix. The UX issue is a fair point but the UX for most is that the packages inside internal just don't show up and are not documented, etc. That would still be the same here. Even better, one would not have to traverse the file structure of a module to find all of the internal "nodes".

EDIT: @bcmills Made a great point that there is a command to run to get all internal packages but imagine the nightmare of tracking how internal packages have changed over the history of the repo. You would be faced with tracking potentially hundreds of file renames or more. If you could configure this sort of thing inside go.mod you would only need to look at the history of that one file to see how internal has changed over time.

I'm trying to form a concrete model of the behavior you are intending in my mind.

What I think you are proposing is that, given the example go.mod file you presented:

internal (
  github.com/org/repo/package
  github.com/org/repo/otherpkg/subpkg
)

...the toolchain should essentially treat both ./package and ./otherpkg/subpkg as if they were named internal, even though they aren't.

I think this would mean that any package under github.com/org/repo can import github.com/org/repo/package, and any package under github.com/org/repo/otherpkg can import github.com/org/repo/otherpkg/subpkg, but no other packages would be able to import them.

This would (I think) match today's treatment if these packages were instead named github.com/org/repo/internal and github.com/org/repo/otherpkg/internal, assuming I'm correctly recalling the current internal package behavior.

Does what I've described above match your intention? Thanks!

@apparentlymart Hi, I appreciate you looking into this humble proposal.

I'm trying to form a concrete model of the behavior you are intending in my mind.

What I think you are proposing is that, given the example go.mod file you presented:

internal (
  github.com/org/repo/package
  github.com/org/repo/otherpkg/subpkg
)

...the toolchain should essentially treat both ./package and ./otherpkg/subpkg as if they were named internal, even though they aren't.

Yes, that seems accurate...

I think this would mean that any package under github.com/org/repo can import github.com/org/repo/package, and any package under github.com/org/repo/otherpkg can import github.com/org/repo/otherpkg/subpkg, but no other packages would be able to import them.

Yes, same as current behavior...

This would (I think) match today's treatment if these packages were instead named github.com/org/repo/internal and github.com/org/repo/otherpkg/internal, assuming I'm correctly recalling the current internal package behavior.

Does what I've described above match your intention? Thanks!

Yes, I believe you captured it. Current behavior says that importing code can only access the internal code if it's within the same tree as the parent of the internal package (from the docs):

When the go command sees an import of a package with internal in its path, it verifies that the package doing the import is within the tree rooted at the parent of the internal directory. For example, a package .../a/b/c/internal/d/e/f can be imported only by code in the directory tree rooted at .../a/b/c. It cannot be imported by code in .../a/b/g or in any other repository.

As I mentioned before, another really cool part of this would be that the module prefix itself would be valid. This would make it possible to mark whole modules as internal without disturbing file structure. For example, you could provide prefix github.com/org/module.

EDIT: I updated the description to clarify, I think if the go.mod block is present, all directories are treated equally and there is no more special treatment of directories named internal. This would be useful in development if an outside test module wanted to import/test that code in a CI scenario. For example, if I wanted to use the Go CLI to modify go.mod and add this internal block so that I can build my test module which imports/tests internal code.