The first part of this proposal seems closely related to #6386.

I'm not sure I understand the second part, but it seems to just be a compiler optimization. That does not need to be a proposal at all. The proposal process is primarily for user visible changes.

I'm not sure I understand the second part, but it seems to just be a compiler optimization. That does not need to be a proposal at all. The proposal process is primarily for user visible changes.

Hmm, in some cases compiler optimizations enable the user to write visibly different code, and this is such a case. Like how tail-recursion optimization or lightweight-goroutines change user behavior.

Whether Go2 decides to adopt the Compiler optimizations for immutable struct assignment proposal or not dictates how I write my code today (for Go1) and when I decode to use pointers throughout the code. And implementing this optimization later will require me to refactor a lot of code, probably introducing bugs in the process.

And implementing this optimization later will require me to refactor a lot of code, probably introducing bugs in the process.

The way to refactor with confidence is to ensure your code has good unit test coverage, not insist that the langauge implements a syntax change before you start to write.

I checked Wikipedia for what capabilities based security means and it looks like most use cases are in hardware, intel iAXP432, Burroughs B5000, etc, or an operating system concept like the capabilities mask on processes in Linux.

I can see why this would be useful, but as this proposal reads today you’re not asking for capabilities on references (I’m guessing this is how it works at a programming language level), instead your asking for much requested of features like immutability and generic types.

Given that this proposal cannot be successful unless you get all the things you ask for in the proposal, and bluntly, when you ask for more than one thing, you risk getting none of them.

My suggestion would be to abandon this proposal and instead contribute to the discussion for generic types and immutablility.

I'm not sure that I understand why the word security is used in the proposal. What is your definition of security here?

For the second proposal you did point out the existing solution:

The type of values you can assign to a const are more expressive, so we now have no excuse to declare module-level var variables which can be (maliciously or not) modified by anyone who imports the module.
(Of course we can declare a global function func GetMyStructP() *MyStruct { ... }) that more or less does the same thing, but nobody does that because it's easier to use var, and what's the point of doing the right thing unless there's a commitment to evolve the language toward a capabilities-based system?).

For the third proposal, there are cases where modifying the stack copy of the struct in a method makes sense, such as for changing parameters before calling another method without effects on the original struct.

If you are providing a struct and you don’t want somebody to edit the contents then the private field plus getter/setter methods is still a solution.

The empty interface case is interesting because assigning the type assertion to a variable gives you a copy and it can be passed as a reference instead of the full struct. Doesn’t emptyInterfaceVar.(pack.MyType).Field = value assign to the original memory though?

An approach may be to have a wrapper struct pointing at your base struct that has no public fields and only has a Copy() method. This way you pass just the pointer down the stack but the person using it can’t get at the original memory.

I still think that running untrusted plugins in a separate process is the right solution. Here's some discussion on how to implement communication: https://stackoverflow.com/questions/9366550/go-inter-process-communication

Imagine writing a web browser (implemented in Golang), but instead of Javascript for websites, you have Golang scripts. When you visit a webpage, the browser loads the Golang script, parses it, filters it, mutates it, then compiles and loads it as a plugin. When you close the tab, the plugin gets released. What's currently missing is the ability to release the plugin as in the last step.

This is a great idea. Since Javascript is being written everywhere with the addition of Node.js, why not Go everywhere? We'd need a jQuery syntax.

Cancellation is a problem the context package handles, but it requires the context to be explicitly checked during processing. Since you are able to make additions to the arbitrary Go scripts you could have the plugin IPC handler trigger a context cancellation when a cancel is sent and have context checks somehow added to the script.

I'm not sure that I understand why the word security is used in the proposal. What is your definition of security here?

One or more dependencies may be malicious. I'd like to defend my process against malicious dependencies.

In a secure language ecosystem, if a module or a function argument passed in or a value returned provides the user with the capability to do something, it's because that is the intent. Ability is equivalent to right.

Please take a look at the top post, I've updated it with clarifications.

Given that this proposal cannot be successful unless you get all the things you ask for in the proposal, and bluntly, when you ask for more than one thing, you risk getting none of them.

• The meta-proposal gives each sub-proposal more meaning. It gives a set of proposals a particular motivation, thus can help guide the discussion for each sub-proposal.
• Each sub-proposal in isolation doesn't make Go a more "secure" language, but the combination of all of them does, much more-so than the sum of its parts. Security is a product of lack-of-insecurities (not an addition of features).
• I'm seeking feedback to see if there are other sub-proposals that I might have missed.
• When you don't ask for something, you generally don't get it. I like moonshots :)

The empty interface case is interesting because assigning the type assertion to a variable gives you a copy and it can be passed as a reference instead of the full struct. Doesn’t emptyInterfaceVar.(pack.MyType).Field = value assign to the original memory though?

If it works, it's because it's the same var. Once you assign/copy it to another var, mutating the other var doesn't affect my var.

An approach may be to have a wrapper struct pointing at your base struct that has no public fields and only has a Copy() method. This way you pass just the pointer down the stack but the person using it can’t get at the original memory.

Yes. My point is that the ability to do that is not sufficient to make the whole ecosystem "secure". If everyone did the above for every structure as needed, it wouldn't be a problem. But we don't, because there's no motivation to, because there is no commitment starting from the language spec and implementation. It's a chicken and egg problem, but I'm not in a position to be hacking on Golang right now, so here is my contribution. :)

For the second proposal you did point out the existing solution:

Please check the top post, I showed why that solution isn't used, and pointed out the example w/ crypto/rand. Examples aren't hard to find, and it's just a matter of time until someone exploits it somewhere.

My suggestion would be to abandon this proposal and instead contribute to the discussion for generic types and immutablility.

I'll make appropriate sub-proposals, but for reasons stated this proposal should exist on its own. That is, each proposal in isolation isn't very motivating.

I still think that running untrusted plugins in a separate process is the right solution. Here's some discussion on how to implement communication: https://stackoverflow.com/questions/9366550/go-inter-process-communication

You're right about that. I think the plugin argument isn't needed to make a case for this proposal (since dependencies can always be malicious) so I will remove that from the proposal.

I'll split this proposal up as suggested, but I'd like to keep this issue as a reference to the whole concept of making Golang more secure against malicious dependencies.

I think the problem of mutable package level state is better solved by not having package level state rather than finding ways to make package level state immutable.

The language could enforce that no module can have exposed variables. That would be an acceptable solution.

The proposal here is not to make module level state "immutable". Consts can still be mutable internally, but the pointers that they refer to cannot be swapped out for another pointer. I'd like for Golang to have these mutable consts (they're so convenient), and to disallow exported vars entirely.

My mistake, I pressed the wrong button.

@jaekwon : I'm confused as to what your threat model is that you're trying to defend against. I think there are two categories: buggy or otherwise inadvertent mistakes, and nefarious code. I think it is reasonable to provide some help to programmers to avoid the former. I think it is a much taller order to do anything about the latter. Go just isn't very well suited to defending against adversaries that can inject code. A few reasons:

1. Package unsafe allows anything. It's pretty easy to forbid direct use of unsafe. It's harder to prevent indirect use of unsafe due to bugs in other packages. See for instance CL 83255 about a strings.Builder bug.
2. Data races. Hard to defend against. See my post on golang-nuts for how use a data race to get around the type system. The Go playground once defended against this by always running with GONUMPROCS=1, but that's not really a general solution. The playground has since moved to a sandboxed environment.
3. No one has spent significant effort hardening the compiler, linker, runtime, etc. against nefarious code attacks. We fix bugs when we find them, but there would have to be a lot more effort put into verification before I trusted our software stack against such attacks.

The threat model is reality, with software such as go-ethereum, tendermint, or docker with complex external dependencies and not enough time, money, or human-capital to audit everything. The threat model is our financial infrastructure being exploited by hackers for personal gain through whatever means possible.

1. it's easy enough to scan dependencies for usage of unsafe.
2. great, we found another bug/subproposal :)
3. is this too significant of an endeavor for Go2 proposals?

Lets do the hard work of making it resilient against nefarious code, and make this be a guiding principle for the design of Go2. It doesn't matter how it's done as long as it's done with the spirit of Go1 -- "simple", "explicit", "orthogonal", "concurrent". I'm suggesting we add "secure". It doesn't have to be perfect, but it should work towards it, and make decisions on language changes and classification of bug/feature accordingly.

Why not? Don't you want Go to be the Go2 language for everyone? It's awesome!

1. it's easy enough to scan dependencies for usage of unsafe.

Right, if you're willing to just say no unsafe altogether, then fine. But that means no reflect, no math, no strings, .... Even if you're willing to whitelist the stdlib (see point 3), what about other popular libraries, like proto?

2. great, we found another bug/subproposal :)

This is one I don't know of any solution for. Except sandboxing a whole process. Which coincidentally also solves point 1 and 3.

3. is this too significant of an endeavor for Go2 proposals?

A concrete proposal about how to be more sure of the correctness of the compiler/runtime/stdlib would be welcome. But they are all moving targets, so it would be complicated. Probably not just technically complicated, but change process complicated.

In any case, without solving point 2, it is all kinda moot. No point in locking your windows when the door is open.

Right, if you're willing to just say no unsafe altogether, then fine. But that means no reflect, no math, no strings, .... Even if you're willing to whitelist the stdlib (see point 3), what about other popular libraries, like proto?

I like the whitelist idea. Yes, protobuf seems worthy to audit, so it could be whitelisted too. I imagine most libraries otherwise don't use unsafe.

This is one I don't know of any solution for. Except sandboxing a whole process. Which coincidentally also solves point 1 and 3.

Yeah, confirmed that it's a problem on my mac. Hmm...

Disclosure: I work at Google, but not related to Go team. Kenton Varda and I had a proposal on this topic long before Go was first released. I think the original proposal is still valid to these days, so I share it here for discussion purpose.

The proposal (nickname gocap) is to introduce the concept of stateless package. A stateless package cannot have mutable state and can only depend on stateless packages. A stateless package is essentially pure functional: it can modify inputs, generate output, consume cpu and memory, but it cannot create side effect besides modification to inputs. math and unicode can be such stateless packages. In this proposal, the inputs become the security capabilities.

This proposal only needs a tiny language change: a simple declarative marks a package as stateless, and the linker verifies it. However, we found the proposal cannot work because Go doesn't have good constant support. Without constants, core packages, such as unicode, are full of mutable states, so there is no way to create useful stateless package.

If Go native supports constants, something equivalent to C++ constexpr, our original proposal would work with very little effort. By constants, I mean I can write code like const x = math.sin(3), which forces the compiler to evaluate the expression and put the result into readonly data segment, so the package can be stateless. Core packages, such as math, can be marked as stateless even they call unsafe code.

Regarding read-only slice et al, I think it is only needed for performance reasons. The caller can always duplicate the slice and not worry about malicious modification by the callee. In Java, this can be achieved by using immutable collections without direct language support.

To introduce read-only data structure at language level, we need to be aware that caller and the callee don't really trust each other. For example, a caller can pass a read-only slice to a callee, but the caller can create a go routine to continuously modify the slice. So the callee sees a read-only but mutable by someone else slice, which seems a bad idea.

We can solve the mutual trust problem by using Rust-like borrow check plus deep immutability to make sure an immutable reference can not be interfered by a mutable reference. This would make the language and language usage significantly more complicated. It also requires massive rewrite of all core libraries to correctly mark every pointer parameter. I don't think it is worthwhile and it is also against Go's philosophy.

Summary: if Go natively supports constants, we can use stateless package to achieve capability-based security with very little effort. A stateless package can only damage the input data structures and consume cpu+memory.

Go is a multi-platform language, most of the systems it supports have no concept of process-based security or proper namespaces. Building these features on top of systems that don't support them sounds silly.

That doesn't make sense to me. Of the things being proposed, Proposal_0 and Proposal_1 are handled by rules of the language, and is not dependent on machine architecture or platform. Proposal_2 is in a sense already implemented as demonstrated, we just need syntactic support. And these proposals would help make any large codebase secure regardless of the architecture or platform.

The data races issue might be solvable by packing things into 64 bits. You'd have to run the code in a 64-bit machine for complete security, but those machines are widely available. See slide 3: https://talks.golang.org/2015/go-gc.pdf

Regarding read-only slice et al, I think it is only needed for performance reasons. The caller can always duplicate the slice and not worry about malicious modification by the callee. In Java, this can be achieved by using immutable collections without direct language support.

Yes, we need it for performance.

Do you always duplicate slices before passing it to any interface object? Often there's an implicit contract that the callee should not modify something. The proposal makes that implicit contract (which would otherwise have to be written as documentation in english) an explicit contract. It's just making the compiler do more useful work to help let you enforce whatever contract that you're supposed to set. It sounds like a bad idea not to think about that contract.

See the broadcast function for a concrete example of this.

To introduce read-only data structure at language level, we need to be aware that caller and the callee don't really trust each other. For example, a caller can pass a read-only slice to a callee, but the caller can create a go routine to continuously modify the slice. So the callee sees a read-only but mutable by someone else slice, which seems a bad idea.

That depends on the context. It might be perfectly ok. It's the job of the code that created the mutable data-structure (or received it from elsewhere) to ensure that it is only accessible for writing in ways such that reading is also safe. Again, see the broadcast function. The function doesn't let anyone else write to it, and by contract it received a slice that it could have written to safely. So you can see for yourself that the implementation is safe, locally. The compiler enforces this contract globally.

We can solve the mutual trust problem by using Rust-like borrow check plus deep immutability to make sure an immutable reference can not be interfered by a mutable reference.

We already have that to some degree w/ non-pointer structs (+ private fields on pointer structs). See Proposal_2, Compiler optimizations for immutable struct assignment. Embedded structs, even when the field type is an interface, as long as the structs are non-pointers, are already deeply immutable unless you hold a pointer to it. Proposal_0 extends this to slices.

The whole point of Proposal_0 and Proposal_2 is that Rust-like control over mutability is not too difficult to achieve (Proposal_0 for slices; Already works for structs, but Proposal_2 makes it better).

This would make the language and language usage significantly more complicated. It also requires massive rewrite of all core libraries to correctly mark every pointer parameter. I don't think it is worthwhile and it is also against Go's philosophy.

These proposals do solve the "mutual trust" problem, and you can see for yourself how simple it is. The only complication is from the introduction of readonly/any modifiers, but you were supposed to be aware of that anyways, so it's actually helping you code. Within your own project, or if you're not using the standard libraries, you don't even have to use the modifiers!

math and unicode can be such stateless packages

Note that the current implementation of the math package has internal state (and modifies it during init) by figuring out at runtime which cpu capabilities are present (which can not be known at compile time) and choosing optimized function implementations accordingly (which might slightly differ in output within some range of precision too). for example:
https://github.com/golang/go/tree/master/src/math/floor_asm.go#L11
https://github.com/golang/go/tree/master/src/math/exp_asm.go#L11

@wora

The proposal (nickname gocap) is to introduce the concept of stateless package. A stateless package cannot have mutable state and can only depend on stateless packages. A stateless package is essentially pure functional: it can modify inputs, generate output, consume cpu and memory, but it cannot create side effect besides modification to inputs. math and unicode can be such stateless packages. In this proposal, the inputs become the security capabilities.

• What happens when what was once a pure dependency decides to be come mutable?
• Can stateless packages use reflect or unsafe? fmt.Sprintf("%p")? try/catch and get a stack trace?
• Can it run goroutines in parallel and introduce dependencies or run into the data-race problem mentioned in this thread?
• If stateless packages aren't deterministic or pure, what's the point?

I'm having a hard time understanding the benefit of this proposal.

This proposal only needs a tiny language change: a simple declarative marks a package as stateless, and the linker verifies it. However, we found the proposal cannot work because Go doesn't have good constant support. Without constants, core packages, such as unicode, are full of mutable states, so there is no way to create useful stateless package.

Now I'm arguing the other side... why not just use a func GetConstantXYZ() ...?

If Go native supports constants, something equivalent to C++ constexpr, our original proposal would work with very little effort. By constants, I mean I can write code like const x = math.sin(3), which forces the compiler to evaluate the expression and put the result into readonly data segment, so the package can be stateless. Core packages, such as math, can be marked as stateless even they call unsafe code.

How do you address the static initialization fiasco? If you have a specific proposal for constexpr-like consts, how does it compare to #23161 ?

Regarding read-only slice et al, I think it is only needed for performance reasons. The caller can always duplicate the slice and not worry about malicious modification by the callee.

The caller could duplicate the slice, but sometimes the caller doesn't want to. Yes, for performance. It's already the case today that developers do not always duplicate slices before passing it onto an unknown object (e.g an Interface, who knows what it'll do) for performance, and they're either doing so with documentation-level contracts (e.g. "CONTRACT: caller must duplicate slice before passing"), or worse, not having a clear contract documented anywhere, because the programmer was too busy breeding cryptokittles.

So yes, we need it for performance, but we need it yesterday for both performance and security.

In Java, this can be achieved by using immutable collections without direct language support.

That's a great point. We could implement a custom XYZSlice type without setters, maybe that will work. But who would actually use this thing without language-level support?

type ROByteSlice struct {
    s []byte
}
func (r ROByteSlice) Get(i int) byte {...}
func (r ROByteSlice) Subslice(s int, e int) ROByteSlice {...}
func (r ROByteSlice) Bytes() []byte {...} // copies
func (r ROByteSlice) Len() int {...}

• Changing a package from stateless to stateful is a breaking change. If a library you use decides to make a breaking change, you either give up on it or accept it by making some changes.
• A stateless package can use any other stateless package. Whether reflect, unsafe, or fmt is stateless depends on their design. For example, you can move Sprintf() into its own package and make it stateless.
• A stateless package can not create go routine by itself. By the caller can pass in a fork function which let callee to create a go routine under caller control.
• If only pure language has a point, people would be using Haskell. This is a design trade off. If we want to be deterministic and pure, people should just use Rust, which already solves the problem.

The benefit of my proposal is it has minimum complexity to achieve practical goal of running untrusted code. Without deep immutability, the caller has to protect inputs from being destroyed by callees. This is an intentional trade off to make the proposal practical.

I am not against solving the mutability issue, such as readonly slice or even deep mutual immutability. I just think it is a very hard problem, and I don't think it is really needed. This issue should be decided by the Go community. I don't have strong take on this.

Regarding the static initializer fiasco, it doesn't exist with stateless package. For stateless package, all constants are resolved by compiler at compile time. If compiler can not handle it, it is build error. There is no static initializer in stateless package.

PS: I was involved with Java static initialization logic. I understand the complexity, which is why I would eliminate it entirely for stateless package.

@wora do you mind creating a gocap proposal separate from this one?
I think I like most of your ideas and they deserve to be mulled over and discussed on their own.

thanks, looking forward to it. happy hols to you too.

math depends on unsafe: https://godoc.org/math?import-graph

If we're going to talk about performance then we need benchmarks based on real systems to prove there's a problem.

A stateless package can not create go routine by itself. By the caller can pass in a fork function which let callee to create a go routine under caller control.

Concurrent algorithms can be done in a stateless way from the caller's perspective, this is a big restriction to place on Go library code. We don't know what performance needs there will be in the future and concurrent algorithms as a possible solution is a strength of Go.

I don't understand a software development model of using untrusted code directly linked into a security-critical program, are there examples that can be more deeply described?

I created a dedicated issue #23267 for Gocap.

Thanks everyone for your valuable feedback.

There is no clear proposal here on which to make a decision. And as @randall77 says above, "Go just isn't very well suited to defending against adversaries that can inject code." It's almost certainly a mistake to try to retrofit security into a language that wasn't designed with it as a first principle.

Please reconsider, @ianlancetaylor . I've studied a few languages and I think Go is very well suited to be a secure language in the way proposed here. All that we need to make code injection safe is to parse it and filter it, e.g. ensure that only certain modules are imported. Even filtering for go-routine calls, while it would be prohibitive for Golang in general, can work very well for many use-cases where logic can be plugged in.

We're determined to make the next iteration of this language be safe in the way described here, and would really prefer not to fork the language. Please reconsider, I'm convinced that it's worth discussing.

Follow this for discussions of a Go fork: https://twitter.com/jaekwon/status/985577190300508160

https://github.com/HogLang/hog

@jaekwon You didn't address the most important point: there is no clear proposal here on which to make a decision.

@ianlancetaylor This largely depends on whether the Go community wants to provide in-process security and at what development cost. If there is no sufficient desire, then a concrete proposal wouldn't make too much difference. I don't know how to sense the desire on this topic.

This largely depends on whether the Go community wants to provide in-process security and at what development cost.

That's circular logic. No one can tell if someone wants to make it happen if there's no proper proposal to evaluate.

@wora I'm not sure precisely what you mean by in-process security, but see @randall77 's comment above: #23157 (comment) .

Certainly in-process security is not going to be a top priority for Go. That is, we are not willing to do it, whatever it is, at any cost. You want to know whether there is "sufficient desire," but it seems impossible to know whether there is sufficient desire without knowing how much it would cost.

In that case, this is a chicken and egg problem. The best way to solve it is to create a fork for Secure Go, and see what we can achieve. Many years ago, I tried to explore the concept of Gocap. Since Go doesn't support constexpr, it seems infeasible to implement such concept.

Much like other design challenges, a convincing prototype is a good way to find the right answer.