The crash is occurring in the assembler implementation for bytes.Equal. It is comparing two byte slices whose length is the same and is greater than 0 and less than 8. It is executing a MOVQ instruction to load the bytes of the second slice. That MOVQ is failing with a SIGSEGV.

The second slice is the key field of the JSON string, as read by unquoteBytes in encoding/json/decode.go. That function always reads the first and last bytes of the slice and normally reads every byte of the slice. In some cases it copies the slice to a newly allocated []byte.

This crash frankly seems completely impossible unless there has already been some memory corruption. You say "potentially there was some data corruption here"; I would have to agree. The Go runtime can't operate normally in the presence of data corruption.

Is this problem repeatable?

It seems repeatable with the bolt db in question (which I'm still trying to get my hands on).
I've setup a test to make sure our usage of boltdb is not causing the panic here and it doesn't seem to be the case.

So, until I can get a copy of the affected db, I can't replicate.

I think this may be due to a misuse of bolt.

The byte slice you are unmarshalling, entry.Value, comes from entries, which is generated in listEntries by directly preserving references to []byte from bolt.

However, the bolt docs state:

Keys and values retrieved from the database are only valid for the life of the transaction. When used outside the transaction, these byte slices can point to different data or can point to invalid memory which will cause a panic.

I believe the correct fix is in listEntries. When you are creating each new entry, make new k and v byte slices of the correct length and copy in the data from bolt.

You will probably want to audit the other uses of bolt for similar bugs.

Something like this (untested, typed into github comment) should do the trick:

func listEntries(tx *bolt.Tx) []*dbEntry {
	var entries []*dbEntry
	b := tx.Bucket(volumeBucketName)
	b.ForEach(func(k, v []byte) error {
		// Make copies of k and v, since they will outlive tx.
		var entry dbEntry
		entry.k = append(entry.k, k...)
		entry.v = append(entry.v, v...)
		entries = append(entries, &entry)
		return nil
	})
	return entries
}

Ah thanks, I was wondering about that but hadn't seen anything, and couldn't repro the issue with the same useage.

Closing based on @josharian's analysis. Please comment if that turns out to be incorrect.