Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

encoding/gob.(*Decoder).Decode: heap grows , out of memory #29806

Closed
JianJunVictory opened this issue Jan 18, 2019 · 1 comment
Closed

encoding/gob.(*Decoder).Decode: heap grows , out of memory #29806

JianJunVictory opened this issue Jan 18, 2019 · 1 comment

Comments

@JianJunVictory
Copy link

operating system

Debian system on the Raspberry Pi 3A

go version

GOARCH="arm"
GOBIN=""
GOCACHE="/home/pi/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/pi/gowork"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go1.11.4"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go1.11.4/pkg/tool/linux_arm"
GCCGO="gccgo"
GOARM="7"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -marm -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build041997765=/tmp/go-build -gno-record-gcc-switches"

go tool pprof http://localhost:12345/debug/pprof/heap

(pprof) top
Showing nodes accounting for 483.73MB, 99.79% of 484.73MB total
Dropped 19 nodes (cum <= 2.42MB)
Showing top 10 nodes out of 46
      flat  flat%   sum%        cum   cum%
  280.14MB 57.79% 57.79%   280.14MB 57.79%  encoding/gob.decString
      87MB 17.95% 75.74%       87MB 17.95%  reflect.unsafe_NewArray
   82.50MB 17.02% 92.76%    82.50MB 17.02%  reflect.New
      24MB  4.95% 97.71%       24MB  4.95%  math/big.nat.make (inline)
    6.08MB  1.25% 98.97%     6.08MB  1.25%  github.com/dappley/go-dappley/core.(*UTXOIndex).DeepCopy
       4MB  0.83% 99.79%        4MB  0.83%  github.com/dappley/go-dappley/vendor/github.com/syndtr/goleveldb/leveldb/memdb.New
         0     0% 99.79%   473.65MB 97.71%  encoding/gob.(*Decoder).Decode
         0     0% 99.79%   473.65MB 97.71%  encoding/gob.(*Decoder).DecodeValue
         0     0% 99.79%   473.65MB 97.71%  encoding/gob.(*Decoder).decOpFor.func2
         0     0% 99.79%   473.65MB 97.71%  encoding/gob.(*Decoder).decOpFor.func3

(pprof) list encoding/gob.\(\*Decoder\).DecodeValue
Total: 484.73MB
ROUTINE ======================== encoding/gob.(*Decoder).DecodeValue in /usr/local/go1.11.4/src/encoding/gob/decoder.go
         0   473.65MB (flat, cum) 97.71% of Total
         .          .    207:
         .          .    208:	dec.buf.Reset() // In case data lingers from previous invocation.
         .          .    209:	dec.err = nil
         .          .    210:	id := dec.decodeTypeSequence(false)
         .          .    211:	if dec.err == nil {
         .   473.65MB    212:		dec.decodeValue(id, v)
         .          .    213:	}
         .          .    214:	return dec.err
         .          .    215:}
         .          .    216:
         .          .    217:// If debug.go is compiled into the program , debugFunc prints a human-readable
(pprof) 

the code:

// UTXOIndex holds all unspent TXOutputs indexed by public key hash.
type UTXOIndex struct {
	index map[string][]*UTXO
	mutex *sync.RWMutex
}

// UTXO contains the meta info of an unspent TXOutput.
type UTXO struct {
	TXOutput
	Txid    []byte
	TxIndex int
}

// NewUTXOIndex initializes an UTXOIndex instance
func NewUTXOIndex() *UTXOIndex {
	return &UTXOIndex{make(map[string][]*UTXO), &sync.RWMutex{}}
}

func deserializeUTXOIndex(d []byte) *UTXOIndex {
	utxos := NewUTXOIndex()
	utxos.mutex.Lock()
	defer utxos.mutex.Unlock()
	decoder := gob.NewDecoder(bytes.NewReader(d))
	err := decoder.Decode(&utxos.index)
	if err != nil {
		logger.WithError(err).Panic("UTXOIndex: failed to deserialize UTXOs.")
	}
	return utxos
}

func (utxos *UTXOIndex) serialize() []byte {
	var encoded bytes.Buffer
	utxos.mutex.Lock()
	defer utxos.mutex.Unlock()
	enc := gob.NewEncoder(&encoded)
	err := enc.Encode(utxos.index)
	if err != nil {
		logger.Panic(err)
	}
	return encoded.Bytes()
}

expect

oom will not happen and heap will not always grow ,when deserializing

result

encoding/gob.(*Decoder).Decode: heap always grow ,so does cum(to view pprof ).

@bradfitz
Copy link
Contributor

Sorry, there's not enough information here to help you.

You'll need to post a complete reproduction, not just a snippet of your coded.

I'll close this for now, but leave a comment if you have a minimal repro.

@golang golang locked and limited conversation to collaborators Jan 18, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants