Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

debug/elf: fatal error: runtime: out of memory #45599

Closed
fangxlmr opened this issue Apr 16, 2021 · 13 comments
Closed

debug/elf: fatal error: runtime: out of memory #45599

fangxlmr opened this issue Apr 16, 2021 · 13 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@fangxlmr
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.15.5 linux/amd64

Does this issue reproduce with the latest release?

YES

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/data/leonxlfang/.cache/go-build"
GOENV="/data/leonxlfang/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/data/leonxlfang/go/pkg/mod"
GONOPROXY="git.code.oa.com,git.woa.com"
GONOSUMDB="git.code.oa.com,git.woa.com"
GOOS="linux"
GOPATH="/data/leonxlfang/go"
GOPRIVATE="git.code.oa.com,git.woa.com"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/data/leonxlfang/.local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/data/leonxlfang/.local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.3"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/data/leonxlfang/Test/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build789926490=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Try to parse an ELF file. The file has valid header but other sections are broken somehow.

Example code:

package main

import (
	"debug/elf"
	"fmt"
	"os"
)

func main() {
	// pathname := "/bin/ls"
	pathname := "./pvrsrvkm.ko"
	f1, err := os.Open(pathname)
	if err != nil {
		panic(fmt.Sprintf("error opening %s: %s", pathname, err))
	}
	f2, err := elf.NewFile(f1)
	if err != nil {
		panic(fmt.Sprintf("error creating new ELF struct: %s", err))
	}
	for _, section := range f2.Sections {
		fmt.Println(section.Name)
	}
}

Example file: pvrsrvkm.ko.zip

What did you expect to see?

NO PANIC AT LEAST! Or panic with informative messages.

readelf command neither panics nor causes out of memory.

readelf pvrsrvkm.ko Output
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              REL (Relocatable file)
  Machine:                           AArch64
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          0 (bytes into file)
  Start of section headers:          804368 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           0 (bytes)
  Number of program headers:         0
  Size of section headers:           64 (bytes)
  Number of section headers:         39
  Section header string table index: 36
readelf: pvrsrvkm.ko: Warning: Section 0 has an out of range sh_link value of 1572864
readelf: pvrsrvkm.ko: Warning: Section 1 has an out of range sh_link value of 3145728
readelf: pvrsrvkm.ko: Warning: Section 2 has an out of range sh_link value of 16842752
readelf: pvrsrvkm.ko: Warning: Section 3 has an out of range sh_link value of 6291456
readelf: pvrsrvkm.ko: Warning: Section 4 has an out of range sh_link value of 7864320
readelf: pvrsrvkm.ko: Warning: Section 5 has an out of range sh_link value of 16842752
readelf: pvrsrvkm.ko: Warning: Section 6 has an out of range sh_link value of 11010048
readelf: pvrsrvkm.ko: Warning: Section 7 has an out of range sh_link value of 12582912
readelf: pvrsrvkm.ko: Warning: Section 8 has an out of range sh_link value of 16842752
readelf: pvrsrvkm.ko: Warning: Section 9 has an out of range sh_link value of 15728640
readelf: pvrsrvkm.ko: Warning: Section 10 has an out of range sh_link value of 17301504
readelf: pvrsrvkm.ko: Warning: Section 11 has an out of range sh_link value of 16842752
readelf: pvrsrvkm.ko: Warning: Section 12 has an out of range sh_link value of 20447232
readelf: pvrsrvkm.ko: Warning: Section 13 has an out of range sh_link value of 22020096
readelf: pvrsrvkm.ko: Warning: Section 14 has an out of range sh_link value of 16842752
readelf: pvrsrvkm.ko: Warning: Section 15 has an out of range sh_link value of 7864320
readelf: pvrsrvkm.ko: Warning: Section 16 has an out of range sh_link value of 106430464
readelf: pvrsrvkm.ko: Warning: Section 17 has an out of range sh_link value of 16842752
readelf: pvrsrvkm.ko: Warning: Section 18 has an out of range sh_link value of 33030144
readelf: pvrsrvkm.ko: Warning: Section 21 has an out of range sh_link value of 620756992
readelf: pvrsrvkm.ko: Warning: Section 23 has an out of range sh_link value of 620756992
readelf: pvrsrvkm.ko: Warning: Section 25 has an out of range sh_link value of 620756992
readelf: pvrsrvkm.ko: Warning: Section 27 has an out of range sh_link value of 620756992
readelf: pvrsrvkm.ko: Warning: Section 30 has an out of range sh_link value of 620756992
readelf: pvrsrvkm.ko: Error: Reading 0x2000000000 bytes extends past end of file for string table

Section Headers:
[Nr] Name Type Address Offset
Size EntSize Flags Link Info Align
[ 0] 00010000: <unkn 0000000000100000 1000001010000
0000000195640000 0000000000000000 xop 1572864 0 470626161077059584
[ 1] NULL 0000000002b40000 00300000
0010000001010000 0010000001010000 xox 3145728 0 3670016
[ 2] NULL 0598000001010000 00000000
0000000000480000 0000000000500000 o 16842752 524288 2359361536
[ 3] 00010000: <unkn 0000000000580000 1000001010000
0000000194180000 0000000000000000 xxop 6291456 0 470626161077059584
[ 4] NULL 0000000001c00000 00780000
0010000001010000 0010000001010000 xox 7864320 0 8388608
[ 5] NULL 0598000001010000 00000000
0000000000900000 0000000000980000 xo 16842752 524288 2393505792
[ 6] 00010000: <unkn 0000000000a00000 1000001010000
00000001a4b80000 0000000000000000 xop 11010048 0 470626161077059584
[ 7] NULL 0000000003540000 00c00000
0010000001010000 0010000001010000 xox 12582912 0 13107200
[ 8] NULL 0598000001010000 00000000
0000000000d80000 0000000000e00000 o 16842752 524288 2393505792
[ 9] 00010000: <unkn 0000000000e80000 1000001010000
00000001a2a00000 0000000000000000 xxop 15728640 0 470626161077059584
[10] NULL 00000000051c0000 01080000
0010000001010000 0010000001010000 xox 17301504 0 17825792
[11] NULL 0598000001010000 00000000
0000000001200000 0000000001280000 xo 16842752 524288 2393505792
[12] 00010000: <unkn 0000000001300000 1000001010000
00000001a08c0000 0000000000000000 xop 20447232 0 470626161077059584
[13] NULL 0000000003e40000 01500000
0010000001010000 0010000001010000 xox 22020096 0 22544384
[14] NULL 0598000001010000 00000000
0000000000000000 0000000000280000 o 16842752 786432 0
[15] 000c0000: <unkn 0000000000500000 c000001010000
0000000000580000 0000000000400000 o 7864320 0 3377699737370624
[16] NULL 0000000000280000 00000000
000e000001010000 000e000001010000 xoxx 106430464 0 524288
[17] NULL 000e000001010000 07c80000
0000000000180000 0000000000200000 o 16842752 917504 139460608
[18] 000e0000: <unkn 0000000001200000 602000001010000
0000000000000000 0000000000000000 xo 33030144 0 398287092062420992
[19] NULL 0000000000000000 00000000
0000000000000000 0000000000000000 0 0 0
[20] 00010000: <unkn 0000000000000000 00400000
00000003ec9c0000 0000000000000000 xx 0 0 262144
[21] 00040000: <unkn 0000000000000000 70a0d900000
000004e738000000 0000000018000000 o 620756992 16777216 134217728
[22] 01000000: <unkn 0000000000000000 3ecdc000000
0000000654000000 0000000000000000 o 0 0 67108864
[23] 04000000: <unkn 0000000000000000 bf1c8000000
0000000d38000000 0000000018000000 p 620756992 50331648 134217728
[24] 01000000: <unkn 0000000000000000 3f330000000
0000000084000000 0000000000000000 o 0 0 67108864
[25] 04000000: <unkn 0000000000000000 bff00000000
0000000150000000 0000000018000000 p 620756992 83886080 134217728
[26] 01000000: <unkn 0000000000000000 3f3b4000000
0000000474000000 0000000000000000 o 0 0 67108864
[27] 04000000: <unkn 0000000000000000 c0050000000
0000000690000000 0000000018000000 p 620756992 117440512 134217728
[28] 07000000: <unkn 0000000000000000 3f828000000
0000000024000000 0000000000000000 o 0 0 67108864
[29] 01000000: <unkn 0000000028000000 3f850000000
0000000020000000 0000000000000000 o 0 0 134217728
[30] 04000000: <unkn 0000000000000000 c06e0000000
0000000060000000 0000001800000000 p 620756992 218103808 34359738378
[31] 00000081: <unkn 0000000000000000 3f87000000000
000028c800000000 0000000000000000 Wx 0 0 34359738368
[32] 0000007c: <unkn 0000000000000000 c074000000000
000029b800000000 0000001800000000 Xx 0 37 34359738380
[33] 00000089: <unkn 0000000000000000 4213800000000
0001451700000000 0000000100000000 Wxxx 0 0 4294967296
[34] 00000098: <unkn 0000000000000000 5664f00000000
0000002900000000 0000000000000000 Wx 0 0 4294967296
[35] 000000aa: <unkn 0000000000000000 5667800000000
000000f000000000 0000000000000000 Wx 0 0 34359738368
[36] 000000b8: <unkn 0000000000000000 5676800000000
0000002000000000 0000000000000000 Wx 0 0 34359738368
[37] 000000b3: <unkn 0000000000000000 c30f800000000
0000004800000000 0000001800000000 Xx 0 37 34359738385
[38] 000000c0: <unkn 0000000000000000 5678800000000
0000007000000000 0000000000000000 Wx 0 0 34359738368
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
L (link order), O (extra OS processing required), G (group), T (TLS),
C (compressed), x (unknown), o (OS specific), E (exclude),
p (processor specific)

There are no section groups in this file.

There are no program headers in this file.

There are no relocations in this file.

The decoding of unwind sections for machine type AArch64 is not currently supported.

No version information found in this file.

What did you see instead?

Program panics with stack trace
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x4db49c, 0x16)
        /data/leonxlfang/.local/go/src/runtime/panic.go:1117 +0x72
runtime.sysMap(0xc004000000, 0x2000000000, 0x5a6ab0)
        /data/leonxlfang/.local/go/src/runtime/mem_linux.go:169 +0xc6
runtime.(*mheap).sysAlloc(0x58e960, 0x2000000000, 0x4293b7, 0x58e968)
        /data/leonxlfang/.local/go/src/runtime/malloc.go:729 +0x1e5
runtime.(*mheap).grow(0x58e960, 0x1000000, 0x0)
        /data/leonxlfang/.local/go/src/runtime/mheap.go:1346 +0x85
runtime.(*mheap).allocSpan(0x58e960, 0x1000000, 0xc000060100, 0x7f92b3607d30)
        /data/leonxlfang/.local/go/src/runtime/mheap.go:1173 +0x609
runtime.(*mheap).alloc.func1()
        /data/leonxlfang/.local/go/src/runtime/mheap.go:910 +0x59
runtime.systemstack(0x462994)
        /data/leonxlfang/.local/go/src/runtime/asm_amd64.s:379 +0x66
runtime.mstart()
        /data/leonxlfang/.local/go/src/runtime/proc.go:1246

goroutine 1 [running]:
runtime.systemstack_switch()
        /data/leonxlfang/.local/go/src/runtime/asm_amd64.s:339 fp=0xc000057a88 sp=0xc000057a80 pc=0x462ac0
runtime.(*mheap).alloc(0x58e960, 0x1000000, 0x101, 0x40)
        /data/leonxlfang/.local/go/src/runtime/mheap.go:904 +0x85 fp=0xc000057ad8 sp=0xc000057a88 pc=0x425065
runtime.(*mcache).allocLarge(0x7f92b35fd108, 0x2000000000, 0x7f92b3600101, 0x203000)
        /data/leonxlfang/.local/go/src/runtime/mcache.go:224 +0x97 fp=0xc000057b30 sp=0xc000057ad8 pc=0x415877
runtime.mallocgc(0x2000000000, 0x4bb3c0, 0x7f92b3608801, 0x10)
        /data/leonxlfang/.local/go/src/runtime/malloc.go:1078 +0x925 fp=0xc000057bb8 sp=0xc000057b30 pc=0x40cf85
runtime.makeslice(0x4bb3c0, 0x2000000000, 0x2000000000, 0x577e00)
        /data/leonxlfang/.local/go/src/runtime/slice.go:98 +0x6c fp=0xc000057be8 sp=0xc000057bb8 pc=0x4489ec
debug/elf.(*Section).Data(0xc000133200, 0xc00007a9c0, 0x4facb8, 0x5a5520, 0x4b67e0, 0xc00001f400)
        /data/leonxlfang/.local/go/src/debug/elf/file.go:105 +0x45 fp=0xc000057c50 sp=0xc000057be8 pc=0x4ab405
debug/elf.NewFile(0x4fa090, 0xc00000e028, 0x0, 0x7f9200000000, 0xc00000e028)
        /data/leonxlfang/.local/go/src/debug/elf/file.go:456 +0xff6 fp=0xc000057ed0 sp=0xc000057c50 pc=0x4ac8b6
main.main()
        /data/leonxlfang/Test/elf.go:16 +0xee fp=0xc000057f88 sp=0xc000057ed0 pc=0x4ae8ae
runtime.main()
        /data/leonxlfang/.local/go/src/runtime/proc.go:225 +0x256 fp=0xc000057fe0 sp=0xc000057f88 pc=0x4355f6
runtime.goexit()
        /data/leonxlfang/.local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000057fe8 sp=0xc000057fe0 pc=0x4647c1
exit status 2
@fangxlmr
Copy link
Author

Root cuase of this specific issue is pretty obvious, but I was wondering if debug package will plan to introduce more strict checking and validation to prevent this (and other similar problems) from happening?

@mknyszek
Copy link
Contributor

In that panic, the debug/elf package is trying to allocator a slice of size ~128 GiB. Why? I'm not sure. That might be a bug.

@mknyszek mknyszek added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Apr 16, 2021
@mknyszek mknyszek added this to the Backlog milestone Apr 16, 2021
@mknyszek
Copy link
Contributor

@ianlancetaylor
Copy link
Contributor

It's a corrupt ELF file. The question is to what extent debug/elf should protect itself against corrupt files. It's hard to know what limits to set, as ELF files can indeed be very very large.

I'm certainly open to suggestions.

@ianlancetaylor ianlancetaylor modified the milestones: Backlog, Unplanned Apr 19, 2021
@dingleshxt
Copy link

We do have some choice:

  1. Using strict checking in debug/elf which leads to complex codes in there
  2. Choosing another elf parser for untrusted input elf

@prattmic
Copy link
Member

When I wrote a loader for untrusted ELF files in Go a few years ago, I avoided using debug/elf for exactly this reason (or, rather, used it only for types/constants). debug/elf is not currently suited to parsing corrupt/malicious ELF files. This is only one of numerous places where it passed values directly from the file to make, easily causing such OOMs.

Should debug/elf try to be safe against malicious ELFs? I'm not sure; that is a pretty high bar to meet, and safety also limits the API to some extent. For example, the crashing function here, Section.Data is not really safe even for a legitimate, very large, ELF, which could contain a multi-GB section that will OOM the process when read in entirety.

@ianlancetaylor
Copy link
Contributor

I am in favor of better checking in debug/elf, where possible.

It occurs to me that in this particular case we can compare the number of sections with the size of the file. If the file is too small, we know it is corrupt before we try to allocate memory.

@fangxlmr
Copy link
Author

fangxlmr commented Apr 21, 2021

I am in favor of better checking in debug/elf, where possible.

Agree. I'd like have debug/elf at least NO (unpredictable) PANIC as well. But as above post said, to what degree and how?

It occurs to me that in this particular case we can compare the number of sections with the size of the file. If the file is too small, we know it is corrupt before we try to allocate memory.

Yeah, I thought this before as well. But the thing is NewFile(r io.ReaderAt) (*File, error) accepts a io.ReaderAt inerface, so how do we exactly know the file size? Read until EOF? Sounds no good.

@ianlancetaylor
Copy link
Contributor

We can have NewFile do a type assertion to see if it is given something that supports Stat, as will be true when NewFile is called from elf.Open. If we want to get really fancy we can use a type assertion to see if it supports Seek, as is likely, and then seek to the end to find the offset.

@gopherbot
Copy link

Change https://golang.org/cl/362255 mentions this issue: debug/elf: read section contents incrementally in Section.Data()

@ianlancetaylor
Copy link
Contributor

More or less a dup of #47653.

@gopherbot
Copy link

Change https://go.dev/cl/413775 mentions this issue: debug/elf: avoid elf.Open crashing on malformed files

@gopherbot
Copy link

Change https://go.dev/cl/408679 mentions this issue: debug/elf: use saferio to read section data

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 13, 2022
@golang golang locked and limited conversation to collaborators Aug 11, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
6 participants