Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gollvm: unable to build etcd #33020

Open
thanm opened this issue Jul 10, 2019 · 9 comments
Open

gollvm: unable to build etcd #33020

thanm opened this issue Jul 10, 2019 · 9 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@thanm
Copy link
Contributor

thanm commented Jul 10, 2019

What version of Go are you using (go version)?

$ go version
go version go1.12.2 gollvm LLVM 9.0.0svn linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

linux/amd64

What did you do?

As reported by yuanting@ict.ac.cn

$ git clone https://github.com/etcd-io/etcd.git
$ cd etcd/ && ./build

What did you expect to see?

Successful compilation

What did you see instead?

go build: when using gccgo toolchain, please pass linker flags using -gccgoflags, not -ldflags
# go.etcd.io/etcd/etcdserver/api/rafthttp
 #0 0x000055ecc4cfa34a llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/home/yt/LLVMsvn/install/bin/llvm-goc+0xa6c34a)
 #1 0x000055ecc4cf8124 llvm::sys::RunSignalHandlers() (/home/yt/LLVMsvn/install/bin/llvm-goc+0xa6a124)
 #2 0x000055ecc4cf8262 SignalHandler(int) (/home/yt/LLVMsvn/install/bin/llvm-goc+0xa6a262)
 #3 0x00007fbbd49d5890 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x12890)
 #4 0x000055ecc44f0940 Export::type_index(Type const*) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x262940)
 #5 0x000055ecc44f0c31 Export::write_type(Type const*) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x262c31)
 #6 0x000055ecc449f2e1 Named_type::do_export(Export*) const (/home/yt/LLVMsvn/install/bin/llvm-goc+0x2112e1)
 #7 0x000055ecc44f10ce Export::write_type_definition(Type const*, int) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x2630ce)
 #8 0x000055ecc44f1390 Export::write_types(int) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x263390)
 #9 0x000055ecc44fcea0 Export::export_globals(std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, std::map, std::allocator >, Package*, std::less, std::allocator > >, std::allocator, std::allocator > const, Package*> > > const&, std::map, std::allocator >, Package*, std::less, std::allocator > >, std::allocator, std::allocator > const, Package*> > > const&, std::__cxx11::basic_string, std::allocator > const&, Import_init_set const&, Bindings const*) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x26eea0)
#10 0x000055ecc4457047 Gogo::do_exports() (/home/yt/LLVMsvn/install/bin/llvm-goc+0x1c9047)
#11 0x000055ecc4441a90 go_parse_input_files(char const**, unsigned int, bool, bool) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x1b3a90)
#12 0x000055ecc442ee69 gollvm::driver::CompileGoImpl::invokeFrontEnd() (/home/yt/LLVMsvn/install/bin/llvm-goc+0x1a0e69)
#13 0x000055ecc4436ee5 gollvm::driver::CompileGoImpl::performAction(gollvm::driver::Compilation&, gollvm::driver::Action const&, llvm::SmallVector const&, gollvm::driver::Artifact const&) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x1a8ee5)
#14 0x000055ecc442950f gollvm::driver::Driver::processAction(gollvm::driver::Action*, gollvm::driver::Compilation&, bool) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x19b50f)
#15 0x000055ecc44296af gollvm::driver::Driver::processActions(gollvm::driver::Compilation&) (/home/yt/LLVMsvn/install/bin/llvm-goc+0x19b6af)
#16 0x000055ecc43cf403 main (/home/yt/LLVMsvn/install/bin/llvm-goc+0x141403)
#17 0x00007fbbd386db97 __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:344:0
#18 0x000055ecc442130a _start (/home/yt/LLVMsvn/install/bin/llvm-goc+0x19330a)
Stack dump:
0.	Program arguments: /home/yt/LLVMsvn/install/bin/llvm-goc -c -O2 -g -m64 -fdebug-prefix-map=/tmp/go-build979597534=/tmp/go-build -gno-record-gcc-switches -fgo-pkgpath=go.etcd.io/etcd/etcdserver/api/rafthttp -o $WORK/b192/_go_.o -I $WORK/b192/_importcfgroot_ etcdserver/api/rafthttp/coder.go etcdserver/api/rafthttp/doc.go etcdserver/api/rafthttp/http.go etcdserver/api/rafthttp/metrics.go etcdserver/api/rafthttp/msg_codec.go etcdserver/api/rafthttp/msgappv2_codec.go etcdserver/api/rafthttp/peer.go etcdserver/api/rafthttp/peer_status.go etcdserver/api/rafthttp/pipeline.go etcdserver/api/rafthttp/probing_status.go etcdserver/api/rafthttp/remote.go etcdserver/api/rafthttp/snapshot_sender.go etcdserver/api/rafthttp/stream.go etcdserver/api/rafthttp/transport.go etcdserver/api/rafthttp/urlpick.go etcdserver/api/rafthttp/util.go

Needs investigation. There have been several changes and bugs in import/export lately; this looks like a new one.

@thanm thanm added this to the gollvm milestone Jul 10, 2019
@thanm thanm self-assigned this Jul 10, 2019
@gopherbot
Copy link

Change https://golang.org/cl/185977 mentions this issue: compiler: fix bug in handling of unordered set during exporting

@thanm
Copy link
Contributor Author

thanm commented Jul 12, 2019

CL 185977 resolves the problem listed above, bit it looks like there is another problem hiding behind that one (I am investigating). Failure mode:

llvm-goc: /go-llvm-materialize.cpp:853: Bexpression* Llvm_backend::materializeComposite(Bexpression*): Assertion vals.size() == numElements' failed.
`

@gopherbot
Copy link

Change https://golang.org/cl/186697 mentions this issue: test: new testcase for gccgo bug

gopherbot pushed a commit to golang/gofrontend that referenced this issue Jul 18, 2019
In CL 183850 a change was made to combine tracking/discovery of
exported types and imported packages during export data generation. As
a result of this refactoring a bug was introduced: the new code can
potentially insert items into the exports set (an unordered_set) while
iterating through the same set, which is illegal according to the spec
for std::unordered_set.

This patch fixes the problem by changing the type discovery phase to
iterate through a separate list of sorted exports, as opposed to
iterating through the main unordered set.  Also included is a change
to fix the code that looks for variables that are referenced from
inlined routine bodies (this code wasn't scanning all of the function
that it needed to scan).

New test case for this problem in CL 186697.

Updates golang/go#33020.

Change-Id: Id9ae5af462bc0819e8c9a8dd038ce9746297ad5d
Reviewed-on: https://go-review.googlesource.com/c/gofrontend/+/185977
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Red-Portal pushed a commit to Red-Portal/gcc that referenced this issue Jul 18, 2019
    
    In CL 183850 a change was made to combine tracking/discovery of
    exported types and imported packages during export data generation. As
    a result of this refactoring a bug was introduced: the new code can
    potentially insert items into the exports set (an unordered_set) while
    iterating through the same set, which is illegal according to the spec
    for std::unordered_set.
    
    This patch fixes the problem by changing the type discovery phase to
    iterate through a separate list of sorted exports, as opposed to
    iterating through the main unordered set.  Also included is a change
    to fix the code that looks for variables that are referenced from
    inlined routine bodies (this code wasn't scanning all of the function
    that it needed to scan).
    
    New test case for this problem in CL 186697.
    
    Updates golang/go#33020.
    
    Reviewed-on: https://go-review.googlesource.com/c/gofrontend/+/185977


git-svn-id: svn+ssh://gcc.gnu.org/svn/gcc/trunk@273564 138bc75d-0d04-0410-961f-82ee72b054a4
gopherbot pushed a commit that referenced this issue Jul 18, 2019
Updates #33020

Change-Id: I82554ef20ea35e0087fd9ecd9548c2dfeacdc617
Reviewed-on: https://go-review.googlesource.com/c/go/+/186697
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@shawndx
Copy link
Contributor

shawndx commented Aug 26, 2019

Enclose a reduced case which could reproduce the issue of Llvm_backend::materializeComposite.

Given a struct type:

type FArg func(args []string) error
type Command struct {
  Name string
  Arg1 FArg
  Arg2 func(args []string) error
}

The named base of FArg is set to "func(args []string)error" which happens to be the type of field 'Arg2', converting the backend representation of the type of Arg1 has a side-effect which sets the btype of Arg2's type, such side-effect result in the post-processing of 'Arg2' being bypassed and remaining 'Command' in 'unresolved placeholder' status, so its opaque llvm type fails to be concretized. So far the issue could be reproduced via 'importing' type definitions only as the exporter seems to do type definitions sharing, perhaps there are other scenarios undiscovered

Tried a fix locally which adds place-holder pointer types created for function type to a list (similar to Type::placeholder_pointers, or they can share the same list) then lets finish_pointer_types takes care the concretizing, it could fix the crash of building etcd (a few errors of undefined symbol observed), wonder there might be a better solution from the perspective of type sharing.

@shawndx
Copy link
Contributor

shawndx commented Aug 26, 2019

Seems that neither .go nor .txt is supported, enclosing the code here.

//t.go
package Args 

type FArg func(args []string) error

type Command struct {
  Name string
  Arg1 FArg 
  Arg2 func(args []string) error 
}
// main.go
package main

import (
	"fmt"
        "mypkg/Args"
)

var (
  cmd = &Args.Command {
    Name: "test",
  }
)

func main() {
  fmt.Println("test main")
}
/* put t.go under $GOPATH
    % go build mypkg/Args 
   % go build main.go

#4  0x0000555555ed47f0 in Llvm_backend::materializeComposite (this=0x55555d054350, comExpr=0x55555d07e710) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/bridge/go-llvm-materialize.cpp:860
#5  0x0000555555edb8e1 in MaterializeVisitor::visitNodePost (this=0x7fffffffd5a0, node=0x55555d07e710) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/bridge/go-llvm-materialize.cpp:1967
#6  0x0000555555ee04c8 in UpdatingNodeWalker<MaterializeVisitor>::walk (this=0x7fffffffd528, node=0x55555d07e710) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/bridge/go-llvm-bnode.h:514
#7  0x0000555555ede344 in update_walk_nodes<MaterializeVisitor> (root=0x55555d07e710, vis=...) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/bridge/go-llvm-bnode.h:530
#8  0x0000555555ed8df5 in Llvm_backend::materialize (this=0x55555d054350, expr=0x55555d07e710, lvalueContext=VE_rvalue) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/bridge/go-llvm-materialize.cpp:2037
#9  0x0000555555de582f in Llvm_backend::makeInitStatement (this=0x55555d054350, bfunction=0x55555d0ce0e0, var=0x55555d07e5e0, init=0x55555d07e710) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/bridge/go-llvm.cpp:1759
#10 0x0000555555de572a in Llvm_backend::init_statement (this=0x55555d054350, bfunction=0x55555d0ce0e0, var=0x55555d07e5e0, init=0x55555d07e710) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/bridge/go-llvm.cpp:1742
#11 0x0000555555de8109 in Llvm_backend::temporary_variable (this=0x55555d054350, function=0x55555d0ce0e0, bblock=0x55555d0a0c00, btype=0x55555d0a74b0, binit=0x55555d07e710, is_address_taken=true, location=...,
    pstatement=0x7fffffffd800) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/bridge/go-llvm.cpp:2305
#12 0x0000555555e813c6 in Heap_expression::do_get_backend (this=0x55555d0642f0, context=0x7fffffffdc30) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/gofrontend/go/expressions.cc:16780
#13 0x0000555555e4d5c1 in Expression::get_backend (this=0x55555d0642f0, context=0x7fffffffdc30) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/gofrontend/go/expressions.cc:543
#14 0x0000555555e5523b in Type_conversion_expression::do_get_backend (this=0x55555d0cd1f0, context=0x7fffffffdc30) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/gofrontend/go/expressions.cc:3960
#15 0x0000555555e4d5c1 in Expression::get_backend (this=0x55555d0cd1f0, context=0x7fffffffdc30) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/gofrontend/go/expressions.cc:543
#16 0x0000555555d6e0d6 in Temporary_statement::do_get_backend (this=0x55555d0cd300, context=0x7fffffffdc30) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/gofrontend/go/statements.cc:628
#17 0x0000555555d6cb25 in Statement::get_backend (this=0x55555d0cd300, context=0x7fffffffdc30) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/gofrontend/go/statements.cc:185
#18 0x0000555555cf945d in Block::get_backend (this=0x55555d0cd290, context=0x7fffffffdd00) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/gofrontend/go/gogo.cc:7065
#19 0x0000555555cfbb11 in Variable::get_init_block (this=0x55555d063d80, gogo=0x55555d05cdd0, function=0x55555d0cddf0, var_decl=0x55555d0a2bf0) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/gofrontend/go/gogo.cc:7821
#20 0x0000555555ce7eaa in Gogo::write_globals (this=0x55555d05cdd0) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/gofrontend/go/gogo.cc:1687
#21 0x0000555555cdfac4 in go_write_globals () at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/gofrontend/go/go.cc:194
#22 0x0000555555ccba93 in gollvm::driver::CompileGoImpl::invokeFrontEnd (this=0x55555d001d90) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/driver/CompileGo.cpp:811
#23 0x0000555555cc89c7 in gollvm::driver::CompileGoImpl::performAction (this=0x55555d001d90, compilation=..., jobAction=..., inputArtifacts=..., output=...)
    at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/driver/CompileGo.cpp:192
#24 0x0000555555ccc8fe in gollvm::driver::CompileGo::performAction (this=0x55555d039dc0, compilation=..., jobAction=..., inputArtifacts=..., output=...) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/driver/CompileGo.cpp:1022
#25 0x0000555555cbcbe5 in gollvm::driver::Driver::processAction (this=0x7fffffffe2d0, act=0x55555d03a3b0, compilation=..., lastAct=false) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/driver/Driver.cpp:546
#26 0x0000555555cbcda7 in gollvm::driver::Driver::processActions (this=0x7fffffffe2d0, compilation=...) at /home/xiaji01/src/gollvm.etcd/llvm/tools/gollvm/driver/Driver.cpp:568

*/

@thanm
Copy link
Contributor Author

thanm commented Aug 26, 2019

Thanks @shawn-xdji for the analysis and the reduced test case. I agree with your analysis, the problem is that the FE is manufacturing a placeholder but then never finalizing it. There was a similar problem a while back fixed in https://go-review.googlesource.com/c/gofrontend/+/51131, I think the fix for this bug is to generalize/extend that change. I will send a CL.

@gopherbot
Copy link

Change https://golang.org/cl/191743 mentions this issue: test: new testcase for gollvm bug

@gopherbot
Copy link

Change https://golang.org/cl/191744 mentions this issue: compiler: generalize cleanup of unresolved placeholder pointer types

gopherbot pushed a commit to golang/gofrontend that referenced this issue Aug 26, 2019
This change extends the work in https://golang.org/cl/51131 to include
placeholder pointer types created for Go function types, which can
also be left dangling/unresolved in some instances. This fixes an
assert in Llvm_backend::materializeComposite.

Test case can be found in https://golang.org/cl/191743.

Updates golang/go#33020.

Change-Id: I66094131916a2a7175f32654a7376d7db568f741
Reviewed-on: https://go-review.googlesource.com/c/gofrontend/+/191744
Reviewed-by: Ian Lance Taylor <iant@golang.org>
kraj pushed a commit to kraj/gcc that referenced this issue Aug 26, 2019
    
    This change extends the work in https://golang.org/cl/51131 to include
    placeholder pointer types created for Go function types, which can
    also be left dangling/unresolved in some instances. This fixes an
    assert in Llvm_backend::materializeComposite.
    
    Test case can be found in https://golang.org/cl/191743.
    
    Updates golang/go#33020.
    
    Reviewed-on: https://go-review.googlesource.com/c/gofrontend/+/191744


git-svn-id: svn+ssh://gcc.gnu.org/svn/gcc/trunk@274935 138bc75d-0d04-0410-961f-82ee72b054a4
gopherbot pushed a commit that referenced this issue Aug 29, 2019
Testcase for a gollvm bug (assert in Llvm_backend::materializeComposite).

Updates #33020.

Change-Id: Icdf5b4b2b6eb55a5b48a31a61c41215b1ae4cf01
Reviewed-on: https://go-review.googlesource.com/c/go/+/191743
Reviewed-by: Ian Lance Taylor <iant@golang.org>
tomocy pushed a commit to tomocy/go that referenced this issue Sep 1, 2019
Testcase for a gollvm bug (assert in Llvm_backend::materializeComposite).

Updates golang#33020.

Change-Id: Icdf5b4b2b6eb55a5b48a31a61c41215b1ae4cf01
Reviewed-on: https://go-review.googlesource.com/c/go/+/191743
Reviewed-by: Ian Lance Taylor <iant@golang.org>
t4n6a1ka pushed a commit to t4n6a1ka/go that referenced this issue Sep 5, 2019
Testcase for a gollvm bug (assert in Llvm_backend::materializeComposite).

Updates golang#33020.

Change-Id: Icdf5b4b2b6eb55a5b48a31a61c41215b1ae4cf01
Reviewed-on: https://go-review.googlesource.com/c/go/+/191743
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@advancedwebdeveloper
Copy link

Hi.
Is it possible to compile etcd now?

Ivan

asiekierka pushed a commit to WonderfulToolchain/gcc-ia16 that referenced this issue May 16, 2022
    
    In CL 183850 a change was made to combine tracking/discovery of
    exported types and imported packages during export data generation. As
    a result of this refactoring a bug was introduced: the new code can
    potentially insert items into the exports set (an unordered_set) while
    iterating through the same set, which is illegal according to the spec
    for std::unordered_set.
    
    This patch fixes the problem by changing the type discovery phase to
    iterate through a separate list of sorted exports, as opposed to
    iterating through the main unordered set.  Also included is a change
    to fix the code that looks for variables that are referenced from
    inlined routine bodies (this code wasn't scanning all of the function
    that it needed to scan).
    
    New test case for this problem in CL 186697.
    
    Updates golang/go#33020.
    
    Reviewed-on: https://go-review.googlesource.com/c/gofrontend/+/185977

From-SVN: r273564
@rsc rsc unassigned thanm Jun 23, 2022
@cagedmantis cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

5 participants