Initial RCA:

function enumerateAttributes constructs clang command line with "-march=" option if the triple passed to capture-fcn-attributes equals to the default triple of the host platform, it works for x86-64 but not aarch64 due to the different behaviors of -march&-mtune/-mcpu between x86_64 and aarch64, -mcpu option is expected on aarch64.

Enclosing a thread on clang-developers for reference.
http://clang-developers.42468.n3.nabble.com/Behavior-of-mcpu-td4064178.html

Thanks for the problem report.

BTW, congratulations on getting an aarch64 version of gollvm working -- I hope you will contribute your work back to the project...

I tried what you described from my x86_64 box, as follows:

$ cd <llvmroot>/tools/gollvm/tools
$ go build capture-fcn-attributes.go
$ ./capture-fcn-attributes -o /tmp/aheader.h -triples aarch64-unknown-linux-gnu
$

and I am seeing something different from what you're seeing-- my output looks like:

// DO NOT EDIT: this file auto-generated by the following command:
//
//    ./capture-fcn-attributes -o /tmp/aheader.h -triples aarch64-unknown-linux-gnu
//
// in combination with clang:
//
//  clang version 10.0.0 (trunk 368990) (llvm/trunk 369004)
//

typedef struct {
  const char *cpu;
  const char *attrs;
} CpuAttrs;

typedef struct {
  const char *triple;
  const CpuAttrs *cpuattrs;
} TripleCpus;

// triple: aarch64-unknown-linux-gnu
static const CpuAttrs attrs0[] = {
  // first entry is default cpu
  { "generic", "+neon" },
  { "cortex-a35", "+aes,+crc,+crypto,+fp-armv8,+neon,+sha2" },
  { "cortex-a53", "+aes,+crc,+crypto,+fp-armv8,+neon,+sha2" },
  { "cortex-a55", "+aes,+crc,+crypto,+dotprod,+fp-armv8,+fullfp16,+lse,+neon,+ras,+rcpc,+rdm,+sha2,+v8.2a" },
  { "cortex-a57", "+aes,+crc,+crypto,+fp-armv8,+neon,+sha2" },
  { "cortex-a65", "+aes,+crc,+crypto,+dotprod,+fp-armv8,+fullfp16,+lse,+neon,+ras,+rcpc,+rdm,+sha2,+v8.2a" },
  { "cortex-a65ae", "+aes,+crc,+crypto,+dotprod,+fp-armv8,+fullfp16,+lse,+neon,+ras,+rcpc,+rdm,+sha2,+v8.2a" },
  { "cortex-a72", "+aes,+crc,+crypto,+fp-armv8,+neon,+sha2" },
  { "cortex-a73", "+aes,+crc,+crypto,+fp-armv8,+neon,+sha2" },
  { "cortex-a75", "+aes,+crc,+crypto,+dotprod,+fp-armv8,+fullfp16,+lse,+neon,+ras,+rcpc,+rdm,+sha2,+v8.2a" },
  { "cortex-a76", "+aes,+crc,+crypto,+dotprod,+fp-armv8,+fullfp16,+lse,+neon,+ras,+rcpc,+rdm,+sha2,+v8.2a" },
  { "cortex-a76ae", "+aes,+crc,+crypto,+dotprod,+fp-armv8,+fullfp16,+lse,+neon,+ras,+rcpc,+rdm,+sha2,+v8.2a" },
  { "cyclone", "+aes,+crypto,+fp-armv8,+neon,+sha2,+zcm,+zcz" },
  { "exynos-m1", "+aes,+crc,+crypto,+fp-armv8,+neon,+sha2" },
  { "exynos-m2", "+aes,+crc,+crypto,+fp-armv8,+neon,+sha2" },
  { "exynos-m3", "+aes,+crc,+crypto,+fp-armv8,+neon,+sha2" },
  { "exynos-m4", "+aes,+crc,+crypto,+dotprod,+fp-armv8,+fullfp16,+lse,+neon,+ras,+rdm,+sha2,+v8.2a" },
  { "exynos-m5", "+aes,+crc,+crypto,+dotprod,+fp-armv8,+fullfp16,+lse,+neon,+ras,+rdm,+sha2,+v8.2a" },
  { "falkor", "+aes,+crc,+crypto,+fp-armv8,+neon,+rdm,+sha2" },
  { "kryo", "+aes,+crc,+crypto,+fp-armv8,+neon,+sha2" },
  { "neoverse-e1", "+aes,+crc,+crypto,+dotprod,+fp-armv8,+fullfp16,+lse,+neon,+ras,+rcpc,+rdm,+sha2,+v8.2a" },
  { "neoverse-n1", "+aes,+crc,+crypto,+dotprod,+fp-armv8,+fullfp16,+lse,+neon,+ras,+rcpc,+rdm,+sha2,+spe,+v8.2a" },
  { "saphira", "+aes,+crc,+crypto,+fp-armv8,+lse,+neon,+ras,+rcpc,+rdm,+sha2,+spe,+v8.3a" },
  { "thunderx", "+aes,+crc,+crypto,+fp-armv8,+neon,+sha2,+spe" },
  { "thunderx2t99", "+aes,+crc,+crypto,+fp-armv8,+lse,+neon,+rdm,+sha2,+v8.1a" },
  { "thunderxt81", "+aes,+crc,+crypto,+fp-armv8,+neon,+sha2,+spe" },
  { "thunderxt83", "+aes,+crc,+crypto,+fp-armv8,+neon,+sha2,+spe" },
  { "thunderxt88", "+aes,+crc,+crypto,+fp-armv8,+neon,+sha2,+spe" },
  { "tsv110", "+aes,+crc,+crypto,+dotprod,+fp-armv8,+fp16fml,+fullfp16,+lse,+neon,+ras,+rdm,+sha2,+spe,+v8.2a" },
  { "", "" } // sentinel
};

const TripleCpus triples[] = {
  { "aarch64-unknown-linux-gnu", &attrs0[0] },
  { "", nullptr } // sentinel
};

Not sure about the quality of this info, but it at least seems plausible. How are you configuring your LLVM build (e.g. cmake args etc)?

Hi Than,

Thanks for looking into the issue.
Just FYI, 3 topics are covered below, thanks.

<1>. Issue of capture-fcn-attributes
I used both clang-6.0 (official release build) and clang-10.0 debug build built from source code, both failed with the following error if running capture-fcn-attributes on x86 machine passing the triple aarch64-unknown-linux-gnu, test.c is same as the .c file under /tmp/*.

Seems to be a cross-compile issue which I'm not sure if gollvm intends to support.

===============================================================
xiaji01@x86:~/data/sample/cpus/x86$ clang -emit-llvm -S -O3 -Xclang -disable-llvm-passes test.c --target=aarch64-unknown-linux-gnu -mcpu=cortex-a35
In file included from ../test.c:1:
In file included from /home/xiaji01/build/clang.dbg/lib/clang/10.0.0/include/inttypes.h:21:
In file included from /usr/include/inttypes.h:25:
/usr/include/features.h:424:12: fatal error: 'sys/cdefs.h' file not found

include <sys/cdefs.h>

       ^~~~~~~~~~~~~

1 error generated.

Guess that might be due to no cross-compile tool chain in my x86 system.

To summarize, we got two issues so far:

1. on aarch64, capture-fcn-attributes -o header.h -triples aarch64-unknown-linux-gnu doesn't work because "-march=" is passed to clang while it expects "-cpu="

2. on x86_64, capture-fcn-attributes -o header.h -triples aarch64-unknown-linux-gnu, doesn't work in some environments if target-specific <sys/cdefs.h> could not be found.

<2> About ArchCpusAttrs.h

May I ask what's the suggestion to incorporate multiple triples? To have multiple ArchCpuAttrs.h, or just one having multiple triples, or let cmake generate it on the fly?

<3> AArch64 porting
Sure, we're perfectly willing to contribute back, so far it's still an experimental work for studying gollvm on aarch64, utilizing llvm-goc only, as the c-abi part is not in place, hopefully what have been changed, mainly in build configuration, tool chain and a few libgo scripts, could be finalized and checked in eventually.

The header issue should be easy to fix, I will send a CL.

Change https://golang.org/cl/190899 mentions this issue: gollvm: avoid use of C headers in capture-fcn-attributes.go

I sent a fix that should allow you to run the tool on x86_64 without having a cross-compile header environment set up.

You asked about multiple triples: this should already be supported (just past a list of triples to the -triples flag).

AArch64 porting -- the C abi is the interesting part. I am happy to provide reviews and suggestions if someone wants to work on it.

Thanks Than, the fix works now, for both cross-compile and generating multiple triples' attrs on x86, I assume ArchCpusAttrs.h will stay pre-generated in gollvm, so plan to add the attr array for aarch64.

And, just FYI, 'llc --version' might outputs a different default triple on x86_64 other than x86_64-unknown-linux-gnu, thus relying comparing the triple argument and the default triple to choose between -mcpu and -march may not work sometimes, but figure it's not a big deal.

============================================================
LLVM (http://llvm.org/):
LLVM version 6.0.0

Optimized build.
Default target: x86_64-pc-linux-gnu
Host CPU: haswell

@thanm
Hi Than,

Are the exported libgo packages, i.e those .gox files, are platform independent, or supposed to support cross-platform importing? Thanks.

Depends on what you mean by "platform-independent" and I guess. For example, consider this package:

package example

import "unsafe"

const IntSize = unsafe.Sizeof(uintptr(0))

The export data for the package will be different depending on whether you are building on a 32-bit architecture vs a 64-bit architecture. On the other hand it's not hard to imagine Go packages whose export data would be pretty much identical across platforms. Something else to keep in mind: export data for gccgo/gollvm now contains bodies of small inlinable functions, meaning that that code could contain assumptions about the host platform/target.

Regarding "cross-platform importing", you could certainly build a package on one platform and then use it on another as experiment, but the toolchain is definitely not designed to support this in any meaningful or comprehensive way (due to issues like the one above).

Thanks for the details, it helps much, especially the inline functions part.

Got a few questions w.r.t aarch64 configuration, hopefully you could help here, thanks for your time.

1. (for the record, this one is just for experiment before the ABI part is available)
   llvm-goc's help message indicates it could generate cross-target code, currently the compiler will look for packages of the specified target, say on x86_64 it imports from /lib64/go// if setting the --target option.
   --target= Generate code for the given target
   is there any change in 'importing' needed, or my use model is not expected?
   Noticed gccgo doesn't have a counterpart option.

2. for 64bit arm platform, llvm's arch is AARCH64 while go uses ARM64, is there any plan to unify them in go?

3. gen-sysinfo.go and runtime-sysinfo.go generated on aarch64 contains type "uint128" and a couple of functions utilizing it, is it expected to define it as '[16]byte' in go? If it is, is mkrsysinfo.sh/mksysinfo.sh a right place to introduce the post-processing (that's what I'm doing now in my experimental change)?

4. Can I assume clang (6.0 or above) or gcc-9 are c compilers gollvm building expects? Because the arm-v8 intrinsics in aeshash.c are only available with them or higher version.

[general remark -- this seems to be turning into a general discussion of gollvm as opposed to bug in capture-fcn-attributes, so maybe it should move to go-nuts mailing list?]

For question 1:

This is somewhat unexplored territory, but in general with GCC tools (both the gcc C and Go compilers) there is no notion of a single compiler that targets two different unrelated architectures-- you have to build and install each GCC cross compiler separately. In other words if you want compilers for x86, Arm, and PPC, you need three separate installs.

With clang the model is that a single executable can target multiple architectures (assuming you didn't disable this at cmake time), and you can simply select the target using a command line option.

For gollvm, I think the hope is that we can use the clang model instead of the GCC model for cross-compilation, but we haven't really fleshed out the driver support for any of that yet. Someone will need to think through the issues and make the necessary changes to the driver so that it can pick up Go libraries (ex: libgo.a) and C libraries (ex: libgcc.a) from the correct locations.

Similarly we'll need to change the libgo build procedure to generate different architecure-specific variants and place them in the correct locations. I can help with these changes if you like.

For question 2:

for 64bit arm platform, llvm's arch is AARCH64 while go uses ARM64, is there any plan to unify them in go?

Currently this is handled by having cmake code that looks at the LLVM cmake variables containing OS/arch and translates them into the Go equivalents (e.g. "amd64" instead of "x86_64"). See /cmake/modules/GoVars.cmake.

For question 3:

gen-sysinfo.go and runtime-sysinfo.go generated on aarch64 contains type "uint128" and a couple of functions utilizing it, is it expected to define it as '[16]byte' in go? If it is, is mkrsysinfo.sh/mksysinfo.sh a right place to introduce the post-processing (that's what I'm doing now in my experimental change)?

Yes, that sounds like the right place to make changes.

For question 4:

Can I assume clang (6.0 or above) or gcc-9 are c compilers gollvm building expect, because the arm-v8 intrinsics in aeshash.c are only available with them or higher version.

That's fine. I think at the moment the contributor guide requires V5.0 or later for Clang, or V6.0 or later of GCC, but we can update it to require later versions for specific targets.

Thanks, Than

Thanks Than.

[general remark -- this seems to be turning into a general discussion of gollvm as opposed to bug
in capture-fcn-attributes, so maybe it should move to go-nuts mailing list?]
Is it a good idea we change the thread's title to make it a general discussion on arm64 build/configuration, as well as tracking submits?

I may look into the cross-compilation requirement after finishing enabling aarch64 build, without abi part at first, trying to refine existing changes, which support building llvm-goc, tools and running unit-tests, to avoid redundancy.

For the goarch issue, the same change as you proposed was made in GoVars.cmake, good to know we'll continue going with that way.

Thanks for the answer to other questions.

Change https://golang.org/cl/194997 mentions this issue: libgo: support build on arm64 linux

Change https://golang.org/cl/194998 mentions this issue: gollvm: support build on arm64 linux

gen-sysinfo.gccgo.go.txt
gen-sysinfo.gollvm.go.txt

Hi Than,
Two patches are associated with this issue, could https://golang.org/cl/194998 be submitted as well? Thanks @thanm