proposal: net: fetch all interfaces with associated addresses in a single call #53660

mandarjog · 2022-07-02T07:31:03Z

Background

At present if we want to get all interface and associated Addresses we write the following

ifaceByName := make(map[string]net.Interface)
addrByName := make(map[string][]net.IP)

interfaces, _ := net.Interfaces()

for i := 0; i < len(interfaces); i++ {
    intf := interfaces[i]
    addrByName[intf.Name] = intf.Addrs()
}

I would like a function that returns this entire information.

Why is it needed ?

intf.Addrs() call is inefficient because it results in getting the entire RIB syscall.NetlinkRIB(syscall.RTM_GETADDR, syscall.AF_UNSPEC) every time. The size of the RIB is proportional to the number of interfaces/Addrs. If on a router you have 1000s of interfaces this O(n^2) behaviour becomes untenable.
When number of interfaces is small this problem does not show up.

details

The flamegraphs show excessive time spent making syscall and repeatedly parsing messages.
I also ran call count using bpftrace

/usr/local/bin/bpftrace -p ${prog-pid}\
    -e 'uprobe:/usr/local/bin/prog:*vmsysmon.*,uprobe:/usr/local/bin/prog:net.int*,uprobe:/usr/local/bin/prog:syscall.* { @[func] = count(); }'

@[prog.(*IfMon).calcStats]: 5922
@[net.interfaceAddrTable]: 6169
@[syscall.Close]: 6318
@[syscall.RawSyscall.abi0]: 10287
@[syscall.Syscall.abi0]: 10583
@[syscall.(*SockaddrNetlink).sockaddr]: 12320
@[syscall.ParseNetlinkRouteAttr]: 16582
@[syscall.recvfrom]: 398880
@[syscall.anyToSockaddr]: 506789
@[syscall.Syscall6.abi0]: 568181
@[syscall.Recvfrom]: 622379
@[syscall.ParseNetlinkMessage]: 655167

calcStats function results in

After I made the change to fetch the entire RIB and associate addresses in a single loop it makes significant change in the call counts and times.

@[syscall.Syscall.abi0]: 2265
@[syscall.recvfrom]: 3330
@[syscall.Recvfrom]: 3334
@[syscall.ParseNetlinkMessage]: 3335
@[syscall.anyToSockaddr]: 3347
@[syscall.Syscall6.abi0]: 3482
@[aviatrix.com/vmsysmon.(*IfMon).calcStats]: 5912
@[syscall.ParseNetlinkRouteAttr]: 18326

Proposed API

Option 1

// InterfaceAddrsWithIndex returns a map from interface index to a list of
// associated unicast addresses.
func InterfaceAddrsWithIndex()(map[int][]Addr, error)  {
	return nil, nil
}

Keeps the flavor of existing InterfaceAddrs API.
Typical use.

func testListInterfaceDetails() {
	interfaces, _ := Interfaces()
	addrsMap, _ := InterfaceAddrsWithIndex()

	for i := 0; i < len(interfaces); i++ {
		intf := interfaces[i]
		if addrs, found := addrsMap[intf.Index]; found {
			fmt.Printf("%v: %v\n", intf, addrs)
		}
	}
}

Option 2

Treat all the information above as a single DB and return it. More optimizations are possible with this API without changing the API signature.

// IntfRecord keeps interface and associated Addresses.
type IntfRecord struct {
	Intf  Interface
	Addrs []Addr
}
type IntfDB map[int]IntfRecord

// InterfaceDB returns a map from interface index to an IntfRecord.
// This is the most efficient way feth the entire Interface database.
func InterfaceDB() (IntfDB, error) {
	return nil, nil
}

The text was updated successfully, but these errors were encountered:

ianlancetaylor · 2022-07-03T00:09:36Z

What specific new API do you propose to address this? Thanks.

mandarjog · 2022-07-03T18:05:20Z

@ianlancetaylor I have updated the issue with the API proposal.

mandarjog · 2022-07-03T18:17:17Z

@ianlancetaylor can you point me to benchmark tests that test with a number of interfaces? I found benchmark tests in interface_tests.go however they operate with a small number of interfaces (perhaps just loopback?). Is there test infra to create a large number of interfaces, managing network namespaces to do that and cleanup ?

seankhliao · 2022-07-03T20:57:11Z

Alternate API: the net.Addrs returned by net.InterfaceAddrs() could expose an additional method for getting the associated interface

ianlancetaylor · 2022-07-03T20:58:16Z

Is there test infra to create a large number of interfaces, managing network namespaces to do that and cleanup ?

Not to my knowledge.

seankhliao · 2022-07-03T21:06:38Z

#51934 is a report for efficiency / race / ignoring failures from k8s

mandarjog · 2022-07-03T21:40:47Z

@seankhliao as you can see in the usage example I have specified, devices and interfaces are snapshots at slightly different times. This means that we should expect to see device indexes returned by 2nd call to be unknown to the interfaces() call. The only way to fix it is to get a full snapshot from the kernel, which i don’t think is available.

mandarjog · 2022-07-04T01:58:58Z

Alternate API: the net.Addrs returned by net.InterfaceAddrs() could expose an additional method for getting the associated interface

interesting idea however,
@seankhliao addresses are a property of an interface not the other way around. So from a modeling perspective it does not make sense.

mandarjog · 2022-07-11T21:22:26Z

@ianlancetaylor any comments? Does a PR need to be implemented for all platforms or can start with Linux only ?

ianlancetaylor · 2022-07-13T03:36:08Z

This will go to review by the proposal review committee as described at https://go.dev/s/proposal.

If we adopt this change, it needs to do something sensible on all platforms.

Thanks.

wjordan · 2022-12-17T23:22:11Z

Alternate API: the net.Addrs returned by net.InterfaceAddrs() could expose an additional method for getting the associated interface

interesting idea however, @seankhliao addresses are a property of an interface not the other way around. So from a modeling perspective it does not make sense.

I think it makes perfect sense, both from a 'modeling perspective' (it's called a 'belongs to' relation) and based on the existing underlying ifaddrmsg structure (which contains an 'interface index' property).

That said, this seems like a Linux-specific issue that may not require any changes to the high-level API to resolve. An alternate platform-implementation-only fix would be for the Linux implementation of net.InterfaceAddrs() (specifically newAddr()) to return a subtype of net.IPNet that contains an additional Index field, which would make it possible for user-level code to associate this Linux-specific address type with interfaces returned by net.Interfaces(), skipping the extra calls to intf.Addrs() as a platform-specific performance optimization.

mandarjog added the Proposal label Jul 2, 2022

gopherbot added this to the Proposal milestone Jul 2, 2022

ianlancetaylor changed the title ~~proposal: net/interface.go fetch all interfaces with associated addresses in a single call~~ proposal: net: fetch all interfaces with associated addresses in a single call Jul 3, 2022

ianlancetaylor added this to Incoming in Proposals (old) Jul 3, 2022

wjordan mentioned this issue Jan 30, 2023

Improve Linux performance of GetAllInterfaces hashicorp/go-sockaddr#47

Open

This was referenced Aug 18, 2023

Improve performance of interface name lookups ema/qdisc#12

Merged

Upgrade github.com/ema/qdisc to v1.0.0 to improve qdisc collector performance prometheus/node_exporter#2779

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

proposal: net: fetch all interfaces with associated addresses in a single call #53660

proposal: net: fetch all interfaces with associated addresses in a single call #53660

mandarjog commented Jul 2, 2022 •

edited

ianlancetaylor commented Jul 3, 2022

mandarjog commented Jul 3, 2022

mandarjog commented Jul 3, 2022

seankhliao commented Jul 3, 2022

ianlancetaylor commented Jul 3, 2022

seankhliao commented Jul 3, 2022

mandarjog commented Jul 3, 2022

mandarjog commented Jul 4, 2022

mandarjog commented Jul 11, 2022

ianlancetaylor commented Jul 13, 2022

wjordan commented Dec 17, 2022

proposal: net: fetch all interfaces with associated addresses in a single call #53660

proposal: net: fetch all interfaces with associated addresses in a single call #53660

Comments

mandarjog commented Jul 2, 2022 • edited

Background

Why is it needed ?

details

Proposed API

Option 1

Option 2

ianlancetaylor commented Jul 3, 2022

mandarjog commented Jul 3, 2022

mandarjog commented Jul 3, 2022

seankhliao commented Jul 3, 2022

ianlancetaylor commented Jul 3, 2022

seankhliao commented Jul 3, 2022

mandarjog commented Jul 3, 2022

mandarjog commented Jul 4, 2022

mandarjog commented Jul 11, 2022

ianlancetaylor commented Jul 13, 2022

wjordan commented Dec 17, 2022

mandarjog commented Jul 2, 2022 •

edited