Skip to content

net/rpc: excessive mutex contention #3947

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dvyukov opened this issue Aug 13, 2012 · 6 comments
Closed

net/rpc: excessive mutex contention #3947

dvyukov opened this issue Aug 13, 2012 · 6 comments
Labels
FrozenDueToAge

Comments

@dvyukov
Copy link
Member

dvyukov commented Aug 13, 2012 

The attached contention profile (collected with http://golang.org/cl/6443115 on
BenchmarkEndToEndAsync) shows great amount of contention in rpc.Server.sendResponse(),
and the test fails to scale beyond 4 cores.
The problem seems to be that the code does reply serizlization under the sending mutex.
Can't the reply be serialized outside of the mutex, and then just binary data send under
the mutex?
I think the similar problem is on receive side, where a single goroutine deserializes
all the messages. Potentially the goroutine can just read-in the message and hand it off
for deserialization. 
In cpu profile I see that gob.Encode() consumes 15% of time and gob.Decode() - 30%. That
seems like too much work under the global mutex, and presumably it may explain bad
scaling.

Attachments:

  1. rpc_contention.svg (24270 bytes)
@robpike
Copy link
Contributor

robpike commented Aug 17, 2012

Comment 1:

This will require some redesign.

Labels changed: added priority-later, removed priority-triage.

Status changed to Accepted.

@robpike
Copy link
Contributor

robpike commented Aug 17, 2012

Comment 2:

Owner changed to @robpike.

@dvyukov
Copy link
Member Author

dvyukov commented Aug 17, 2012

Comment 3:

I have a prototype for this:
http://golang.org/cl/6461075/diff/10001/src/pkg/net/fd_linux.go
It runs significantly better on some workloads and somewhat worser on another (mainly
GOMAXPROCS=1 case).
It completely eliminates any global mutexes and epoll_ctl() on fast paths.

@robpike
Copy link
Contributor

robpike commented Aug 17, 2012

Comment 4:

Is this prototype CL relevant to this issue, or did you mean to attach it to a different
issue?

@dvyukov
Copy link
Member Author

dvyukov commented Aug 18, 2012

Comment 5:

You are right, I meant to attach it to a different non-existent issue.
The situation with net/rpc may be not that bad. I realized that the benchmark uses a
single connection for all requests. That's why it show great contention on the
connection. At least we must take into account other cases - several high-throughput
connections, a lot of connections with low activity.
Perhaps a program that is meant to use a single connection  can just use several
connections and manual load-balancing to reduce contention.

@rsc
Copy link
Contributor

rsc commented Jul 30, 2013

Comment 6:

Status changed to Retracted.

@golang golang locked and limited conversation to collaborators Jun 24, 2016
@rsc rsc unassigned robpike Jun 22, 2022
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@rsc @dvyukov @robpike @gopherbot