Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crypto/sha256: optimize sha256 implementation #34037

Closed
WangLeonard opened this issue Sep 3, 2019 · 3 comments
Closed

crypto/sha256: optimize sha256 implementation #34037

WangLeonard opened this issue Sep 3, 2019 · 3 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Performance
Milestone

Comments

@WangLeonard
Copy link
Contributor

What version of Go are you using (go version)?

$ go version
`go version go1.12.4 linux/amd64`

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
GOARCH="amd64"  

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial
and my cpu is Intel(R) Xeon(R) E5-2690 v3 @ 2.60GHz(support avx2)
openssl version is OpenSSL 1.0.2g 1 Mar 2016

What did you do?

I test the performance in the following three cases:

  1. Call the openssl library with C
  2. Call the crypto library with Go
  3. Call the openssl library with Go's Cgo

For different length strings, do 10,000,000 sha256 calculations.
The result is as follows, the unit of data is second.

length of string openssl golang cgo
1 2.93 4.47 4.43
16 2.96 4.43 4.38
128 6.93 9.21 8.53
511 18.77 23.95 21.89
512 18.37 23.14 20.63
1023 34.08 42.51 37.38
1024 33.61 41.74 36.76
4096 125.56 152.72 133.35
8192 248.05 301.27 262.73

What did you expect to see?

Test results in different cases are similar.

What did you see instead?

It can be seen that when the length of the string is small, openssl is more than 30% faster than golang.
When string length is long, using cgo to call openssl is also 10% faster than golang.

Here is my demo (does not contain loops and time statistics), please indicate if there is an error, thank you.

Call the openssl library with C (need to install openssl-dev)

#include <stdio.h>
#include <openssl/sha.h>
#include <string.h>

void sha256String(char *string, unsigned char *digest)
{
    SHA256_CTX sha256;
    SHA256_Init(&sha256);
    SHA256_Update(&sha256, string, strlen(string));
    SHA256_Final(digest, &sha256);
}
int main() {
    unsigned char digest[SHA256_DIGEST_LENGTH];
    sha256String("a",digest);
    return 0;
}
// gcc -c -g -o sha256.o sha256.c
// gcc sha256.o -lssl -lcrypto -g -O0 -o sha256

golang and cgo

package main

//#cgo LDFLAGS:-lssl -lcrypto
//#include <stdio.h>
//#include <openssl/sha.h>
//#include <string.h>
//
//void sha256String(char *string, unsigned char *digest,int strlen)
//{
//SHA256_CTX sha256;
//SHA256_Init(&sha256);
//SHA256_Update(&sha256, string, strlen);
//SHA256_Final(digest, &sha256);
//}
import "C"
import (
	"crypto/sha256"
	"fmt"
	"unsafe"
)

func sha256StringCgo(b []byte)[]uint8{
	output:=make([]uint8,32,32)
	inputa:=(*C.char)(unsafe.Pointer(&b[0]))
	inputb:=(*C.uchar)(unsafe.Pointer(&output[0]))
	inputc:=C.int(len(b))
	C.sha256String(inputa,inputb,inputc)
	return output
}

func sha256StringGo(b []byte)[]uint8{
	h:=sha256.New()
	h.Write(b)
	ans:=h.Sum(nil)
	return ans
}

func main(){
	fmt.Println(sha256StringCgo([]byte("a")))
	fmt.Println(sha256StringGo([]byte("a")))
}

So can we optimize the sha256 of golang? Because this is a hot function in my application, I am concerned about the time efficiency of sha256.

Here are some preliminary analysis, I hope it can provide some help.
The longest running function in the test, in golang is $GOROOT/src/crypto/sha256/sha256block_amd64.s block avx2 case,
in openssl is sha256_block_data_order_avx2.
You can download openssl 1.0.2g source file, make , then can found it in openssl-1.0.2g/crypto/sha/sha256-x86_64.s.
sha256-x86_64.s.zip

Since I am not very familiar with assembly, I don't know how to optimize.
Look forward to your reply, thank you.

@katiehockman katiehockman changed the title crypto: can we refer to OpenSSL to optimize sha256 implementation? crypto: optimize sha256 based on OpenSSL implementation Sep 3, 2019
@katiehockman katiehockman added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Sep 3, 2019
@katiehockman katiehockman added this to the Unplanned milestone Sep 3, 2019
@katiehockman
Copy link
Contributor

/cc @FiloSottile

@katiehockman katiehockman changed the title crypto: optimize sha256 based on OpenSSL implementation crypto/sha256: optimize sha256 based on OpenSSL implementation Sep 3, 2019
@FiloSottile FiloSottile changed the title crypto/sha256: optimize sha256 based on OpenSSL implementation crypto/sha256: optimize sha256 implementation Sep 3, 2019
@FiloSottile
Copy link
Contributor

The Go standard library places higher value on reducing complexity, so it doesn't aim to be the fastest at implementing primitives. If this makes it unsuitable for your use case, please do share some details, as we take that into account when deciding how much complexity to take. See also https://golang.org/design/cryptography-principles.

There are more aggressively optimized third-party implementations of SHA-256 you might want to consider instead, like https://github.com/minio/sha256-simd.

@WangLeonard
Copy link
Contributor Author

Ok, I will find a computer that support for avx512 to verify the performance of sha256-simd, Thank you very much.

@golang golang locked and limited conversation to collaborators Apr 12, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Performance
Projects
None yet
Development

No branches or pull requests

4 participants