Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime/cgo: Memory RSS has been growing abnormally when call cgo and go #43160

Closed
bluce-ldg opened this issue Dec 13, 2020 · 4 comments
Closed

runtime/cgo: Memory RSS has been growing abnormally when call cgo and go #43160

bluce-ldg opened this issue Dec 13, 2020 · 4 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Backlog

Comments

@bluce-ldg
Copy link

bluce-ldg commented Dec 13, 2020 •
edited

What version of Go are you using (go version)?

$ go version
go1.15.6 linux/amd64

Does this issue reproduce with the latest release?

yes, reproduce with version 1.13、1.15

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=".gitlab.com,.gitee.com"
GONOSUMDB=".gitlab.com,.gitee.com"
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=".gitlab.com,.gitee.com"
GOPROXY="https://goproxy.cn,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.google.cn"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build926124830=/tmp/go-build -gno-record-gcc-switches"

What did you do?

demo code 
package main

/*

#cgo CFLAGS: -I${SRCDIR}/so -fpermissive -std=c99 -std=c++11  -Wall -w -O0 -fpie -pthread -pedantic

//#cgo LDFLAGS: -Wl,-rpath-link=${SRCDIR}/so -L${SRCDIR}/so  -ldl -lm -lcpp_clustersdk

#cgo LDFLAGS: -Wl,-rpath-link=${SRCDIR}/so -L${SRCDIR}/so  -ldl -lm

//#cgo LDFLAGS: -Wl,-rpath-link=${SRCDIR}/so -I${SRCDIR}/so -L${SRCDIR}/so -ldl -lm -lcpp_clustersdk -lcwcluster

#include <stdbool.h>

#include <stdlib.h>

#include <stdio.h>

//#include <string.h>

//#include <dlfcn.h>


char* _gTestFeatures = NULL;

char* _gHashs = NULL;

char* _gAttrs = NULL;

bool g_init = false;

long g_testSize = 100000;

long g_testCount = 1000000;


typedef struct CMemTest_

{

long _index;

long _count;


char* pFeas;
char* pHashs;
char* pAttrs;



}CMemTest;


CMemTest* gMemTest = NULL;


void PushMemTest(CMemTest* obj)

{

if(!obj->pFeas)

{

obj->pFeas = (char*)malloc(384g_testSize);

obj->pHashs = (char)malloc(64g_testSize);

obj->pAttrs = (char)malloc(832g_testSize);

}


    memcpy(obj->pFeas+obj->_index*384, &_gTestFeatures[obj->_index*384], 384);
    memcpy(obj->pHashs+obj->_index*64, &_gHashs[obj->_index*64], 64);
    memcpy(obj->pAttrs+obj->_index*8*32, &_gAttrs[obj->_index*8*32], 8*32);
    obj->_index++;
    obj->_count++;
    if (obj->_index >= g_testSize)
    {
        obj->_index = 0;
    }



}


CMemTest* AllocMemTest()

{

if(gMemTest != NULL)

abort();


gMemTest = (CMemTest*)malloc(sizeof(CMemTest));
gMemTest->_index = 0;
gMemTest->_count = 0;
gMemTest->pFeas = NULL;
gMemTest->pHashs = NULL;
gMemTest->pAttrs = NULL;



}


void FreeMemTest(CMemTest* obj)

{

if(obj->pFeas) free(obj->pFeas);

if(obj->pHashs) free(obj->pHashs);

if(obj->pAttrs) free(obj->pAttrs);

free(obj);

}


int c_Master_TestMem1()

{

if (g_init == false)

{

_gTestFeatures = (char*)malloc(384*(g_testSize+1));

_gHashs = (char*)malloc(64*(g_testSize+1));

_gAttrs = (char*)malloc(832(g_testSize+1));

for (long i = 0; i < g_testSize; i++)

{

for (long j = 0; j < 384; j++)

{

_gTestFeatures[i384+j] = ij;

}

for (long j = 0; j < 64; j++)

{

_gHashs[i64+j] = ij;

}

for (long j = 0; j < 832; j++)

{

_gAttrs[i832+j] = ij;

}

}


// 		new std::thread(& {

// 			uint64_t timeCount = 0;

// 			while (1)

// 			{

// #ifndef _WIN32

// 					malloc_trim(0);

//                     printf("-----------------\n");

// #endif

// 				std::chrono::seconds dura(1);

// 				std::this_thread::sleep_for(dura);

//             }

// 		});

g_init = true;

}


if(gMemTest == NULL)
{
     gMemTest = AllocMemTest();
}

printf("====================================== gMemTest count:%lld =============================\n", gMemTest->_count);

if (0 == (gMemTest->_count % g_testCount))
{
    FreeMemTest(gMemTest);
    gMemTest = NULL;
	printf("\n\n\n\n====================================== reset, press anykey continue =============================\n\n\n\n\n");
	//getchar();

     gMemTest = AllocMemTest();
  //  sleep(1);
}

for (long i = 0; i < 10000; i++)
{
    //if ((i % 1000) == 0)
    //{
    //    //usleep(1000*100);
    //}
    PushMemTest(gMemTest);
}
return 0;



}

*/

import "C"

import (

"fmt"

"net/http"

_ "net/http/pprof"

"os"

)


type DBFeatureJournalnfo struct {

InstType int

FaceId   int64

UserId   string

PriorLabel     int64

GroupIdMapAlgo int32

Feature            []byte

ShortFeatureA      []byte

ShortFeatureB      []byte

ImageUrl           string

Qualityscore       []byte

FailedModelingInfo string

AddScoreFlag       bool

AddScore           float32

Time               int64

Status             int32

TrackId            int64

TrackBeginTime     int64

TrackEndTime       int64

ExtData            string

}


var rawFea = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"


func GetMemTest() []DBFeatureJournalnfo {

results := make([]DBFeatureJournalnfo, 10000)

result := DBFeatureJournalnfo{}

result.InstType = 1

result.FaceId = 1

result.UserId = "1111111111111111111111111111111111111111111111111"

result.PriorLabel = 1

result.GroupIdMapAlgo = 1


//result.Feature = append(result.Feature, rawFea[0:384]...)
//result.Feature = append(result.Feature, rawFea[0:384]...)

result.Feature = make([]byte, 384, 384*2)
//copy(result.Feature, "11111111111fgdgggggggggggggggggggggggggsgsfddddddddddddddddddddddddddddddddddddddddddddd")
result.Qualityscore = make([]byte, 32*8, 32*8*2)
result.ImageUrl = "ffffffffffffffffffffffffff"

for i := 0; i < 10000; i++ {
	results = append(results, result)
}
return results



}


func main() {

go func() {

http.ListenAndServe("0.0.0.0:8080", nil)

}()


//go func() {
//	http.ListenAndServe("0.0.0.0:8080", nil)
//	debug.FreeOSMemory()
//	time.Sleep(time.Second*5)
//}()

fmt.Println("------------------------------------------")
N := 100000000

for i := 0; ; i++ {
	// step 1 call cgo test function
	C.c_Master_TestMem1()
	// step 2 call go test function
	test := GetMemTest()
	if test == nil {
		os.Exit(1)
	}
}



}

What did you expect to see?

when both call cgo test and go test, It would taking 167Mb of RSS and should be able to stabilize around 170Mb,instead of memory increase several times.

What did you see instead?

  1. We can see from the code，when only call cgo test function(step 1), not call go test function(step 2), It would taking ~142Mb of RSS(resident set size memory) .
    2.when only call go test function(step 2), not call cgo test function(step 1), It would taking ~33Mb of RSS(resident set size memory) .
    3.when both call cgo test and go test, It would taking 167Mb of RSS and then grow to >700Mb
    4.In addition, Using the windows platform，RSS is normal , not as high as linux
@toothrot toothrot changed the title Memory RSS has been growing abnormally when call cgo and go cgo: Memory RSS has been growing abnormally when call cgo and go Dec 17, 2020
@toothrot toothrot changed the title cgo: Memory RSS has been growing abnormally when call cgo and go runtime/cgo: Memory RSS has been growing abnormally when call cgo and go Dec 17, 2020
@toothrot toothrot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Dec 17, 2020
@toothrot toothrot added this to the Backlog milestone Dec 17, 2020
@toothrot
Copy link
Contributor

/cc @ianlancetaylor

@ianlancetaylor
Copy link
Contributor

I don't understand what "abnormally" means here. The measure of RSS depends a great deal on what the kernel does and what other processes are running. If the memory does eventually stabilize, then that may be expected. If memory use keeps increasing, there may be a bug in the way that the program is using cgo.

It's not surprising that a program that uses cgo uses more memory than a program that does not use cgo.

@ianlancetaylor ianlancetaylor added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Dec 17, 2020
@ianlancetaylor
Copy link
Contributor

It's unlikely we'll be able to say anything useful without a test case that we can use to reproduce the problem.

It's likely that you will get better answers on a forum. See https://golang.org/wiki/Questions.

@gopherbot
Copy link

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@golang golang locked and limited conversation to collaborators Jan 17, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
4 participants
@toothrot @ianlancetaylor @gopherbot @bluce-ldg