cmd/pprof: If the file under the current path contains a colon, go tool pprof will parse it into a url and will not work properly. #63924

cuishuang · 2023-11-03T07:02:42Z

What version of Go are you using (`go version`)?

$ go version

go version devel go1.22-6dd7462a04 Fri Oct 13 17:10:31 2023 +0000 darwin/arm64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE='on'
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/myname/Library/Caches/go-build'
GOENV='/Users/myname/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/myname/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/myname/go'
GOPRIVATE=''
GOPROXY=''
GOROOT='/Users/myname/mypath/go'
GOSUMDB='off'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/Users/myname/mypath/go/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='devel go1.22-6dd7462a04 Fri Oct 13 17:10:31 2023 +0000' # I compiled it myself, it can be ignored and has nothing to do with the version.
GCCGO='gccgo'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='1'
GOMOD='/Users/myname/mypath/go/src/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/9t/839s3jmj73bcgyp5x_xh3gw00000gn/T/go-build3891298911=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

I helped a colleague write a sample code to obtain the pprof file when the memory exceeds the threshold.

package main

import (
	"fmt"
	"os"
	"runtime"
	"runtime/pprof"
	"time"
)

func main() {

	ticker := time.NewTicker(10 * time.Second)

	go func() {
		for {
			time.Sleep(1e9)
			var m runtime.MemStats
			runtime.ReadMemStats(&m)

			memUsage := m.Sys / 1024 / 1024 

			if memUsage > 1024 {
				fmt.Printf("memory usage more than 1GB：%dMB\n", memUsage)

				// save pprof file
				filepath := "/xxx/yyy/path/mem" + "." + strconv.Itoa(int(time.Now().Unix()))

				f, err := os.Create(filepath)
				if err != nil {
					fmt.Println("err1:", err)
				}
				defer f.Close()

				if err2 := pprof.WriteHeapProfile(f); err2 != nil {
					fmt.Println("err2:", err2)
				}

				fmt.Println("pprof data has saved in:", filepath)

			} else {
				fmt.Printf(memory usage is：%dMB\n", memUsage)
			}
		}
	}()

	for {
		select {
		case <-ticker.C:
		
			var a = make([]byte, 1073741824)
			_ = a
			fmt.Println("ticker：", time.Now())
		}
	}

	select {}

}

I tested it locally and everything was fine. He made certain modifications and then released it to the production environment. When the memory surge occurred, this program obtained the pprof file as expected, but what was confusing and surprising was that when using go tool pprof thefile, an error was reported:

Fetching profile over HTTP from http://mem_2023-11-01_14:23:16/debug/pprof/profile

profile.pprof.2023-11-03_14:23:16: Get "http://mem_2023-11-01_14:23:16/debug/pprof/profile": dial tcp: lookup mem_2023-11-01_14:23: no such host

I spent a lot of time trying to research, and one of his modifications was to change the timestamp in the file path to time.Now().Format("2006-01-02_15:04:05") in order to be more intuitive.

That is

filepath := "/xxx/yyy/path/mem" + "." + strconv.Itoa(int(time.Now().Unix())) ---> filepath := "/xxx/yyy/path /mem" + "." + time.Now().Format("2006-01-02_15:04:05")

After adding debug information several times, it was found that it was the adjustURL method in internal/driver/fetch.go. Because the file name contained a colon, url.Parse parsed the file path into a url. . .

This is confusing and counter-intuitive. After many experiments, I think we can consider determining whether there is a file with the same name in the directory. If so, treat it as a file instead of a URL.

(We only need to consider the case of executing go tool pprof in the directory where the profile is located. In the case of go tool pprof a/b/mem_pprof_file_2006_01_02_15:04:05, it will not be recognized as a URL now)

What did you expect to see?

I want to be able to use pprof normally without renaming the file

What did you see instead?

Fetching profile over HTTP from http://mem_2023-11-01_14:23:16/debug/pprof/profile

profile.pprof.2023-11-03_14:23:16: Get "http://mem_2023-11-01_14:23:16/debug/pprof/profile": dial tcp: lookup mem_2023-11-01_14:23: no such host

The text was updated successfully, but these errors were encountered:

gopherbot · 2023-11-03T07:05:35Z

Change https://go.dev/cl/539595 mentions this issue: cmd/pprof: fix exception when file type contains colon

cuishuang · 2023-11-03T07:09:32Z

To fix this problem, we also need to update google/pprof. I have already raised an issue google/pprof#816

rhysh · 2023-11-03T13:29:42Z

I don't see which specific go tool pprof [...] command you ran that didn't work.

Have you tried prefixing the file with ./ to specify that it's a file in the local directory?

Here's how it works (and fails) on my machine. First creating a small demo profile:

$ go1.21 version                                             
go version go1.21.3 darwin/arm64

$ go1.21 test context -o /dev/null -memprofile './localhost:8080'
ok  	context	0.366s

$ ls -al ./localhost:8080
-rw-r--r--  1 rhys  wheel  350 Nov  3 06:24 ./localhost:8080

Then triggering what I think is the behavior you're describing:

$ go1.21 tool pprof -top localhost:8080                        
Fetching profile over HTTP from http://localhost:8080/debug/pprof/profile
localhost:8080: Get "http://localhost:8080/debug/pprof/profile": dial tcp [::1]:8080: connect: connection refused
failed to fetch any source profiles

Then working around that by adding a ./ prefix to the file name:

$ go1.21 tool pprof -top ./localhost:8080                      
Type: alloc_space
Time: Nov 3, 2023 at 6:24am (PDT)
Showing nodes accounting for 512.20kB, 100% of 512.20kB total
      flat  flat%   sum%        cum   cum%
  512.20kB   100%   100%   512.20kB   100%  runtime.malg
         0     0%   100%   512.20kB   100%  runtime.newproc.func1
         0     0%   100%   512.20kB   100%  runtime.newproc1
         0     0%   100%   512.20kB   100%  runtime.systemstack

cherrymui · 2023-11-03T16:13:14Z

As @rhysh pointed out, it may be hard to for the tool to distinguish a file named something like localhost:8080 and a URL in general. If adding a leading ./ works, I think that is a reasonable workaround to disambiguate. Does this work for you? Thanks.

cuishuang · 2023-11-06T02:18:54Z

As @rhysh pointed out, it may be hard to for the tool to distinguish a file named something like localhost:8080 and a URL in general. If adding a leading ./ works, I think that is a reasonable workaround to disambiguate. Does this work for you? Thanks.

I think from usage habits, when the target source is a file in the current directory, many developers are likely to omit ./

The current processing method I modified is that if the file contains a colon and will be processed by url.Parse, check whether a file with the same name exists before requesting. If there is no localhost:8080 file locally, it will be treated as a URL normally. But if a file named localhost:8080 does exist, I think the user is more likely to want to handle it as a file.

I don't see which specific go tool pprof [...] command you ran that didn't work.

Have you tried prefixing the file with ./ to specify that it's a file in the local directory?

Here's how it works (and fails) on my machine. First creating a small demo profile:

$ go1.21 version                                             
go version go1.21.3 darwin/arm64

$ go1.21 test context -o /dev/null -memprofile './localhost:8080'
ok  	context	0.366s

$ ls -al ./localhost:8080
-rw-r--r--  1 rhys  wheel  350 Nov  3 06:24 ./localhost:8080

Then triggering what I think is the behavior you're describing:

$ go1.21 tool pprof -top localhost:8080                        
Fetching profile over HTTP from http://localhost:8080/debug/pprof/profile
localhost:8080: Get "http://localhost:8080/debug/pprof/profile": dial tcp [::1]:8080: connect: connection refused
failed to fetch any source profiles

Then working around that by adding a ./ prefix to the file name:

$ go1.21 tool pprof -top ./localhost:8080                      
Type: alloc_space
Time: Nov 3, 2023 at 6:24am (PDT)
Showing nodes accounting for 512.20kB, 100% of 512.20kB total
      flat  flat%   sum%        cum   cum%
  512.20kB   100%   100%   512.20kB   100%  runtime.malg
         0     0%   100%   512.20kB   100%  runtime.newproc.func1
         0     0%   100%   512.20kB   100%  runtime.newproc1
         0     0%   100%   512.20kB   100%  runtime.systemstack

Thank you for taking the time to test and reply. But I think from usage habits, when the target source is a file in the current directory, many developers are likely to omit ./

cuishuang · 2023-11-06T02:24:50Z

As @rhysh pointed out, it may be hard to for the tool to distinguish a file named something like localhost:8080 and a URL in general. If adding a leading ./ works, I think that is a reasonable workaround to disambiguate. Does this work for you? Thanks.

Thank you very much for your reply. I think from usage habits, when the target source is a file in the current directory, many developers are likely to omit ./

The current processing method I modified is that if the file contains a colon and will be processed by url.Parse, check whether a file with the same name exists before requesting. If there is no localhost:8080 file locally, it will be treated as a URL normally. But if a file named localhost:8080 does exist, I think the user is more likely to want to handle it as a file. (Of course, in most cases the file name is unlikely to be localhost:8080, but like mem_2023-11-01_01:01:00. If this type is processed as a URL, it is best to avoid it)

Since google/pprof also requires almost the same changes, I am also paying attention to the suggestions of the maintainers of this project. In addition, the two fetch methods in go and google/pprof have almost the same logical judgment, which seems to be able to streamline some code.

cuishuang · 2023-12-05T11:00:40Z

To fix this problem, we also need to update google/pprof. I have already raised an issue google/pprof#816

Merged

cuishuang · 2023-12-05T11:06:31Z

As @rhysh pointed out, it may be hard to for the tool to distinguish a file named something like localhost:8080 and a URL in general. If adding a leading ./ works, I think that is a reasonable workaround to disambiguate. Does this work for you? Thanks.

Relevant changes have been merged into google/pprof.

According to communication and suggestions with the pprof maintainer, if src is a file that exists in the current directory, it will be treated as a file (even if it contains a colon).

For details please refer to google/pprof#817

I think it's time to move on to the next step.

But I feel that this CL will change the relevant logic of pprof to a certain extent (although it is very small). Should I submit a proposal and request the Go team for review?

Thanks.

gopherbot · 2024-02-02T08:58:29Z

Change https://go.dev/cl/547236 mentions this issue: cmd: go get github.com/google/pprof v0.0.0-20240130152714-0ed6a68c8d9e and revendor

cuishuang · 2024-03-18T06:13:52Z

related pictures:

gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Nov 3, 2023

cherrymui added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 3, 2023

cherrymui added this to the Backlog milestone Nov 3, 2023

cherrymui added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Nov 3, 2023

seankhliao removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jan 28, 2024

mknyszek assigned cherrymui Jan 31, 2024

gopherbot closed this as completed in c5698e3 May 1, 2024

dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels May 4, 2024

dmitshur modified the milestones: Backlog, Go1.23 May 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/pprof: If the file under the current path contains a colon, go tool pprof will parse it into a url and will not work properly. #63924

cmd/pprof: If the file under the current path contains a colon, go tool pprof will parse it into a url and will not work properly. #63924

cuishuang commented Nov 3, 2023

gopherbot commented Nov 3, 2023

cuishuang commented Nov 3, 2023

rhysh commented Nov 3, 2023

cherrymui commented Nov 3, 2023

cuishuang commented Nov 6, 2023

cuishuang commented Nov 6, 2023

cuishuang commented Dec 5, 2023

cuishuang commented Dec 5, 2023

gopherbot commented Feb 2, 2024

cuishuang commented Mar 18, 2024

cmd/pprof: If the file under the current path contains a colon, go tool pprof will parse it into a url and will not work properly. #63924

cmd/pprof: If the file under the current path contains a colon, go tool pprof will parse it into a url and will not work properly. #63924

Comments

cuishuang commented Nov 3, 2023

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

gopherbot commented Nov 3, 2023

cuishuang commented Nov 3, 2023

rhysh commented Nov 3, 2023

cherrymui commented Nov 3, 2023

cuishuang commented Nov 6, 2023

cuishuang commented Nov 6, 2023

cuishuang commented Dec 5, 2023

cuishuang commented Dec 5, 2023

gopherbot commented Feb 2, 2024

cuishuang commented Mar 18, 2024

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?