Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: trim go.sum without adding module requirements #29800

Open
F21 opened this issue Jan 17, 2019 · 7 comments
Open

cmd/go: trim go.sum without adding module requirements #29800

F21 opened this issue Jan 17, 2019 · 7 comments
Labels
FeatureRequest modules NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Milestone

Comments

@F21
Copy link

F21 commented Jan 17, 2019

What version of Go are you using (go version)?

go version go1.11.4 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64" 
GOBIN=""
GOCACHE="/home/user/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/user/go" 
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo" 
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/d/14-Workspace/go/github.com/F21/somerepo/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2" 
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build348706889=/tmp/go-build -gno-record-gcc-switches"

What did you do?

  1. Create go.mod: go mod init github.com/F21/somerepo.
  2. Populate go.mod and go.sum by running go test ./....
  3. One of the dependencies github.com/uber/jaeger-client-go conflicts with another dependency, so I bumped it to master: go get github.com/uber/jaeger-client-go@master.
  4. There are now 2 entries for github.com/uber/jaeger-client-go in my go.sum:
github.com/uber/jaeger-client-go v2.15.0+incompatible h1:NP3qsSqNxh8VYr956ur1N/1C1PjvOJnJykCzcD5QHbk=
github.com/uber/jaeger-client-go v2.15.0+incompatible/go.mod h1:WVhlPFC8FDjOFMMWRy2pZqQJSXxYSwNYOkTr/Z6d3Kk=
github.com/uber/jaeger-client-go v2.15.1-0.20190116124224-6733ee486c78+incompatible h1:b2A4zBpOpzqZoSlqlJ3OOe0yjd577TnQIHG1/KN7U8E=
github.com/uber/jaeger-client-go v2.15.1-0.20190116124224-6733ee486c78+incompatible/go.mod h1:WVhlPFC8FDjOFMMWRy2pZqQJSXxYSwNYOkTr/Z6d3Kk=

The github.com/uber/jaeger-client-go v2.15.0+incompatible entries are unnecessary as the project is now pinned to github.com/uber/jaeger-client-go v2.15.1-0.20190116124224-6733ee486c78+incompatible (current master).

In #26381, the ability to trim go.sum was added to go mod tidy. However, go mod tidy is something I avoid as it pulls in all test dependencies for all my dependencies and their dependencies for all platforms. Unfortunately, go mod tidy has burnt me quite a few times and causes a lot more problems than it solves.
See:

What did you expect to see?

I'd like to be able to trim unneeded entries from go.sum without having to run go mod tidy as its current behavior causes a lot more problems than it solves.

What did you see instead?

N/A.

@bcmills bcmills changed the title Ability to trim go.sum without running go mod tidy cmd/go: ability to trim go.sum without running go mod tidy Jan 17, 2019
@bcmills bcmills changed the title cmd/go: ability to trim go.sum without running go mod tidy cmd/go: trim go.sum without adding module requirements Jan 22, 2019
@bcmills bcmills added this to the Unplanned milestone Jan 22, 2019
@bcmills bcmills added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Jan 22, 2019
@ghasemloo
Copy link

It would be nice to have a way to cleanup go.sum from stuff that are not relevant for the building the module and only keep the minimal required.

The current behavior adds hashes for all the stuff in one's go root which one might not want to expose to others (privacy) when pushing the module to a repo. Right now I need to manually cleanup the go.sum file before pushing it to the repo.

relevant bug: #26381

@qbiqing
Copy link

qbiqing commented Dec 24, 2020

Any update on this? Am I right in understanding that there is still no way to trim hashes from go.sum which belong to packages that have been removed from go.mod?

@jayconrod
Copy link
Contributor

@qbiqing go mod tidy does that, in addition to adding missing requirements in go.mod and removing unnecessary requirements.

This issue is about having separate functionality to trim sums from go.sum that go mod tidy would add, including test dependencies of dependencies and modules that only provide packages on platforms that aren't supported by the module author.

Since this is a very narrow case, and since having additional hashes in go.sum doesn't have any build or run-time cost, we haven't prioritized it.

@jayconrod
Copy link
Contributor

@ghasemloo

The current behavior adds hashes for all the stuff in one's go root which one might not want to expose to others (privacy) when pushing the module to a repo. Right now I need to manually cleanup the go.sum file before pushing it to the repo.

go mod tidy doesn't add sums for anything outside the current module and its dependencies. After running go mod tidy, go.sum contains sums for modules providing packages needed to build packages in the main modules, as well as tests of those packages, on any platform. It doesn't depend on anything in GOROOT or GOPATH outside the main module, so there should be no privacy issue. go mod tidy has more info.

@ghasemloo
Copy link

I wrote some code to do this.
I will upload to a github project and share the link.

@ghasemloo
Copy link

here is the hacky way I did it:

gosumtrim/gosumtrim.go

// Binary gosumtrim trims the go.sum file based on the given go.mod file.
package main

import (
	"bufio"
	"flag"
	"fmt"
	"io"
	"os"
	"strings"

	"github.com/golang/glog"
)

var (
	modFile = flag.String("mod", "", "input go.mod file")
	sumFile = flag.String("sum", "", "input go.sum file")
	outFile = flag.String("out", "", "output go.sum file")
)

func main() {
	flag.Parse()

	modIn, err := os.Open(*modFile)
	if err != nil {
		glog.Exitf("Open(%q) failed: %v", *modFile, err)
	}
	sumIn, err := os.Open(*sumFile)
	if err != nil {
		glog.Exitf("Open(%q) failed: %v", *sumFile, err)
	}
	sumOut, err := os.Create(*outFile)
	if err != nil {
		glog.Exitf("Create(%q) failed: %v", *outFile, err)
	}

	if err := trim(modIn, sumIn, sumOut); err != nil {
		glog.Exitf("Trim() failed: %v", err)
	}
	if err := sumOut.Close(); err != nil {
		glog.Exitf("os.Close() fialed: %v", err)
	}
}

func trim(mod io.Reader, sum io.Reader, o io.Writer) error {
	m := bufio.NewReader(mod)
	s := bufio.NewReader(sum)

	// Read till we get to the require line.
	for {
		line, err := m.ReadString('\n')
		if err != nil {
			return fmt.Errorf("reading mod file failed: %v", err)
		}
		glog.Infof("line: %v", line)
		if line == "require (\n" {
			break
		}
	}

	// Process require lines and add dependencies into used map.
	used := make(map[string]bool)
	for {
		line, err := m.ReadString('\n')
		line = strings.Trim(line, "\n \t")
		if err != nil {
			return fmt.Errorf("reading input go.mod failed: %v", err)
		}
		glog.Infof("line: %v", line)
		if line == ")" {
			break
		}
		used[line] = true
	}

	glog.Infof("used: %v", used)

	// Filter sum->out, dropping what is not used in mod.
	for {
		line, err := s.ReadString('\n')
		if err == io.EOF {
			break
		}
		glog.Infof("line: %v", line)
		if err != nil {
			return fmt.Errorf("reading input go.sum failed: %v", err)
		}
		parts := strings.Split(strings.TrimSpace(line), " ")
		if len(parts) != 3 {
			return fmt.Errorf("expecting 3 parts per go.sum line, got %v", parts)
		}
		if !used[strings.Join(parts[:2], " ")] {
			continue
		}
		if _, err := o.Write([]byte(line)); err != nil {
			return fmt.Errorf("writing output go.sum failed: %v", err)
		}
	}
	return nil
}

gosumtrim/gosumtrim_test.go

package main

import (
	"bytes"
	"io"
	"io/ioutil"
	"os"
	"testing"

	"github.com/google/go-cmp/cmp"
)

const (
	root       = ".../gosumtrim/" // module root
	inModFile  = root + "testdata/go.mod.in"
	inSumFile  = root + "testdata/go.sum.in"
	outSumFile = root + "testdata/go.sum.out"
)

func TestTrim(t *testing.T) {
	mod := mustOpen(t, inModFile)
	sum := mustOpen(t, inSumFile)
	out := mustOpen(t, outSumFile)
	want, err := ioutil.ReadAll(out)
	if err != nil {
		t.Fatalf("ioutil.ReadAll(out) failed: %v", err)
	}

	b := bytes.NewBuffer(nil)
	if err := trim(mod, sum, b); err != nil {
		t.Fatalf("trime() failed: %v", err)
	}
	got := b.Bytes()

	t.Logf("got:\n%v", string(got))
	t.Logf("want:\n%v", string(want))

	if diff := cmp.Diff(string(got), string(want)); diff != "" {
		t.Fatalf("trime() returned:\n%s", diff)
	}
}

func mustOpen(t *testing.T, fn string) io.Reader {
	t.Helper()
	r, err := os.Open(fn)
	if err != nil {
		t.Fatalf("os.Open(%q) failed: %v", fn, err)
	}
	return r
}

@ghasemloo
Copy link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
FeatureRequest modules NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Projects
None yet
Development

No branches or pull requests

6 participants