Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/go: unclear error when SumDB tree note is corrupted #38348

Open
nicolascamilo opened this issue Apr 10, 2020 · 17 comments
Open

cmd/go: unclear error when SumDB tree note is corrupted #38348

nicolascamilo opened this issue Apr 10, 2020 · 17 comments
Labels
modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unplanned

Comments

@nicolascamilo
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.14.2 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
set GO111MODULE=on
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\Nicolas\AppData\Local\go-build
set GOENV=C:\Users\Nicolas\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS= -mod=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=D:\Mis Documentos\Programacion\Go
set GOPRIVATE=
set GOPROXY=direct
set GOROOT=C:\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=NUL
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\Nicolas\AppData\Local\Temp\go-build045229398=/tmp/go-build -gno-record-gcc-switches

What did you do?

I'm trying to use go mod tidy command. Since my computer turned off because of a blackout it never worked again

What did you expect to see?

Succesful go.mod file

What did you see instead?

For every project and import I'm seeing this error:

github.com/sirupsen/logrus: github.com/sirupsen/logrus@v1.5.0: verifying module: github.com/sirupsen/logrus@v1.5.0: initializing sumdb.Client: reading tree note: malformed note
note:

 github.com/dgrijalva/jwt-go: github.com/dgrijalva/jwt-go@v1.0.2: verifying module: github.com/dgrijalva/jwt-go@v1.0.2: initializing sumdb.Client: reading tree note: malformed note
note:

I know I just copied 2 libraries, but It's happening with every one
@bcmills bcmills added modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Apr 10, 2020
@bcmills bcmills added this to the Go1.15 milestone Apr 10, 2020
@bcmills
Copy link
Contributor

bcmills commented Apr 10, 2020

Probably you have a corrupted file somewhere, but that diagnostic isn't giving you enough information to find and fix it.

@FiloSottile, what method would you recommend to repair this, and can we either do that automatically or add detail to the error message?

CC @katiehockman @rsc @jayconrod @matloob

@bcmills bcmills changed the title gomod initializing sumdb.Client: reading tree note: malformed note error cmd/go: unclear error when SumDB tree note is corrupted Apr 10, 2020
@katiehockman
Copy link
Contributor

@nicolascamilo for now in order to unblock you, can you try clearing your module cache to see if that resolves the issue? You can do that by running go clean -modcache

If that fixes your specific problem, then we should still leave this issue open for the investigation that @bcmills mentioned above.

@FiloSottile
Copy link
Contributor

@nicolascamilo Can you share with us the contents of $(go env GOPATH)/pkg/sumdb/sum.golang.org/latest? Then you can delete that file and it should unblock you.

$ cat "$(go env GOPATH)/pkg/sumdb/sum.golang.org/latest"
go.sum database tree
1072016
5rxqXjwvPB0445OEz5nw+s/gdsfoR0EUt3FEqMzbtCc=

— sum.golang.org Az3grhjle9b69brGvNYM1ZPKtFcpzaeKwQHomx3L7S5WuvtNRoVIM/KgmpsbfQSMEwqNFmoVR/gizfLYnOhtS7sUJgk=

@bcmills If the corrupted file is GOPATH/pkg/sumdb/sum.golang.org/latest, it would be important to understand how it came to be corrupted, because if an attacker can cause that they can downgrade the security guarantees significantly. If it's any other file, we can blow it away.

@bcmills
Copy link
Contributor

bcmills commented Apr 10, 2020

@FiloSottile

it would be important to understand how it came to be corrupted, because if an attacker can cause that they can downgrade the security guarantees significantly.

Per the original issue description:

Since my computer turned off because of a blackout it never worked again

So I think the scenario here is “local power cut with a non-journaled filesystem”, which I assume is beyond our threat model for the sumdb...

@FiloSottile
Copy link
Contributor

Fair enough, totally fine to delete here. I am a bit reticent to have us delete it automatically because we wouldn't hear about other ways it might go corrupt. Let's punt on it until the next time?

@bcmills
Copy link
Contributor

bcmills commented Apr 10, 2020

Ok. Putting on hold until / unless we get more reports.

@bcmills bcmills modified the milestones: Go1.15, Unplanned Apr 10, 2020
@nicolascamilo
Copy link
Author

nicolascamilo commented Apr 12, 2020 •
edited

@nicolascamilo for now in order to unblock you, can you try clearing your module cache to see if that resolves the issue? You can do that by running go clean -modcache

If that fixes your specific problem, then we should still leave this issue open for the investigation that @bcmills mentioned above.

Tried that already, nothing happens.

@nicolascamilo Can you share with us the contents of $(go env GOPATH)/pkg/sumdb/sum.golang.org/latest? Then you can delete that file and it should unblock you.

$ cat "$(go env GOPATH)/pkg/sumdb/sum.golang.org/latest"
go.sum database tree
1072016
5rxqXjwvPB0445OEz5nw+s/gdsfoR0EUt3FEqMzbtCc=

— sum.golang.org Az3grhjle9b69brGvNYM1ZPKtFcpzaeKwQHomx3L7S5WuvtNRoVIM/KgmpsbfQSMEwqNFmoVR/gizfLYnOhtS7sUJgk=

@bcmills If the corrupted file is GOPATH/pkg/sumdb/sum.golang.org/latest, it would be important to understand how it came to be corrupted, because if an attacker can cause that they can downgrade the security guarantees significantly. If it's any other file, we can blow it away.

This is the result:

go.sum database tree
1077497
wBavjUxZoXIpqgvc3T3tEi+XaKQOOgdQ+YP5OR0Anyw=

— sum.golang.org Az3griPfGhP2UeDZ/87y3uE8qKV68nDWey+KoJ3y9se5HyipCsyRzt4vGnxhYrdZfNarDmh62xfaz12cbzztfGMCLQE=

Update: Apparently is working right now. I'd love to provide more information but I have no actually clue what I've done, but I guess it was my problem and not something you guys have to solve. I can't use go mod tidy now with the libs I'm working (I was able before this problem) but I guess that's just another issue.

Again, I'm deeply sorry for I can't provide any new hint about this problem.

@sajallimbu
Copy link

@nicolascamilo Can you share with us the contents of $(go env GOPATH)/pkg/sumdb/sum.golang.org/latest? Then you can delete that file and it should unblock you.

$ cat "$(go env GOPATH)/pkg/sumdb/sum.golang.org/latest"
go.sum database tree
1072016
5rxqXjwvPB0445OEz5nw+s/gdsfoR0EUt3FEqMzbtCc=

— sum.golang.org Az3grhjle9b69brGvNYM1ZPKtFcpzaeKwQHomx3L7S5WuvtNRoVIM/KgmpsbfQSMEwqNFmoVR/gizfLYnOhtS7sUJgk=

@bcmills If the corrupted file is GOPATH/pkg/sumdb/sum.golang.org/latest, it would be important to understand how it came to be corrupted, because if an attacker can cause that they can downgrade the security guarantees significantly. If it's any other file, we can blow it away.

Deleting the file named latest worked for me!

@FiloSottile
Copy link
Contributor

Deleting the file named latest worked for me!

Can you share what error you were seeing?

@mreddimasi
Copy link

Hi,

Not sure, if it helps, but was facing the same issue, the following steps actually resolved the issue for me (Windows 10 - go version go1.14.4 windows/amd64)

The file %GOPATH%\pkg\sumdb\sum.golang.org\latest did not exist when I was got this error

Steps:

  1. touch %GOPATH%\pkg\sumdb\sum.golang.org\latest
  2. Ran the build got the following output

go: finding module for package github.com/gomodule/redigo/redis                                                                                                                                                          
go: finding module for package github.com/go-sql-driver/mysql                                                                                                                                                            
go: finding module for package github.com/jinzhu/gorm                                                                                                                                                                    
go: finding module for package github.com/google/uuid                                                                                                                                                                    
go: finding module for package github.com/gin-gonic/gin                                                                                                                                                                  
go: finding module for package github.com/dgrijalva/jwt-go                                                                                                                                                               
go: finding module for package github.com/nats-io/nats.go                                                                                                                                                                
go: finding module for package github.com/gin-contrib/cors                                                                                                                                                               
go: downloading github.com/jinzhu/gorm v1.9.15                                                                                                                                                                           
go: downloading github.com/go-sql-driver/mysql v1.5.0                                                                                                                                                                    
go: downloading github.com/google/uuid v1.1.1                                                                                                                                                                            
go: downloading github.com/gin-contrib/cors v1.3.1                                                                                                                                                                       
go: downloading github.com/gomodule/redigo/redis v0.0.0-do-not-use                                                                                                                                                       
go: downloading github.com/gin-gonic/gin v1.6.3                                                                                                                                                                          
go: downloading github.com/dgrijalva/jwt-go v1.0.2                                                                                                                                                                       
go: downloading github.com/gomodule/redigo v1.8.2                                                                                                                                                                        
go: downloading github.com/nats-io/nats.go v1.10.0                                                                                                                                                                       
internal\middlewares\auth.go:12:2: github.com/dgrijalva/jwt-go@v1.0.2: verifying module: github.com/dgrijalva/jwt-go@v1.0.2: initializing sumdb.Client: reading tree note: malformed note                                
note:                                                                                                                                                                                                                    
                                                                                                                                                                                                                         
internal\app\app.go:10:2: github.com/gin-contrib/cors@v1.3.1: verifying module: github.com/gin-contrib/cors@v1.3.1: initializing sumdb.Client: reading tree note: malformed note                                         
note:                                                                                                                                                                                                                    
                                                                                                                                                                                                                         
internal\app\app.go:11:2: github.com/gin-gonic/gin@v1.6.3: verifying module: github.com/gin-gonic/gin@v1.6.3: initializing sumdb.Client: reading tree note: malformed note                                               
note:                                                                                                                                                                                                                    
                                                                                                                                                                                                                         
internal\conf\configuration.go:12:2: github.com/go-sql-driver/mysql@v1.5.0: verifying module: github.com/go-sql-driver/mysql@v1.5.0: initializing sumdb.Client: reading tree note: malformed note                        
note:                                                                                                                                                                                                                    
                                                                                                                                                                                                                         
cmd\merchant-backend\main.go:9:2: github.com/gomodule/redigo/redis@v0.0.0-do-not-use: verifying module: github.com/gomodule/redigo/redis@v0.0.0-do-not-use: initializing sumdb.Client: reading tree note: malformed note 
note:                                                                                                                                                                                                                    
                                                                                                                                                                                                                         
internal\helpers\utils.go:19:2: github.com/google/uuid@v1.1.1: verifying module: github.com/google/uuid@v1.1.1: initializing sumdb.Client: reading tree note: malformed note                                             
note:                                                                                                                                                                                                                    
                                                                                                                                                                                                                         
cmd\merchant-backend\main.go:10:2: github.com/jinzhu/gorm@v1.9.15: verifying module: github.com/jinzhu/gorm@v1.9.15: initializing sumdb.Client: reading tree note: malformed note                                        
note:                                                                                                                                                                                                                    
                                                                                                                                                                                                                         
internal\conf\configuration.go:10:2: github.com/nats-io/nats.go@v1.10.0: verifying module: github.com/nats-io/nats.go@v1.10.0: initializing sumdb.Client: reading tree note: malformed note                              
note:
  1. go clean -modcache
  2. Ran go build again, with the same error message as in Step 2.
  3. rm %GOPATH%\pkg\sumdb\sum.golang.org\latest
  4. Ran go build again this time successfully with the following output
go: finding module for package github.com/gomodule/redigo/redis                         
go: finding module for package github.com/jinzhu/gorm                                   
go: finding module for package github.com/go-sql-driver/mysql                           
go: finding module for package github.com/gin-gonic/gin                                 
go: finding module for package github.com/nats-io/nats.go                               
go: finding module for package github.com/dgrijalva/jwt-go                              
go: finding module for package github.com/gin-contrib/cors                              
go: finding module for package github.com/google/uuid                                   
go: downloading github.com/nats-io/nats.go v1.10.0                                      
go: downloading github.com/go-sql-driver/mysql v1.5.0                                   
go: downloading github.com/dgrijalva/jwt-go v1.0.2                                      
go: downloading github.com/gin-contrib/cors v1.3.1                                      
go: downloading github.com/gomodule/redigo v1.8.2                                       
go: downloading github.com/google/uuid v1.1.1                                           
go: downloading github.com/gin-gonic/gin v1.6.3                                         
go: downloading github.com/gomodule/redigo/redis v0.0.0-do-not-use                      
go: downloading github.com/jinzhu/gorm v1.9.15                                          
go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible                         
go: found github.com/gomodule/redigo/redis in github.com/gomodule/redigo v1.8.2         
go: found github.com/jinzhu/gorm in github.com/jinzhu/gorm v1.9.15                      
go: found github.com/gin-contrib/cors in github.com/gin-contrib/cors v1.3.1             
go: found github.com/gin-gonic/gin in github.com/gin-gonic/gin v1.6.3                   
go: found github.com/go-sql-driver/mysql in github.com/go-sql-driver/mysql v1.5.0       
go: found github.com/nats-io/nats.go in github.com/nats-io/nats.go v1.10.0              
go: found github.com/dgrijalva/jwt-go in github.com/dgrijalva/jwt-go v3.2.0+incompatible
go: found github.com/google/uuid in github.com/google/uuid v1.1.1                       
go: downloading github.com/mattn/go-isatty v0.0.12                                      
go: downloading github.com/gin-contrib/sse v0.1.0                                       
go: downloading github.com/jinzhu/inflection v1.0.0                                     
go: downloading github.com/ugorji/go v1.1.7                                             
go: downloading github.com/nats-io/nkeys v0.1.4                                         
go: downloading github.com/go-playground/validator/v10 v10.2.0                          
go: downloading gopkg.in/yaml.v2 v2.2.8                                                 
go: downloading github.com/golang/protobuf v1.3.3                                       
go: downloading github.com/ugorji/go/codec v1.1.7                                       
go: downloading github.com/nats-io/jwt v0.3.2                                           
go: downloading github.com/nats-io/nuid v1.0.1                                          
go: downloading golang.org/x/crypto v0.0.0-20200323165209-0ec3e9974c59                  
go: downloading github.com/go-playground/universal-translator v0.17.0                   
go: downloading github.com/leodido/go-urn v1.2.0                                        
go: downloading github.com/go-playground/locales v0.13.0         
```                       ` 

I can see the latest file had been created after this step.

@atlas-comstock
Copy link

Thanks @mreddimasi . I met this issue, and this does solve the issue.

@vli-platform
Copy link

Thanks @mreddimasi . I also got the same error and your solution helped me.

@exodustx0
Copy link

For what it's worth, I'll add that I similarly had a power outage (as in, my PC at some point deadlocked on all processes, Go or otherwise, including in the middle of a go get, so I was forced to cut the power), and I too got the "malformed note" error, which went away after deleting the %GOPATH%\pkg\sumdb\sum.golang.org\latest file (which was just 187 null bytes at that point). However, unlike the comments above me, another error took its place:

go: downloading github.com/hajimehoshi/oto/v2 v2.0.0
SECURITY ERROR
go.sum database server misbehavior detected!

old database:
        go.sum database tree
        7494163
        oOF65d3/MJZ5c1hjrktwPzGAPxhg7myKELBEqBK4gm8=

        — sum.golang.org Az3grmCvlocGJUpU2KWCCTzTMGZ/B+qyaGxoesMGbruKmiHiVax9N/UqyJ9gkALuSDZyKI2J9hu2DMUnd37LcSbmkwg=

new database:
        go.sum database tree
        7797325
        3B75PrlHOu76bf9kJUniWb4/d2xLX7oJCkCm0URIB50=

        — sum.golang.org Az3grvS7Ch4euazi+BB2bRhhpVmffiefMwk797VmmVbpvFGI9SRLKruR635hVEE2YS4QvqIzmuOjmYP1tkhFEdzI8QE=

proof of misbehavior:
        evhMbFcgHraUjA7LoYBgALFsih/yafHjiJIfuHnoTV4=    internal error: generated inconsistent proof
SECURITY ERROR
go.sum database server misbehavior detected!

old database:
        go.sum database tree
        7493958
        FlqjTFRXLZ4NQKfa50iQygh2OsnzxOuRSDf9fzwapAw=

        — sum.golang.org Az3griuSFkEngPLvClbWM4YETXJ9SXj0eaEqhGHFFG2F4+OblSQV0s7r4k4ssQPhSTvU4f5+ofnip76W7CavZ4c8oAY=

new database:
        go.sum database tree
        7797325
        3B75PrlHOu76bf9kJUniWb4/d2xLX7oJCkCm0URIB50=

        — sum.golang.org Az3grvS7Ch4euazi+BB2bRhhpVmffiefMwk797VmmVbpvFGI9SRLKruR635hVEE2YS4QvqIzmuOjmYP1tkhFEdzI8QE=

proof of misbehavior:
        etMZn0ytt7DnXmFpE++/UuKSsP0vveP0pbSOZAn2g5U=    internal error: generated inconsistent proof

I'm also running on Windows, in case it's relevant.

Apologies if this new error deserves a separate issue; this seemed relevant enough to me to warrant adding to this issue, but I'm also rather new to Go.

@myk4040okothogodo

This comment was marked as duplicate.

Sign in to view
@bhutfless
Copy link

Tools environment: GOPATH=D:\GitHub\PCS\Go
Installing 1 tool at D:\GitHub\PCS\Go\bin in module mode.
goimports

Installing golang.org/x/tools/cmd/goimports@latest FAILED
{
"killed": false,
"code": 1,
"signal": null,
"cmd": "C:\Program Files\Go\bin\go.exe install -v golang.org/x/tools/cmd/goimports@latest",
"stdout": "",
"stderr": "go: downloading golang.org/x/tools v0.1.12\ngo: golang.org/x/tools/cmd/goimports@latest: golang.org/x/tools@v0.1.12: verifying module: golang.org/x/tools@v0.1.12: initializing sumdb.Client: reading tree note: malformed note\nnote:\ngo.sum database tree\r\n11339752\r\nzLpBTbjreUQ7+qcGuHJd6mS5PmzKd5glETqA44RyzCs=\r\n\r\n— sum.golang.org Az3grlA0jMpae7qUTMsKqrrNGa13uOqrsqFDLTaJvwDkqP6o5xs0BVMg2MXehnc4SKEt6Ksl8vLy5V5abSpF0GF53AI=\r\n"
}

1 tools failed to install.

goimports: failed to install goimports(golang.org/x/tools/cmd/goimports@latest): Error: Command failed: C:\Program Files\Go\bin\go.exe install -v golang.org/x/tools/cmd/goimports@latest
go: downloading golang.org/x/tools v0.1.12
go: golang.org/x/tools/cmd/goimports@latest: golang.org/x/tools@v0.1.12: verifying module: golang.org/x/tools@v0.1.12: initializing sumdb.Client: reading tree note: malformed note
note:
go.sum database tree
11339752
zLpBTbjreUQ7+qcGuHJd6mS5PmzKd5glETqA44RyzCs=

— sum.golang.org Az3grlA0jMpae7qUTMsKqrrNGa13uOqrsqFDLTaJvwDkqP6o5xs0BVMg2MXehnc4SKEt6Ksl8vLy5V5abSpF0GF53AI=

@bhutfless
Copy link

This solved the issue for me ...

C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional>type D:\GitHub\PCS\Go\pkg\sumdb\sum.golang.org\latest
go.sum database tree
11339752
zLpBTbjreUQ7+qcGuHJd6mS5PmzKd5glETqA44RyzCs=

ΓÇö sum.golang.org Az3grlA0jMpae7qUTMsKqrrNGa13uOqrsqFDLTaJvwDkqP6o5xs0BVMg2MXehnc4SKEt6Ksl8vLy5V5abSpF0GF53AI=

C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional>del D:\GitHub\PCS\Go\pkg\sumdb\sum.golang.org\latest

C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional>go install golang.org/x/tools/cmd/goimports@latest
go: downloading golang.org/x/tools v0.1.12
go: downloading golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f
go: downloading golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4

@NeverlandMJ
Copy link

when i go the same error, this worked for me:
go env -w GOSUMDB="sum.golang.org"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests