Allow go get from https://*.onion with self-signed certificates #41076

AnimusPEXUS · 2020-08-27T18:42:52Z

For some reasons I can not use clearnet for some of My golang packages.
So I'm trying to host my packages using gitea and tor hidden services.

calling

export all_proxy="socks5://127.0.0.1:9050"
export http_proxy=$all_proxy
export https_proxy=$all_proxy
go get -u -v 'onionsite.onion/project/package'

results in x509: certificate signed by unknown authority
server hosts self-signed certificate

probably it is ok to allow serf-signed .onion certificates for golang packages hosts

also, trying to use --insecure flag, result in Could not resolve host error

The text was updated successfully, but these errors were encountered:

FiloSottile · 2020-08-27T20:11:17Z

Using GOINSECURE (which unlike --insecure lets you specify that you only want to skip checks for those domains) and GOPRIVATE (to bypass proxy.golang.org and sumdb) should work. If it doesn't and you think it's a Go issue, please provide more details of how you used those settings and the result, including environment and output.

AnimusPEXUS · 2020-08-27T21:58:51Z

Looks like something strange happening:
looks like GOINSECURE and GOPRIVATE doesn't change go command behavior.

now I've modified above script so:

export all_proxy="socks5://127.0.0.1:9050"
export http_proxy=$all_proxy
export https_proxy=$all_proxy

ONION_DOMAIN='someonion.onion'

export GOINSECURE=$ONION_DOMAIN
#export GOPRIVATE=$ONION_DOMAIN
#export GONOPROXY=$ONION_DOMAIN
#export GONOSUMDB=$ONION_DOMAIN
"$@"

I've saved it to ./soxy file and I'm trying to get repository using following command: ./soxy go get -u -v 'onionsite.onion/project/package'
only adding --insecure flag make some change and I getting Could not resolve host with it and x509: certificate signed by unknown authority without it.

as You asked about environment:

my go version is go version go1.14.6 linux/amd64
my environment is 64-bit Fedora 32
also I use this docker-compose to setup hidden gitea instance https://github.com/AnimusPEXUS/hiddengitea
following is my environment variables after above script:

GOINSECURE=someonion.onion
GONOPROXY=someonion.onion
GONOSUMDB=someonion.onion
GOPATH=/home/animuspexus/gopath_main:/home/animuspexus/gopath_work
GOPRIVATE=someonion.onion
GOROOT=/usr/lib/golang
all_proxy=socks5://127.0.0.1:9050
http_proxy=socks5://127.0.0.1:9050
https_proxy=socks5://127.0.0.1:9050

some standard Fedora 32 variables

FiloSottile · 2020-08-27T22:00:34Z

What's the output of go env and what is the full go get output?

AnimusPEXUS · 2020-08-27T22:07:47Z

BASH=/bin/bash
BASHOPTS=checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_LINENO=([0]="0")
BASH_SOURCE=([0]="/home/animuspexus/if/soxy")
BASH_VERSINFO=([0]="5" [1]="0" [2]="17" [3]="1" [4]="release" [5]="x86_64-redhat-linux-gnu")
BASH_VERSION='5.0.17(1)-release'
COLORTERM=truecolor
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
DESKTOP_SESSION=gnome
DIRSTACK=()
DISPLAY=:1
EUID=1000
GDMSESSION=gnome
GDM_LANG=en_US.UTF-8
GJS_DEBUG_OUTPUT=stderr
GJS_DEBUG_TOPICS='JS ERROR;JS LOG'
GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/459e68c4_36b2_4f2b_a140_bb42256e8444
GNOME_TERMINAL_SERVICE=:1.78
GOINSECURE=someonion.onion
GONOPROXY=someonion.onion
GONOSUMDB=someonion.onion
GOPATH=/home/animuspexus/gopath_main:/home/animuspexus/gopath_work
GOPRIVATE=someonion.onion
GOROOT=/usr/lib/golang
GROUPS=()
HISTCONTROL=ignoreboth
HISTSIZE=1000
HOME=/home/animuspexus
HOSTNAME=localhost.localdomain
HOSTTYPE=x86_64
IFS=$' \t\n'
INVOCATION_ID=126bf759904c448698c391253a06f5fb
JOURNAL_STREAM=8:48861
LANG=en_US.UTF-8
LESSOPEN='||/usr/bin/lesspipe.sh %s'
LOADEDMODULES=
LOGNAME=animuspexus
LS_COLORS='rs=0:di=38;5;33:ln=38;5;51:mh=00:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=01;37;41:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;40:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.zst=38;5;9:*.tzst=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.wim=38;5;9:*.swm=38;5;9:*.dwm=38;5;9:*.esd=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.mjpg=38;5;13:*.mjpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.webp=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:'
MACHTYPE=x86_64-redhat-linux-gnu
MAIL=/var/spool/mail/animuspexus
MANAGERPID=1660
MANPATH=:
MC_SID=38677
MC_TMPDIR=/var/tmp/mc-animuspexus
MODULEPATH=/etc/scl/modulefiles:/etc/scl/modulefiles:/etc/scl/modulefiles:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/modulefiles
MODULEPATH_modshare=/usr/share/modulefiles:1:/usr/share/Modules/modulefiles:1:/etc/modulefiles:1
MODULESHOME=/usr/share/Modules
MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl
MODULES_RUN_QUARANTINE=LD_LIBRARY_PATH
OLDPWD=/home/animuspexus/tmp
ONION_DOMAIN=someonion.onion
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH=/home/animuspexus/.local/bin:/home/animuspexus/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/animuspexus/gopath_main/bin:/home/animuspexus/gopath_work/bin:/home/animuspexus/gopath_main/bin:/home/animuspexus/gopath_work/bin
PIPESTATUS=([0]="0")
PPID=38677
PS4='+ '
PWD=/home/animuspexus/tmp/go3
QT_IM_MODULE=ibus
SDL_VIDEO_MINIMIZE_ON_FOCUS_LOSS=0
SESSION_MANAGER=local/unix:@/tmp/.ICE-unix/1792,unix/unix:/tmp/.ICE-unix/1792
SHELL=/bin/bash
SHELLOPTS=braceexpand:hashall:interactive-comments
SHLVL=3
SSH_AGENT_PID=1747
SSH_AUTH_SOCK=/run/user/1000/keyring/ssh
STEAM_FRAME_FORCE_CLOSE=1
TERM=xterm-256color
UID=1000
USER=animuspexus
USERNAME=animuspexus
VTE_VERSION=6003
WINDOWPATH=2
WISECONFIGDIR=/usr/share/wise2/
XAUTHORITY=/run/user/1000/gdm/Xauthority
XDG_CURRENT_DESKTOP=GNOME
XDG_DATA_DIRS=/home/animuspexus/.local/share/flatpak/exports/share/:/var/lib/flatpak/exports/share/:/usr/local/share/:/usr/share/
XDG_MENU_PREFIX=gnome-
XDG_RUNTIME_DIR=/run/user/1000
XDG_SESSION_CLASS=user
XDG_SESSION_DESKTOP=gnome
XDG_SESSION_TYPE=x11
XMODIFIERS=@im=ibus
_=GONOSUMDB=someonion.onion
all_proxy=socks5://127.0.0.1:9050
http_proxy=socks5://127.0.0.1:9050
https_proxy=socks5://127.0.0.1:9050
_module_raw () 
{ 
    unset _mlshdbg;
    if [ "${MODULES_SILENT_SHELL_DEBUG:-0}" = '1' ]; then
        case "$-" in 
            *v*x*)
                set +vx;
                _mlshdbg='vx'
            ;;
            *v*)
                set +v;
                _mlshdbg='v'
            ;;
            *x*)
                set +x;
                _mlshdbg='x'
            ;;
            *)
                _mlshdbg=''
            ;;
        esac;
    fi;
    unset _mlre _mlIFS;
    if [ -n "${IFS+x}" ]; then
        _mlIFS=$IFS;
    fi;
    IFS=' ';
    for _mlv in ${MODULES_RUN_QUARANTINE:-};
    do
        if [ "${_mlv}" = "${_mlv##*[!A-Za-z0-9_]}" -a "${_mlv}" = "${_mlv#[0-9]}" ]; then
            if [ -n "`eval 'echo ${'$_mlv'+x}'`" ]; then
                _mlre="${_mlre:-}${_mlv}_modquar='`eval 'echo ${'$_mlv'}'`' ";
            fi;
            _mlrv="MODULES_RUNENV_${_mlv}";
            _mlre="${_mlre:-}${_mlv}='`eval 'echo ${'$_mlrv':-}'`' ";
        fi;
    done;
    if [ -n "${_mlre:-}" ]; then
        eval `eval ${_mlre}/usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash '"$@"'`;
    else
        eval `/usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash "$@"`;
    fi;
    _mlstatus=$?;
    if [ -n "${_mlIFS+x}" ]; then
        IFS=$_mlIFS;
    else
        unset IFS;
    fi;
    unset _mlre _mlv _mlrv _mlIFS;
    if [ -n "${_mlshdbg:-}" ]; then
        set -$_mlshdbg;
    fi;
    unset _mlshdbg;
    return $_mlstatus
}
module () 
{ 
    _module_raw "$@" 2>&1
}
scl () 
{ 
    if [ "$1" = "load" -o "$1" = "unload" ]; then
        eval "module $@";
    else
        /usr/bin/scl "$@";
    fi
}
switchml () 
{ 
    typeset swfound=1;
    if [ "${MODULES_USE_COMPAT_VERSION:-0}" = '1' ]; then
        typeset swname='main';
        if [ -e /usr/share/Modules/libexec/modulecmd.tcl ]; then
            typeset swfound=0;
            unset MODULES_USE_COMPAT_VERSION;
        fi;
    else
        typeset swname='compatibility';
        if [ -e /usr/share/Modules/libexec/modulecmd-compat ]; then
            typeset swfound=0;
            MODULES_USE_COMPAT_VERSION=1;
            export MODULES_USE_COMPAT_VERSION;
        fi;
    fi;
    if [ $swfound -eq 0 ]; then
        echo "Switching to Modules $swname version";
        source /usr/share/Modules/init/bash;
    else
        echo "Cannot switch to Modules $swname version, command not found";
        return 1;
    fi
}

without --insecure

~/if/soxy go get -u -v 'someonion.onion/name/repo'
unrecognized import path "someonion.onion/name/repo": https fetch: Get "https://someonion.onion/name/repo?go-get=1": x509: certificate signed by unknown authority

with --insecure

~/if/soxy go get -u -v --insecure 'someonion.onion/name/repo'
get "someonion.onion/name/repo": found meta tag get.metaImport{Prefix:"someonion.onion/name/repo", VCS:"git", RepoRoot:"https://someonion.onion/name/repo.git"} at //someonion.onion/name/repo?go-get=1
someonion.onion/name/repo (download)
# cd .; git clone -- https://someonion.onion/name/repo.git /home/animuspexus/gopath_main/src/someonion.onion/name/repo
Cloning into '/home/animuspexus/gopath_main/src/someonion.onion/name/repo'...
fatal: unable to access 'https://someonion.onion/name/repo.git/': Could not resolve host: someonion.onion
package someonion.onion/name/repo: exit status 128

FiloSottile · 2020-08-27T22:12:28Z

This is not the go tool erroring out, the go tool can reach the HTML page, which points it to https://someonion.onion/name/repo.git. That's passed to git clone, which fails because presumably it's not setup to work with the SOCKS5 proxy.

Not sure why GOINSECURE isn't working though. Can you post the output of go env (not env)?

AnimusPEXUS · 2020-08-27T22:16:01Z

presumably it's not setup to work with the SOCKS5 proxy

thanks, I'll look into this.

~/if/soxy go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/animuspexus/.cache/go-build"
GOENV="/home/animuspexus/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE="someonion.onion"
GONOPROXY="someonion.onion"
GONOSUMDB="someonion.onion"
GOOS="linux"
GOPATH="/home/animuspexus/gopath_main:/home/animuspexus/gopath_work"
GOPRIVATE="someonion.onion"
GOPROXY="direct"
GOROOT="/usr/lib/golang"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
GCCGO="/usr/bin/gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build302437077=/tmp/go-build -gno-record-gcc-switches"

FiloSottile · 2020-08-27T22:33:34Z

Ah, GOINSECURE only works in module mode, while you might still be using GOPATH here. I think the go tool is working as intended, and you need to resolve your issue with git.

AnimusPEXUS · 2020-08-28T20:29:11Z

If somebody interested, I've managed this working

git simply requires this in config:

[http "https://*.onion"]
   sslVerify = false
   proxy = "socks5h://127.0.0.1:9050"

also I used following script to run go programs over tor (note: GOINSECURE and GOPRIVATE not needed if go modules not used):

#!/bin/bash

export all_proxy="socks5://127.0.0.1:9050"
export http_proxy=$all_proxy
export https_proxy=$all_proxy

"$@"

also I used ~/.netrc file to provide http username:password

AnimusPEXUS changed the title ~~Allow go get from https:// with self-signed certificates~~ Allow go get from https://*.onion with self-signed certificates Aug 27, 2020

FiloSottile added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Aug 27, 2020

FiloSottile closed this as completed Aug 27, 2020

golang locked and limited conversation to collaborators Aug 29, 2021

gopherbot added the FrozenDueToAge label Aug 29, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow go get from https://*.onion with self-signed certificates #41076

Allow go get from https://*.onion with self-signed certificates #41076

AnimusPEXUS commented Aug 27, 2020 •

edited

FiloSottile commented Aug 27, 2020

AnimusPEXUS commented Aug 27, 2020 •

edited

FiloSottile commented Aug 27, 2020

AnimusPEXUS commented Aug 27, 2020

FiloSottile commented Aug 27, 2020

AnimusPEXUS commented Aug 27, 2020 •

edited

FiloSottile commented Aug 27, 2020

AnimusPEXUS commented Aug 28, 2020 •

edited

Allow go get from https://*.onion with self-signed certificates #41076

Allow go get from https://*.onion with self-signed certificates #41076

Comments

AnimusPEXUS commented Aug 27, 2020 • edited

FiloSottile commented Aug 27, 2020

AnimusPEXUS commented Aug 27, 2020 • edited

FiloSottile commented Aug 27, 2020

AnimusPEXUS commented Aug 27, 2020

FiloSottile commented Aug 27, 2020

AnimusPEXUS commented Aug 27, 2020 • edited

FiloSottile commented Aug 27, 2020

AnimusPEXUS commented Aug 28, 2020 • edited

AnimusPEXUS commented Aug 27, 2020 •

edited

AnimusPEXUS commented Aug 27, 2020 •

edited

AnimusPEXUS commented Aug 27, 2020 •

edited

AnimusPEXUS commented Aug 28, 2020 •

edited