net/http: which not support FQDN url ( domain contains dot at the end )，it should be fixed #67107

zengyuxing007 · 2024-04-29T10:52:45Z

Go version

go1.21.5

Output of `go env` in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/root/go/pkg/mod'
GOOS='linux'
GOPATH='/root/go'
GOPROXY='https://goproxy.cn'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.5'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build391041608=/tmp/go-build -gno-record-gcc-switches'

What did you do?

package main
import (
    "bufio"
    "fmt"
    "net/http"
)
func main() {
    resp, err := http.Get("http://www.baidu.com./index.html")
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()
    fmt.Println("Response status:", resp.Status)
    scanner := bufio.NewScanner(resp.Body)
    for i := 0; scanner.Scan() && i < 5; i++ {
        fmt.Println(scanner.Text())
    }
    if err := scanner.Err(); err != nil {
        panic(err)
    }

go run httpclient.go ，but it return 403 .

When we look at the requested http message we can see that the host field is populated with "www.baidu.com."

111.165.121.180.60696-220.181.038.150.00080: GET /index.html HTTP/1.1
Host: www.baidu.com.
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip
220.181.038.150.00080-111.165.121.180.60696: HTTP/1.1 403 Forbidden
Server: bfe
Date: Mon, 29 Apr 2024 10:50:30 GMT
Content-Length: 0
Content-Type: text/plain; charset=utf-8

What did you see happen?

Response status: 403 Forbidden

What did you expect to see?

http response 200

The text was updated successfully, but these errors were encountered:

seankhliao · 2024-04-29T11:20:53Z

That's a server error. you get the same response with curl.

zengyuxing007 · 2024-04-30T01:25:25Z

That's a server error. you get the same response with curl.

curl --version
curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.1 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
Release-Date: 2018-01-24
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL

@seankhliao curl returns ok. maybe some versions of curl don't.

curl www.baidu.com./index.html -v
*   Trying 220.181.38.150...
* TCP_NODELAY set
* Connected to www.baidu.com (220.181.38.150) port 80 (#0)
> GET /index.html HTTP/1.1
> Host: www.baidu.com
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Cache-Control: private, no-cache, no-store, proxy-revalidate, no-transform
< Connection: keep-alive
< Content-Length: 2381
< Content-Type: text/html
< Date: Tue, 30 Apr 2024 01:21:07 GMT
< Etag: "588604c8-94d"
< Last-Modified: Mon, 23 Jan 2017 13:27:36 GMT
< Pragma: no-cache
< Server: bfe/1.0.8.18
< Set-Cookie: BDORZ=27315; max-age=86400; domain=.baidu.com; path=/
<
<!DOCTYPE html>
<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=http://s1.bdstatic.com/r/www/cache/bdorz/baidu.min.css><title>百度一下，你就知道</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=百度一下 class="bg s_btn"></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>新闻</a> <a href=http://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>地图</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>视频</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>贴吧</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&amp;tpl=mn&amp;u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>登录</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登录</a>');</script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">更多产品</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>关于百度</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>&copy;2017&nbsp;Baidu&nbsp;<a href=http://www.baidu.com/duty/>使用百度前必读</a>&nbsp; <a href=http://jianyi.baidu.com/ class=cp-feedback>意见反馈</a>&nbsp;京ICP证030173号&nbsp; <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html>
* Connection #0 to host www.baidu.com left intact

The main reason why the go client doesn't work is that the host sent by the client doesn't remove the dot at the end of the domain.

zengyuxing007 · 2024-04-30T01:26:49Z

please reopen it. @seankhliao

seankhliao · 2024-04-30T05:17:05Z

see curl/curl#8290
not sending a trailing dot is considered a bug

zengyuxing007 · 2024-05-07T06:12:50Z

see curl/curl#8290 not sending a trailing dot is considered a bug

@seankhliao

Thanks for u reply, I read the issue(curl/curl#8290) , and from the multiple replies to the discussion, it doesn't say that the DOT should be kept.

And removing the dot under SNI is an RFC requirement.

seankhliao closed this as not planned Won't fix, can't repro, duplicate, stale Apr 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

net/http: which not support FQDN url ( domain contains dot at the end )，it should be fixed #67107

net/http: which not support FQDN url ( domain contains dot at the end )，it should be fixed #67107

zengyuxing007 commented Apr 29, 2024

seankhliao commented Apr 29, 2024

zengyuxing007 commented Apr 30, 2024

zengyuxing007 commented Apr 30, 2024

seankhliao commented Apr 30, 2024

zengyuxing007 commented May 7, 2024 •

edited

net/http: which not support FQDN url ( domain contains dot at the end )，it should be fixed #67107

net/http: which not support FQDN url ( domain contains dot at the end )，it should be fixed #67107

Comments

zengyuxing007 commented Apr 29, 2024

Go version

Output of go env in your module/workspace:

What did you do?

What did you see happen?

What did you expect to see?

seankhliao commented Apr 29, 2024

zengyuxing007 commented Apr 30, 2024

zengyuxing007 commented Apr 30, 2024

seankhliao commented Apr 30, 2024

zengyuxing007 commented May 7, 2024 • edited

Output of `go env` in your module/workspace:

zengyuxing007 commented May 7, 2024 •

edited