Issue with processing http://127.1 URL

[v1ktor0t]

A Tweeter user mentioned that, when running the CTF challenge locally, requests such as:

curl 'http://127.0.0.1/webhook?url=http://127.1' 

return the following error message:

dial tcp: lookup 127.1: no such host

It looks like the URL http://127.1 was interpreted as a hostname instead of an IP address and Go tried to resolve it.

To easily debug the issue, it would be helpful to have:

• the code of the web server for the example, if it was changed from the one on the blog
• Go and OS version used to run the web server
• debug output of the library. This can be enabled by adding EnableDebugLogging(true) to the config

Additionally, does this behavor happen with a payload like http://127.0.0.1 or any other variant of the loopback IP is used, or is it just http://127.1 causing it?

[padovah4ck]

Hi there, here all the information. If you need something more let me know!
(I'm not much of a golang developer :/ just a beginner)

OS version

$ cat /proc/version  
Linux version 5.14.0-kali4-amd64 ([email protected]) (gcc-10 (Debian 10.3.0-12) 10.3.0, GNU ld (GNU Binutils for Debian) 2.37) #1 SMP Debian 5.14.16-1kali1 (2021-11-05)

go version

$ go version
go version go1.19.2 linux/amd64

go env

$ go env                                                                                                                                                                                                                              
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/kali/.cache/go-build"
GOENV="/home/kali/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/kali/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/kali/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go-1.19"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.19/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19.2"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/kali/safeurl_src/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2203160740=/tmp/go-build -gno-record-gcc-switches"

Http Server Code

package main

import (
	"fmt"
	"io"
	"io/ioutil"
	//"safeurl/safeurl"
	"github.com/doyensec/safeurl"
	"log"
	"net/http"
	"sync"
)

/* http */

func getRoot(w http.ResponseWriter, r *http.Request) {
	fmt.Printf("got / request\n")
	fmt.Printf("serving request: %v\n", r.URL.Path)
	io.WriteString(w, "This is my website!\n")
}

func getWebhook(w http.ResponseWriter, r *http.Request) {
	fmt.Printf("got /webhook request\n")

	hasUrl := r.URL.Query().Has("url")
	url := r.URL.Query().Get("url")
	
	body, err := ioutil.ReadAll(r.Body)
	if err != nil {
		fmt.Printf("could not read body: %s\n", err)
	}	

	fmt.Printf("url=%s \nbody:\n%s\n",
	url, body)

	config := safeurl.GetConfigBuilder().SetBlockedIPs("164.92.85.153").EnableDebugLogging(true).Build()
    client := safeurl.Client(config)
	
	if hasUrl {
		resp, err := client.Get(url)
		if err != nil {
			//fmt.Println(fmt.Errorf("request return error: %v", err))
			w.WriteHeader(http.StatusBadRequest)
			io.WriteString(w, fmt.Errorf("%v", err).Error() )
			return
		}

		if resp != nil {
			// read response body
			defer resp.Body.Close()

			if resp.StatusCode == http.StatusOK {
				bodyBytes, err := io.ReadAll(resp.Body)
				if err != nil {
					log.Fatal(err)
					fmt.Errorf("request return error: %v", err)
				}
				bodyString := string(bodyBytes)
				//fmt.Println(bodyString)
				io.WriteString(w,bodyString)
			}
		}
	}
	//io.WriteString(w, "\n\nwebhook, HTTP!\n")
}

func startHTTPServer() {

	http.HandleFunc("/", getRoot)
	http.HandleFunc("/webhook", getWebhook)

	const port = 8081

	fmt.Printf("test http server started. listening on: %v\n", port)
	http.ListenAndServe(fmt.Sprintf(":%v", port), nil)
}

func main() {
	wg := new(sync.WaitGroup)
	wg.Add(1)

	go func() {
		startHTTPServer()
		wg.Done()
	}()

	wg.Wait()
}

Curl outputs

$ curl 'http://127.0.0.1:8081/webhook?url=http://127.1'    
Get "http://127.1": dial tcp: lookup 127.1: no such host                                     
                                                                                                                                                                                                       
$ curl 'http://127.0.0.1:8081/webhook?url=http://127.0.1'  
Get "http://127.0.1": dial tcp: lookup 127.0.1: no such host                                                                                                                                                                                                                                            

$ curl 'http://127.0.0.1:8081/webhook?url=http://0.0.0'  
Get "http://0.0.0": dial tcp: lookup 0.0.0: no such host                                                                                                                                                                                                                                            

$ curl 'http://127.0.0.1:8081/webhook?url=http://0.0'  
Get "http://0.0": dial tcp: lookup 0.0: no such host  

$ curl 'http://127.0.0.1:8081/webhook?url=http://[::1]'    
Get "http://[::1]": dial tcp [::1]:80: ipv6 blocked. connection to [::1]:80 dropped                                                                                                                                                                                                                                            

$ curl 'http://127.0.0.1:8081/webhook?url=http://127.0.0.1'
Get "http://127.0.0.1": dial tcp 127.0.0.1:80: ip: 127.0.0.1 not found in allowlist 

$ curl 'http://127.0.0.1:8081/webhook?url=http://0.0.0.0'  
Get "http://0.0.0.0": dial tcp 0.0.0.0:80: ip: 0.0.0.0 not found in allowlist  

curl 'http://127.0.0.1:8081/webhook?url=http://localhost'
Get "http://localhost": dial tcp [::1]:80: ipv6 blocked. connection to [::1]:80 dropped 

Server debug output

$ go run testing/server_chall.go
test http server started. listening on: 8081
got /webhook request
url=http://127.1 
body:

[safeurl] calling proxied Get...
[safeurl] calling proxied Do...

got /webhook request
url=http://127.0.1 
body:

[safeurl] calling proxied Get...
[safeurl] calling proxied Do...

got /webhook request
url=http://0.0.0 
body:

[safeurl] calling proxied Get...
[safeurl] calling proxied Do...

got /webhook request
url=http://0.0 
body:

[safeurl] calling proxied Get...
[safeurl] calling proxied Do...

got /webhook request
url=http://[::1] 
body:

[safeurl] calling proxied Get...
[safeurl] calling proxied Do...
[safeurl] connection to address: [::1]:80
[safeurl] ipv6 is disabled

got /webhook request
url=http://127.0.0.1 
body:

[safeurl] calling proxied Get...
[safeurl] calling proxied Do...
[safeurl] connection to address: 127.0.0.1:80
[safeurl] ip: 127.0.0.1 found in blocklist

[safeurl] calling proxied Get...
[safeurl] calling proxied Do...
got /webhook request
url=http://0.0.0.0 
body:

[safeurl] calling proxied Get...
[safeurl] calling proxied Do...
[safeurl] connection to address: 0.0.0.0:80
[safeurl] ip: 0.0.0.0 found in blocklist

got /webhook request
url=http://localhost 
body:

[safeurl] calling proxied Get...
[safeurl] calling proxied Do...
[safeurl] connection to address: [::1]:80
[safeurl] ipv6 is disabled
[safeurl] connection to address: 127.0.0.1:80
[safeurl] ip: 127.0.0.1 found in blocklist

[v1ktor0t]

@padovah4ck thanks. i believe that's all i need. at first glance, everything seems pretty normal. i'll take a close look later today.

[v1ktor0t]

I did some digging around in thenet/http code, and there does seem to be some differences in how DNS resolution is performed on different OSs. Before doing any DNS resolution, Go first tries to decide on the resolution order (https://github.com/golang/go/blob/master/src/net/conf.go#L127). There are OS specific configurations which ultimately caused the behavior you were seeing.

For example, when running on Kali, the same setup as you, I am able to reproduce the issue. In this scenario, Go uses the hostLookupFilesDNS (https://github.com/golang/go/blob/master/src/net/dnsclient_unix.go#L526) lookup. This lookup first checks the system's /etc/hosts file. If no result is found, it validates the hostname and tries to perform DNS resolution. The resolution, however, happens only when the hostname is valid (https://github.com/golang/go/blob/master/src/net/dnsclient_unix.go#L595). Well, it the case of 127.1, the hostname is not valid, which is why you get the 127.1: no such host error.

If we take MacOS as another example, Go has a special case, where it forces a hostLookupCgo DNS lookup order. This will internally call C's getaddrinfo function (https://github.com/golang/go/blob/master/src/net/cgo_unix.go#L135), with 127.1 as the input. Because the getaddrinfo call succeeds, the lookup is considered a success. The next step would be to establish a connection, where safeurl steps in and blocks it. This is the behavior of the live CTF challenge.

All that being said, the behavior you're seeing does not have any security implications. It's fails before safeurl has a chance to perform the final checks, so no DNS or HTTP requests are made.

We'll have a discussion if it makes sense to introduce a default configuration which would force the same DNS behavor on all OSs. Given the developer comments in the initConfVal function (https://github.com/golang/go/blob/master/src/net/conf.go#L44), I do fear that such a change might cause more problems in the long run.

Until then, I'll leave the issue open.

[padovah4ck]

Thanks @v1ktor0t that's crystal clear! I didn't mean to address a security concern with this :)
I was just trying to reproduce the challenge and was expecting and hoping that behaves accordingly.
Anyway, I've found a way to use netdns=cgo https://pkg.go.dev/net#hdr-Name_Resolution

export GODEBUG=netdns=go    # force pure Go resolver
export GODEBUG=netdns=cgo   # force native resolver (cgo, win32)

Now all works as "expected" and go uses native resolver :)

Thanks again.