http

This is a historical learning note and may contain outdated or incomplete understanding.

1. What It Is

You can simply use http.Get, http.Post, and http.PostForm; underneath they all call client.Do.
Client is safe for concurrent use.
Constants predefine status codes and methods.
Use NewRequest to construct a Request. This object has various methods for setting headers, parameters, request body, and so on. Use Client to send the request; the returned value is a Response, which exposes various fields.

2. Usage

2.1. http builder

package builder

import (
	"bytes"
	"context"
	"net/http"
	"time"
)

const DefaultTimeout = 5

type HttpBuilder struct {
	method      string
	url         string
	headers     map[string]string
	queryParams map[string]string
	body        string
	timeout     int
}

func NewHttpBuilder() *HttpBuilder {
	return &HttpBuilder{
		headers:     make(map[string]string),
		queryParams: make(map[string]string),
	}
}

func (h *HttpBuilder) Method(method string) *HttpBuilder {
	h.method = method
	return h
}

func (h *HttpBuilder) Url(url string) *HttpBuilder {
	h.url = url
	return h
}

func (h *HttpBuilder) QueryParam(key string, value string) *HttpBuilder {
	h.queryParams[key] = value
	return h
}

func (h *HttpBuilder) Body(body string) *HttpBuilder {
	h.body = body
	return h
}

func (h *HttpBuilder) Timeout(timeout int) *HttpBuilder {
	h.timeout = timeout
	return h
}

// Build the HTTP client and request the target address.
// When error is not nil, remember to release the returned response and CancelFunc resources with defer after use.
func (h *HttpBuilder) DoRequest() (*http.Response, context.CancelFunc, error) {
	req, err := http.NewRequest(h.method, h.url, bytes.NewBuffer([]byte(h.body)))
	if err != nil {
		return nil, nil, err
	}

	// Assemble headers.
	for k, v := range h.headers {
		req.Header.Set(k, v)
	}

	// Assemble query parameters.
	query := req.URL.Query()
	for k, v := range h.queryParams {
		query.Add(k, v)
	}
	req.URL.RawQuery = query.Encode()

	// Set timeout.
	timeout := DefaultTimeout
	if h.timeout > 0 {
		timeout = h.timeout
	}
	ctx, cancel := context.WithTimeout(context.Background(), time.Duration(timeout)*time.Second)
	req = req.WithContext(ctx)

	// Build the client and send the request.
	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		defer cancel()
		return nil, nil, err
	}

	return resp, cancel, nil
}

Test

func main() {
	resp, cancel, err := builder.NewHttpBuilder().
		Method(http.MethodGet).
		Url("https://www.baidu.com/s").
		QueryParam("wd", "test").
		DoRequest()
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()
	defer cancel()

	fmt.Println("response Status:", resp.Status)
	fmt.Println("response Headers:", resp.Header)
	body, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		panic(err)
	}
	ioutil.WriteFile("test.html", body, 0644)
}

3. HTTP Internals

http.Client contains a Transport, whose structure is as follows:

type Transport struct {

The Transport used by a default &http.Client{} is DefaultTransport:

var DefaultTransport RoundTripper = &Transport{
	Proxy: ProxyFromEnvironment,
	DialContext: (&net.Dialer{
		Timeout:   30 * time.Second,
		KeepAlive: 30 * time.Second,
		DualStack: true,
	}).DialContext,
	ForceAttemptHTTP2:     true,
	MaxIdleConns:          100,
	IdleConnTimeout:       90 * time.Second,
	TLSHandshakeTimeout:   10 * time.Second,
	ExpectContinueTimeout: 1 * time.Second,
}

DisableKeepAlives is not set, so it defaults to false, which means HTTP Keep-Alive is enabled; that is, persistent HTTP connections are enabled.

MaxIdleConns is 100, meaning the connection pool can contain at most 100 idle connections. MaxIdleConnsPerHost is 0, so it uses DefaultMaxIdleConnsPerHost=2, meaning the pool keeps two idle connections for each Host (IP:Port). In other words, the HTTP connection pool is enabled.

MaxConnsPerHost is 0, meaning the number of connections is unlimited. If a new HTTP request finds no idle connection, a new connection is created; under high concurrency this can result in many TIME_WAIT connections.

A tRPC HTTP client uses a transport like this:

var StdHTTPTransport = &stdhttp.Transport{
	Proxy: stdhttp.ProxyFromEnvironment,
	DialContext: (&net.Dialer{
		Timeout:   30 * time.Second,
		KeepAlive: 30 * time.Second,
		DualStack: true,
	}).DialContext,
	ForceAttemptHTTP2:     true,
	MaxIdleConns:          200,
	IdleConnTimeout:       90 * time.Second,
	TLSHandshakeTimeout:   10 * time.Second,
	MaxIdleConnsPerHost:   100,
	MaxConnsPerHost:       200,
	ExpectContinueTimeout: time.Second,
}

An HTTP connection pool and HTTP persistent connections are different concepts.

An HTTP persistent connection means an HTTP Request and HTTP SubRequest reuse the same TCP connection.

An HTTP connection pool means putting HTTP connections into a pool for reuse. Such a connection can only be a persistent connection, because a short-lived connection closes its underlying TCP connection after use. Putting that closed connection into a pool and then taking it out again would be useless. Therefore only persistent HTTP connections can be placed in the pool for reuse.

1. What It Is

2. Usage

2.1. http builder

3. HTTP Internals

4. References

Discussion