3.3 Go Build Process

1. Text Files

package main

import "fmt"

func main() {
	fmt.Println("hello world")
}

After it is saved, this file is stored on disk in binary form. Open it with Vim and enter :%!xxd. The leftmost column contains addresses, the middle contains the hexadecimal ASCII codes corresponding to the text characters, and the rightmost contains the text characters. The ASCII table can be queried with man ascii.

A file like this, where binary data is converted through a character table into something human-readable, is called a text file. Files that are not readable this way are binary files, such as images and videos.

2. Process of Building an Executable

This source file is transformed into an executable object file through a series of steps such as compilation, assembly, and linking.

3. Compilation

Go compiler source code: Compiler executable:

3.1. Compilation Process

3.1.1. Front End

3.1.1.1. Lexical Analysis

The scanner converts a sequence of characters into a sequence of tokens. All tokens in Go are located here: The scanner source code is located here: The most important part is the Next function.

func (s *scanner) next() {
// ……

redo:
	// Get the next unparsed character, skipping subsequent spaces, carriage returns, newlines, and tab characters
	c := s.getr()
	for c == ' ' || c == '\t' || c == '\n' && !nlsemi || c == '\r' {
		c = s.getr()
	}

	// token start
	s.line, s.col = s.source.line0, s.source.col0

	if isLetter(c) || c >= utf8.RuneSelf && s.isIdentRune(c, true) {
		s.ident()
		return
	}

    // Enter a large switch-case statement and match different cases. Eventually a Token is parsed and the corresponding line and column numbers are recorded, completing one parsing step
	switch c {
    // ……

	case '\n':
		s.lit = "newline"
		s.tok = _Semi

	case '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
		s.number(c)
		
	// ……
	
   default:
		s.tok = 0
		s.error(fmt.Sprintf("invalid character %#U", c))
		goto redo
	return

assignop:
	if c == '=' {
		s.tok = _AssignOp
		return
	}
	s.ungetr()
	s.tok = _Operator
}

3.1.1.2. Syntax Analysis

The token sequence is converted into a syntax tree whose nodes are expressions. For example, slice[i] = i * (2 + 6):

3.1.1.3. Semantic Analysis

Correct syntax does not mean correct semantics. For example, the statement pointer * pointer can be syntactically valid but semantically invalid. Go’s semantic checks mainly include type checking and rewriting the abstract syntax tree.

3.1.2. Back End

The syntax tree is converted into intermediate code. Intermediate code is generally independent of the target machine and runtime environment. Common forms include three-address code and P-code. Go’s representation is SSA.

4. Linking

The object files produced by the compiler are linked into an executable. The resulting file is divided into various sections, such as the data section, code section, and BSS section, which are loaded into memory at runtime.

Linker source code: Linker executable:

5. Go Bootstrap Flow

5.1. Using GDB

For the Hello World file above, compile it with go build -gcflags "-N -l" -o hello hello.go, enter gdb hello to enter debugging mode, and execute info files. You can see that the entry address is 0x4568e0; enter b *0x4568e0. You can see that the entry file is C:/software/Go/src/runtime/rt0_windows_amd64.s, as follows:

TEXT _rt0_amd64_windows(SB),NOSPLIT,$-8
	JMP	_rt0_amd64(SB) # jump to TEXT _rt0_amd64_windows_lib_go(SB),NOSPLIT,$0

Continue with b _rt0_amd64. You can see that _rt0_amd64 jumps to C:/software/Go/src/runtime/asm_amd64.s, line 15, as follows:

TEXT _rt0_amd64(SB),NOSPLIT,$-8
	MOVQ	0(SP), DI	// argc
	LEAQ	8(SP), SI	// argv
	JMP	runtime·rt0_go(SB)

Continue with b runtime·rt0_go. runtime·rt0_go jumps to C:/software/Go/src/runtime/asm_amd64.s, line 89, as follows:

TEXT runtime·rt0_go(SB),NOSPLIT,$0
    #......
    # Initialize the absolute path of the executable file
	CALL	runtime·args(SB)
	# Initialize the CPU count and memory page size
	CALL	runtime·osinit(SB)
	# Initialize command-line arguments, environment variables, GC, stack space, memory management, all P instances, HASH algorithms, etc.
	CALL	runtime·schedinit(SB)

	// Function to run on the main goroutine
	MOVQ	$runtime·mainPC(SB), AX		// entry
	PUSHQ	AX
	PUSHQ	$0			// arg size
	// Create a new goroutine, bind it to runtime.main, place it in P's local queue, and wait for scheduling
	CALL	runtime·newproc(SB)
	POPQ	AX
	POPQ	AX

	// Start M and begin scheduling goroutines
	CALL	runtime·mstart(SB)

	CALL	runtime·abort(SB)	
	RET

	MOVQ	$runtime·debugCallV1(SB), AX
	RET

DATA	runtime·mainPC+0(SB)/8,$runtime·main(SB)
GLOBL	runtime·mainPC(SB),RODATA,$8

Continue by looking at the three major functions in the initialization process: runtime·args, runtime·osinit, and runtime·schedinit. runtime1.go line 60:

func args(c int32, v **byte) {
    // Organize command-line arguments
	argc = c
	argv = v
	sysargs(c, v)
}

os_windows.go line 389:

func osinit() {
	asmstdcallAddr = unsafe.Pointer(funcPC(asmstdcall))
	usleep2Addr = unsafe.Pointer(funcPC(usleep2))
	switchtothreadAddr = unsafe.Pointer(funcPC(switchtothread))

	setBadSignalMsg()

	loadOptionalSyscalls()

	disableWER()

	initExceptionHandler()

	stdcall2(_SetConsoleCtrlHandler, funcPC(ctrlhandler), 1)

	timeBeginPeriodRetValue = osRelax(false)

    // Determine the number of CPU cores
	ncpu = getproccount()

	physPageSize = getPageSize()

	stdcall2(_SetProcessPriorityBoost, currentProcess, 1)
}

proc.go line 529:

func schedinit() {
	// raceinit must be the first call to race detector.
	// In particular, it must be done before mallocinit below calls racemapshadow.
	_g_ := getg()
	if raceenabled {
		_g_.racectx, raceprocctx0 = raceinit()
	}

    // Maximum system-thread count limit
	sched.maxmcount = 10000

	tracebackinit()
	moduledataverify()
	// Stack initialization
	stackinit()
	// Memory allocator initialization
	mallocinit()
	// Scheduler-related initialization
	mcommoninit(_g_.m)
	cpuinit()       // must run before alginit
	alginit()       // maps must not be used before this call
	modulesinit()   // provides activeModules
	typelinksinit() // uses maps, activeModules
	itabsinit()     // uses activeModules

	msigsave(_g_.m)
	initSigmask = _g_.m.sigmask

    // Process command-line arguments and environment variables
	goargs()
	goenvs()
	// Process GODEBUG and GOTRACEBACK debugging-related environment-variable settings
	parsedebugvars()
	// Garbage collector initialization
	gcinit()

	sched.lastpoll = uint64(nanotime())
	// Determine the number of Ps. It is determined by the CPU core count and the GOMAXPROCS environment variable
	procs := ncpu
	if n, ok := atoi32(gogetenv("GOMAXPROCS")); ok && n > 0 {
		procs = n
	}
	// Adjust the number of Ps
	if procresize(procs) != nil {
		throw("unknown runnable goroutine during bootstrap")
	}

	// For cgocheck > 1, we turn on the write barrier at all times
	// and check all pointer writes. We can't do this until after
	// procresize because the write barrier needs a P.
	if debug.cgocheck > 1 {
		writeBarrier.cgo = true
		writeBarrier.enabled = true
		for _, p := range allp {
			p.wbBuf.reset()
		}
	}

	if buildVersion == "" {
		// Condition should never trigger. This code just serves
		// to ensure runtime·buildVersion is kept in the resulting binary.
		buildVersion = "unknown"
	}
	if len(modinfo) == 1 {
		// Condition should never trigger. This code just serves
		// to ensure runtime·modinfo is kept in the resulting binary.
		modinfo = ""
	}
}

After initialization completes, runtime.main is called.

func main() {
	g := getg()

	// Racectx of m0->g0 is used only as the parent of the main goroutine.
	// It must not be used for anything else.
	g.m.g0.racectx = 0

	// Stack-size limit
	if sys.PtrSize == 8 {
	    // 1 GB on 64-bit
		maxstacksize = 1000000000
	} else {
	    // 250 MB on 32-bit
		maxstacksize = 250000000
	}

	// Allow newproc to start new Ms.
	mainStarted = true

	if GOARCH != "wasm" { // no threads on wasm yet, so no sysmon
		// Start system background monitoring (related to periodic garbage collection and concurrent scheduling tasks)
		systemstack(func() {
			newm(sysmon, nil)
		})
	}

	// Lock the main goroutine onto this, the main OS thread,
	// during initialization. Most programs won't care, but a few
	// do require certain calls to be made by the main thread.
	// Those can arrange for main.main to run in the main thread
	// by calling runtime.LockOSThread during initialization
	// to preserve the lock.
	lockOSThread()

	if g.m != &m0 {
		throw("runtime.main not on m0")
	}

    // Execute all init functions in the runtime package
	doInit(&runtime_inittask) // must be before defer
	if nanotime() == 0 {
		throw("nanotime returning zero")
	}

	// Defer unlock so that runtime.Goexit during init does the unlock too.
	needUnlock := true
	defer func() {
		if needUnlock {
			unlockOSThread()
		}
	}()

	// Record when the world started.
	runtimeInitTime = nanotime()

    // Execute garbage-collector background operations
	gcenable()

	main_init_done = make(chan bool)
	if iscgo {
		if _cgo_thread_start == nil {
			throw("_cgo_thread_start missing")
		}
		if GOOS != "windows" {
			if _cgo_setenv == nil {
				throw("_cgo_setenv missing")
			}
			if _cgo_unsetenv == nil {
				throw("_cgo_unsetenv missing")
			}
		}
		if _cgo_notify_runtime_init_done == nil {
			throw("_cgo_notify_runtime_init_done missing")
		}
		// Start the template thread in case we enter Go from
		// a C-created thread and need to create a new thread.
		startTemplateThread()
		cgocall(_cgo_notify_runtime_init_done, nil)
	}

    // Execute init functions for all packages other than runtime
	doInit(&main_inittask)

	close(main_init_done)

	needUnlock = false
	unlockOSThread()

	if isarchive || islibrary {
		// A program compiled with -buildmode=c-archive or c-shared
		// has a main, but it is not executed.
		return
	}
	// Execute the main.main function
	fn := main_main // make an indirect call, as the linker doesn't know the address of the main package when laying down the runtime
	fn()
	if raceenabled {
		racefini()
	}

	// Make racy client program work: if panicking on
	// another goroutine at the same time as main returns,
	// let the other goroutine finish printing the panic trace.
	// Once it does, it will exit. See issues 3934 and 20018.
	if atomic.Load(&runningPanicDefers) != 0 {
		// Running deferred functions should not take long.
		for c := 0; c < 1000; c++ {
			if atomic.Load(&runningPanicDefers) == 0 {
				break
			}
			Gosched()
		}
	}
	if atomic.Load(&panicking) != 0 {
		gopark(nil, nil, waitReasonPanicWait, traceEvGoStop, 1)
	}

    // Execution ends; return the exit status code
	exit(0)
	for {
		var x *int32
		*x = 0
	}
}

5.2. Using go tool

go tool objdump hello > hello.asm

5.3. Overall Flowchart

6. Go Startup Flow

• runtime.rt0_go
  • Runtime type checking
  • Determines two important runtime constants: the number of processor cores and the physical memory page size
• schedinit
  • Completes initialization of the entire program runtime, including components such as the scheduler, execution stack, memory allocator, scheduler, and garbage collector

6.1. g0

• runtime·rt0_go allocates stack space for the main thread’s g0
• The main thread is bound to m0, m0 is bound to g0, and then to p0. In this way GPM starts running
• Create the main goroutine, put it into p0’s local runnable queue, and finally enter scheduling through the schedule function
• When execution reaches main.main, if there is a line of code like this:

go func() {
    // Work to do
}()

• A goroutine is started, which is eventually converted into the newproc function
  • newproc1 is called on the g0 stack
• Put the newly created goroutine into p’s local runnable queue

7. References

• How a Go Program Starts Running | qcrao
• 5.3 Go Program Startup Bootstrap | Go Language Essentials