Manabie Tech Blog

Sharing the humble technical knowledge we’re using to improve education

Test Coverage of Go Services during Integration Tests

Test Coverage of Go Services during Integration Tests

In Golang, getting code coverage with go test is easy. But it’s still rather hard for integration tests.

Here I want to introduce the method we used at Manabie to measure the code covered by our integration tests across many microservices.

About our integration tests

At Manabie we use Kubernetes for container orchestration. To perform integration tests, we deploy our services, and then run a test container with a go program with a whole lot of integration tests.

On dev environments, we start up minikube, deploy the services, and then run the test container as well. In CI, we start a vcluster instead of a minikube.

Collecting coverage in this setting is a little more complicated, but doable. The main idea is to compile the services with go test -cover instead of go build, and then get the services to exit in a timely manner after testing is finished.

What’s Coming Up

Part 1:

  • compile services with go test -c,
  • add an http killswitch

Part 2:

  • run services & run tests
  • stop the services by calling the http killswitch endpoint

Part 3:

  • collect coverage reports from containers
  • merge them with gocovmerge.
  • get a code coverage percent.

Conclusion & Example Code

Part I: Compiling services with go test instrumentation

We want to run our service with go test so we can use it’s cover flags to enable code coverage output.

To do this, we need to first create a test function that works like our func main(), so that we can use go test to run our server.

For example:

func main() {


func run() {

func main() {

so that we can write a test like this that starts the server:

func TestRun(t *testing.T) {

For the coverage to be output, TestRun function needs to finish. We can’t just kill the process, or go test will not output a coverage profile.

Because the service does not know when the tests are complete, we have to set up a mechanism to stop it remotely. We can set up a simple HTTP server. When it gets a request, it will gracefully terminate our service. Later, we can call it with curl.

Our Kill HTTP server calls a context’s CancelFunc when it receives a request.

type killServer struct {
	server http.Server
	cancel context.CancelFunc

func newKillServer(addr string, cancel context.CancelFunc) *killServer {
	return &killServer{
		server: http.Server{
			Addr: addr,
		cancel: cancel,

func (s *killServer) Start() {
	s.server.Handler = s

	err := s.server.ListenAndServe()
	if err != nil {
		fmt.Println("KillServer Error:", err)

func (s *killServer) ServeHTTP(w http.ResponseWriter, r *http.Request) {

  // cancel the context

Our TestRun uses the same context to run the service.

func TestRun(t *testing.T) {
	ctx, cancel := context.WithCancel(context.Background())
	killServer := newKillServer(":19999", cancel)
	go killServer.Start()  
	go runService(ctx)



To test out our killswitch locally, we can compile the service with: go test -c ./ -cover -covermode=count -coverpkg=./... to create a .test binary, and run it with:

./my-service.test -test.coverprofile my-service.out

In another terminal, we run some tests and then kill it with the HTTP signal.

./ # here we could make test calls to the server.

curl localhost:19999 # and then kill the remote server when were done

Finally, a file named my-service.out should be created with coverage information from our server.

Using go test -c to build test binaries is a nice little trick I learned about recently. It plays nicely with containers too.

Part 2: Running in our Kubernetes Cluster

Now we can add them to our Dockerfile. Also, we’re going to add a little wrapper script to restart them endlessly. If we let the entrypoint process end, our container and coverage files will be deleted after the killswitch is called, which is not what we want.


# this script passes arguments into server.test with
# the addition of -test.coverprofile=cover.out

while true; do
    echo "Service started in coverage mode"
    /server.test -test.coverprofile=cover.out "$@" || exit 1;
    echo "Server restarting.."

We need a Dockerfile with the server.test binary, curl, and the restart script.

FROM alpine


RUN apk --no-cache add curl

COPY ./ /
COPY ./server.test /server.test


Next, build the new docker image and deploy it in your kubernetes cluster, possibly with helm. Make sure you don’t so override the Dockerfile’s entrypoint. However, you can pass args as usual into the deployment container, which will pass into the server.test command.

Next, run your tests. In our case, we built an integration testing program named gandalf that we run in our cluster:

helm install gandalf ./deployments/gandalf
kubectl exec -it gandalf -- /gandalf-tests

Coverage will be recorded by the server.test.

Part 3: Collecting Coverage

Now to generate, download and merge our coverage.

To output the coverage we just have to send an HTTP request to our killserver on port 19999.

kubectl exec my-service -- curl -s localhost:19999

Then copy the file to our local filesystem:

kubectl exec -i my-service -- cat /cover.out > cover/my-service.out

Do this for each service being tested, and then merge the coverage profiles together.

go install

gocovmerge cover/*.out > cover/merged.cov

# Output Total Coverage 
go tool cover -func=cover/merged.cov | grep -E '^total\:' | sed -E 's/\s+/ /g'

Note: We use a .cov extention on the merged.cov to make it easy to use cover/*.out in scripting.

Inspect the contents of your new merged.cov file and try not to get drunk with power.


Now our build pipeline prints out our integration test coverage percent collected from all our go services, and combined into a final figure.

For us at Manabie, we output this percent in our CI logs, and we set a rule that prevents pull requests from being mergeable if they decrease this percent.

Example on Github

About ds0nt
A developer who loves making things with golang, customizing his linux, and connecting too many displays


comments powered by Disqus