showing coverage of functional tests without blind spots

Question

I have a production golang code and functional tests for it written not in golang. Functional tests run compiled binary. Very simplified version of my production code is here: main.go:

package main

import (
    "fmt"
    "math/rand"
    "os"
    "time"
)

func main() {
    rand.Seed(time.Now().UTC().UnixNano())
    for {
        i := rand.Int()
        fmt.Println(i)
        if i%3 == 0 {
            os.Exit(0)
        }
        if i%2 == 0 {
            os.Exit(1)
        }
        time.Sleep(time.Second)
    }
}

I want to build coverage profile for my functional tests. In order to do it I add main_test.go file with content:

package main

import (
    "os"
    "testing"
)

var exitCode int

func Test_main(t *testing.T) {
    go main()
    exitCode = <-exitCh
}

func TestMain(m *testing.M) {
    m.Run()
    // can exit because cover profile is already written
    os.Exit(exitCode)
}

And modify main.go:

package main

import (
    "flag"
    "fmt"
    "math/rand"
    "os"
    "runtime"
    "time"
)

var exitCh chan int = make(chan int)

func main() {
    rand.Seed(time.Now().UTC().UnixNano())
    for {
        i := rand.Int()
        fmt.Println(i)
        if i%3 == 0 {
            exit(0)
        }
        if i%2 == 0 {
            fmt.Println("status 1")
            exit(1)
        }
        time.Sleep(time.Second)
    }
}

func exit(code int) {
    if flag.Lookup("test.coverprofile") != nil {
        exitCh <- code
        runtime.Goexit()
    } else {
        os.Exit(code)
    }
}

Then I build coverage binary:

go test -c -coverpkg=.  -o myProgram

Then my functional tests run this coverage binary, like this:

./myProgram -test.coverprofile=/tmp/profile
6507374435908599516
PASS
coverage: 64.3% of statements in .

And I build HTML output showing coverage:

$ go tool cover -html /tmp/profile -o /tmp/profile.html
$ open /tmp/profile.html

Problem

Method exit will never show 100% coverage because of condition if flag.Lookup("test.coverprofile") != nil. So line os.Exit(code) is kinda blind spot for my coverage results, although, in fact, functional tests go on this line and this line should be shown as green.

On the other hand, if I remove condition if flag.Lookup("test.coverprofile") != nil, the line os.Exit(code) will terminate my binary without building coverage profile.

How to rewrite exit() and maybe main_test.go to show coverage without blind spots?

The first solution that comes into mind is time.Sleep():

func exit(code int) {
        exitCh <- code
        time.Sleep(time.Second) // wait some time to let coverprofile be written
        os.Exit(code)
    }
}

But it's not very good because will cause production code slow down before exit.

Answer 1

As per our conversation in the comments our coverage profile will never include that line of code because it will never be executed.

With out seeing your full code it is hard to come up with a proper solutions however there a few things you can do to increase the coverage with out sacrificing too much.

func Main and TestMain

Standard practice for GOLANG is to avoid testing the main application entry point so most professionals extract as much functionality into other classes so they can be easily tested.

GOLANG testing framework allows you to test your application with out the main function but in it place you can use the TestMain func which can be used to test where the code needs to be run on the main thread. Below is a small exert from GOLANG Testing.

It is sometimes necessary for a test program to do extra setup or teardown before or after testing. It is also sometimes necessary for a test to control which code runs on the main thread. To support these and other cases, if a test file contains a function: func TestMain(m *testing.M)

Check GOLANG Testing for more information.

Working example

Below is an example (with 93.3% coverage that we will make it 100%) that tests all the functionality of your code. I made a few changes to your design because it did not lend itself very well for testing but the functionality still the same.

package main

dofunc.go

import (
    "fmt"
    "math/rand"
    "time"
)

var seed int64 = time.Now().UTC().UnixNano()

func doFunc() int {
    rand.Seed(seed)
    var code int
    for {
        i := rand.Int()
        fmt.Println(i)
        if i%3 == 0 {
            code = 0
            break
        }
        if i%2 == 0 {
            fmt.Println("status 1")
            code = 1
            break
        }
        time.Sleep(time.Second)
    }
    return code
}

dofunc_test.go

package main

import (
    "testing"
    "flag"
    "os"
)

var exitCode int

func TestMain(m *testing.M) {
    flag.Parse()
    code := m.Run()
    os.Exit(code)
}

func TestDoFuncErrorCodeZero(t *testing.T) {
    seed = 2

    if code:= doFunc(); code != 0 {
        t.Fail()
    }
}

func TestDoFuncErrorCodeOne(t *testing.T) {
    seed = 3

    if code:= doFunc(); code != 1 {
        t.Fail()
    }
}

main.go

package main

import "os"

func main() {
    os.Exit(doFunc());
}

Running the tests

If we build our application with the cover profile.

$ go test -c -coverpkg=. -o example

And run it.

$ ./example -test.coverprofile=/tmp/profile

Running the tests

1543039099823358511
2444694468985893231
6640668014774057861
6019456696934794384
status 1
PASS
coverage: 93.3% of statements in .

So we see that we got 93% coverage we know that is because we don't have any test coverage for main to fix this we could write some tests for it (not a very good idea) since the code has os.Exit or we can refactor it so it is super simple with very little functionality we can exclude it from our tests.

To exclude the main.go file from the coverage reports we can use build tags by placing tag comment at the first line of the main.go file.

//+build !test

For more information about build flags check this link: http://dave.cheney.net/2013/10/12/how-to-use-conditional-compilation-with-the-go-build-tool

This will tell GOLANG that the file should be included in the build process where the tag build is present butNOT where the tag test is present.

See full code.

//+build !test

package main

import "os"

func main() {
    os.Exit(doFunc());
}

We need need to build the the coverage application slightly different.

$ go test -c -coverpkg=. -o example -tags test

Running it would be the same.

$ ./example -test.coverprofile=/tmp/profile

We get the report below.

1543039099823358511
2444694468985893231
6640668014774057861
6019456696934794384
status 1
PASS
coverage: 100.0% of statements in .

We can now build the coverage html out.

$ go tool cover -html /tmp/profile -o /tmp/profile.html

Answer 2

In my pkglint project I have declared a package-visible variable:

var exit = os.Exit

In the code that sets up the test, I overwrite it with a test-specific function, and when tearing down a test, I reset it back to os.Exit.

This is a simple and pragmatic solution that works well for me, for at least a year of extensive testing. I get 100% branch coverage because there is no branch involved at all.