The Windows virtual machine that Travis runs the integration tests on
seems to be slower than the other machines. We try to increase the
stability of the tests by cutting the number of parallel running suites
in half. This will come at the cost of longer execution time but
hopefully with a better stability in return.
To remove the need to have an extra make goal for the Windows itests, we
instead add the flag windows=1 that sets the make variable EXEC_SUFFIX
to properly add the ".exe" suffix to all executable names.
There is a setting to control how often the garbage collector is run.
Apparently this is a tradeoff between CPU and memory usage. If we can
limit the memory being used in that way, this allows us to use multiple
worker again, so overall this shouldn't be much slower than before.
This commit adds an integration test that runs on a Windows virtual
machine on Travis. The tests run inside of a "Git Bash" environment
which supports the same command line syntax as a proper bash but doesn't
have all the tooling installed. Some tools also behave differently on
Windows. Therefore we also have to simplify the command to upload the
logs to termbin and remove the upload to file.io on Windows because both
the find and tar command don't work as expected.
To free up build in Travis, we decided to run the non-flaky parts of
the CI pipeline in GitHub Workflows/Actions only. The integration tests
on the other hand are removed from GitHub because individual actions
cannot be restarted there which caused us to restart the whole workflow
if one test was flaky.
This split should give us the best of both worlds: Fast run of small
checks, linting and unit tests with an easy overview of what failed in
the PR directly. And more free build slots on Travis to do more advanced
integration tests on other architectures and/or operating systems. And
the option to restart a single flaky integration test on Travis.
By default, Travis-CI does `git clone --depth=50` which limits the
repo to the last 50 commits. Because we do linting against an old
commit, it cannot be found any more. We remove the limit completely
to not run into this problem again.
This PR introduces staging to our travis pipeline. Currently all
instances perform:
- compilation of lnd
- linting
- compilation and installation of btcd binaries
- installation of bitcoind binaries
In total this adds about 3 minutes to each of our 5 instances, resulting in
roughly 12 minutes of redundant execution time. Additionally, if if a build
fails to compile or lint we detect this 5 separate times, consuming precious
instances from other builds.
We alleviate this by adding an initial Build phase, which runs a single
instance performing the actions above. This has the benefit of quickly sanity
checking the pr before moving on to the more expensive unit or integration test
suites, and failing faster for common mistakes. It also warms up the build
caches for the Test stage in one fell swoop.
For instance, if 5 people push changes at the same time, they can all get
immediate feedback regarding compilation or linting issues, and potentially
save hours waiting for other people's test to finish or fail before finding out
they had a spelling error. This doesn't alleviate all possible issues, e.g. the
5 instances may already be consumed by test suites, but it does make a sizable
step towards minimizing time-to-failure in common scenarios.
In this commit, we update Travis to start building against the newly
released go 1.13. Additionally, we'll now utilize the new `trimpath` to
the `go build` and `go install` commands. This new flag serves to remove
all file system paths from the compiled Go executable, which will make
our binaries more reproducible.
Before this commit, if for example the linter failed, then we would go
on to all the other tests rather than halting. We fix this by instead
chaining the relevant commands, and eliminating the LINT env variable
in the build matrix.
This commit sets the GOCACHE environment variable, and enables caching
on travis.
GOCACHE will tell the compiler (only go 1.10+) where to store build and
test artifacts. Caching this directory will significantly speed up
succeeding builds and test runs.
We also cache vendor/ as this will speed up the call to 'make dep'.
A few dependencies in the GOPATH are cached, as calls to 'go get' for
these will be sped up. Note that we don't cache the lnd directory, as it
will conflict with the changes in the PR being built.
This commit distributes the CI tests into 3 independend builds, by
splitting the integration test run and unit test coverage.
To better handle the extra cases, we define a build matrix with the
three build types (RACE and LINT, ITEST, COVER).
Instead of calling 'make travis' directly, we call each step. This lets us
better track how much time is spent on each.
Also note that we execute 'itest-only' instead of 'itest', and instead
execute the dependencies (btcd, build) manually first.
We explicitly set the ITEST environment variable, for readability, and
define a new COVER. This is currently true when ITEST=true to keep the
existing build configuration, but will later be configured to be
independent.
Joost Jager
Oliver Gugger
Olaoluwa Osuntokun
Andras Banki-Horvath
Johan T. Halseth
Wilmer Paulino
Conner Fromknecht
Harald Nordgren