nostrils is a Python script written to easily compare the results of test suites across different git refs (commits/branches/tags).
It will run the tests suite on the required commits, build any extension if needed (that is, run the build_ext target of the setup.py) and print out a diff of the test results. This comes handy when the repo you are working on already has failing tests, possibly many of them, and you want to quickly isolate any regression caused by your modifications.
nostils runs tests for the two commits in parallel. Moreover, it caches test results, so if it is called again on some given ref (typically master!), it will fetch them instead than rerunning the test suite (this behaviour can be overridden with the –force argument).
Usage
This is the typical usage:
nostrils mybranch
It will compare the results of tests on mybranch with the results on master.
You can specify a different benchmark commit as follows (this will compare with the previous commit in mybranch):
nostrils mybranch~1 mybranch
In order to specify which test command to run, you should create a file ~/.config/nostrils/test_commands.py containing only a Python dictionary which assigns a command to each repo, such as:
{
'/home/myuser/repo/pandas' : './test_fast.sh'
}
If at any time you want to run a different command without messing with the config file, you can o so by setting the environmental variable TESTCMD, as in:
TESTCMD=nosetests nostrils mybranch