[Scons-users] best practices: unit tests

[R0b0t1]

On Thu, Feb 1, 2018 at 4:05 PM, Mats Wichmann <mats at wichmann.us> wrote:
>
> I have a few "is there a Best Known Method" type questions that I'll
> dribble out over a few messages (that is, if I can keep my focus, I keep
> getting dragged off onto other things).
>
> This one has several interesting hits on the internet, to my
> recollection - wiki pages, blogs, stackexchange, etc. The ones I've
> looked at over several months didn't seem terribly comprehensive, though.
>
> Unit tests are a key part of many software projects these days, and
> often a part of acceptance criteria in code review systems. scons itself
> follows this model - I see tests running every time you submit a pull
> request on github.
>
> What do people do here? Integrate the running of tests into the scons
> builds? Keep them external?  If integrated, how?  On a smaller project I
> suspect it could make sense to integrate the test tightly into the
> build, such that a builder doesn't even report success until the
> matching unit test has also passed - does anyone actually do that?  In a
> more complex project, the ordering can get confusing: e.g. if several
> different directories each build a shared library and a unit test
> program for one shared library may require use of others of the shared
> libraries, you can't bind it that simply.  What then - e.g. if you ought
> to wait for the whole project to finish building before running the
> various tests?  In the case of the project I'm working on, tests are
> built with googletest, which is told on invocation to generate a result
> file and the CI system swings through and collects the various result
> files - thus the pass/fail decision is external to scons - but the
> launching of the tests is not.  Indeed, it's big and complex enough that
> running the unit tests is optional, which several of the CI builds do,
> but a developer might not when testing a patch initially, as it could
> slow them down too much.
>
> Again... I realize that if it works, it's not wrong, I'm just curious
> for thoughts on best practices, I assume others have had to think
> through this question?
>

Unit tests are a mixed bag. If your program's behavior can be defined
precisely and is limited in scope, unit tests may save you time; on
the other hand, if your program's behavior is not limited in scope
(i.e. it has lots of user interaction or takes continuous inputs) unit
tests may not save you time.

The reason for this is that, in general, it is not possible to prove
what something is by saying what it is not, so you can not prove your
program is "correct" by showing various ways it is known to not be
incorrect.

Using a precise example: If I pick an integer and let you guess what
it is and report whether your guess was correct, in the worst case,
you will spend forever guessing to find the number I picked, because
there are an infinite number of integers.

For this reason, view software projects that rely heavily on tests to
prove they are consistent[1] with a great deal of suspicion.


It sounds like referring you to an existing system would help - so,
you might look at Ant and Maven and the structure of large Java
projects.[2] There are typically src/ and test/ directories in the
project. The former is built as it is a dependency for everything in
the latter, and the latter should not have any dependencies between
themselves (or at least I've never seen any tests that depend on each
other, but it could happen).

Enumerating the tests and tracking their completion is quite a bit of
work, so there are frameworks or harnesses to do this, if it is not
part of the build tool. This is where my knowledge stops. Perl has
good unit test frameworks, and I have seen some that use Makefiles.

Cheers,
     R0b0t1


[1]: https://github.com/ruby/ruby
[2]: https://github.com/scala/scala