What to Test

Seems there are three different kinds of tests worth doing:

• Type 1: Small unit tests testing individual pieces of functionality. For example, for each script that we ship, we should have a set of tests making sure that (1) the logging works; (2) its NOTICEs are generated; and (3) all exported options trigger the corresponding change in behavior (e.g., a whitelist indeed suppresses the corresponding NOTICE).

  We have already decided to use BTest as the main driver for these tests, and I suggest to create a directory structure like this for the unit tests:

  testing/unit/Makefile         -> will run all tests
  testing/unit/btest.cfg        -> BTest configuration
  testing/unit/scripts/conn/*   -> Tests for conn.bro
  testing/unit/scripts/http/*   -> ...
  testing/unit/scripts/ftp/*
  ...
  

  I.e., one sub-directory per Zeek script, with all its tests going in there.

  Typically, testing a script will require a trace to run Zeek with. For these unit tests, traces should be very small and must be non-sensitive. Let’s place them all in single directory, say testing/unit/traces so that they can be potentially reused across tests.

  Establishing the baselines for these unit tests should be pretty straight-forward by just looking at the output: because the input traces are small, one can just see whether things look as expected.

  Along with having tests for all the scripts, we will also add tests for other Zeek components. There are already some tests for the new logging framework, which can go into testing/unit/logging; and the existing tests for the remote communication code go into testing/unit/communication. (Note that the latter are currently in testing/istate and not using BTest yet. But I think I know how to port them over to BTest and plan to do so soon.)

  (See the branch topic/robin/logging-internal for an initial btest.cfg in testing/btest/btest.cfg)

• Type 2: Running a complex Zeek configuration on large comprehensive traces, and making sure all output matches with a previously established baseline. Two questions here are: (1) what’s the complex configuration, and (2) what trace(s) to use.

  Regarding (1), let’s just load as many of the default scripts as make sense (some scripts do not, such as contents.bro, which dumps the payload of every connection into a file).

  Regarding (2), I think we should use (at least) two different traces, one public and one private. The public one can be used with NMI builds (see below) and also by external contributors; the private one is internal to one of our environments.

  For the public trace, the recent Patents-scenario might actually be a good candidate (or a subset of it). See:

  http://digitalcorpora.org/corpora/scenarios/m57-patents-scenario https://domex.nps.edu/corp/scenarios/2009-m57/net/

  The private trace we need to capture somewhere where we’ll have a machine available for continuous testing. I’m thinking this will be a "really private" trace with access just to people authorized for the environment it’s coming from (note that the 1.5 test suite uses a model of semi-private trace: it’s not public, but we are rather liberal in terms of who can get it; but building such a trace that at the same time goes beyond other public traces seems hard).

  These large-trace tests can also be driven using BTest, and we can store the corresponding configuration in testing/trace (or some nicer name). However the traces and the baselines shouldn’t be part of the testing/* tree due to their size. Not quite sure yet where to store them and how to then get them; we’ll need to figure that out.

  Establishing baselines for these large traces is more difficult than for the small unit tests as we can’t just manually look at the output anymore. We also can’t just rely on Zeek 1.5 to tell us what’s correct because we’re changing the output formats. So, I think we’ll need to do some random consistency checks (e.g., pick a few connections and examine them manually) and otherwise just declare the output to be correct at some point to establish the initial baseline. From then on, we’ll be able to catch all changes. Doing this initial baseline probably waits best until we have most of the small unit tests in place; they will ensure that the output is at least basically correct.

  (Note that the "private trace" model is the only type of testing we are currently doing already with 1.5. However, the traces used for that are very old and it would be good to get a new ones).

• Type 3: Full end-to-end configure/build/install/run tests. This is to make sure that a complete Zeek distribution works all the way from downloading to starting Zeek, including using broctl. I don’t think BTest is the right tool for this, it’s probably better to write individual shell scripts executing the steps of process (our NMI scripts seem to already do much of this) There are two variations that need testing:

  • Zeek standalone
  • Zeek cluster

  Both need to make sure that BroControl works, including all of its commands that can be reasonably tested.

  We can store the relevant scripts in, e.g., testing/build.

• Type 4: Online live tests. We should set up a live system (and ideally a test cluster) that always runs the current master and devel live, watching for crashes, CPU usage, memory leak, etc.

  I’m afraid this will be quite tricky to get right (e.g., what do we one if one crashes…?) but it will very helpful long-term.

How/when to Test

So now we have four kinds of tests: I’ll call them the unit tests; trace tests; build tests; and live tests.

It should generally be possible to run all these locally on whatever development machine one is on (exception: the private-trace test because the trace may not be available). For the unit tests, that should just mean going to testing/unit and running make there; that will then produce local output telling how things went. For the trace tests, it depends on where the traces are stored, but there should also be some easy way to get everything that’s required to the local machine and then start the tests from testing/trace. For the build tests, a driver script in testing/build can run the individual scripts. For the live tests, it’s mainly about providing the same Zeek/BroControl configuration, say in testing/live, and perhaps any additional scripts it may need.

The first three types of tests should also be run automatically and regularly on NMI and inside the private test environment. Doing so can be triggered nightly by cron, plus we need some way to force a full test immediately if required (login via SSH is ok). Results are then either emailed out (beware of large outputs and potentially private data!) or presented in some web-based GUI.

For the NMI tests, let’s make sure we run at least the following configurations on all platforms:

• Standard ./configure
• ./configure --enable-brov6
• ./configure --enable-debug

The git branches that need automatic testing are master and devel.

Also, all tests should be runnable with releases as well (both tarballs and binary).

How to Update Baselines

Need to figure out a process between maintainers of the baseline and folks making changes to the code triggering updates.