VTK-5.0.0/Testing.txt

How to run and create tests....

In creating tests there are a few goals to strive for.

1) Increase coverage of the code to help detect defects

2) Don't mess up the tester's directories. If a test wants to write out a
   file, first make sure the directory is writable, and if it is, then write
   out the files, and then delete them. So that the directory is left in its
   original state.

3) Don't require excessive disk space or CPU time. Most tests that run on a
   512 cubed volume will work just as well on a 64 cubed volume. The 64 cubed
   volume will require less disk space for the data, less CPU time to
   process, and much less CPU time to purify. The same applies to valid
   images. Typically there isn't any need for a 600 by 600 pixel valid
   image. A 300 by 300 image will work fine and require less storage and less
   CPU time to compute the image difference.

4) Try to use existing data files. If you can create a test using a data file
   that is already present, use that data file. Don't add a new data file
   unless it is critical to test the functionality you want tested. If you do
   need to add a new data file consider its size. If it is a large data file
   can you produce a lower resolution version of it and check that in instead?

The current status is that the new tree has about 100 tests and is achieving
about 43 percent coverage. These tests require about three minutes to run on
a 1 GHz Pentium and currently require about ten megabytes of disk space for
the data files and valid images. 


How to run a test
--------------------------

The easiest way to run a test is using ctest which is included with
cmake. You must make sure your build of VTK was done with BUILD_TESTS set to
on when running cmake. Once your build is complete you can run tests in a few
different ways. For example; you can cd to the top of your binary tree and
run ctest. This will run every test in VTk and show you the results. If you
only want to run specific tests you can use ctest -R <regexp> and provide a
regular expression or ctest -I start,end,stride,extra ... Run ctest --help to
see a more complete listing of options.

How to create a new test
---------------------------

The easiest approach is to look at an existing test and use that as a
starting point. The tests are kept in Testing sub-directories under each main
package. (e.g. VTK/Graphics/Testing) In Testing there will be sub-directories
for each type of test (Cxx Tcl Python Java). For C++ tests you need to write
the test and then add it to the CMakeLists.txt file in Testing/Cxx. For
Tclexamples you would write a Tcl script, and then add it to the
CMakeLists.txt file in Testing/Tcl. Just follow the examples already in the
tree. Most tests take a few common arguments:

  -D /path/to/VTKData
  -V relative/path/to/valid/image
  
Tcl scripts also accept:

  -A /path/to/vtktcl/package

Typically these arguments are optional and the test will try to run without
them. If the -V option is provided the test will try to perform image
differences between the valid image and the rendered image. If the valid
image specified with -V doesn't exist it will create it.


Running a Tcl example
-----------------------

The most common way to run a Tcl example is to first set TCLLIBPATH to
/path/to/VTK/Wrapping/Tcl so that the examples can find the vtktcl
package. On UNIX you can do:

setenv TCLLIBPATH /home/you/VTK/Wrapping/Tcl

or on windows set TCLLIBPATH to c:/your/VTK/Wrapping/Tcl note that UNIX style
slashes are used on windows as well. Once TCLLIBPATH is set you can run the
example as

cd VTK/Graphics/Testing/Tcl
/path/to/VTK/bin/vtk yourScript.tcl 

It will look for VTKData to be at the same level as VTK on your disk. If it
isn't you will need to provide the -D argument.

/path/to/VTK/bin/vtk yourScript.tcl -D /path/to/VTKData

to run the example as a regression test you should make the example the first
argument to VTK/Common/Testing/Tcl/rtImageTest.tcl. For example

/path/to/VTK/bin/vtk /path/to/VTK/Common/Testing/Tcl/rtImageTest.tcl yourScript.tcl -D /path/to/VTKData -V Baseline/Graphics/yourScript.png

That is a long command line but normally Dart will take care of it for you.


VTK-Python tests
------------------

Writing and running VTK-Python tests is documented in
Wrapping/Python/README.txt.  There are two types of Python tests, the
old tests and the newer tests that use unit testing approach
(unittest).  The newer tests can be run with a -h or --help argument.
This provides information on the available command line options.  The
other options for the newer tests are slightly different from the Tcl
test scripts.  Please read Wrapping/Python/README.txt for more details
on the newer tests.  The older tests should have the same command line
options that the Tcl scripts have.  Test writers are encouraged to use
the new approach (unittest) for the tests.


Producing a dashboard
-------------------------

If you want to test your build of VTK you can produce a dashboard. To do this
you will need to have cmake installed on your system (which you need anyhow
to build VTK).  Then when you run CMake, make sure BUILD_TESTING is on. Then
you can do

make Experimental

on UNIX to perform a build, run the tests, and submit the result to the
dashboard. If you just want to run the tests you can do

make ExperimentalTest

on windows there are targets in the workspace for Experiemental and
ExperimentalTest