Disclaimer

Well yeah, this specification is far from complete. Currently it's pretty much a bunch of thoughts, not really a specification. However if you see it's potential and have something to add please either edit the page (if you can) or leave a comment.

Additional files

Besides this specification there are also:
  • The mind map describing the initial ideas for the verifier. Can be retrieved from the Subversion repository. Mindmap was created with FreeMind.

Introduction

It’s a tool to perform verification calculations on the different parts of the dnAnalytics library. Main goal is to test the matrix methods, solvers and other numerical methods which have an approximate outcome (i.e. methods that cannot be fully unit tested due to their approximate nature).

What are verification calculations:
Calculations of which the outcome is known which are used to verify that the calculation algorithm is implemented correctly.

The goal of the tool is to provide a way to verify the accuracy (and performance?) of the numerical algorithms in the dnAnalytics library, both for a single run and over time as the code gets changed (so as to track improvements in accuracy, or losses thereof). While the initial idea is to test the solvers etc. it would be nice to keep the implementation slightly flexible so that it's easy to adapt when we add new algorithms to the library.

Suggested workflow

  1. Create a single test implementation that can use different data sets.
  2. Get the input data and the known-good result data.
  3. Run tests.
    1. The user should be able to select specific tests, i.e. select tests to run or not run.
    2. There should be a way to indicate an order in the tests. It is probably more logical to run the simple tests first to flush out major errors quickly.
  4. Test system compares calculated results with known-good result data and stores information about comparison.
  5. Show results to user and allow user to review how changes in source affect accuracy.

Note that tests could be long running, ranging from a few seconds to a few hours or more. This means we’ll have to deal with threading (to not lock up the system) and send updates to the UI (to let the user know we’re still alive).

Data handling and storage

Data handling will be the tricky part. We are dealing with input data and output data. For the input data we’ll need to allow multiple datasets per test. The current questions are:
  • How will the input data be obtained? Will it be stored in files in a specific directory or will the data be embedded in the source code (much like unit tests)?
  • If the data is not embedded in the source code how should the developer specify which data should be used?
  • How will we provide the testing methods / objects with the data? Directly through a stream or ...?

Once the input data is located and loaded the test can be run. After the test has run we should have some output data. Now the following questions arise:
  • How does the developer indicate what the output data is?
  • How will we process the output data? A comparison must be performed but how? Should we ask the developer to write a comparison method or will there be default implementations?
  • How will we store the final comparison / results? And what data will we store?

User interface

We’ll need a GUI and a console version. The console version is meant for automatic running so we’ll need to think about the command line arguments carefully.

Some issues for the test running phase:
  • How do we deal with updates to the UI. We'll at least need a progress indicator to show the user where we are at. For the GUI we'll probably need to be a bit more specific than just letting the user know we're running test number XX. It would be nice to have more details, e.g. location in the test, location in the algorithm etc.

Technical details

Obviously there are a ton of testing frameworks (both open and closed source) around. The first question is if we can use any of these systems as the base for this application. The current answer is "Not a clue". Both NUnit (which is used as the unit testing framework for dnAnalytics) and MbUnit are build on top of testing frameworks which could possibly be used (although there may be licensing issues).
The long and short of this is that we'll have to think / research the use of unit testing frameworks a bit and some questions that need answering are:
  • Can we use either NUnit or MbUnit from a license perspective. The verifier will be released under the same license as the dnAnalytics library so we could have trouble on this front.
  • How well would the unit testing framework deal with large / long running tests
  • Would we need to create our own user interface or could we work with their UI's (GUI and console)

Last edited Sep 26, 2008 at 1:57 AM by pvandervelde, version 4

Comments

No comments yet.