TESTS

About testing in the hledger project. For how to run the tests, see Developer workflows.

Kinds of tests

1. Unit tests

   Unit tests exercise small chunks of functionality. In hledger, that means a function. So, many of our functions have one or more unit tests. These are mostly in hledger-lib, with a few in hledger.

   Our unit tests use the tasty test runner, tasty-hunit HUnit-style tests, and some helpers from Hledger.Utils.Test, such as:

   • tests and test aliases for testGroup and testCase
   • assert* helpers for constructing various kinds of assertions

   We would like our unit tests to be:

   • easy to read (clear, concise)
   • easy to write (low boilerplate, low cognitive load)
   • easy to maintain (easy to edit, easy to refactor, robust)
   • easy to associate with the code under test (easy to view/jump between code & test, easy to estimate coverage)
   • and scalable (usable for all devs, easy to run and select, suitable for small/large modules/packages).

   Here's the current pattern (let us know if you see a better way):

   module Foo (
     ...
     tests_Foo -- export this module's and submodules' tests
   )
   where
   import Hledger  -- provides Hledger.Utils.Test helpers
   import Bar      -- submodules, providing tests_Bar etc.
   import Baz
   
   functionA = ...
   functionB = ...
   functionC = ...
   functionD = ...
   
   tests_Foo = tests "Foo" [ -- define tests at the end of each module
   
      -- a group of several named tests for functionA
      tests "functionA" [
        test "a basic test"           $ assertBool "" SOMEBOOL
       ,test "a pretty equality test" $ SOMEEXPR @?= EXPECTEDVALUE
       ,test "a pretty parsing test"  $ assertParseEq PARSER INPUT EXPECTEDRESULT
       ,test "a multiple assertions test" $ do
         A @?= B
         doSomeIO
         C @?= D
       ]
   
      -- a single test containing multiple unnamed assertions for functionB
     ,test "functionB" $ do
        assertBool "" BOOL
        EXPR @?= VALUE
   
     ,tests_Bar            -- aggregate submodule tests
     ,tests_Baz
     ]
   

   Here are some real-world examples (search for tests_).

   The unit tests are shipped as part of the hledger executable, and can always be run with the --test flag (hledger --test).

2. Doc tests

   Like unit tests, but defined inside functions' haddock documentation, in the style of a GHCI transcript. These test functionality, provide usage examples in the API docs, and test those examples, all at once. They are a bit more finicky and slower than unit tests; running them requires a separate build of hledger-lib. See doctest for more.

3. Functional tests

   Functional tests test the overall functioning of the program. For hledger, that means running hledger with various inputs and options and checking for the expected output. This exercises functionality in the hledger and hledger-lib packages. We do this with shelltestrunner. Tests are defined in files named *.test under hledger/test/, grouped by component (command or topic name). For more about these, see the README there.

   When these test for an error, they usually match just a key phrase of the message. The layout of error messages is tested separately in hledger/test/errors/, which has one full-message example of each kind of error message (see ERRORS).

   hledger-web has its own server tests and browser tests; see hledger-web's tests below.

4. Code tests

   We have some tests aimed at testing eg code quality, generally runnable via just. Eg just haddocktest (haddock generation), just embedtest (embedded files), just cabalfilestest (cabal file syntax).

5. Package test suites

   Haskell tools like stack and cabal recognise test suites defined in a package's cabal file (or package.yaml file). These can be run via stack test, cabal test etc., and they are required to build and pass by services like Stackage. Here are the current hledger package test suites:

   package	test suite	what it runs
   hledger-lib	unittest	hledger-lib's unit tests
   hledger-lib	doctest	doctests
   hledger	unittest	builtin test command (hledger's + hledger-lib's unit tests)
   hledger-ui
   hledger-web	apptest	hledger-web's server tests

hledger-web's tests

hledger-web has two test suites: server tests, which check what the server sends, and browser tests, which check what a real browser does with it.

So a change to what the server sends (a header, a page's HTML) is covered by the server tests, and a change to hledger.js or to how pages behave in a browser needs the browser tests.

Coverage

This means how thoroughly the code is tested - both in breadth (are all parts of the code tested at least a little ?) and in depth (are all possible code paths, states, situations tested ?).

Our current test coverage can be summarised like so:

There are ways to generate detailed coverage reports for haskell unit tests, at least. It would be useful to set this up for hledger.

How to run tests

See Developer workflows for the main commands (just functest, just doctest, just test, stack test etc.), and just h test for all test-related scripts.

Also, the unit tests are built in to the hledger executable, and can be run any time with the --test flag:

$ hledger --test             # run all unit tests
$ hledger --test=balance     # run tests with "balance" in their name