Software testing

What is software testing?

What is software testing?

🤔 What is the actual motivation for testing a code?

  • Compare results with expectations
  • Detect unknown bugs
  • Prevent known bugs to reappear
  • Secure code changes
What is software testing?

Tests should run ...

... automaticallyon every push to the repository
... quicklywithin seconds
... regularlyevery night
... reproduciblywith constant input and output
... realiablyin the same stable environment
Testing strategies

Testing strategies

There is an enormous number of different testing strategies.

💡 Many of them can be applied to scientific software development.

Testing strategies

Decades ago: Manual testing

  1. Implement a new feature
  2. Run the software
  3. Save the result as a reference
  4. Compare with the reference result next time

👎 Cumbersome and unreliable

Testing strategies

Important testing strategies

Functional testing checks what the software produces

  • Unit tests
  • Integration tests
  • Regression tests

Non-functional testing checks how the software obtains its results

  • Performance tests
  • Scaling tests
Testing strategies

Unit tests

Bottom-up idea: Test every function with various input and compare the output with expected results.

from numpy import pi
from pytest import approx

def surface(radius, dimension):
    if dimension == 2 and radius > 0:
        return 2 * pi * radius
    elif dimension == 3:
        return 4 * pi * radius**2
    return 0

def test_surface():
    assert surface(2.0, 2) == approx(4 * pi)
Testing strategies

Off topic

assert surface(2.0, 2) == approx(4 * pi)

🤔 How is approx implemented to provide "nearly equal" here?

  • approx returns an object that implements __eq__
  • float returns NotImplemented and is thus skipped
Testing strategies

Code coverage

Code coverage is the amount of code executed by tests vs. the total amount of code.

  • Line coverage: executed code lines
  • Branch coverage: reached if clauses, for loops, ...
  • Function coverage: called functions

✅ Try to reach 100% code coverage!

Testing strategies
from numpy import pi
from pytest import approx

def surface(radius, dimension):
    if dimension == 2 and radius > 0:
        return 2 * pi * radius
    elif dimension == 3:
        return 4 * pi * radius**2
    return 0

def test_surface():
    assert surface(2.0, 2) == approx(4 * pi)
    assert surface(-2.0, 2) == 0
    assert surface(2.0, 3) == approx(16 * pi)
    for dimension in [0, 1, 4]:
        assert surface(2.0, dimension) == 0
Testing strategies

⚠️ Be careful with your test cases:

from numpy import pi
from pytest import approx

def surface(radius, dimension):
    if dimension == 2 and radius > 0:
        return 2 * pi * radius
    elif dimension == 3:
        return 4 * pi * radius**3    # Bug here!
    return 0

def test_surface():
    assert surface(1.0, 1) == 0
    assert surface(1.0, 2) == approx(2 * pi)
    assert surface(1.0, 3) == approx(4 * pi)

💡 100% code coverage does not guarantee the absence of bugs!

Testing strategies

Corner cases

Think of possible corner cases in your unit tests:

def int_from_user_input(input):
    try:
        return int(input)
    except ValueError:
        return 0

def test_int_from_user_input():
    assert int_from_user_input("7") == 7
    assert int_from_user_input("-7") == -7
    assert int_from_user_input("7\n") == 7

    assert int_from_user_input("- 7") == 0
    assert int_from_user_input("7.0") == 0
    assert int_from_user_input("7,0") == 0
    assert int_from_user_input("seven") == 0
Testing strategies

Treat test code as production code

Invest time to maintain your test code! Violation of the DRY principle may be acceptable:

def test_int_from_user_input():
    assert int_from_user_input("7") == 7
    assert int_from_user_input("7\n") == 7

    assert int_from_user_input("- 7") == 0
    assert int_from_user_input("7.0") == 0
    assert int_from_user_input("7,0") == 0
    assert int_from_user_input("seven") == 0

instead of

def test_int_from_user_input():
    for input in ["7", "7\n"]:
        assert int_from_user_input(input) == 7

    for input in ["- 7", "7.0", "7,0", "seven"]:
        assert int_from_user_input(input) == 0
Testing strategies

Testability

If your code does not respect the SOC principle, it will be hard to test:

from numpy import pi
from pytest import approx

class Circle:
    def __init__(self, radius):
        self.radius = radius
        self.previous_area = 0

    def area(self):
        result = pi * self.radius**2
        if result > self.previous_area:
            self.previous_area = result
        return self.previous_area

def test_circle():
    circle = Circle(2.0)
    assert circle.area() == approx(4 * pi)
    circle.radius = 1.0
    assert circle.area() == approx(pi)   # AssertionError

💡 If you fail creating a unit test, maybe the reason is bad code design.

Testing strategies

Typical steps:

  1. Implement a new feature
  2. Write tests for it that pass

🤔 What about writing the tests first?

Testing strategies

Test-driven development

  1. Write tests ("specification") for the new feature that all fail
  2. Implement the feature
  3. Run all tests
  4. Go back to 2. until all tests pass

💡 Test-driven development enforces detailed planning: What are the input and output of the function, which corner cases exist, what happens in case of errors?

Testing strategies

💡 Interpret a unit test as a specification of the function!

Testing strategies

Integration tests

Not all components can be tested with unit tests:

  • File input/output
  • Console input/output
  • Parallelization (Threading, MPI)
  • Library calls

Integration tests or system tests run the whole code in a defined environment with realistic input data.

Testing strategies

🤔 Can I omit unit tests and focus on integration tests?

No! Integration tests can never be tuned to call all statements in the code.
If an integration test fails, you will have a hard time to find the root cause.

Testing strategies

Regression tests

For every bug in your code:

  1. Write a failing integration test and/or unit test
  2. Fix the bug
  3. Observe the regression test pass

✅ Build up a test suite of regression tests as you proceed extending your code.

Pytest

Pytest

Unit tests are often a collection of assert statements.

Good practice:

  • Put unit tests into separate files, but close to the production code
  • Use a testing framework to run the tests automatically

💡 Pytest can detect and execute tests:

  • File names must be test_* or *_test
  • Function names must have the prefix test
Pytest

How to run Pytest

pip install pytest

Detect and run all tests in all subdirectories:

pytest .

Additional plugins are available to extend the functionality.

Example: Print the test coverage

pip install pytest-cov

pytest --cov .
Pytest

Example

def function_with_return():
    return 0.123

def function_with_exception():
    raise RuntimeError

import pytest

def test_function_with_return():
    assert function_with_return() == pytest.approx(0.123)

def test_function_with_exception():
    with pytest.raises(RuntimeError):
        function_with_exception()
Pytest

Fixtures

✅ Use fixtures to generate test input (cf. Factory pattern)

class Bird:
    def can_fly(self):
        return True

import pytest
from bird import Bird

@pytest.fixture(scope="module")
def bird():
    return Bird()

def test_can_fly(bird):
    assert bird.can_fly()
Pytest

Fixtures

Pytest provides several built‑in fixtures, such as tmp_path and tmp_path_factory (see the hints for Milestone 1 in today’s exercise).

Additional fixtures can also be made available via external plugins (e.g., the mocker fixture from the pytest‑mock plugin).

Pytest

Mocking and spying

Unit testing sometimes requires replacing an object or function with a mock.
A mock implements only what is needed to let the unit test pass.

def get_radius(circle):
    return circle.radius

import pytest

class CircleMock:
    radius = 10

@pytest.fixture
def circle():
    return CircleMock()   # Use mock here instead of full Circle class

def test_get_radius(circle):
    assert get_radius(circle) == 10

A spy can record all calls of a certain function. The original code is still being executed.

Pytest

Mocking and spying

pip install pytest-mock

class Bird:
    def __init__(self, species):
        self.species = species

    def print_species(self):
        print(self.species)

    def identify(self):
        self.print_species()

import pytest
from bird import Bird

@pytest.fixture
def bird():
    return Bird('puffin')

def test_identify(mocker, bird):
    mocker.patch('builtins.print')            # Replace print function with a no-op
    spy = mocker.spy(bird, 'print_species')   # Record all calls of print_species
    bird.identify()
    spy.assert_called_once()
    print.assert_called_once_with('puffin')
Unittest

Unittest

  • Built-in testing framework unittest is an alternative to Pytest
  • More verbose syntax: boilerplate code and test classes
  • Otherwise fine for typical Python projects

💡 See this blog post for a comparison

class Bird:
    def can_fly(self):
        return True

import unittest
from bird import Bird

class TestBird(unittest.TestCase):
    def setUp(self):
        self.bird = Bird()

    def test_can_fly(self):
        self.assertTrue(self.bird.can_fly())

if __name__ == '__main__':
    unittest.main()
Further reading

Further reading

https://www.geeksforgeeks.org/types-software-testing/
https://www.geeksforgeeks.org/how-to-use-pytest-for-unit-testing/
https://media.readthedocs.org/pdf/pytest/latest/pytest.pdf
https://www.geeksforgeeks.org/difference-between-pytest-and-unittest/