Testing & Evaluation

The goal of testing is to ensure that the software that we produce meets our objectives when deployed into the environment in which it will be used, and when faced with real-world contraints.

Objective Test components and systems together
Activities Produce a testing strategy to ensure completeness; produce integration and system tests.
Outcome Test strategy; integration tests and system tests.

The goal of testing is to ensure that the software that we produce meets our objectives when deployed into the environment in which it will be used, and when faced with real-world constraints.

Why do we test?

We consider testing to be an integral part of software production. Why do we test?

  1. To gain confidence in the correctness of your results.
  2. To gain confidence that you are handling edge cases and errors properly, which will result in a better user experience.
  3. To produce an improved design, usually as a by-product of having written tests.

As a matter of course, it’s also concerned with identifying deficiencies and flaws in software, and helping to determine how they should be addressed. In an iterative model, testing impacts implementation (by providing iterative feedback during the implementation process), and by providing input into planning phases, where we may need to fix bugs or plan product changes.

How do we test?

We can identify three types of testing:

  1. Unit testing: tests operating at the class level (or smallest functional unit), which are meant to check the validity of low-level interfaces and systems.
  2. Integration testing: testing across multiple classes or functional units, to check interaction between objects.
  3. Functional testing: this is testing functionality from the perspective of the user; end-to-end feature testing. Sometimes called System testing.

Types of testing Types of testing

Unit tests are particularly important and are meant to exercise the interface of a single class or module.

  • Unit tests should be very quick to execute and report results.
  • They should be restricted in scope to a class or entity, and dependencies should be eliminated for testing.
  • They should be integrated into our development workflow, so that they are routinely executed (e.g. prior to commits).

Unit tests should always include “black box” tests. You should always test the interface by examining expected/actual results from a given input. NEVER write tests that rely on knowledge of the inner workings of a class. You should be able to interchange classes with identical interfaces and expect identical results.

Info

Software entropy is the tendency of software to become slower and more error prone over time as we continue to make changes. Unit testing helps overcome software entropy by ensuring that what worked before continues to work now.

We also need to want to confirm that the software works in the environment where it will be deployed. We may need to have multiple staged configurations where we will perform integration tests, to confirm that various systems work together, and functional tests to check user-level correctness.

When should we test?

Traditional views were that testing should be done after implementation. This is costly. Testing is more useful when done earlier in the process.

Different tests are suitable for different parts of the development process:

  • Unit Tests: done during implementation, when you are working on a class.
  • Integration Tests: done during implementation, when you want to ensure that classes work together.
  • System Tests: done when features are complete and merged, to ensure that the system continues working.

When to test? When to test?

In the previous chapter, we discussed Unit Testing and the value of writing tests as you develop (TDD). We’ll continue talking about testing that is done post-implementation. Specifically, let’s discuss Integration testing.

Info

We used junit for unit testing and we’ll also use it for integration testing. Although it’s described as a unit-testing framework, JUnit is perfect suitable for doing broader scoped tests, as we’ll see.

Resources

Martin Fowler. 2006. TestDouble. https://martinfowler.com/bliki/TestDouble.html

Lisa Crispin, Janet Gregory. 2008. Agile Testing: A Practical Guide for Testers and Agile Teams. Addison-Wesley Professional. ISBN 978-0321534460.

Vladimir Khorikov. 2020. Unit Testing Principles, Practices, and Patterns. Manning Publishing. ISBN 978-1617296227.

Glenford J. Myers, Corey Sandler, Tom Badgett. 2011. The Art of Software Testing. Wiley. ISBN 978-1118031964.