# Testing Strategies

We can characterize tests in terms of the type of testing, types of tests and phase where they are executed (see Types of Automation Testing: A Beginner’s Guide).

Types of Testing

There are two distinct types of requirements that we can identify when specifying a project: functional requirements, related to features or product requirements, and non-functional requirements or qualities of the system that we can measure.

• Functional: test the features and functonality; determine usability, fit-for-purpose.
• Non-functional: examine qualities of a system: performance, scalability, robustness.

When testing applications, both of these are valid. You should be measuring against the initial requirements that you defined for this product, to ensure that requirements are being met.

Types of Tests

There are different purposes to run tests.

• Smoke tests: functional tests only cover crucial functionality; a quick "sanity check" to ensure that the build ran, for instance.
• Integration tests: tests interrelated functionalities to ensure that they work together.
• Regression tests: tests that ensure that software hasn't gotten "worse" over time (degraded performance, or functionality no longer working).
• Security tests: test the system for any vulnerabilities.
• Performance tests: non-functional tests that evaluate measurable criteria like performance, stability.
• Acceptance tests: functional tests that must be met for software to meet the minimum acceptable bar for users.

Phase of Testing

Finally, there are different areas of testing.

• Unit test: small functional tests, designed to check behaviour at the class-level.
• API test: test the "outer edges" of your architecture; check that APIs work as expected.
• UI test: check that interaction with the user interface functions properly. May be done by-hand, or automated (see mocking).

Are we expected to write all of these tests?!? No, at least not in this course.

You are expected to write a reasonable subset: unit tests and API tests for your architecture. You should be running these tests every time you build, so they can be considered both smoke tests and integration tests (if you run after integrating branches).

We’re going to use JUnit, a popular testing framework to create and execute tests. It can be installed in number of ways: directly from the JUnit home page, or one of the many package managers for your platform.

We will rely on Gradle to install it for us as a project dependency. Include this line in the dependencies of your build.gradle.kts configuration file.

dependencies {
    testImplementation('org.jetbrains.kotlin:kotlin-test')
}

Unit testing involves writing low-level, class-specific tests.

A unit test is a test that meets the following three requirements [Khorikov 2020]:

• Verifies a single unit of behavior,
• Does it quickly, and
• Does it in isolation from other tests.

Unit tests should target classes or other low-level components in your program. They should be small, very focused, and quick to execute and return results.

Testing in isolation means removing the effects of any dependencies on other code, libraries or systems. This means that when you test the behaviour of your class, you are assured that nothing else is contributing to the results that you are observing.

This is also why designing cohesive, loosely coupled classes is critical: it makes testing them so much easier if they work independently!

Unit tests are just Kotlin functions that execute and check the results returned from other functions. Ideally, you would produce one unit or more unit tests for each function. You would then have a set of unit tests to check all of the methods in a class, and multiple sets of units tests to cover all of your implementation classes.

Your goal should be to have unit tests for every critical class and function that you produce.

You will NOT likely get 100% code coverage. You should focus on testing entities/data models, and related features. You do NOT have to unit test user interfaces directly (there are ways to do it, but they are outside-scope of this course).

Test-Driven Development (TDD) is a strategy introduced by Kent Beck, which suggests writing tests first, before you start coding. You write tests against expected behaviour, and then write code which works without breaking the tests. TDD suggests this process:

1. Think about the function (or class) that you need to create.
2. Write tests that describe the behaviour of that function or class. As above, start with valid and invalid input.
3. Your test will fail (since the implementation code doesn’t exist yet). Write just enough code to make the tests pass.
4. Repeat until you can’t think of any more tests to write.

Running out of tests means that there is no more behaviour to implement... and you’re done, with the benefit of fully testable code.

https://medium.com/@tunkhine126/red-green-refactor-42b5b643b506

There are some clear benefits to testing early:

• Early bug detection. You are building up a set of tests that can verify that your code works as expected.
• Better designs. Making your code testable often means improving your interfaces, having clean separation of concerns, and cohesive classes. Testable code is by necessity better code.
• Confidence to refactor. You can make changes to your code and be confident that the tests will tell you if you have made a mistake.
• Simplicity. Code that is built up over time this way tends to be simpler to maintain and modify.

Although you aren't required to use TDD, you are required to write extensive working unit tests!

A typical unit test uses a very particular pattern, known as the arrange, act, assert pattern. This suggests that each unit test should consist of these three parts:

1. Arrange: bring the system under test (SUT) to the starting state.
2. Act: call the method or methods that you want to test.
3. Assert: verify the outcome of the above action. This can be based on return values, or some other conditions that you can check.

Note that your Act section should have a single action, reflecting that it's testing a single behaviour. If you have multiple actions taking place, that's a sign that this is probably an integration test (see below). As much as possible, you want to ensure that you are writing minimal-scope unit tests.

Another anti-pattern is an if statement in a test. If you are branching, it means that you are testing multiple things, and you should really consider breaking that one test into multiple tests instead.

A unit test is just a Kotlin class, with annotated methods that tell the compiler to treat the code as a test. It’s best practice to have one test class for each implementation class that you want to test. E.g. class Main has a test class MainTest. This test class would contain multiple methods, each representing a single unit test.

Tests should be placed in a special test folder in the Gradle directory structure. When building, Gradle will automatically execute any tests that are placed in this directory structure.

Android has two test folders: src/test is used for tests that can run on your development machine, and src/androidTest is used for tests that need to run on an Android device (e.g. the AVD where you are testing).

Your unit tests should be structured to create instances of the classes that they want to test, and check the results of each function to confirm that they meet expected behaviour.

For example, here's a unit test that checks that the Model class can add an Observer properly. The addObserver() method is the test (you can tell because it's annotated with @Test). The @Before method runs before the test, to setup the environment. We could also have an @After method if needed to tear down any test structures.

class ObserverTests() {
    lateinit var model: Model
    lateinit var view: IView

    class MockView() : IView {
        override fun update() {
        }
    }

    @Before
    fun setup() {
        model = Model()
        view = MockView()
    }

    @Test
    fun addObserver() {
        val old = model.observers.count()
        model.addObserver(view)
        assertEquals(old+1, model.observers.count())
    }

Gradle will automatically execute and display the test results when you build your project.

Lets walkthrough creating a test.

To do this in IntelliJ, select a class in the editor, press Alt-Enter, and select "Create Test" from the popup menu. This will create a unit test using JUnit, the default test framwork. There is a detailed walkthrough on the IntelliJ support site.

You can also just create the test files by-hand. Just make sure to save them in the correct location!

The convention is to name unit tests after their class they're testing, with "Test" added as a suffix. In this example, we're creating a test for a Model class so the test is automatically named ModelTest. This is just a convention—you can name it anything that you want.

We've manually added the addition() function. We can add as many functions as we want within this class. By convention, they should do something useful with that particular class.

Below, class ModelTest serves as a container for our test functions. In it, we have two unit tests that will be automatically executed when we built. NOTE that you would need more than these two tests to adequately test this class—this is just an example.

import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Before
import org.junit.Test

class ModelTests {
    lateinit var model: Model

    @Before
    fun setup() {
        model = Model()
        model.counter = 10
    }

    @Test
    fun checkAddition() {
        val original = model.counter
        model.counter++
        assertEquals(original+1, model.counter)
    }

    @Test
    fun checkSubtraction() {
        val original = model.counter
        model.counter--
        assertEquals(original-1, model.counter)
    }

    @After
    fun teardown() {
    }
}

The kotlin.test package provides annotations to mark test functions, and denote how they are managed:

In our test, we call utility functions to perform assertions of how the function should successfully perform.

Tests will be run automatically with gradle build or we can execute gradle test to just execute the tests.

$ gradle test
BUILD SUCCESSFUL in 760ms
3 actionable tasks: 3 up-to-date

Gradle will report the test results, including which tests - if any - have failed.

You can also click on the arrow beside the test class or name in the editor. For example, clicking on the arrow in the gutter would run this addObserver() test.

Tests shouldn’t verify units of code. Rather, they should verify units of behaviour: something that is meaningful for the problem domain and, ideally, something that a business person can recognize as useful. The number of classes it takes to implement such a unit of behaviour is irrelevant. — Khorikov (2020)

Code coverage is a metric comparing the number of lines of code with the number of lines of code that have unit tests covering their execution. In other words, what “percentage” of your code is tested?

This is a misleading statistic at the best of times (we can easily contrive cases where code will never be executed or tested).

TDD would suggest that you should have 100% unit test coverage but this is impractical and not that valuable. You should focus instead on covering key functionality. e.g. domain objects, critical paths of your source code.

One recommendation is to look at the coverage tools in IntelliJ, which will tell you how your code is being executed, as well as what parts of your code are covered by unit tests. Use this to determine which parts of the code should be tested more completely.

See also

• https://www.jetbrains.com/help/idea/code-coverage.html
• https://www.jetbrains.com/help/idea/running-test-with-coverage.html

A unit test is a test that meets these criteria (from the previous chapter):

• Verifies a single unit of behaviour (typically a class),
• Does it quickly, and
• Does this in isolation from other dependencies and other tests.

An integration test is a test that fails to meet one or more of these criteria. In other words, if you determine that you need to test something outside the scope of a unit test, it's considered an integration test (typically because it's integrating behaviours from multiple components). Performance tests, system tests, etc. are all kinds of integration tests.

There are a lot of different nuanced tests that are used in computer science, but we'll focus on building generic integration tests.

Typically, an integration test is one where you leave in selected dependencies so that you can test the combination of classes together. Integration tests are also suitable in cases where it is difficult to completely remove a dependency. This can happen with some critical, external dependencies like an external database.

This diagram demonstrates how unit tests primarily test the domain model, or business logic classes. Integration tests focus on the point where these business logic classes interact with external systems or dependencies.

Note that in this diagram, we're also identifying code that we should not bother testing. Trivial code is low complexity, and typically has no dependencies or external impact, so it doesn't require extensive testing. Overcomplicated code likely has so many dependencies that it's nearly impossible to test - and it should likely be refactored into something similar and more manageable before you attempt to add tests to it.

When discussing unit tests, we suggested that you should focus on core classes and their behaviours. This is reasonable for unit tests.

We can expand this to suggest that you should “check as many of the business scenario’s edge cases as possible with unit tests; use integration tests to cover one happy path, as well as any edge cases that can’t be covered by unit tests.”

A "happy path" in testing is a successful execution of some functionality. In other words, once your unit tests are done, you should write an integration tests that exercises the functionality that a customer would likely exercise if they were using your software with common features and a common workflow. Focus on that first and only add more integration tests once you have the main execution path identified and tested.

The primary purpose of integration tests is to exercise dependencies, so that should be your main goal. Your main integration test (the "happy path test") should exercise all external dependencies (libraries, database etc). If it cannot satisfy this requirement, add more integration tests to satisfy this constraint.

Here's some guidelines for creating integration tests.

1. Make domain model boundaries explicit.

Try to always have an explicit, well-known place for the domain model in your code base. The domain model is the collection of domain knowledge about the problem your project is meant to solve. This can be a separate class, or even a package reserved for the model.

2. Reduce the number of layers of abstraction

Try to have as few layers of indirection as possible. In most backend systems, you can get away with just three: the domain model, application services layer (controllers), and infrastructure layer. Having excessive layers of abstraction can lead to bloated and confusing code.

3. Eliminate circular dependencies

Try to keep dependencies flowing in a single direction. i.e. don't have class A call into class B, and class B call back into class A with results. Circular dependencies like this create a huge cognitive load for the person reading the code, and make testing much more difficult.

https://xkcd.com/2200