Testing

When testing, we execute a set of test cases. A test case specifies how to perform a test. At a minimum, it specifies the input to the software under test (SUT) and the expected behavior.

Test cases can be determined based on the specification, reviewing similar existing systems, or comparing to the past behavior of the SUT.

For each test case we do the following:

1. Feed the input to the SUT
2. Observe the actual output
3. Compare actual output with the expected output

A test case failure is a mismatch between the expected behavior and the actual behavior. A failure indicates a potential defect (or a bug), unless the error is in the test case itself.

Testability is an indication of how easy it is to test an SUT. As testability depends a lot on the design and implementation. You should try to increase the testability when you design and implement a software. The higher the testability, the easier it is to achieve a better quality software.

A simple way to semi-automate testing of a CLI(Command Line Interface) app is by using input/output re-direction.

• First, we feed the app with a sequence of test inputs that is stored in a file while redirecting the output to another file.
• Next, we compare the actual output file with another file containing the expected output.

Let us assume we are testing a CLI app called AddressBook. Here are the detailed steps:

1. Store the test input in the text file input.txt.

   add Valid Name p/12345 valid@email.butNoPrefixadd Valid Name 12345 e/valid@email.butPhonePrefixMissing

2. Store the output we expect from the SUT in another text file expected.txt.

   Command: || [add Valid Name p/12345 valid@email.butNoPrefix]Invalid command format: add ​Command: || [add Valid Name 12345 e/valid@email.butPhonePrefixMissing]Invalid command format: add 

3. Run the program as given below, which will redirect the text in input.txt as the input to AddressBook and similarly, will redirect the output of AddressBook to a text file output.txt. Note that this does not require any code changes to AddressBook.

   java AddressBook < input.txt > output.txt

   • The way to run a CLI program differs based on the language.
     e.g., In Python, assuming the code is in AddressBook.py file, use the command
     python AddressBook.py < input.txt > output.txt

   • If you are using Windows, use a normal command window to run the app, not a Power Shell window.

   More on the > operator and the < operator extra

   A CLI program takes input from the keyboard and outputs to the console. That is because those two are default input and output streams, respectively. But you can change that behavior using < and > operators. For example, if you run AddressBook in a command window, the output will be shown in the console, but if you run it like this,

   java AddressBook > output.txt 

   the Operating System then creates a file output.txt and stores the output in that file instead of displaying it in the console. No file I/O coding is required. Similarly, adding < input.txt (or any other filename) makes the OS redirect the contents of the file as input to the program, as if the user typed the content of the file one line at a time.

   Resources:

   • Using command redirection operators in Windows

4. Next, we compare output.txt with the expected.txt. This can be done using a utility such as Windows FC (i.e. File Compare) command, Unix diff command, or a GUI tool such as WinMerge.

   FC output.txt expected.txt

Note that the above technique is only suitable when testing CLI apps, and only if the exact output can be predetermined. If the output varies from one run to the other (e.g. it contains a time stamp), this technique will not work. In those cases we need more sophisticated ways of automating tests.

A test driver is the code that ‘drives’ the SUT for the purpose of testing i.e. invoking the SUT with test inputs and verifying the behavior is as expected.

public class PayrollTestDriver {    public static void main(String[] args) throws Exception {​        //test setup        Payroll p = new Payroll();​        //test case 1        p.setEmployees(new String[]{"E001", "E002"});        // automatically verify the response        if (p.totalSalary() != 6400) {            throw new Error("case 1 failed ");        }​        //test case 2        p.setEmployees(new String[]{"E001"});        if (p.totalSalary() != 2300) {            throw new Error("case 2 failed ");        }​        //more tests...​        System.out.println("All tests passed");    }}

JUnit is a tool for automated testing of Java programs. Similar tools are available for other languages and for automating different types of testing.

@Testpublic void testTotalSalary(){    Payroll p = new Payroll();​    //test case 1    p.setEmployees(new String[]{"E001", "E002"});    assertEquals(6400, p.totalSalary());​    //test case 2    p.setEmployees(new String[]{"E001"});    assertEquals(2300, p.totalSalary());​    //more tests...}

Most modern IDEs has integrated support for testing tools. The figure below shows the JUnit output when running some JUnit tests using the Eclipse IDE.

If a software product has a GUI component, all product-level testing (i.e. the types of testing mentioned above) need to be done using the GUI. However, testing the GUI is much harder than testing the CLI (command line interface) or API, for the following reasons:

• Most GUIs can support a large number of different operations, many of which can be performed in any arbitrary order.
• GUI operations are more difficult to automate than API testing. Reliably automating GUI operations and automatically verifying whether the GUI behaves as expected is harder than calling an operation and comparing its return value with an expected value. Therefore, automated regression testing of GUIs is rather difficult.
• The appearance of a GUI (and sometimes even behavior) can be different across platforms and even environments. For example, a GUI can behave differently based on whether it is minimized or maximized, in focus or out of focus, and in a high resolution display or a low resolution display.

Moving as much logic as possible out of the GUI can make the GUI testing easier. That way, you can bypass the GUI to test the rest of the system using automated API testing. While this still requires the GUI to be tested, the number of such test cases can be reduced as most of the system has been tested using automated API testing.

There are testing tools that can automate GUI testing.

Test coverage is a metric used to measure the extent to which testing exercises the code i.e., how much of the code is 'covered' by the tests.

Here are some examples of different coverage criteria:

• Function/method coverage : based on functions executed e.g., testing executed 90 out of 100 functions.
• Statement coverage : based on the number of line of code executed e.g., testing executed 23k out of 25k LOC.
• Decision/branch coverage : based on the decision points exercised e.g., an if statement evaluated to both true and false with separate test cases during testing is considered 'covered'.
• Condition coverage : based on the boolean sub-expressions, each evaluated to both true and false with different test cases. Condition coverage is not the same as the decision coverage.

• Path coverage measures coverage in terms of possible paths through a given part of the code executed. 100% path coverage means all possible paths have been executed. A commonly used notation for path analysis is called the Control Flow Graph (CFG).
• Entry/exit coverage measures coverage in terms of possible calls to and exits from the operations in the SUT.

Measuring coverage is often done using coverage analysis tools. Most IDEs have inbuilt support for measuring test coverage, or at least have plugins that can measure test coverage.

Coverage analysis can be useful in improving the quality of testing e.g., if a set of test cases does not achieve 100% branch coverage, more test cases can be added to cover missed branches.

Dependency injection is the process of 'injecting' objects to replace current dependencies with a different object. This is often used to inject stubs to isolate the SUT from its dependencies so that it can be tested in isolation.

class Logic {    Storage s;​    Logic(Storage s) {        this.s = s;    }​    String getName(int index) {        return "Name: " + s.getName(index);    }}

interface Storage {    String getName(int index);}

class DatabaseStorage implements Storage {​    @Override    public String getName(int index) {        return readValueFromDatabase(index);    }​    private String readValueFromDatabase(int index) {        // retrieve name from the database    }}

Logic logic = new Logic(new DatabaseStorage());String name = logic.getName(23);

@Testvoid getName() {    Logic logic = new Logic(new DatabaseStorage());    assertEquals("Name: John", logic.getName(5));}

class StorageStub implements Storage {​    @Override    public String getName(int index) {        if(index == 5) {            return "Adam";        } else {            throw new UnsupportedOperationException();        }    }}

@Testvoid getName() {    Logic logic = new Logic(new StorageStub());    assertEquals("Name: Adam", logic.getName(5));}

A Foo object normally depends on a Bar object, but we can inject a BarStub object so that the Foo object no longer depends on a Bar object. Now we can test the Foo object in isolation from the Bar object.

Polymorphism can be used to implement dependency injection, as can be seen in the example given in [Quality Assurance → Testing → Unit Testing → Stubs] where a stub is injected to replace a dependency.

class Logic {    Storage s;​    Logic(Storage s) {        this.s = s;    }​    String getName(int index) {        return "Name: " + s.getName(index);    }}

interface Storage {    String getName(int index);}

class DatabaseStorage implements Storage {​    @Override    public String getName(int index) {        return readValueFromDatabase(index);    }​    private String readValueFromDatabase(int index) {        // retrieve name from the database    }}

Logic logic = new Logic(new DatabaseStorage());String name = logic.getName(23);

@Testvoid getName() {    Logic logic = new Logic(new DatabaseStorage());    assertEquals("Name: John", logic.getName(5));}

class StorageStub implements Storage {​    @Override    public String getName(int index) {        if(index == 5) {            return "Adam";        } else {            throw new UnsupportedOperationException();        }    }}

@Testvoid getName() {    Logic logic = new Logic(new StorageStub());    assertEquals("Name: Adam", logic.getName(5));}

class Payroll {    private SalaryManager manager = new SalaryManager();    private String[] employees;​    void setEmployees(String[] employees) {        this.employees = employees;    }​    void setSalaryManager(SalaryManager sm) {       this. manager = sm;    }​    double totalSalary() {        double total = 0;        for(int i = 0;i < employees.length; i++){            total += manager.getSalaryForEmployee(employees[i]);        }        return total;    }}​​class SalaryManager {    double getSalaryForEmployee(String empID){        //code to access employee’s salary history        //code to calculate total salary paid and return it    }}

class PayrollTest {    public static void main(String[] args) {        //test setup        Payroll p = new Payroll();        p.setSalaryManager(new SalaryManagerStub()); //dependency injection        //test case 1        p.setEmployees(new String[]{"E001", "E002"});        assertEquals(2500.0, p.totalSalary());        //test case 2        p.setEmployees(new String[]{"E001"});        assertEquals(1000.0, p.totalSalary());        //more tests ...    }}​​class SalaryManagerStub extends SalaryManager {    /** Returns hard coded values used for testing */    double getSalaryForEmployee(String empID) {        if(empID.equals("E001")) {            return 1000.0;        } else if(empID.equals("E002")) {            return 1500.0;        } else {            throw new Error("unknown id");        }    }}

Test-Driven Development(TDD)_ advocates writing the tests before writing the SUT, while evolving functionality and tests in small increments. In TDD you first define the precise behavior of the SUT using test cases, and then write the SUT to match the specified behavior. While TDD has its fair share of detractors, there are many who consider it a good way to reduce defects. One big advantage of TDD is that it guarantees the code is testable.

Note that TDD does not imply writing all the test cases first before writing functional code. Rather, proceed in small steps:

1. Decide what behavior to implement.
2. Write test cases to test that behavior.
3. Run those test cases and watch them fail.
4. Implement the behavior.
5. Run the test case.
6. Keep modifying the code and rerunning test cases until they all pass.
7. Refactor code to improve quality.
8. Repeat the cycle for each small unit of behavior that needs to be implemented.