Testing

Software companies can spend a lot of time and money developing their software.  They cannot afford to release poor quality or flawed software. It is therefore critical that software is tested thoroughly before release.

It seems a little obvious and simplistic to suggest that testing is only necessary to identify errors.  Testing is the process used to quality assure computer software.  In particular, it checks that software is fit for purpose – i.e. it satisfies the specification. Broadly speaking, this involves evaluating functionality and performance:

  • Functionality – Does the software include all the features that it’s supposed to?
  • Performance – Is the software fast, reliable, robust and easy to use?

Systematic Testing

Systematic testing strategies often reflect the hierarchical structure of the software and adopt a ‘bottom-up‘ approach. The smallest components are ‘unit tested’ in isolation as they are developed. At the next level up, larger components are tested to check that the smaller components which form them integrate properly. This process is repeated up the successive levels until the system is tested as a whole entity.

Systematic testing involves the creation of a test plan which specifies the testing activities to be undertaken throughout different phases of the development cycle.

Creating a  Test Plan

Shortly after the software specification is produced (early in the design phase) a test plan is written. It details every aspect of the testing activities to be undertaken during and after the development of the application. It typically includes the following:

  • Type of test
  • Input of test
  • Expected output
  • Actual output

Comprehensive Testing

Ensures that all (functional) requirements of the program are tested

It is impossible to test a piece of software under every possible condition and input.  However, it is possible to devise an extensive set of test cases which are representative of all conditions.

This example shows the test table that you could create for a program that calculates the grade that you would get when you input a test mark.

A pass        70 and 100
B pass        60 – 69
C pass        50 – 59
Fail              0 – 49

Debugging

No matter how careful you are when you write program code, your programs are likely to have errors, or bugs, that prevent them from running the way you intended. Debugging is the process of locating and fixing errors in programs.

Program errors can be classified in the following way:

Syntax errors

Syntax errors are ‘mistakes’ in the grammatical syntax of the code.  In simple terms, the source code has not been written in a way that conforms to the strict rules of the language. Syntax errors are ‘flagged up’ during translation from source code to executable binary code (or object code).

Common examples of syntax errors are:

  • missing punctuation & brackets
  • misspelling of reserved commands and function names
  • unclosed constructs like ifs & loops
  • wrong number or type of parameters.

Execution Errors

Also known as run-time errors, these are only detected during program execution. They cause abnormal execution or termination of the program.  Example are

  • division by a variable whose value is 0
  • trying to access an array entry which doesn’t exist (“list index out of range” in Python)
  • a built-in function whose parameter is the wrong type e.g. int(num)
  • where the parameter number is a text value entered by the user
  • insufficient space on computer RAM
  • file path when reading a file does not exist
  • when writing to a file, a file with the same name already exists
  • attempting to open a file when already open

Logic Errors

Also known as semantic errors, these are the hardest to detect.  Logic errors result in the program functioning incorrectly such as producing a result for a complex calculation which is not correct.

These errors are usually the result of poor design and often require rigorous testing to detect. Dry runs at the design stage can help prevent many logic errors.

Debugging Tools

There are a variety of techniques used to identify execution and logic errors.  Some are manual methods used to check the design, while others use debugging tools available within the software development environment.

Manual Methods: Dry Runs and Trace Tables

Dry run testing is usually a ‘paper and pencil’ exercise carried out to identify logic errors.  The process involves manually stepping through an algorithm using purposely chosen sample data to record the values of variables in a trace table as the algorithm unfolds. No software is involved in this process. Dry runs are most suited to testing smaller components of the software at the design phase before they are coded.

Trace tables are frequently used when conducting dry runs on program components. A trace table is used to record the values of variables throughout the flow of an algorithm. Each column in the table represents a different variable and each row records the value of each variable at a certain point in program flow.  Trace tables are particularly useful for seeing how variables change through iterations of a loop.

Environment Debugging Tools: Trace Facility, Breakpoints and Watches

Many programming environments provide a variable trace facility. While dry runs and trace tables are useful at the design stage, a trace facility is very useful to help when debugging code. The program environment does the work for you in keeping track of every variable as the program executes.  You can step through each line of code and check local and global variable values before the program executes the next line.

Breakpoints provide a good way of being more precise about what areas of code to examine with the variable trace facility. A breakpoint is a marker placed at a line of code in a program to halt execution at that point. Breakpoints can be placed where a programmer suspects there is a bug. Placing a breakpoint on the first line inside a loop for example is a good way of examining how the variables being manipulated inside the loop change after each loop repetition.

A watchpoint is a way of targeting a specific variable to keep track of during program execution. The simplest watchpoint could be set up to halt execution when the specified variable changes. The code which caused that change gets highlighted so it can be examined. Alternatively a conditional watchpoint could be set to halt execution when the variable value matches the condition that has been set.  The old way of setting a watchpoint was to insert a print command to ‘echo’ the value of a variable.