Logic Testing

Overview

The McCabe Cyclomatic Complexity metric and resulting control flow graphs describe the minimum set of tests to cover logic combinations in the code.  The control flow graphs depict the logical structure of the analyzed code.

Test plans are decomposed into a set of test cases. To be valid, each test case must have an expected result that is used to validate against actual results during the test suite's execution.

Some clarification:

• The unit test plan defines modules, which may be functions, routines, or files.

• White box testing – directly testing code constructs.

• White box tests from easiest to most extreme:

  • Statement – not safe

  • basis path – fail-safe, standard requirement for ISO 9001 and CMM.

  • Path – all independent logic combinations of the program from beginning to end are tested. With rigor, this may exercise edge cases.

  • Branch – validating all resulting outcomes for each logic statement.

  • Multiple Condition Decision Coverage (MC/DC) – conditions are atomic and cover all the logic statement outcomes.  The logic statements are the decisions.  The highest level of safety in automotive (ASIL 4) requires this testing since it ensures that each condition in a decision has been tested independently. MC/DC testing allows only one change between test cases. (hint: truth tables ensure one change at a time).

Basis Path Testing:

The McCabe Flow Graphs result in the minimum set of white box tests to exercise logic. Each test case uniquely exercises a basis path. For example, a complexity of 10 has ten unique test cases, each exercising a unique basis path.

Unit Testing

·         Exercises logic combinations within a module

·         Validates test cases within a module by matching actual against expected results for each test case.

·         A 100% unit test coverage of the basis paths results in fail-safe code for the module under test.

Integration Testing

·         Exercises logic between modules, resulting in inter-module communication.

·         Will exercise logic within a module given the set of conditions from the calling module.

·         Uncover issues between modules.

·         Uncover potential software dead code

o   Test coverage results in code that is not executed.

o   This is typically caused by a logic test in the calling module duplicating the logic test for the dead code segment.

·         A 75% integration test coverage of basis paths results in fail-safe code for the modules under test.  This includes all logic between and within modules under test.

Summary

·         Basis path tests guarantee that all statement tests are executed. However, statement tests do not guarantee that all logic tests are performed.

·         Statement execution is a light test with no logic testing intent.  In practice, I have seen very few statement tests with expected results.

·         Exercising statements may check algorithms for accuracy, but these test cases must include expected results.

·         Logic mistakes cause most failures at the unit test level. Therefore, my white box testing is driven by logic testing.

·         Logic combinations – interaction between logic statements.  The Basis Path testing is an example of complete test coverage for these constructs.

o   Unit tests – logic contained within a module.

o   Integration tests – logic between modules and including all internal module logic.

·         Safe modules are attained by having 100% unit test basis path coverage and 75% integration basis path coverage.

·         After performing basis path testing, I have never experienced a logic failure with subsequent tests (e.g., feature, function, requirements…). 

·         Therefore, I would claim it is highly improbable that a blocker or severe logic bug would be uncovered in subsequent tests.

For further information or discussion, please book a consultation.