June 24,1996
TEXT SUMMARY
------------
THE ART OF SOFTWARE TESTING
Book By: Glenford J. Myers
Summary By : Glenn Francis
Contents:
1. The Psychology & Economics of program testing
2. Program Inspections,Walkthroughs,Reviews
3. Test-Case Design
4. Module Testing
5. High-Order Testing
6. Debugging
7. Test Tools & Techniques
------------------------------------------------------------------------
1.Psychology & Economics of Testing:
----------------------------------
Testing is the process of executing a program with the intent of
finding errors.
Errors are present if a program does not do what it is supposed to
do and also if a program does what it is not supposed to do.
It is not possible to test a program to find ALL errors.Therefore
the tester will have to make assumptions about the program and the
the manner in which TEST CASES will be designed.
1.1 Black-Box Testing: (Data-Driven, Input/Output testing)
-------------------------------------------------------
The tester is interested in finding cases in which the program
does not behave according to its Specifications.
The tester is not concerned about the internal workings of
the black box.
To find all errors Exhaustive Input Tesing is used with
all possible Valid & Invalid inputs.
This is not possible because:
1. One cannot test a program to guarantee
it is error free.
2. Economics or Cost is a factor.
Therefore the Objective is:
To maximize the number of errors found
by performing a finite number of test cases.
1.2 White-Box Testing: (Logic-Driven testing)
------------------------------------------
The tester derives test data from the internal structure of the
program by examining the program's logic and not the specification.
Here Exhaustive Path testing is performed.A tool used here is the
Control-Flow graph.
It fails to detect missing paths and data sensitivity errors.
Exhaustive input testing is superior to Exhaustive path testing,
but both are infeasible.Combining elements of both methods provides
a reasonable strategy.
Testing Principles:
-------------------
1. A necessary part of a test case is a definition of the expected
output or result.
2. A programmer should avoid attempting to test his or her own
program.
3. A programming organization should not test its own programs.
4. Thoroughly inspect the results of each test.
5. Test cases must be written for valid and unexpected as well
as valid and expected input conditions.
6. Examining a program to see if it does not do what it is supposed
to do is only half of the battle.The other half is seeing wether
the program does what it is not supposed to do.
7. Avoid throw-away test cases unless the program is a throw-away
program.
8. Do not plan a testing effort under the tacit assumption that
no errors will be found.
9. The probability of the existence of no more errors in a section
of a program is proportional to the number of errors already
found in that section.
-----------------------------------------------------------------------
2. Program Inspections,Walkthroughs,Reviews : (Human testing methods)
-------------------------------------------------------------------
These involve a team of people and require preparatory work.
Code Inspections:
-----------------
It is a set of procedures and error-detection techniques for
group code reading.
A Team consists of a QA engineer, programmer,software designer,
and software tester.
Steps: 1. Programmer narrates step by step the logic of the program.
2. Program is analyzed with respect to a checklist of
historically common programming errors.
3. Discuss minor problems and fixes that may lead to
major problems and fixes in the program flowchart.
4. The type of errors to look for are: Data reference Errors,
Data declaration errors,Computation Errors,Comparison errors,
Control Flow errors,Interface errors,Input/Output errors.
2. Program Inspections,Walkthroughs,Reviews : (Human testing methods)
--------------------------------------------------------------------
Walkthroughs:
------------
Uninterrupted meeting of one to two hours for a team of three
to five people; experienced programmer,language expert,new programmer,
maintenance programmer,programmer from another project.
The participants must play "Computer" .A small set of test cases,
representative of inputs and expected outputs for the program
and module are implemented.The test data are walked thorough the
logic of the program mentally.And the state of the program is monitored
on paper or the black board.This gets the process of questioning
started.Comments are to be directed to the program and not the
programmer.
Desk Checking:
-------------
It is a one person program inspection or walkthrough.
3. Test-Case Design
------------------
The key issue is: What subset of all possible test cases has the
highest probability of detecting errors.
The methologies are:
-- Black Box Testing
-Equivalence Partitioning
-Boundary-value Analysis
-Cause-effect Graphing
-Error Guessing
-- White Box Testing
-Statement Coverage
-Decision Coverage
-Decision/Condition Coverage
-Multiple-Condition Coverage
The recommended procedure is to develop test cases using black-box
methods and then develop supplementary test cases as necessary
using white-box methods.
White-box testing or Logic-Coverage Testing is concerned with the
logic or source code of the program.Writing enough test caes such
that every statement is executed at least once is a necessary but
not sufficient condition for a reasonable white box test.
Decision coverage requires one to write enough test cases
such that each decision has a true or false outcome at least once.
Condition coverage requires that one write enough test cases so that
each condition in a decision takes on all possible outcomes at least once.
Decision/Condition coverage requires sufficient test cases so that:
-Each condition in a decision takes on all possible
outcomes at least once.
-Each decision takes on all possible outcomes at least once.
-Each point of entry is invoked at least once.
Multiple-condition coverage requires that one write sufficient test
cases so that all possible combinations of condition outcomes
in each decision and all points of entry are invoked at least once.
Black-box testing:
Equivalence partitioning: (Input/Output equivalences)
-Reduces the number of test cases to be developed.And it covers
a large set of possible test cases to tell us about the presence
or absence of errors above the specific input set of values.
Boundary-value Analysis: (Over,Beneath,and on edge of I/O equivalences)
-Select one or more elements so that it is on the edge of the
equivalence class.Consider both input and output equivalences.
Cause-Effect graphing:
Selects a high yield set of test cases.It points out incompleteness,
and ambiguities in a specification.The specification is divided
into workable pieces.A cause is an input condition, and an effect
is an output condition.The specification is transformed into a
Boolean graph or cause-effect graph.
THE OVERALL STRATEGY: (Reasonable Strategy)
=====================
This combines various methologies into a set of useful test cases.
Points to consider when doing this are:
- For a combination of input conditions start with cause-effect
graphing.
- Use boundary value analysis for input/output boundaries to yield
a set of supplementary test cases.
- Identify a set of valid and invalid equivalences for input/output,
and supplementary test cases.
- Use error guessing to add additional test cases.
- Examine the program's logic and use either, decision coverage,
condition coverage, decision/condition coverage or multiple
condition coverage.
- If the coverage condition has not been satisfied, add sufficient
test cases to met this criteria.
4. Module Testing : (Unit Testing)
--------------------------------
This is a process of testing the individual subprograms,subroutines,
and procedures within a program.
For this two types of information must be available:
- A specification for the Module
- Source Code for the Module
This is largely white-box testing.And the first step is to list
the conditional decisions in the program.The last step is to
explore multi-condition coverage and have sufficient number of
test cases for all possible combinations of conditions evoked
at least once.
Two approaches to module testing are incremental and non-incremental
testing.Incremental testing involves either Top Down or Bottom Up testing.
Testing each module independently is non-incremental testing.While
combining each module prior to test is incremental testing.
Testing each module requires a special Driver Module and one or
more Stub modules.
5. High-Order Testing
--------------------
A software error is present when the program does not do what
its end user reasonably expects it to do.
The Testing Cycle is structured to model the software development cycle.
It is applicable to contract software products that are intended
for wide usage.
5.1 Function Testing : Attempts to find discrepencies between a
program and its external specification.
5.2 System Testing : It compares a program to its origional objectives.
5.2.1 Facility Testing: Determines wether each function in the objective
is actually implemented.
5.2.2 Volume Testing: Subjecting the program to heavy volumes of data.
5.2.3 Stress Testing: Subjecting the program to heavy loads of stress.
5.2.4 Security Testing: Devise test cases to subvert security checks.
5.2.5 Performance Testing: Verifying response times,and throughput.
5.2.6 Configuration Testing: If possible verify each hardware/software
configuration.
5.2.7 Compatibility/Conversion Testing:
5.2.8 Installibility Testing:
5.2.9 Reliability Testing: Down Time objective
5.2.10 Recovery Testing:
5.2.11 Serviceablilty Testing:
5.2.12 Documentation Testing:
5.2.13 Procedure Testing:
5.3 Acceptance Testing: Comparing the program to its initial requirements.
5.4 Installation Testing:
5.5 Test Planning & Control: planning, monitoring and controlling
6. Debugging
-----------
Debugging consists of : (1) Determining the exact Nature and
Location of the error.
(2) Fixing the error by providing a solution
Debugging methods are: (1) Brute Force
(2) Induction
(3) Deduction
(4) Backtracking
(5) Testing
6.1 Brute Force Debugging
- Debugging with a storage dump
- Debugging with print statement scattered through the program
- Debugging with automated tools
People who use thinking rather than a set of helps, exhibit
superior performance.
6.2 Debugging by Induction
This is a process that locates the data, organizes it and devises
a hypothesis.This hypothesis must then be proved.
A clue structure may be used for what,where,when and to what extent
for catagories of Is and Is not.
6.3 Debugging by deduction
Determine the possible causes and use data to eliminate causes.
Refine the remaining cause into a hypothesis and prove it.
6.4 Debugging by backtracking
This involves backtracking the incorrect results through the
logic of the program.
6.5 Debugging by testing
This thinking type debugging involves using test cases.
There are 2 types of test cases.Cases that expose a previously
undetected error.And cases that provide useful information
in debugging to locate an error.For an undetected error cases
tend to cover many conditions per test case.But test cases for
locating a specific error cover a single condition for each
test case.
Debugging Principles are:
- Think
-Use debugging tools as a second resort
-Avoid experimenting
-There is more than one error
-Fix the error not the symptom
Debugging error analysis provides valuable feedback to improve
future design and testing processes.
Analyze,when the error was made,who made the error,what was done
incorrectly,how could the error been prevented,why wasn,t the error
detected earlier and how was the error found.
7. Test Tools & Techniques
-----------------------
7.1 Module Driver Tools
It is a small program that feeds test case inputs to the module.
AUT(Automated Unit Test) encodes test case inputs in a non-
procedural language and stores them in a database.It also
compare the actual output to the expected output.
7.2 Static Flow Analysis Tools
It is an automated code inspection tool that is a program
that investigates another program.
7.3 Test Coverage monitors
These are used in white box testing.They monitor the program
during execution to provide statistics on :
--Statemaent coverage
--Decision coverage
7.4 Mathematical Proof of Correctness
This uses the method of inductive assertions to develop a
set of theories about the program.
7.5 Program Correctness Provers
This involves automating the process of formula manipulations
and tedious clerical work.
7.6 Symbolic Execution
A program is executed with variables taking on symbolic
and numeric values.
7.7 Test Data Generators
Generates random streams of input data.
But one cannot design test cases for :
--Equivalence partitioning
--Boundary value analysis
7.8 Environmental Simulators
7.9 Sneak-Circuit Analysis
Detects flaws in software structures
7.10 Predictive Models
For the knowledge of the number of errors remaining in
program.
7.11 Program Library Systems
Automated way of managing code,documentation and test cases.
7.12 Interactive Debugging Tools
7.13 Program State Monitors
******************** Please feel free to copy this Document***************