读书笔记-The art of software testing

Chapter 1: Self-assessment test

Software testing is a process, or a series of processes, designed to make sure computer code does what it was designed to do and that it does not do anything unintended.

Software should be predictable and consistent, offering no surprises to users.

The program reads three integer values from an input dialog. The

three values represent the lengths of the sides of a triangle. The

program displays a message that states whether the triangle is scalene,

isosceles, or equilateral.

1. Do you have a test case that represents a valid scalene triangle? (Note that test cases such as 1,2,3 and 2,5,10 do not warrant a “yes” answer because there does not exist a triangle having these dimensions.)

2. Do you have a test case that represents a valid equilateral triangle?

3. Do you have a test case that represents a valid isosceles triangle?(Note that a test case representing 2,2,4 would not count because it is not a valid triangle.)

4. Do you have at least three test cases that represent valid isosceles triangles such that you have tried all three permutations of two equal sides (such as, 3,3,4; 3,4,3; and 4,3,3)?

5. Do you have a test case in which one side has a zero value?

6. Do you have a test case in which one side has a negative value?

7. Do you have a test case with three integers greater than zero such that the sum of two of the numbers is equal to the third? (That is, if the program said that 1,2,3 represents a scalene triangle, it would contain a bug.)

8. Do you have at least three test cases in category 7 such that you have tried all three permutations where the length of one side is equal to the sum of the lengths of the other two sides (for example, 1,2,3; 1,3,2; and 3,1,2)?

9. Do you have a test case with three integers greater than zero such that the sum of two of the numbers is less than the third (such as 1,2,4 or 12,15,30)?

10. Do you have at least three test cases in category 9 such that you have tried all three permutations (for example, 1,2,4; 1,4,2; and 4,1,2)?

11. Do you have a test case in which all sides are zero (0,0,0)?

12. Do you have at least one test case specifying non integer values (such as 2.5,3.5,5.5)?

13. Do you have at least one test case specifying the wrong number of values (two rather than three integers, for example)?

14. For each test case did you specify the expected output from the program in addition to the input values?

Chapter 2: The Psychology and Economics of Program Testing

Software testing is a technical task,but it also involves some important considerations of economics and human psychology.

Creating test cases for all of these possibilities is impractical.the tester’s attitude may be more important than the actual process itself.

Adding value through testing means raising the quality or reliability of the program. Raising the reliability of the program means finding and removing errors.

Testing is the process of executing a program with the intent of finding errors.

Two of the most prevalent strategies include black-box testing and white-box testing.

 

Black-Box Testing

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

One important testing strategy is black-box, data-driven, or input/output driven testing.

Your goal is to be completely unconcerned about the internal behavior and structure of the program. Instead, concentrate on finding circumstances in which the program does not behave according to its specifications.

If you want to use this approach to find all errors in the program,the criterion is exhaustive input testing, making use of every possible input condition as a test case.

To be sure of finding all such errors, you have to test using not only all valid inputs, but all possible inputs.

This discussion shows that exhaustive input testing is impossible.Two implications of this are that (1) you cannot test a program to guarantee that it is error free and (2) a fundamental consideration in program testing is one of economics.

White-Box Testing

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Another testing strategy, white-box or logic-driven testing, permits you to examine the internal structure of the program. This strategy derives test data from an examination of the program’s logic

the analog is usually considered to be exhaustive path testing.

Determining the number of unique logic paths is the same as determining the total number of unique ways of moving from point a to point b

 

The first is that an exhaustive path test in no way guarantees that a program matches its specification.

Second, a program may be incorrect because of missing paths.Exhaustive path testing, of course, would not detect the absence of necessary paths.

Third, an exhaustive path test might not uncover data-sensitivity errors.

if (a-b < c)
System.out.println("a-b < c");

Detection of this error, however, is dependent upon the values used for a and b and would not necessarily be detected by just executing every path through the program.

 

Software Testing Principles

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Principle 1: A necessary part of a test case is a definition of the expected output or result.

A test case must consist of two components:
1. A description of the input data to the program.
2. A precise description of the correct output of the program for that set of input data.

 

Principle 2: A programmer should avoid attempting to test his or her own program

After a programmer has constructively designed and coded a pro-gram, it is extremely difficult to suddenly change perspective to look at the program with a destructive eye.

It implies that testing is more effective and successful if someone else does it.

debugging is more efficiently performed by the original programmer.

 

Principle 3: A programming organization should not test its own programs.

A  programming  organization or a project manager is largely measured on the ability to produce a program by a given date and for a certain cost.

One reason for this is that it is easy to measure time and cost objectives, but it is extremely difficult to quantify the reliability of a program.

 

Principle 4:Thoroughly inspect the results of each test.

Put another way, errors that are found on later tests are often missed in the results from earlier tests.

 

Principle 5:Test cases must be written for input conditions that are invalid and unexpected, as well as for those that are valid and expected.

There is a natural tendency when testing a program to concentrate on  the  valid  and expected  input  conditions, at  the  neglect  of the invalid and unexpected conditions.

test cases representing unexpected and invalid input conditions seem to have a higher error-detection yield than do test cases for valid input conditions.

 

Principle 6: Examining a program to see if it does not do what it is supposed to do is only half the battle; the other half is seeing whether the program does what it is not supposed to do.

 

Principle 7: Avoid throwaway test cases unless the program is truly a throwaway program

Saving test cases and running them again after changes to other components of the program is known as regression testing.

 

Principle 8: Do not plan a testing effort under the tacit assumption that no errors will be found.

This is a mistake project managers often make and is a sign of the use of the incorrect definition of testing—that is, the assumption that testing is the process of showing that the program function correctly.

 

Principle 9:The probability of the existence of more errors in a section of a program is proportional to the number of errors already found in that section.

Another way of stating this principle is to say that errors tend to come in clusters.

If a particular section of a program seems to be much more prone to errors than other sections, then this phenomenon tells us that, in terms of yield on our testing investment, additional testing efforts are best focused against this error-prone section.

 

Principle 10:Testing is an extremely creative and intellectually challenging task.

It is probably true that the creativity required in testing a large program exceeds the creativity required in designing that program.

Summary

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

As you proceed through this book, keep in mind these three important principles of testing:

• Testing is the process of executing a program with the intent of finding errors.

• A good test case is one that has a high probability of detecting an as yet undiscovered error.
• A successful test case is one that detects an as yet undiscovered error.

 

Chapter 3 Program Inspections,Walkthroughs,and Reviews

Human testing techniques are quite effective in finding errors—so much so that every programming project should use one or more of these techniques.

You should apply these methods between the time the program is coded and the time when computer-based testing begins.

Here’s an important note: Because the involvement of humans results in  less  formal  methods  than  mathematical  proofs  conducted  by  a computer, you may feel skeptical that something so simple and informal  can be  useful.  Just the  opposite  is  true. These  informal techniques  don’t  get  in  the  way  of successful  testing; rather,  they substantially contribute to productivity and reliability in two major ways.

First, it is generally recognized that the earlier errors are found, the lower the costs of correcting the errors and the higher the probability of correcting the errors correctly. Second, programmers seem to experience  a  psychological  change  when  computer-based  testing commences.

 

Inspections and Walkthroughs

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

The two primary human testing methods are code inspections and walkthroughs.

Inspections and walkthroughs involve a team of people reading or visually inspecting a program.

The climax  is  a “meeting of the minds,” at a participant conference. The objective of the meeting is to find errors but not to find solutions to the errors. That is, to test,not debug.

Inspections and walkthroughs are more effective, again because people other than the program’s author are involved in the process.

Another advantage of walkthroughs, resulting in lower debugging (error-correction) costs, is the fact that when an error is found it is usually  precisely  located  in  the  code.

In  addition,  this  process  frequently exposes a batch of errors, allowing the errors to be corrected later en masse.

These methods  generally  are  effective in finding  from  30 to  70 percent of the logic-design and coding errors in  typical programs.

The implication is that inspections/walkthroughs and computer-based testing are complementary;error-detection efficiency will suffer if one or the other is not present.

program modifications  also  should  be  subjected  to  these  testing  processes  as  well  as regression testing techniques.

 

Code Inspections

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

A code inspection is a set of procedures and error-detection techniques for group code reading.

An inspection team usually  consists  of four people.  One  of the four people plays the role of moderator.

The duties of the moderator include
• Distributing materials for, and scheduling, the inspection session
• Leading the session
• Recording all errors found
• Ensuring that the errors are subsequently corrected

The moderator is like a quality-control engineer. The second team member is the programmer. The remaining team members usually are the program’s designer (if different from the programmer) and a test specialist.

During the session, two activities occur:

1. the simple act of reading aloud a program to an audience seems to be a remarkably effective error-detection technique.

2. The program is analyzed with respect to a checklist of historically common programming errors

The time and location of the inspection should be planned to avoid all outside interruptions.

Note that for the inspection process to be effective, the appropriate attitude must be established: The objective of the inspection is to find errors in the program, thus improving the quality of the work.

 

An Error Checklist for Inspections

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

An important part of the inspection process is the use of a checklist to examine  the program for common  errors.

…….参考中文列表

 

Walkthrough

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

The code walkthrough the  procedures are  slightly  different,  and  a  different  error-detection  technique  is employed.

the procedure in the meeting is different. Rather than simply reading the program or using error checklists, the participants “play computer.”

That is, the test data are walked through the logic of the program. The state of the program (i.e., the values of the variables) is monitored on paper or whiteboard.

In most walkthroughs, more errors are found during  the  process  of questioning  the  programmer  than  are  found directly by the test cases themselves.

 

Desk Checking

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

A desk check can be viewed as a one-person inspection or walkthrough: A person reads a program, checks it with respect to an error list, and/or walks test data through it.

In short, desk checking may be more valuable than doing nothing at all, but it is much less effective than the inspection or walkthrough.

 

Peer Ratings

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

it is related to the idea of code reading.

Peer rating is a technique of evaluating anonymous programs in terms of their overall quality, maintainability, extensibility, usability, and clarity. The purpose of the technique is to provide programmer self-evaluation.

 

Summary

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

In fact, most programming projects should include the following human testing techniques:
• Code inspections using checklists
• Group walkthroughs
• Desk checking
• Peer reviews

 

Chapter 4 Test Case design

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

The obvious strategy, then, is to try to make tests as complete as possible.

Given  constraints  on  time  and  cost,  the  key  issue  of testing becomes

What subset of all possible test cases has the highest probability of detecting the most errors?

You can develop a reasonably rigorous test by using certain  black-box-oriented  test-case-design  methodologies  and  then supplementing these test cases by examining the logic  of the  program, using white-box methods.

 

“If you thought designing and coding that program was hard, you ain’t seen nothing yet.”

 

White-Box Testing

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

White-box testing is concerned with the degree to which test cases exercise or cover the logic (source code) of the program.

逻辑覆盖原则:

A stronger logic-coverage criterion is known as decision coverage or branch coverage.

each branch direction must be traversed at least once.

Examples of branch or decision statements are switch, do-while, and if-else statements. Multiway GOTO statements

decision coverage requires that each decision have a true and a false outcome, and that each statement be executed at least once.

This discussion considers only two-way decisions or branches and has  to  be  modified  for  programs  that  contain  multi-way  decisions.

A criterion that is sometimes stronger than  decision coverage is condition coverage.

condition coverage usually is superior to decision coverage in that it may (but does not always) cause every individual condition in a decision to be executed with both outcomes, whereas decision 

The  obvious  way  out of this  dilemma  is  a  criterion  called decision/condition coverage.

 

It requires sufficient test cases that each condition in a decision takes on all possible outcomes at least once, each decision takes on all possible outcomes at least once, and each point of entry is invoked at least once.

 

在存在循环的情况下，多重条件覆盖准则所需要的测试用例的数量通常会远远小于其路径的数量

 

等价类

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

确定这个子集的一种方法，就是要意识到一个精心挑选的测试用例还应具备另外两个特性：
1. 严格控制测试用例的增加，减少为达到“合理测试”的某些既定日标而必
须设计的其他测试用例的数量。
2. 它覆盖了大部分其他可能的测试用例。也就是说，它会告诉我们，使用或
不使用这个特定的输入集合，哪些错误会被发现，哪些会被遗漏掉。

 

第一个特性意味着，每个测试用例都必须体现尽可能多的不同的输入情况，以使最大限度地减少测试所需的全部用例的数量。

第二个特性意味着应该尽量将程序输入范围进行划分，将其划分为有限数量的等价类，这样就可以合理地假设（但是，显然不能绝对肯定）测试每个等价类的代表性数据等同于测试该类的其他任何数据

使用等价划分方法设计测试用例主要有两个步骤：（1）确定等价类；（2）生成测试用例。

 

经验证明，考虑了边界条件的测试用例与其他没有考虑边界条件的测试用例相比，具有更高的测试回报率。所谓边界条件，是指输入和输出等价类中那些恰好处于边界、或超过边界、或在边界以下的状态。

边界值分析方法与等价划分方法存在两方面的不同：
1. 与从等价类中挑选出任意一个元素作为代表不同，边界值分析需要选择一
个或多个元素，以便等价类的每个边界都经过一次测试。
2. 与仅仅关注输入条件（输入空间）不同，还需要考虑从结果空间（输出等
价类）设计测试用例。