Normal Program Development

Normally, when programmers write programs, there are errors in about 2% to 5% of all lines of code. Of course there are differences based on programming language, program complexity, and other factors; but the range is still fairly small.

The error rates on this page are based on code inspection. After a module is developed, it is code inspected by a team. First, the team members study the program independently, looking for errors. Then, they meet as a team to redo the process. There are limits on program size, lines examined per hour, and other factors, because as shown below, hasty inspections.

The error rate usually does not depend on program size, allowing us to compare error rates across programs of different sizes.

Note that programmers are not being sloppy. Even with very careful development, there will be a fair number of errors.
 

Study

Detail

Error Rate

Akiyama [1971]

17,052 lines of assembler code. 6 modules average 2,842 lines.

2.0%

Barnard & Price [1994]

Project 1. 9.3 KLOC covering 23 inspections. Average module size is 343 LOC. Preparation rate 194 lines/hour. Inspection 172 lines/hour.

10.6%

Barnard & Price [1994]

Project 2. 22.5 KLOC. Average module size is 409 LOC. Preparation rate 122 lines/hour. Inspection 155 lines/hour.

8.9%

Barnard & Price [1994]

72% more errors found when modules are smaller. 

Basili & Perricone [1993]

Projects at software engineering laboratory. 90 KLOC FORTRAN. Inspection rate (LOC/hr.): 50 / 150 / 200

1.6%

1.2%

0.6%

Basili & Selby [1986]

20 KLOC FORTRAN. Mixture of new and old code.

1.1%
Beizer (1990) Several studies

1%-3%

Boehm (1987) Cites several studies

3%-8.5%

Bush (1994) Jet Propulsion Lab. 300 code inspections. Per inspection. 28 hours of staff time per inspection. 16 errors found on average, percentage of major errors.

25%

Cohen [2006] 300 code inspections at Cisco Systems. Computer-assisted lightweight code review. Sixty-one inspections found no errors. In others, the defect density ranged from 1% to 13%. Cohen also describes other results from extensive lightweight code review data at Cisco Systems. 3.2%

Fagan [1976]

Aetna Life and Casualty, 4,439 lines of non-comment code. Found during code inspection.

3.8%

Fagan [1976]

6 small modules, 71 to 323 LOC

2.7%

Graden & Horsley [1986]

Major telecommunications project at ATT. 2.5 million LOC over 8 software releases.

3.7%

Haley [1996]

Effects of process improvement over time, 1988 / 1995

3.0%

1.0%

Jones [1986]

Five systems at AT&T

5.0% - 9.5%

Jones [1998]

Percent of time in defect removal. 27%-34% (increasing by size) for COBOL or C (p. 179), 22%-36% by industry. Based on sample of 84 projects in 27 organizations. Insufficient time was planned for debugging and testing in every project (p. 149).

Jones [1998]

(p 497, Table 21.8), Errors per 100 lines of code / per function point

Visual Basic

1.1%

0.21

Java

1.2%

0.50

COBOL

1.4%

1.50

FORTRAN

1.6%

1.68

C

2.0%

2.50

Average

1.5%

1.40

Jones [1998]

Defects per function point. High-severity / Low severity / Total

1.30 (26%)

3.70

5.00

Madachy [1996]

Litton. 150 inspections, 7,162 defects, 276,422 LOC (1,842 LOC/inspection) 4,612 hours, 59 LOC/HR. Defect rate.

2.6%

McCormick [1983]

AT&T portion of Integrated Corporate Information System. 20 modules, 7 KLOC. 3.8% found during code inspection, 0.6% afterward.

4.4%

Moranda [1975]

3-4 KLOC FORTRAN program

2.7%-3.6%

O'Neill [1994]

National Software Quality Experiment. 13% of all errors were major. 44% of all logic errors were major. Error rate for major errors / minor errors / total errors.

0.26%

1.76%

2.02%

Ruby, Dana & Biche [1975]

Student subjects.

2.0%

Russell [1991]

Ultralarge-scale development study. Effects of inspection speed. 150 lines per hour / 450 LPH, 750 LPH

3.7%

1.5%

0.8%

Schulmeyer [1999]

Satellite communications system, 1993. 83.5 KLOC inspected, 1,540 code units (56 LOC/unit) Issues/KLOC Major (1,180) / Minor (1,580)/ Total (2,760)

1.4%

1.9%

3.3%

Spencer [1993]

First year of code inspection / Second year of code inspection. Effect of experience.

2.9%

5.3%

Strauss & Ebenau [1994]

PBX software development project. 7,500 lines of C code. Hasty/non-hasty code inspection.

1.3%

2.0%

Strauss & Ebenau [1994]

Major faults

2.8%

Strauss & Ebenau [1994]

Microcomputer operating system. Pascal. 4,549 LOC. Average module is 198 LOC. Major errors / Total errors. (25% major)

1.4%

5.7%

Strauss & Ebenau [1994]

Electronics firm: several programs. Major defects.

1.5% - 2.0%

Thayer, Lipow & Nelsen [1978]

74,588 lines of Jovial code in 25 modules (average size 2,983 lines)

2.7%

Weller [1993]

Major defects in new code

1.9%

Weller [1993]

Overall, more than 6,000 code inspections. / < 200 lines/hr, 3-person teams / <200 lines per hour, 4-person teams / 200 lines per hour, 3-person teams / 200 lines per hour, 4-person teams

1.9%

2.4%

3.1%

1.2%

2.5%

Weller [1993]

Project B. 12 KLOC

2.3%

Weller [1993]

Project D. 18 KLOC. Major defects

1.4%

Zage & Zage [1993]

21 programs with 23,732 LOC. Average 1130. Total/Easy / Moderate / Difficult

12.6%

4.3%

13.6%

15.0%

Copyright 1997-2008 Raymond R. Panko.