The Art of Lean Software Development

CHAPTER ONE

Why Lean?

                                    Optimism is an occupational hazard of programming.

                                                                              — Kent Beck

精益软件开发艺术

第一章

为什么要使用精益开发方法？

乐观是编程行业的大忌

The practice of software development has been plagued with shockingly low success rates for decades. At the same time, the number of software-driven products and services continues to increase dramatically each year. If these were the only two trends, we would be heading for disaster.

Fortunately, agile software development methods have been demonstrating that higher success rates are possible. And Lean techniques (which have dramatically increased manufacturing success for over 50 years) are now being applied to software development and validating the successes of Agile.

The Lean principles and the mindset of Lean thinking have proved remarkably applicable to improving the productivity and quality of just about any endeavor. Lean has been successfully applied to manufacturing, distribution, supply chain, product development, banking, engineering, the back office, and much more. However, only in the last few years have the Lean principles and techniques been applied to software development.

In this chapter we will give you more details on the problems that continue to plague software development, as well as an overview of Agile software development and the origins of Lean and its unique approach to improving any kind of process.

The Problem with Software Development

Have you ever worked on a software development project that:

* Went over schedule?

* Went over budget?

* Didn’t meet the needs of the customer?

* Was canceled?

If you said no, you are probably fresh out of school and are still     working on your first project. If you said yes, you’re not alone…not by a     long shot!

The actual statistics are shocking.

The 1994 CHAOS Report from the Standish Group is the landmark study of IT project failure. By 1994, the Standish Group had studied over 8,000 software development projects and found that only 16% were successful. This means that 84% of the projects either failed outright or had serious problems. In 2004, after 10 years of study, the number of projects included had increased to 40,000, and the success rate had improved to 29%. Although this is a significant increase, it’s still nothing to brag about.

Can you think of any other industry that has such a staggeringly low success rate?

The CHAOS Study

The Standish Group survey included large, medium, and small companies across major industry segments: banking, securities, manufacturing, retail, wholesale, health care, insurance, services, and local, state, and federal organizations (365 companies in total). Over a 10-year period, it studied over 40,000 projects. In addition, it used focus groups and personal interviews to provide a qualitative context for the survey results.

The CHAOS study classified projects into three categories:

Project Succeeded

The project was completed on time and on budget with all the features and functions that were originally specified.

Project Challenged

The project was completed, but went over budget, over the time estimate, and offered fewer features and functions than originally specified.

Project Failed

The project was canceled at some point during the development cycle (the study actually called this Project Impaired).

Figure 1-1 shows the percent of projects in each of these categories over a 10-year period.

CHAOS study data

As you can see, the success rate improved over time, but at a glacial pace: 1.3% a year. At this rate, successful projects won’t even break the 50% mark until the year 2020!

In fairness, some experts disagree with the definition of success used by the CHAOS study. They point out that many projects that have been classified as challenged have gone on to deliver very successful products. However, even if you take this into account, the success rate still leaves a lot of room for improvement.

The Waterfall Method

So the question is: why is the failure rate so high? A large part of the blame can be traced back to the widespread adoption of the Waterfall method.

The Waterfall model of software development divides development into a set of distinct phases that are performed sequentially: requirements, design, implementation, testing, deployment, and maintenance (see Figure1-2 ).

Development is seen as flowing steadily downward (like a waterfall) through these phases. Each phase has a beginning and an end, and once you move onto the next phase, you never go back (just as water does not flow uphill).

Waterfall model

The Waterfall method has an appealing air of simplicity. Managers like the series of fixed milestones that seem to make it easy to track a project’s progress. However, this manageability is an illusion because this model does not allow for change.

Software development is a highly dynamic process and change is inevitable. During implementation you’ll find problems with the design. Customers don’t know precisely what they want ahead of time, and that will change the requirements.

Studies show that the Waterfall method is positively correlated with reduced productivity, increased defect rates, and project failure. Why, then, was the Waterfall method so widely promoted, and how did it become so entrenched in the face of so much evidence that it did not work?

A historical accident

Winston Royce first described the Waterfall method in a 1970 paper titled, “Managing the Development of Large Software Systems” (Technical Papers of the Western Electronic Show and Convention). This paper is often cited as if it validates the Waterfall model, but it actually does the opposite. People who use Royce’s paper to support the Waterfall method must not have read this paper carefully because it explicitly says that the Waterfall method “is risky and invites failure.” The paper then proceeds to advocate an iterative development style.

The Waterfall method would likely have slowly faded away, but in the 1980s it became the Department of Defense (DoD) standard for the development and procurement of software with the release of DOD-STD-2167. Eventually, the DoD realized that the Waterfall method was not working and in 1994 replaced DOD-STD-2167 with MIL-STD-498, which supports iterative development.

However, the damage was done, and a strong mindset with a bias toward Waterfall development had become ingrained. The Lightweight methods of the 1990s and the Agile methods of the early 2000s have started to turn this around, but there is a long way to go. Many people who are not familiar with the evidence are still staunch supporters of the Waterfall method.

The Agile Success Story

The Agile software development methodologies were the first attempts to improve the software development situation, with Lean coming onto the software scene much later.

Most of the Agile methodologies, like Extreme Programming (XP) and Scrum, actually predate the Agile Manifesto (which we will discuss shortly). In fact, using the term “Agile” to refer to these methods of software development was coined at the famous Snowbird meeting that created the Agile Manifesto.

In the 1990s there was a growing dissatisfaction with the prevalent heavy software development methodologies and processes. Using these processes did not solve any of the endemic problems of software development: high project failure rate, low software quality, and generally unhappy customers.

This spawned a number of alternative methodologies, including XP, Scrum, Dynamic Systems Development Method (DSDM), Crystal, and Feature Driven Development (FDD), that were collectively known as Lightweight methods. The term “Lightweight” was meant to distinguish them from the predominant heavyweight methods of the time. Their creators were not happy with the term “Lightweight” because it seemed to imply that these methods were less comprehensive or less important.

It became increasingly apparent that these Lightweight methods had a lot in common with each other. So, in February of 2001, a group of 17 of the leading independent thinkers about software development gathered at the Snowbird ski resort in Utah to try to find common ground. The roster of this two-day gathering included many of the most prominent     personalities in software development at the time: Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham, Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, Jon Kern, Brian Marick, Robert C. Martin, Steve Mellor, Ken Schwaber, Jeff Sutherland, and Dave Thomas.

Three significant things came out of the now-famous Snowbird gathering: the decision to use the term “Agile,” the Agile Manifesto, and the Agile Alliance. The Agile Alliance (http://www.agilealliance.org/) is a nonprofit organization that exists to further the development and dissemination of information regarding Agile processes.

As stated earlier, no one was happy with the term “Lightweight.” Alistair Cockburn articulated this nicely when he said, “I don’t mind the methodology being called light in weight, but I’m not sure I want to be referred to as a ‘Lightweight’ attending a ‘Lightweight methodologists’ meeting. It sounds like a bunch of skinny, feebleminded people trying to remember what day it is.” And thus, we now have “Agile.”

The Agile Manifesto

The Agile Manifesto (formally called the Manifesto for Agile Software Development ) is a model of simplicity. Written by Kent Beck et al., it eloquently states the four core values that form the philosophical bedrock of all Agile methodologies:

We are uncovering better ways of developing software by doing it and helping others do it.

Through this work we have come to value:

* Individuals and interactions over processes and tools

* Working software over comprehensive documentation

* Customer collaboration over contract negotiation

* Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

The wording of the Agile Manifesto was carefully crafted to communicate the intended message. By saying, “We are uncovering better ways of developing software,” they showed that they didn’t have all the answers and were still on a learning journey of discovery. Just as important was the next part, “by doing it and helping others do it,” which meant that they were actively engaged in using these “better ways” (no ivory-tower theoretical advice here) and were sharing what they learned with others (not telling people what to do).

Each of the four value statements in the Agile Manifesto ists two things and states that both items are important, but the first item is higher priority. Martin Fowler and Jim Highsmith described this very succinctly in their August 2001 article in Dr. Dobb’s Journal itled “The Agile Manifesto”:

The Alliance recognizes the importance of process and tools, with the additional recognition that the interaction of skilled individuals is of even greater importance. Similarly, comprehensive documentation is not necessarily bad, but the primary focus must remain on the final product—delivering working software. Therefore, every project team needs to determine for itself what documentation is absolutely essential.

Contract negotiation, whether through an internal project charter or external legal contract, isn’t a bad practice, just an insufficient one. Contracts and project charters may provide some boundary conditions within which the parties can work, but only through ongoing collaboration can a development team hope to understand and deliver what the client wants.

Plans are important, but they shouldn’t be rigid, unchanging plans. The ability to respond to changes is critical to the success of most software development projects.

In addition to enumerating the four Agile values, the authors of the Agile Manifesto further refined what they meant in those value statements by listing a number of principles that they follow as a direct result of those values:

* Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

* Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.

* Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

* Business people and developers work together daily throughout the project.

* Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

* The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

* Working software is the primary measure of progress.

* Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

* Continuous attention to technical excellence and good design enhances agility.

* Simplicity—the art of maximizing the amount of work not done—is essential.

* The best architectures, requirements, and designs emerge from self-organizing teams.

* At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

Agile Methodologies

There are a fair number of formal software development methodologies that fall within the Agile camp. While they vary in their specific activities and artifacts, they all use short time-boxed iterations that deliver working software at the end of each iteration (the length of the iterations varies between methodologies). Figure 1-3 shows the typical Agile process.

Typical Agile process

All Agile methodologies share a number of core practices (either formally or informally). These core practices also support Lean software development and are the subject of the second half of this book.

Some of the more commonly used Agile methodologies include:

* Scrum

* XP

* Crystal

* FDD

* Unified Process (UP)

* DSDM

These methods all began at different times in the 1990s as a response to the failure of the Waterfall method. There has been a great deal of cross-fertilization of ideas and techniques between these methods.

Scrum has successfully taken the Agile approach and stripped it down to the essentials. The result is one of the simplest and easiest-to-implement Agile methodologies that still provides the benefits of Agile software development.

Crystal is actually a family of methodologies created by Alistair Cockburn. The actual processes and practices vary depending on a project’s size and criticality or complexity.

UP is generally considered to be one of the more “heavyweight” of the Agile processes, although it was intended to be tailored and not implemented in a heavy manner.

Regardless, this has led to a number of variants, including the Rational Unified Process (RUP), the Agile Unified Process (AUP), and the Enterprise Unified Process (EUP).

DSDM is more formal than most of the other Agile methods, fully specifying many different roles, processes, and artifacts. A notable feature is the prioritizing of requirements according to the MoSCoW rules:

M          Must-have requirements

S          Should have if at all possible

C          Could have, but not critical

W          Won’t have this time, but potentially later