In November, I spoke at several Agile Journal events in the Northeast about the benefits of using Agile and how they transcend simply “going faster”. I have already posted the presentation, but like most power point slides, it does not stand that well on its own, so I wanted to write up a summary of my thoughts on this topic for the ages, as well as to develop my own thinking. I share these thoughts with you now.

Personally, I find this question one of the most loaded within the profession, “so how much (faster/cheaper/in some way better) will using Agile make my projects?”

This is certainly a legitimate question, but there’s frequently an undercurrent that indicates a very narrow perspective on what one is willing to invest in such a process. I do not mean to imply that there are not tactical performance improvements to be gained from using Agile, they are legion and well documented. Rather, if we focus purely on simply spinning the wheels faster, we will surrender a significant amount of potential value and ultimately limit the success of such an initiative.

Just the Numbers

Before diving too far into the details of these “other benefits”, let me speak to the first question of tactical improvements. Mike Cohn put out a pretty authoritative presentation on the topic of performance improvements, it is available to all and licensed under the creative commons. Let me share a few highlights from sources he found, as well as some I’ve come across:

• According to QSM’s Agile Impact Report in 2008, Agile teams are 37% faster to market, 16% more productive and, most interestingly, able to support crashing project schedules by up to 50% without seeing geometric growth in defects.

• Several different organizations conducted surveys in 2008 asking respondents to identify the benefits they have realized from using Agile. These include VersionOne’s 2008 State of Agile Report, Tech Target’s 2008 Agile Survey, and Dr. Dobb’s 2008 Agile Adoption Rate Survey. I have included a summary of the responses.
  

• The Standish Group’s infamous CHAOS Report has shown a marked improvement in project success rates since 1994, having moved from 16% (1994) to 29% (2004), and many within that project have specifically highlighted Agile as one of major sources of that improvement

Faster Isn’t Enough

I am not one to  begrudge someone for desiring to spend less money in order to build software faster and with less defects, but I can’t help but think of that old joke about the expedition in the Amazon rain forest. After several difficult days in the forest, the team begins to make progress. They are moving much faster and are covering quite a bit of ground. One of the men is curious how fast they’re going. He climbs up a tree, surveys the surroundings, and shouts back down, “we’re going in the wrong direction”. Someone shouts back, “but we’re making great time!”

Indeed, this is the key challenge, and potential benefit, that the values and practices of the Agile Manifesto seek to address. It’s not about improving our throughput by 10%, it’s about building fundamentally better software in a collaborative way that maximizes the value of our business partners based on their circumstances. Let’s talk through a few dimensions in which to consider how Agile can help us.

Financial Performance

Many people discuss the value of time to market in the abstract, but sometimes it’s good to offer a concrete example. Imagine two projects, one delivered after 1 year (let’s call this one “Traditional”) and the other incrementally beginning at month 3 (let’s call this one “Agile”), with the following assumptions:

• Both projects yield $3 million every year, once they are up & running.
• Both projects will take 1 year of project work to complete.
• Both projects run perfectly and at the same rate, meaning the product is fully built after 1 year at the exact same cost.
• Upfront costs are $100k and the team costs $1.25M a year to run (20% of that in future years to support).
• The cost of capital is 15%, charged over 6 periods every year

What we see is that the Agile project, under the exact same circumstances, will have a NPV value of nearly double the traditional project, due entirely to the value of earning cash flow early. I must give credit for this spread sheet to John Scumniotales, who first put together a similar scenario here. I have modified his assumptions slightly to make it more conservative, you can see my version here.

Other than the return, there are a few other points that bear mentioning in this model. Note that the Agile project has a significantly lower financial exposure ($900k vs $1.4M) and that it reaches that exposure earlier than the traditional project (month 8 vs. month 16). This means that the Agile project will bear less financial risk, and we will know if we are going to break even much sooner. Now, of course, this argument is predicated upon teams being able to plan a product that can launch early and incrementally. This is not a given, and brings us to the next point that early feedback helps us build a better suited product to whatever the need its trying to fit.

Product Quality

I must be careful when I say “quality” as many of us have been conditioned to think that simply means “less bugs”. As we saw earlier, this is a real benefit of using Agile, but it is not what I’m talking about now. What I’m thinking about are those projects that were “paper successes”, where it is brought in on time, on budget, and too spec, but unfortunately it doesn’t actually solve the underlying problem. These are a pernicious challenge for IT professionals. My very first big project was delivered according to the plan, but it didn’t do what the business needed. Now my manager told me that it wasn’t my fault. They should have given us better requirements. This explanation never sat well with me, and it is one of the things that has led me to embrace the high levels of collaboration and empirical feedback championed by techniques like Scrum.

Let me offer another example, in 1994 Fox won broadcast rights for the NHL. They wanted to increase viewership and surveys of potential viewers showed that new comers had a hard time keeping track of the puck. Focus groups indicated that these people would like a way to better track the puck as it moved around, and responded very favorably to the idea of visually highlighting the puck on television screens. The network was under a tight deadline for the NHL All-Star game in 1996. A team was commissioned to achieve the task with $2 million. The outcome was far from certain, even up to a week of the event. The team tried a variety of technologies including radar before settling on a series of infrared LEDs to be contained within the puck and a relay of sensors around the ring hooked up to quite a bit of processing power in order to add visual effects when the puck was in the air or traveling faster than 70 miles per hour. Ultimately the team delivered on time and it was a technical success. Unfortunately FoxTrax has moved on to hockey infamy. As one reviewer lamented, “I watched a hockey game Saturday night and an episode of ‘Star Trek’ broke out” (St. Louis Post-Dispatch, 1/22/96).

While FoxTrax remained somewhat popular with people new to the sport, die hard fans hated it. The technology was used for about two years, and then retired shortly before ABC got broadcast rights in 1998 (more details are available here). Thus, we are faced with an ambitious technical project that was delivered, but was ultimately a market failure. Here’s where we need to be careful about the fine print, when we see things like the CHAOS report show project success rates of about 30%, they are measuring based on technical measures (budget, time, scope). They do not take into account situations like this, in fact, this project would be defined as a “success”. If we look towards Agile to simply make us “go faster”, we completely surrender the opportunity to leverage our incremental and iterative approach in order to make a better product and avoid situations like this. This is what I mean when I refer to product quality.

First, let me give a good non-IT example that I find most interesting. The America’s Cup is a series of sailing races held throughout the world. It is so named as the first ship to win the race was named the “America”, as well as the fact that the USA has the longest sporting record, having won the contest for 125 years from 1857 to 1983. For those not familiar with the sport, it is an intense contest, not only in the performance of the sailors, but also in the design and construction of ships. The only team to beat the Americans twice is the team from New Zealand and their boat “Black Magic”. What’s most interesting is the way they went about building this ship. While the Americans had invested immense amounts of resources in the best computer modeling available at the time, the Kiwis put their designers in the the shipyard with the those responsible for building the boat. They settled on a design and build two ships. Next they allocated 6 months to race the two whereby they would use one ship as a control and the other as a variable to test out different modifications. This is an elegant demonstration of using iterations to design a product based on empirical results, the performance of a boat with a given design modification. Nobody knew what the boats would look like coming out of that six month window, but it certainly was impressive. When the boat first debuted in 1995, it not only won the cup 5-0, but only lost one of the 43 races entered that year (more details here).

The Challenge of Iterative Delivery

I really do enjoy this example, but it illustrates a fundamental challenge we face when trying to run Agile projects: namely, how do we iterate on a product? Jeff Patton first highlighted this dynamic (to me at least) in a post and subsequent presentation he gave about embracing uncertainty. He articulates that there are two different things we do when we break apart a product: we can break it into increments, which would be a distinct and discrete component such as an engine, power train, and brakes, and we can break things into iterations, which are revised versions of a complete whole, such as the 2009 Civic and the 2010 Civic. If we focus too much on increments, then we lose the ability to actually iterate over an entire product and make improvements. This is frequently seen in those projects where someone says “we want to get this right the first time” or people are upset they must revisit a component that was already built out. If we plan our projects around building each part once, we will invariably fail. A truly Agile project, must be planned around iterating over an entire product, because we know that we won’t get it right the first time and the sooner we get the first time done, the sooner we can get feedback to maker it better.  This is born out in numerous research such as the CHAOS report, which shows that 80% of a system’s functionality is implemented in a subsequent release and 66% of all features are rarely or never used (think Microsoft Office). Now, in reality we have to blend approaches, but the fundamental point is that the sooner we get a functional product, the sooner we can get meaningful feedback to build a better product and financial benefit for the company. I will offer one more non-technical example.

Imagine a house, if we build a house increments (the foundation, the walls, the roof, etc.) then we don’t have something we can live in until the end. We also face maximum financial exposure upfront, and must make all design decisions at that time. Whereas, if we build a home and expand on iteratively (a dormer, a garage, a new wing), we can continue to refine and enhance the house all the while having a place to live in simultaneously. Of course this is not a perfect example, as software is infinitely more malleable than wood, brick and mortar, but I hope the analogy demonstrates the concept.

Personally, I think this is one of the most exciting and interesting things about Agile projects, as we must be creative in the way we frame them in order to take advantage of iterations. My first “aha’ moment in this field was with a financial service company building a new webpage. As time went on, they became increasingly frustrated over performance. We had not surfaced it at first, but the page’s performance was a primary concern. So we set up a new feedback loop, and ran a very crude performance test in the development environment with every hourly build. It did not extrapolate perfectly to a true performance environment, but it gave us an excellent tool to engage and discuss with the customer about just how much they wanted that next feature, as well as how much it would cost them. We had total transparency into how much each new component impacted the performance of the page and this led to a website that looked remarkably different than what people originally imagined, but it met their primary concern, and was better suited to their needs than what they originally conceived.

Responsiveness to Change

By now, I hope you all agree that we won’t ever get our requirements right up front and its a fool’s quest to try to do so. But, even if we were to have a magic wand that would give us perfect requirements, the exponential nature of change makes it a moot point. Several educators put together a video titled “Did You Know?” discussing the challenges facing modern educators. It has been wildly popular and I would recommend you see the latest version here. This presentation discusses several points that merit further exploration, but let me focus on just one, the adoption of new technologies. We are now at a point where industries can change in what used to be the cycle time of a traditional project.

Ultimately, this is our last benefit to be realized from Agile practices: the ability to change. If we work in a clear manner where we’re building just enough to meet our immediate goal, we undertake good technical practices like test driven development and we have good, frequent feedback on our product, we are in an excellent position to make adjustments to the vision of our product based on the reality around us. Now this is not necessarily efficient, and it doesn’t helps us with project success metrics if we’re looking to measure things as being on budget, on time and to the original specification, but it is profoundly valuable nonetheless. In fact, this is where we see virtuous cycles. As we saw earlier, Agile projects have less financial exposure, thus there is less at stake if we need to change path, this can be profoundly valuable when we talk about the human condition and our propensity to avoid realizing a loss. Second, we discussed how Agile helps us refine the vision of a product as we go if we don’t have a certain product vision, then change becomes significantly easier.

Conclusion

As I look at the history of project management, I can’t help but see how the legacy of scientific management as defined by Frederick Taylor has been a double edged sword. On one hand, he was the pioneer of using empirical feedback. He dubbed the term “scientific management” and proposed that all tasks be designed and managed based on verifiable measures. Unfortunately he was also the person who said that there should be a distinction between those who define the work and those who execute it. Sadly, I see this legacy in so much of what we do today in IT and it is an expensive piece of mental baggage. It implies that the truly valuable people must be “managers” thereby creating a career track that encourages people to give up their technical skills. It creates a schism between those who will do the work and those who will plan it, ensuring that there will never be full buy in. And ultimately, I think the spirit of that Mr. Taylor is in the question “how much faster will Agile make me”, as it implies that we are simply running a factory line when we build software and its simply a matter of getting the line faster. Some people have been very successful with this mental model, but my experience has been that it sacrifices a significant amount of value for the team and makes for a much less enjoyable workplace.

Thanks and I’d love to know your thoughts!