Let’s bust those myths!

Myth: Agile Doesn’t Work for Projects in the Highly Regulated Medical Environment.  (The reason usually given is that FDA regulations require detailed requirements prior to project approval; hence, waterfall.  However, in reality, you can develop in phases, with small incremental sets of requirements and the FDA requires only enough documentation to demonstrate your process.)

Truth: Abbott Labs overcame medical device regulation and stringent Class 3 certification and developed the m2000 Real-time PCR Diagnostics System, a human blood analysis tool, with four agile teams.  Compared to the prior methodology in use, this project resulted in a less cumbersome process, fewer defects, a reduction in costs of 43%, and a reduction in cycle time of 25%.

(Rasmussen, R., Hughes, T., Jenks, J. R., & Skach, J. (2009). Adopting agile in an FDA regulated environment. Proceedings of the Agile 2009 Conference (Agile 2009), Chicago, Illinois, USA, 151-155)

Myth: Agile Doesn’t Work in Government

Truth: The FBI overcame a CMMI level 3, ISO 9001, government-mandated document-driven waterfall life cycle and developed the Domestic Terrorist Database & Data Warehouse with three agile teams.  Compared to the prior methodology in use, this project resulted in significant improvements in release planning, developer satisfaction, and a focus on the true goal: “to catch bad guys.”

(Babuscio, J. (2009). How the FBI learned to catch bad guys one iteration at a time. Proceedings of the Agile 2009 Conference (Agile 2009), Chicago, Illinois, USA, 96-100.)

For another example, the U.S. Department of Defense developed the Strategic Knowledge Integration Website utilizing three agile teams.  Compared to the prior methodology in use, this project resulted in improved quality, fewer defects, better teamwork, and a 200% productivity increase.

(Fruhling, A., McDonald, P, & Dunbar, C. (2008). A case study: Introducing extreme programming in a U.S. government system development project. Proceedings of the 41st Annual Hawaii International Conference on System Sciences (HICSS 2008), Waikaloa, Big Island, Hawaii,USA, 464-473.))

Myth: Agile Doesn’t Work for Large Products

Myth: Agile Doesn’t Work with Distributed Teams

Truth: Google’s AdWords product busts both of these myths.  With 20 teams and 140 people across 5 different countries, this large agile program was a groundbreaking success at Google and resulted in more predictable releases, higher quality, and an improved ability to accommodate changes, as compared to the prior methodology in use.

(Striebeck, M. (2006). Ssh: We are adding a process. Proceedings of the Agile 2006 Conference (Agile 2006), Minneapolis, Minnesota, USA, 193-201)

Myth: Agile Doesn’t Work in the Regulated Telecom environment

Truth: British Telecom moved their entire IT department to agile, starting with 2000 people from 2004-2007.  This large transformation resulted in an improvement from 10% value stream effectiveness to 55%, created an attitude of delivering real value to the business through IT, and shifted the company’s perception of IT from a service provider to an integral part of the business solutions.

(http://www.agilistapm.com/casestudy-british-telecom/. http://scalingsoftwareagility.files.wordpress.com/2008/06/scrumbt-v14.pdf)

Myth: Agile Doesn’t Work for Client-based projects

Myth: Agile Doesn’t Work for Fixed Price projects

Myth: Agile Doesn’t Work well when integrating a Third Party Product

Truth: I coached an agile team at a prominent consulting company through a project with a client who was a well known record label.  They built a new, fully rebranded, eCommerce website using open source CMS and Search engine, and a third party eCommerce provider.  The site included product bundling, integrated music player, and social networking integration.  It was implemented using Scrum/XP with a single team of about 12 people over 5 months.  The result was an award-nominated site that improved conversion rates dramatically, ultimately profitable for and considered a strong success for both the agency and the client.

Myth: Agile Doesn’t Work for Manufacturing Vehicles

Truth: Wikispeed developed a 4 passenger, 100 mpg, street-legal road car in 3 months using modular, off-the-shelf, carbon-fiber body construction, with no capital investment, and no paid employees.  Agile processes were utilized with a single international team.  The project went beyond the prototype phase and cars are available online.

(http://www.solutionsiq.com/the-agile-ceo/bid/51480/Agile-Innovation-or-How-to-Design-and-Build-a-100-MPG-Road-Car-in-3-Months)

What else ya got?

(note: leads for some of these case studies came from David Rico’s presentation on Lean & Agile Project Management for Large Programs & Projects)

How many of you have seen this situation? After an iteration sprint, the team sits down for a show and tell and the product owner marvels at what has been created in the iteration. This cycle continues for several iterations and then as the team nears release a number of other stakeholders begin to look at the product and express serious concerns. Or worse yet, the product launches and reveals a major miscalculation of market conditions. My own example was working with a transactional system where the team had mapped out a wonderful guided interaction with help text, supporting content and even confirmation prompts. A new user trying to undertake this transaction would have no problem at all figuring it out. Of course, the challenge was that most users weren’t new. In fact, most users were veterans who would need to execute this transaction many times in a single day. Their primary concern was not guides or a slick interface, but performance.

Now even with poor feedback, such a situation would come to a head once the product launched and produced negative reviews. In an Agile environment, at least this would have consumed less time, perhaps several months instead of years. But still, is that good enough? Agile allows for a lot of mistakes, but that doesn’t mean we should have no diligence. We do not allow poor code quality into our system arguing we can catch it in the next release. Similarly, we should not accept poor quality in the feedback we receive as we incrementally build a product. The example above is an example of a poor feedback loop. Much like a properly run retrospective or scrum, effective iteration reviews and demonstrations require skill and expertise to be effective. Simply walking through a piece of functionality on a projector in front of the team is not necessarily enough; we need to look back to our business case to properly demonstrate our progress.

In waterfall, teams face the chronic risk of working towards the wrong metric. Delivering the various SDLC artifacts in a timely fashion does not guarantee a successful project. Agile teams are ideally measured based on the results of their demonstrable product, but what if the demonstration is not a realistic portrayal of what the system needs to do in the market place. Every few weeks it is great to show flashy new interfaces or functionality, but if the system is being demonstrated for the sake of demonstration, then not only are we failing to getting valuable feedback, but we may even get poor feedback that would send the team in an unproductive direction. In the example I cited before, going through a single transaction in such detail lead to wrangling over how much content and support to show, as well as concerns over fringe cases that were making the component much more complex.

So how does one address such a situation? The immediate reaction is to launch often and early. Nothing focuses a team and gets the attention of stakeholders like an impending release. But, this is not always a real option. While we may have a product that is “potentially shippable”, what if it is part of a larger service offering that is simply not ready? We can’t get feedback from our customers if we don’t have a complete enough product for them to actually use. The business constraints may mean that a team must work for months, or maybe a year before they have a viable product that can be launched, how do we ensure valuable feedback prior to that launch? The rules of Scrum dictate that ultimately the product owner is responsible for delivering the business case, and I agree. My concern is that sometimes we as a team assume that is  the sole responsibility of the product owner and is independent of the team. In this case, people invariably focus on how they feel they will be measured: the show and tell demonstration. As the old adage goes, “be careful what you measure, as that is what you will get”.

If the team can not clearly measure and communicate business value delivered, management will seek another measure to gauge progress. Sadly, the most likely culprit is story points. I am always amazed at how quickly people, who had no prior concept of the idea of a story point – a sizing measure only relative to work done by a single team – will begin demanding that a team deliver some arbitrary number of points. Just as the entire product is the joint responsibility of the team, I would argue that demonstrating and measuring business value is the responsibility of the whole team. If we were to look at the case I mentioned before, the team could have built test cases around how quickly someone could run through 10 transactions in rapid succession or run internal betas with larger groups of stakeholders within the organization as the show and tell.

The potential value of having working product every iteration is immense and we should not waste it. As business and technology partners work closer together in Agile teams, there is shared responsibility in both directions. Business partners give up the illusion of a contract-based, “guaranteed” delivery, but in return we as a team must do everything we can to show them what the product can do – and not do – at each given iteration. This calls for creativity in precisely what we test, how we measure system performance and what we demonstrate at the end of an iteration.

Poor feedback can not only be counterproductive, it can be dangerous. Let me offer one example outside of the IT world. 1975, Pepsi started the “Pepsi Challenge”; people were given tastes of Coca-Cola and Pepsi and asked which product they preferred. Even though Coke had a much larger market share, the majority of people favored Pepsi in these tests. Coke conducted their own surveys and found similar results. For several years they developed a recipe based on taste tests and ultimately in 1985 they launched New Coke. After mixed early reviews, the market turned against New Coke so harshly that in less than 3 months they were forced to relaunch the original recipe as “Coca-Cola Classic” and within a year New Coke had a market share of approximately 3%. Experts now argue the sip test was an incorrect measurement of customer preference; people don’t drink a few sips of soda, they drink a whole bottle. Unfortunately by this flawed measure, New Coke did quite well. It won taste tests over Pepsi. Unfortunately, that is not how the market ultimately values soda (Gladwell, 2005). This story did have a happy ending for Coca-Cola, as sales of Coca-Cola Classic exceeded prior sales, but that does not change the fact that this was a huge embarassment and waste of resources for Coke.

Adding Scope and Budget? No Problem. Removing Scope? Forget About It
As one might expect, both parties must agree to changes to a signed FFFS contract. Much effort goes into documenting and approving prospective changes to previously agreed-to scope. However, it’s one thing to add scope. A vendor is likely to welcome increased scope because it means more revenue and margin. Furthermore, once a project is underway the vendor has a much better understanding of the project itself, further reducing (but not eliminating) the vendor-side risk. But have you ever tried reducing scope? No Way! Imagine you contracted a firm to deliver features A, B, C, D and E. Half way through the 12 month project, you realize that feature D offers no value to your or your customer. If your contract was based on Time and Materials (aka. hourly) you would simply stop further work on feature D; incur the cost already spent on the feature and possibly spend some effort removing any related code from the product. You save the time and money you were otherwise going to spend on it. On an FFFS contract – you’re locked in! You are going to get that feature whether you like it or not. The only hope of having your vendor agree to remove feature D is to replace it with features F, G and H — all intended to increase the revenue and margin on the contract.

On the surface it appears that the FFFS contract is the perfect solution for any cost-conscious customer — you get the things you want at a price agreed up-front. As an extra bonus, all the project risk is transferred to the Vendor. If the vendor made an error in estimating during the contract phase, the vendor is contractually bound to deliver the agreed-upon requirements even if the effort is greater than originally estimated. Perfect! Also, now buyers can compare bids on a level playing field. If three vendors are bidding on delivering the same set of functionality — the right vendor is the one who comes in at the cheapest price. Perfect!

Perfect?

Maybe not so perfect.