How many of you have seen this situation? After an iteration sprint, the team sits down for a show and tell and the product owner marvels at what has been created in the iteration. This cycle continues for several iterations and then as the team nears release a number of other stakeholders begin to look at the product and express serious concerns. Or worse yet, the product launches and reveals a major miscalculation of market conditions. My own example was working with a transactional system where the team had mapped out a wonderful guided interaction with help text, supporting content and even confirmation prompts. A new user trying to undertake this transaction would have no problem at all figuring it out. Of course, the challenge was that most users weren’t new. In fact, most users were veterans who would need to execute this transaction many times in a single day. Their primary concern was not guides or a slick interface, but performance.

Now even with poor feedback, such a situation would come to a head once the product launched and produced negative reviews. In an Agile environment, at least this would have consumed less time, perhaps several months instead of years. But still, is that good enough? Agile allows for a lot of mistakes, but that doesn’t mean we should have no diligence. We do not allow poor code quality into our system arguing we can catch it in the next release. Similarly, we should not accept poor quality in the feedback we receive as we incrementally build a product. The example above is an example of a poor feedback loop. Much like a properly run retrospective or scrum, effective iteration reviews and demonstrations require skill and expertise to be effective. Simply walking through a piece of functionality on a projector in front of the team is not necessarily enough; we need to look back to our business case to properly demonstrate our progress.

In waterfall, teams face the chronic risk of working towards the wrong metric. Delivering the various SDLC artifacts in a timely fashion does not guarantee a successful project. Agile teams are ideally measured based on the results of their demonstrable product, but what if the demonstration is not a realistic portrayal of what the system needs to do in the market place. Every few weeks it is great to show flashy new interfaces or functionality, but if the system is being demonstrated for the sake of demonstration, then not only are we failing to getting valuable feedback, but we may even get poor feedback that would send the team in an unproductive direction. In the example I cited before, going through a single transaction in such detail lead to wrangling over how much content and support to show, as well as concerns over fringe cases that were making the component much more complex.

So how does one address such a situation? The immediate reaction is to launch often and early. Nothing focuses a team and gets the attention of stakeholders like an impending release. But, this is not always a real option. While we may have a product that is “potentially shippable”, what if it is part of a larger service offering that is simply not ready? We can’t get feedback from our customers if we don’t have a complete enough product for them to actually use. The business constraints may mean that a team must work for months, or maybe a year before they have a viable product that can be launched, how do we ensure valuable feedback prior to that launch? The rules of Scrum dictate that ultimately the product owner is responsible for delivering the business case, and I agree. My concern is that sometimes we as a team assume that is  the sole responsibility of the product owner and is independent of the team. In this case, people invariably focus on how they feel they will be measured: the show and tell demonstration. As the old adage goes, “be careful what you measure, as that is what you will get”.

If the team can not clearly measure and communicate business value delivered, management will seek another measure to gauge progress. Sadly, the most likely culprit is story points. I am always amazed at how quickly people, who had no prior concept of the idea of a story point – a sizing measure only relative to work done by a single team – will begin demanding that a team deliver some arbitrary number of points. Just as the entire product is the joint responsibility of the team, I would argue that demonstrating and measuring business value is the responsibility of the whole team. If we were to look at the case I mentioned before, the team could have built test cases around how quickly someone could run through 10 transactions in rapid succession or run internal betas with larger groups of stakeholders within the organization as the show and tell.

The potential value of having working product every iteration is immense and we should not waste it. As business and technology partners work closer together in Agile teams, there is shared responsibility in both directions. Business partners give up the illusion of a contract-based, “guaranteed” delivery, but in return we as a team must do everything we can to show them what the product can do – and not do – at each given iteration. This calls for creativity in precisely what we test, how we measure system performance and what we demonstrate at the end of an iteration.

Poor feedback can not only be counterproductive, it can be dangerous. Let me offer one example outside of the IT world. 1975, Pepsi started the “Pepsi Challenge”; people were given tastes of Coca-Cola and Pepsi and asked which product they preferred. Even though Coke had a much larger market share, the majority of people favored Pepsi in these tests. Coke conducted their own surveys and found similar results. For several years they developed a recipe based on taste tests and ultimately in 1985 they launched New Coke. After mixed early reviews, the market turned against New Coke so harshly that in less than 3 months they were forced to relaunch the original recipe as “Coca-Cola Classic” and within a year New Coke had a market share of approximately 3%. Experts now argue the sip test was an incorrect measurement of customer preference; people don’t drink a few sips of soda, they drink a whole bottle. Unfortunately by this flawed measure, New Coke did quite well. It won taste tests over Pepsi. Unfortunately, that is not how the market ultimately values soda (Gladwell, 2005). This story did have a happy ending for Coca-Cola, as sales of Coca-Cola Classic exceeded prior sales, but that does not change the fact that this was a huge embarassment and waste of resources for Coke.

Adding Scope and Budget? No Problem. Removing Scope? Forget About It
As one might expect, both parties must agree to changes to a signed FFFS contract. Much effort goes into documenting and approving prospective changes to previously agreed-to scope. However, it’s one thing to add scope. A vendor is likely to welcome increased scope because it means more revenue and margin. Furthermore, once a project is underway the vendor has a much better understanding of the project itself, further reducing (but not eliminating) the vendor-side risk. But have you ever tried reducing scope? No Way! Imagine you contracted a firm to deliver features A, B, C, D and E. Half way through the 12 month project, you realize that feature D offers no value to your or your customer. If your contract was based on Time and Materials (aka. hourly) you would simply stop further work on feature D; incur the cost already spent on the feature and possibly spend some effort removing any related code from the product. You save the time and money you were otherwise going to spend on it. On an FFFS contract – you’re locked in! You are going to get that feature whether you like it or not. The only hope of having your vendor agree to remove feature D is to replace it with features F, G and H — all intended to increase the revenue and margin on the contract.

On the surface it appears that the FFFS contract is the perfect solution for any cost-conscious customer — you get the things you want at a price agreed up-front. As an extra bonus, all the project risk is transferred to the Vendor. If the vendor made an error in estimating during the contract phase, the vendor is contractually bound to deliver the agreed-upon requirements even if the effort is greater than originally estimated. Perfect! Also, now buyers can compare bids on a level playing field. If three vendors are bidding on delivering the same set of functionality — the right vendor is the one who comes in at the cheapest price. Perfect!

Perfect?

Maybe not so perfect.