However, we were really struggling with the adoption of these practices. We lacked the experience needed to implement the practices successfully and didn’t know how and where to make improvements.  We also had many people resistant over the changes because they didn’t see the value and long term vision of why these new practices should be adopted. Like many companies I have seen who take this approach, many had decided after some time that Agile cannot work in “our organization” and was only good in “theory and not in practice”. They were ready to call it a failure and go back to what they were comfortable with. However, I know we still had the same problems that have plagued many organizations – projects were taking too long, deliverables didn’t meet end user expectations, and we cut corners on quality to avoid going over budget and missing deadlines dictated by the business.  I wasn’t ready to give up quite yet, but knew something was missing in our approach.

Needing help, I sought some advice from those outside of the organization that had been successful in applying Agile. What I discovered was we were too focused on HOW to apply the practices without understand WHY we were using them. We needed some guidelines to help us understand why we were implementing practices to better understand where we can make improvements and still achieve the anticipated results. As I work with teams and organizations new to Agile, I find that not many have thought about implementing Agile this way.  They figure they just need a process methodology, a tool, and a mandate from upper management that we are going “Agile” and everything should come together. That will put you quickly on the road to failure.

Teams need to understand the overarching goals, values and principles that will guide them in their adoption and constant improvement of valuable Agile practices.  Fortunately, the Agile Manifesto (www.agilemanifesto.org) provides these elements.  Let’s explore the Manifesto in more detail.  The goal is simply stated, “We are uncovering better ways of developing software by doing it and helping others do it.”  On its own, the goal is too vague without having more clarification through values and principles.  Let’s read on, “Through this work we come to value: individuals and interactions over processes and tools; working software over comprehensive documentation; customer collaboration over contract negotiation; and responding to change over following a plan. That is, while there is value in the items on the right, we value the items on the left more.”  Now, we have more of a framework around four critical areas: Communication, Delivery, Collaboration and Learning.

While I had read the Agile Manifesto early on in my learning about Agile, I had not realized that there was another part of the Manifesto that usually was not contained in the books and other resources.  There are twelve principles (on right, with more details at  http://agilemanifesto.org/principles.html) that were also developed to help teams understand the why’s of implementing the various practices that have been developed.  Given that Agile is considered an empirical process for continual learning, those responsible for developing the Agile Manifesto wanted to make sure people had some specific guidelines in how they implement, inspect and adapt the practices in their organization.  While I have seen specific practices change over the years, I find that these principles have endured the test of time and can be applied against any process, policy, activity, artifact, or tool to determine if the expected outcomes will be achieved.

Looking back, I knew we were at a critical point and could have made the (easy) decision to go back to our old ways or create a new methodology that would eventually been a failure.  We would not have been successful in our adoption of Agile had it not been for seeking outside help from a person who was experienced and could help us focus on results as guided by unchanging goals, values and principles. Now, when I train and coach teams that are adopting Agile I start with introducing them to the Agile Manifesto and encourage them to put them up in their cubicles or in their team area.  As they try to adapt Agile to fit their organization, they can use the Manifesto as a test to see if the improvement will get the anticipated results and make necessary changes if the test fails.  If you are unsure where to go next, seek help from an outside coach as I did that can help you get on the right path and gets the results you are looking for.

An Agile team is like an engine. An engine needs four basic things to work well:

1. Feed it lots of fuel and air at the right moment.
2. Ignite the fuel and air at the right moment to generate power.
3. Transfer power to the drive line to do some useful work.
4. Remove waste gasses at the right moment to maintain volumetric efficiency on the next stroke.

Agile teams need similar things:

1. Feed them valuable ideas to implement at the right moment.
2. Implement high-quality software at the right moment to realize value.
3. Transfer that value out to the customer so that they can do their work.
4. Receive feedback at the right moment to maintain the delivery of value on the next iteration.

Extending the metaphor:

• Scrum is a bigger engine. It is theoretically capable of generating higher output, if you give it all of the things above.
• XP (or “technical practices” if you prefer) is a well tuned ignition system. You need it to maximize efficiency and clean power output.
• Kanban says, “Find the bottleneck and eliminate it.” That is great advice (For engines too) but maybe not specific enough to tell us how to make a Civic beat a Ferrari. [Note: I deliberately diminished the importance of finding and eliminating bottlenecks here. To be clear, I think it is one of the most important skills a coach (or engine builder) can have.]
• Effective product ownership and related business practices are your intake plenum. The fastest way to sap an engine’s power is to starve it here (That’s why they call it the “throttle.”)
• Your direct line to your customer is like your exhaust system. The tighter and twistier and more packed with crap it is the less power you will get.

To go fast you need all of these parts to work together. So, start with a bigger engine. Sure, why not? Start with more efficient spark. Sure, why not? If you want to get faster you’ve got to understand the whole system, and it more or less doesn’t matter where you start.

First there is MEI which means Manufacturing End Item and represents all the components needed to make the final assembly of what you the customer order. Second there is the CEI or customer end item which is what you buy. These two terms are core to the shopping on the web. When you select the stuff you want on your, iPhones,  personalized bathrooms, or your next auto all work because of MEI and CEI.  The choices you have for building your computer, like disks, and memory, different optical drives not to mention the skins you can wrap them in all reflect MEI’s or Manufacturing End Items. They can be combined because of an extensive Quality integration effort that assures all the bits do fit and will work properly. When they are stuck together they fill your order which defines DONE for your Customer End Item or CEI. So having multiple definitions of DONE can actually add value, as long as you pay attention to the quality needed to integrate all the parts at the end.

Don’t worry, manufacturing has kept up with the times as more and more manufacturing has taken to Modular Manufacturing. In fact in this global economy entire manufacturing systems are designed to be modular so that not only the parts are broken into smaller and smaller levels of DONE but so are the manufacturing steps. For those interested see “A hybrid methodology for synthesis of Petri net models for manufacturing systems”(http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=143353). So in a very real sense the high tech geek world we live in is about three generations behind the guys on the factory floor because most of what they are doing to determine discrete points of Done is to base it on measurable value Pretty cool huh?  Oh yeah they have been using some form of a task board and dependent demand planning  in a pull mode (AKA Kanban) for about 500 years.

Now to make this happen each step has its own QA, QC and Test criteria, patterns and harnesses. This means that if someone down stream figures a way to get folks to want people a choice in the type of metal used in their iPhone 8 antenna, the manufacturing step that makes the antenna will be ready to provision the web page where people choose the metal for the antenna – and the time to market will be the speed at which you can key in the changes to the web page.

So how close are we – software – are becoming the choke point in this whole innovation stream? We could be, if we insist on  sticking with what we are comfortable and wait to the end of the cycle to get the work tested and then have problems logged; wait until the next meeting to get needed changes through some form of CCB; wait for an optimally utilized Product Owner to have time to approve the work, and then have to wait in line while an understaffed and over committed QA group hand crafts test cases 12 timezones away to start this cycle all over again.

If however we develop defined criteria for each step of the process and, like the modular manufacturing world, base our breakdown on what is valuable to the ‘on-line’ shopper mindset.  Who knows what could happen?  Perhaps discussion that don’t get into what done is.

Of course, he’s absolutely right, and it leads nicely to one of my favorite topics that I’ve been wanting to blog about:

Agile software development is about learning. We work from a premise that the final solution is not known (or knowable) until we have delivered it, and that, therefore, we must collaborate with customers to build and define a working solution incrementally. Along the way we learn about the problem space that we are exploring and we incorporate that knowledge and our own expertise to drive towards a heuristic solution.

That may seem obvious, but it has an interesting implication. All software development processes/methodologies tend to obsess over the question of how much we can get done per unit time. However, if we know that we are engaged in learning, and we understand that learning means sometimes getting it wrong in order to adjust and get it right, then how much we can do is the wrong question. The right question is how long does it take before we can find out that we are wrong. In other words, what is the delay before our next opportunity to learn?

Scrum’s notion of “Done” is not really about being done, but about getting to the point where we can get real feedback to learn from. The same is true for the Lean notion of “Cycle Time.” Each cycle is an opportunity to learn. If there is any chance that we will get it wrong then we need at least two cycles. In fact, the likelihood that we will ever get it right is directly related to the number of opportunities we have to learn before we stop. I’ll let you chew on that for a while.

[Note: the assertion in the last paragraph originally said that the likelihood of getting it right within some period of time was inversely related to the length of our learning cycles. Hopefully, the way I wrote it above is somewhat clearer. The notion of keeping learning cycles small is a natural and important consequence of the above. You will hear more from me about that at some point.]

It is a peculiar question, because if we weren’t practicing Scrum the answer would be obvious. In fact, the Lean and Extreme Programming communities more or less agree on what “done” means without any real discussion: Done == deployed in production, in the hands of real customers.

When Scrum folks talk about the “Definition of Done,” however, they mean something slightly different. There is an important assumption here: software can reach an intermediate state called “done” where it is not yet “released,” but is “potentially shippable.”

Potentially shippable is an equally peculiar idea. In theory it ought to mean that to release or not to release is purely a business decision at this point. The software is complete, accepted, and tested, but it is not released. In practice it often means that the main development activities have been completed but there is some amount of intangible work to be done that may or may not fit inside of a Sprint (e.g. QA testing, deployment to various environments, formal release process, etc.) This is a poor, but common, way to address the “Definition of Done.”

Potentially shippable software is actually a useful idea. From a technical excellence point of view the goal should be continuous delivery, but from a business perspective continuous delivery might not be feasible. In fact, in some domains big-bang delivery actually has inherent value (e.g. any COTS, especially commercial games, and various forms of online media.) [actually, even in those environments continuous delivery is useful, but only after the initial release hits its date...] What we want is for continuous delivery to be attainable, so that it becomes a business decision not to do it. Thus, XP has long had the concept of always releasable software.

So, how do Scrum teams define done? I suppose the real answer is, however they want to, but I suggest the following: Start with everything that it would take to release the software to production. Then, to the extent that it is infeasible to do all that stuff, scale back to something that can be accomplished within a Sprint. Then, continuously improve (i.e. inspect and adapt) until the definition is once again everything that it would take to release the software to production.

Test-Driven Development is a disciplined approach to software design that provides a mechanism for programmers to create working software in tiny increments of no more than a few lines at a time. The reason that this is important is that Agile approaches to software development more or less require us to deliver working software in small increments that fit inside a two-week iteration (give or take.) In order to have working software that meets some customer need in such a short time frame we need a way to create even smaller increments of software, see them work, build a little more, see that work, etc., and improve the design as we go so that we don’t create a mess.

The way that Test-Driven Development accomplishes this is by leaning on unit testing frameworks to build a tiny test that specifies a tiny increment of behavior. We write that test before we have written the production code that will make that test pass. Then we write the code, taking care to do no more than is absolutely necessary to make our test pass. Finally, we take a step back and look for indications that our design could be improved, the main one being duplication that is created by bits of code that do or say much the same thing. When we see such duplication we refactor it using techniques and tools that allow us to change the code safely without breaking the test.

We repeat this process over and over. Each cycle takes no more than a minute or two, and each cycle builds incrementally on top of the prior ones until we have created some working feature that the customer has asked for. All the while we are seeing all of the small increments we have built work, via the tests. All the while we are seeking and exploiting small opportunities to improve and simplify the design of the code. And, in the end we have demonstrable, working software that is capable of filling some customer need. All we have to do is integrate and deploy it and we can safely move on to the next thing on our list.

That is more or less it. So, now that we have talked about what Test-Driven Development is, let’s take a moment to discuss what it is not:

Test-Driven Development is not about testing, not really. The purpose of tests is to verify the quality and functionality of software. The purpose of what we are doing is to set an expectation of what we need to do next and have a way to quantify when that expectation has been met. We get verification as a result. That is: later we can run the tests and see them pass and be assured that no regression has happened (At least in the behavior that we specified.) But, that is not the goal that we set out with. We set out with the goal of driving small increments of code that actually work. We needed a way to define what those small increments were, and the test framework gave us a harness in which to run tiny increments of code, set expectations about how the code would work, and see those expectations satisfied.

It is important to realize that Test-Driven Development is not about testing, or quality, because it is important to realize that quality is not achieved through TDD alone. Several studies have shown that software produced with TDD tends to have less bugs, but that is a side-effect of TDD and not the goal per se. In order to have quality we need to be able to verify that the software works the way the customer intends. We need a way to verify that it looks good, is usable, accomplishes work that the customer needs it to do, performs adequately, is free from obvious defects, etc. Test-Driven Development is not about these things.

Quality, in Agile, is achieved by working in small increments, by collaborating to set clear expectations up front, by using Test-Driven Development to build the software, and by performing various kinds of testing to assure that the product meets the goals we set early and continuously throughout the process. Test-Driven Development is only about how we build the software. We need other techniques to assure that the software meets our customers’ needs.

For example, the most recent class was very intense with the team asking me some really hard and crucial questions.  Then they dropped the bomb. “Hey Mike, you act like Scrum is a Silver Bullet.” Arghhh! I HATE THAT. I don’t know how many times people get that impression and how many times I have repeated the litany, “Scrum doesn’t solve anything it shows you what is happening in your organization”.  Well not this time. What jumped from my lips was “Scrum is Not a silver bullet, Scrum is a silver mirror!”.

The next day, one of the class members reported out that my ‘catch phrase’ had really worked.

Huh? The class was right behind me in asking for an explanation.

It seems he left the class last night and went back to work (we are such a bunch of OCD wonks) where is boss was talking about Scrum not being the Silver bullet he, the boss, had expected. Our teammate then popped the phrase “Scrum is Not a silver bullet, it is a silver mirror!”. This stopped the boss in his tracks as he realized that Scrum was just that, a high definition reflection of all the things that were actually going on. And if memory serves the attendee went on to say the conversation went from Scrum not meeting expectations to what was coming off the mirror.

The class, being a great one, started kicking this around. One of the comments emerged from someone saying “Talk to the hand” and became “talk to the silver mirror in my hand”.

Maybe a good ScrumMaster is a person who can have people talk to the mirror in their hand.

My previous post about re-engineering did not discuss what to do in these circumstances. I would recommend the following:

1. Think really hard about it: is there a value proposition here? Or does the existing product do what we need it’s just not that great? Think about it again.
2. If you are certain that there is value then start a new project to replace the existing product.
3. This is the most important step. Forget everything about the existing product. Eliminate it from your brain.
4. Develop the new product in the Agile way. Find out what the customer wants to do and build software that enables them to do it. Do it incrementally and get as much feedback as you can.
5. Always, always challenge your assumptions. It is tempting to assume that the product has to work like the old one, or that you should do something that you wanted to do in the old one but never did. Forget it. Build what the customer needs.

It is worth repeating (again) that this is not a situation to enter lightly. You already have a product that is designed to meet this need. There is a real good chance you are throwing good money after bad. However, if you are going to do it, do it right — forget what you think you know. Start from the beginning. Learn at every turn.

Re-engineering is the activity of changing (hopefully to improve) the structure and design of software. It is distinct from refactoring which is a specific technique. Refactoring can be used to re-engineer, but it can also be used to incrementally improve the design as we add new capabilities. Re-engineering does not have to follow the contract of refactoring (small incremental design changes that preserve observable behavior.) In fact, re-engineering is often done to improve a system by changing its behavior, but not necessarily its user functionality.

If the last sentence in the preceding paragraph doesn’t scare you then I have not yet done the job I intend to do.

Re-engineering is not product development, because it is not primarily concerned with creating useful functionality for customers. Re-engineering can be customer driven, but it tends to be somewhat indirectly customer driven. It tends to be concerned with things that are traditionally considered architectural properties: performance, scalability, security, quality, usability, consistency, etc.  The customer might be saying, “This app does what I need, but it is too slow.” Something like that may drive us to re-engineer.

Re-engineering is highly problematic from an Agile point of view. We are not producing functionality. We are not working from user facing stories. There may be business value to what we are doing (such as improving the performance of our application) but it is not directly measurable business value in most cases.

Re-engineering is also too often driven by technical concerns within the team (or its management). We didn’t build it as well as we wish we had. We have learned new things now, and we want to incorporate them.

Learning is the key. Agile is all about learning. The feedback mechanisms in Agile processes are designed to help us learn as we go. The intent is that we will apply this learning immediately rather than waiting until some future (“post mortem”) time to re-evaluate.

For a functional Agile team these learning mechanisms exist at multiple (somewhat redundant) levels: TDD has tiny learning cycles that are as small as a few seconds to a few minutes. Each time a test passes or fails we have learned something. Continuous Integration is a learning cycle (Especially if we have an Informative Build.) When the build passes or fails we are meant to learn something. Stories, in the form of Card-Conversation-Confirmation are learning cycles. Releases, and the consequent customer feedback, are learning cycles. Each time we do these things we learn, and each time we learn we are meant to immediately attempt to improve.

Re-engineering is a poor business proposition. We are spending additional money on a product that is already functional. The return on that investment is hard to anticipate. We can tell the customers that we have improved the product in some ways that they may or may not be able to appreciate (or even notice.) Therefore, the risk is very high, and the best case outcome is the the product does a slightly better job doing what it already does.

Agile is not compatible with re-engineering. Agile is an alternative to re-engineering. Agile expects us to refactor our design and to incorporate our changing understanding (i.e. learning) as we develop our product, not after the fact.

The pdf of the presentation can be found here: