First, programming is automating. If you are a programmer what you do is tell computers how to do things for people. That is automation. If you need to do things you should take advantage of the talent you have for automating and automate the things you need to do.

Second, as a programmer your time is expensive. Also, the things you program are complex and the opportunity to make mistakes is ever present. For both of these reasons you don’t want to be doing the same things over and over again. Any time you are going to do the same thing again you should automate it.

I want all the programmers out there to be very suspicious of doing things that are similar to things they have done before. When you write some code that is similar to code you have written before you are creating duplication. You should find that duplication and refactor to eliminate it.

Similarly, when you have to do some activity that you have done before, such as testing for regressions, deploying an application onto a server, uploading data to a database, creating some report, or any of the many administrative tasks you do every day, you should automate it. That way you will spend less time doing it over and over and you will have fewer opportunities to make mistakes.

Faster and higher quality for the small cost of a little programming (Ostensibly what you want to be doing anyway.) How does that sound?

However, this only makes sense if we don’t value learning. If we don’t value learning then we can assert that we who know, know. We know because we have learned, and we can share what we know with those less fortunate than us. They should be grateful to have benefited from our wisdom and to have been spared the many trials that we have endured.

If we value learning then we value systems that can learn, but that implies that we must let go. We must let go of the idea that what we have learned is what others should learn. Their experiences are necessarily different than ours. If they succeed it is their success. If they fail it is not our failure.

The quality of a good coach/teacher/parent, is to provide everything that is needed to succeed, to motivate, to encourage, to endure, and to get the hell out of the way.

There are two things we ought to be talking about: 1) How do we change our organizations to optimize the delivery of value? 2) How do we make the lives of people involved in software development better?

XP, Scrum, Lean, Kanban, Agile, and whatever else are collections of ideas and practices each united by strong principles and all designed to help us figure out how to solve some of these problems. Each is limited by its point of view. None is a panacea. None will fit your situation “out-of-the-box” without considerable thought and adaptation.

So, use these ideas and these practices. Do so thoroughly and unapologetically. But, I don’t care what you think Agile is or is not, whether you think it is good or bad, whether you think it works or doesn’t work, whether you think I am doing it right or doing it wrong. It doesn’t matter.

An Agile team is like an engine. An engine needs four basic things to work well:

1. Feed it lots of fuel and air at the right moment.
2. Ignite the fuel and air at the right moment to generate power.
3. Transfer power to the drive line to do some useful work.
4. Remove waste gasses at the right moment to maintain volumetric efficiency on the next stroke.

Agile teams need similar things:

1. Feed them valuable ideas to implement at the right moment.
2. Implement high-quality software at the right moment to realize value.
3. Transfer that value out to the customer so that they can do their work.
4. Receive feedback at the right moment to maintain the delivery of value on the next iteration.

Extending the metaphor:

• Scrum is a bigger engine. It is theoretically capable of generating higher output, if you give it all of the things above.
• XP (or “technical practices” if you prefer) is a well tuned ignition system. You need it to maximize efficiency and clean power output.
• Kanban says, “Find the bottleneck and eliminate it.” That is great advice (For engines too) but maybe not specific enough to tell us how to make a Civic beat a Ferrari. [Note: I deliberately diminished the importance of finding and eliminating bottlenecks here. To be clear, I think it is one of the most important skills a coach (or engine builder) can have.]
• Effective product ownership and related business practices are your intake plenum. The fastest way to sap an engine’s power is to starve it here (That’s why they call it the “throttle.”)
• Your direct line to your customer is like your exhaust system. The tighter and twistier and more packed with crap it is the less power you will get.

To go fast you need all of these parts to work together. So, start with a bigger engine. Sure, why not? Start with more efficient spark. Sure, why not? If you want to get faster you’ve got to understand the whole system, and it more or less doesn’t matter where you start.

Of course, he’s absolutely right, and it leads nicely to one of my favorite topics that I’ve been wanting to blog about:

Agile software development is about learning. We work from a premise that the final solution is not known (or knowable) until we have delivered it, and that, therefore, we must collaborate with customers to build and define a working solution incrementally. Along the way we learn about the problem space that we are exploring and we incorporate that knowledge and our own expertise to drive towards a heuristic solution.

That may seem obvious, but it has an interesting implication. All software development processes/methodologies tend to obsess over the question of how much we can get done per unit time. However, if we know that we are engaged in learning, and we understand that learning means sometimes getting it wrong in order to adjust and get it right, then how much we can do is the wrong question. The right question is how long does it take before we can find out that we are wrong. In other words, what is the delay before our next opportunity to learn?

Scrum’s notion of “Done” is not really about being done, but about getting to the point where we can get real feedback to learn from. The same is true for the Lean notion of “Cycle Time.” Each cycle is an opportunity to learn. If there is any chance that we will get it wrong then we need at least two cycles. In fact, the likelihood that we will ever get it right is directly related to the number of opportunities we have to learn before we stop. I’ll let you chew on that for a while.

[Note: the assertion in the last paragraph originally said that the likelihood of getting it right within some period of time was inversely related to the length of our learning cycles. Hopefully, the way I wrote it above is somewhat clearer. The notion of keeping learning cycles small is a natural and important consequence of the above. You will hear more from me about that at some point.]

It is a peculiar question, because if we weren’t practicing Scrum the answer would be obvious. In fact, the Lean and Extreme Programming communities more or less agree on what “done” means without any real discussion: Done == deployed in production, in the hands of real customers.

When Scrum folks talk about the “Definition of Done,” however, they mean something slightly different. There is an important assumption here: software can reach an intermediate state called “done” where it is not yet “released,” but is “potentially shippable.”

Potentially shippable is an equally peculiar idea. In theory it ought to mean that to release or not to release is purely a business decision at this point. The software is complete, accepted, and tested, but it is not released. In practice it often means that the main development activities have been completed but there is some amount of intangible work to be done that may or may not fit inside of a Sprint (e.g. QA testing, deployment to various environments, formal release process, etc.) This is a poor, but common, way to address the “Definition of Done.”

Potentially shippable software is actually a useful idea. From a technical excellence point of view the goal should be continuous delivery, but from a business perspective continuous delivery might not be feasible. In fact, in some domains big-bang delivery actually has inherent value (e.g. any COTS, especially commercial games, and various forms of online media.) [actually, even in those environments continuous delivery is useful, but only after the initial release hits its date...] What we want is for continuous delivery to be attainable, so that it becomes a business decision not to do it. Thus, XP has long had the concept of always releasable software.

So, how do Scrum teams define done? I suppose the real answer is, however they want to, but I suggest the following: Start with everything that it would take to release the software to production. Then, to the extent that it is infeasible to do all that stuff, scale back to something that can be accomplished within a Sprint. Then, continuously improve (i.e. inspect and adapt) until the definition is once again everything that it would take to release the software to production.

Test-Driven Development is a disciplined approach to software design that provides a mechanism for programmers to create working software in tiny increments of no more than a few lines at a time. The reason that this is important is that Agile approaches to software development more or less require us to deliver working software in small increments that fit inside a two-week iteration (give or take.) In order to have working software that meets some customer need in such a short time frame we need a way to create even smaller increments of software, see them work, build a little more, see that work, etc., and improve the design as we go so that we don’t create a mess.

The way that Test-Driven Development accomplishes this is by leaning on unit testing frameworks to build a tiny test that specifies a tiny increment of behavior. We write that test before we have written the production code that will make that test pass. Then we write the code, taking care to do no more than is absolutely necessary to make our test pass. Finally, we take a step back and look for indications that our design could be improved, the main one being duplication that is created by bits of code that do or say much the same thing. When we see such duplication we refactor it using techniques and tools that allow us to change the code safely without breaking the test.

We repeat this process over and over. Each cycle takes no more than a minute or two, and each cycle builds incrementally on top of the prior ones until we have created some working feature that the customer has asked for. All the while we are seeing all of the small increments we have built work, via the tests. All the while we are seeking and exploiting small opportunities to improve and simplify the design of the code. And, in the end we have demonstrable, working software that is capable of filling some customer need. All we have to do is integrate and deploy it and we can safely move on to the next thing on our list.

That is more or less it. So, now that we have talked about what Test-Driven Development is, let’s take a moment to discuss what it is not:

Test-Driven Development is not about testing, not really. The purpose of tests is to verify the quality and functionality of software. The purpose of what we are doing is to set an expectation of what we need to do next and have a way to quantify when that expectation has been met. We get verification as a result. That is: later we can run the tests and see them pass and be assured that no regression has happened (At least in the behavior that we specified.) But, that is not the goal that we set out with. We set out with the goal of driving small increments of code that actually work. We needed a way to define what those small increments were, and the test framework gave us a harness in which to run tiny increments of code, set expectations about how the code would work, and see those expectations satisfied.

It is important to realize that Test-Driven Development is not about testing, or quality, because it is important to realize that quality is not achieved through TDD alone. Several studies have shown that software produced with TDD tends to have less bugs, but that is a side-effect of TDD and not the goal per se. In order to have quality we need to be able to verify that the software works the way the customer intends. We need a way to verify that it looks good, is usable, accomplishes work that the customer needs it to do, performs adequately, is free from obvious defects, etc. Test-Driven Development is not about these things.

Quality, in Agile, is achieved by working in small increments, by collaborating to set clear expectations up front, by using Test-Driven Development to build the software, and by performing various kinds of testing to assure that the product meets the goals we set early and continuously throughout the process. Test-Driven Development is only about how we build the software. We need other techniques to assure that the software meets our customers’ needs.

My previous post about re-engineering did not discuss what to do in these circumstances. I would recommend the following:

1. Think really hard about it: is there a value proposition here? Or does the existing product do what we need it’s just not that great? Think about it again.
2. If you are certain that there is value then start a new project to replace the existing product.
3. This is the most important step. Forget everything about the existing product. Eliminate it from your brain.
4. Develop the new product in the Agile way. Find out what the customer wants to do and build software that enables them to do it. Do it incrementally and get as much feedback as you can.
5. Always, always challenge your assumptions. It is tempting to assume that the product has to work like the old one, or that you should do something that you wanted to do in the old one but never did. Forget it. Build what the customer needs.

It is worth repeating (again) that this is not a situation to enter lightly. You already have a product that is designed to meet this need. There is a real good chance you are throwing good money after bad. However, if you are going to do it, do it right — forget what you think you know. Start from the beginning. Learn at every turn.

Re-engineering is the activity of changing (hopefully to improve) the structure and design of software. It is distinct from refactoring which is a specific technique. Refactoring can be used to re-engineer, but it can also be used to incrementally improve the design as we add new capabilities. Re-engineering does not have to follow the contract of refactoring (small incremental design changes that preserve observable behavior.) In fact, re-engineering is often done to improve a system by changing its behavior, but not necessarily its user functionality.

If the last sentence in the preceding paragraph doesn’t scare you then I have not yet done the job I intend to do.

Re-engineering is not product development, because it is not primarily concerned with creating useful functionality for customers. Re-engineering can be customer driven, but it tends to be somewhat indirectly customer driven. It tends to be concerned with things that are traditionally considered architectural properties: performance, scalability, security, quality, usability, consistency, etc.  The customer might be saying, “This app does what I need, but it is too slow.” Something like that may drive us to re-engineer.

Re-engineering is highly problematic from an Agile point of view. We are not producing functionality. We are not working from user facing stories. There may be business value to what we are doing (such as improving the performance of our application) but it is not directly measurable business value in most cases.

Re-engineering is also too often driven by technical concerns within the team (or its management). We didn’t build it as well as we wish we had. We have learned new things now, and we want to incorporate them.

Learning is the key. Agile is all about learning. The feedback mechanisms in Agile processes are designed to help us learn as we go. The intent is that we will apply this learning immediately rather than waiting until some future (“post mortem”) time to re-evaluate.

For a functional Agile team these learning mechanisms exist at multiple (somewhat redundant) levels: TDD has tiny learning cycles that are as small as a few seconds to a few minutes. Each time a test passes or fails we have learned something. Continuous Integration is a learning cycle (Especially if we have an Informative Build.) When the build passes or fails we are meant to learn something. Stories, in the form of Card-Conversation-Confirmation are learning cycles. Releases, and the consequent customer feedback, are learning cycles. Each time we do these things we learn, and each time we learn we are meant to immediately attempt to improve.

Re-engineering is a poor business proposition. We are spending additional money on a product that is already functional. The return on that investment is hard to anticipate. We can tell the customers that we have improved the product in some ways that they may or may not be able to appreciate (or even notice.) Therefore, the risk is very high, and the best case outcome is the the product does a slightly better job doing what it already does.

Agile is not compatible with re-engineering. Agile is an alternative to re-engineering. Agile expects us to refactor our design and to incorporate our changing understanding (i.e. learning) as we develop our product, not after the fact.

Ron Jeffries suggests that the Scrum Alliance ought to just drop the word certification. He asserts that sending people to classes is useful because they get exposed to ideas that they may not be adequately exposed to otherwise. However, going to a class should not be called “certification,” because it is imprecise and alienates parts of the community it means to serve. Ron is right.

Tobias Mayer has complained that the new Certified Scrum Developer course is no better than the Certified Scrum Master course since both are merely a couple days of training during which a human being can only retain so much. Tobias is wrong for a number of reasons. First of all, he’s not really in a position to judge something he’s never seen or done. He is also wrong because the CSD class has the one thing that all of the official Scrum Alliance classes have lacked to date – relevance to software development.

Attendees of a CSD class have to write software. This is supposed to have something to do with Agile, and the Manifesto says we value working software. So, maybe we should write some? Tobias is right to say that this isn’t enough, but it is the very first step in the right direction… ever.

I believe that it is absolutely valuable to attend a CSM or CSD course. The CSM course ought to be called, “Intro to Scrum: Do Not Try This At Home.” The CSD course ought to be called, “Intro to Extreme Programming: No Really, Someone Might Get Hurt.” The two of them together are barely a starting point. They are a little bit better than reading a book and trying to get someone to pay you to attempt what it says (The way many of us learned, including myself.) They still aren’t enough by themselves.

To be able to do this stuff well, I think you need at least the following things:

1. A commitment to do your very best and always strive to do a bit better each time.
2. Some experience attempting these practices in a setting where mistakes don’t matter: a user group, a class, doing katas on your own or with friends or coworkers.
3. Access to people who have gone before you so that you can listen and learn from their experiences, and a willingness to do so.
4. Plenty of time for these ideas to ferment and turn into something useful.