It has been a while since I wrote the first part of this post. Sorry for the delay.

In the first part of my post about essential complexity I said that there would be a second part where I explained the approach to managing complexity on Agile teams. However, I also gave the answer in the first part. The answer is Test-Driven Development.

The diagram below shows the usual way that complexity is managed on software projects:

When we start building the solution we have certain ideas about what we are going to need to do and how we are going to solve upcoming problems. We build those ideas into the approach to our solution, but along the way we discover new problems and assumptions. So, we build those onto the solution we started with. By the time that we approach our first release we have created something that contains a certain amount of accidental complexity.

Our solution is represented by the black arrow in the diagram. The essential complexity is represented by the gray arrow. In order to make our solution perform well and respond to change we attempt to drive the complexity down through refactoring. Unfortunately, at this point the solution already contains too much complexity to refactor easily. This is the familiar situation of “legacy code.”

TDD recommends something radically different as represented by the following diagram:

We create a solution that is naively, terribly, overly simple. We know that this solution won’t solve all of our problems, but it solves the problem that we have immediately posed to it (As represented by the tests.) In order to arrive at a solution that contains our essential complexity we continue to add more and more tests that ask further questions of our system. Then we work diligently to ensure that the system is no more complex than necessary to continue to pass these tests. We use refactoring right from the beginning to achieve this.

Working in this way we are able to drive from a system that is too simple to one that is just complex enough. The real beauty comes from the realization that we can often get considerable business value from incomplete solutions. That is to say that our customers benefit when we deliver solutions that do not manage to tackle the essential complexity of the problem posed to us but merely a useful subset of it. Further, our overly simple solution, which has value to our customers, is very lightweight and easy to change. So, we can add features to it at a blistering rate, and continue to learn and get feedback.

There is another advantage to this approach. Our system is always tested, and therefore testable. Those tests provide some assurance that the features we have implemented still work. So, we are always able to make changes safely and quickly. The only bugs that such a system can produce are the result of tests that we forgot to write because we didn’t think of them. However, it is trivially simple to add those tests, and generally quite easy to add the minimum functionality needed to make them pass.

Wait a minute… that sounds just like what we said we do when we add a new feature. In fact, it is. Discovering bugs is just discovering essential complexities that we have yet to address adequately. If we have a lot of complexity to manage when we do that then it is difficult, but if the majority of our mistakes are a result of a solution that is too naively simple then it is almost trivially easy. This is the power of TDD.

[Update: I changed the size of the images so that they don't bork the whole page flow.]

This is an important topic, and I intend to approach it un-gently. Apologies in advance…

Software Architecture, at its worst, is the art of creating solutions looking for problems. Unfortunately, it has the ability to achieve its worst if we are not diligent.

There is not, has never been, and will never be a role called “Architect” on Scrum or XP teams. Scrum says that Product Owners own problems, “the what”, and Teams own solutions to those problems, “the how”. The Whole Team needs to decide how to best solve the problems it faces with the simplest solution possible in a short time frame. Therefore, the Whole Team owns the architecture of the solution.

Recently, at a client, it was pointed out that if teams are allowed to solve problems however they see fit, and if multiple teams solve the same problem different ways, then no one will know what the best solution is. Chaos will ensue.

Interesting problem. Let me take a stab at it:

1. I don’t have any problem with chaos per se. It might even be good for you.
2. If the organization has multiple teams that are capable of solving the same problem, over and over, in simple effective ways, without learning anything new, then we may have a problem. It’s a problem I would love to have.
3. Given that we mostly don’t have this problem (Not anywhere that I have looked) perhaps we should try to foster teams that can create simple solutions, learning as they go, and then solve that problem when we get to it.

Reuse is mostly a red herring. There is a distinct lack of evidence that we can design for reuse and create a solution that is sufficiently simple to be flexible and responsive to change. Since Agile teams value the latter, it may not be in their best interest to attempt the former. Instead, what we tend to do is to build solutions that are more complex than the problem at hand requires (Remember essential complexity?) That complexity is like a weight we have to carry with us everywhere we go. It effects the nimbleness with which we adapt to change.

When we “architect” we bring in that accidental complexity in the name of things like reuse, performance, scalability, testability, security, etc. Not that those goals aren’t worthwhile, but they are things which are best seen in the context of what our users need to do rather than as ends to themselves. Generally, from the user’s perspective, it needs to work and do something useful first and foremost.

This is where I go back to the idea of essential complexity: first build a solution that is too simple to work. That is zero architecture. Then build that solution up incrementally, massaging and caring for the design along the way, until you have something that solves the problems you set out to solve. That is evolutionary architecture. Anything bigger or more upfront than that is probably creating solutions looking for problems.

Thanks to everyone from the Mass Bay PMI Chapter for coming to see me speak about Agile in the Enterprise. It was a great discussion and I thoroughly enjoyed it. I have made my slides available from tonight’s presentation, they can be downloaded here.

Also, several people expressed some interest in local Agile groups so that they could learn more. I would point out three specific ones that have monthly meetings and support vibrant communities of both learners and practitioners:

• Agile Bazaar
• Agile Boston
• Nashua Scrum Club

Yesterday I had the opportunity to attend the “Agile Comes to You” event in Waltham. It was great to see so many people interested in learning more about Agile and I had some great conversations. Here is a copy of the slides I used

Discipline in Agile Teams

The other day I had an interesting thought. I’ m not sure what precipitated it exactly, but there were several things that led me to this idea I’ve been mulling in my head. Perhaps it was Alistair Coburn’s keynote at Agile 2009 where he said that Agile as a distinct entity was gone; if it was once an iceberg, it has since melted and is now just part of the ocean. It could have been Jeff Sutherland’s presentation where he points out that 84% of IT organizations are self-reporting to use Scrum. Or perhaps it was simply working with a current client when they were asking for my help to come up with very clear guidelines about the number of acceptance criteria that should be allowed for a single story. Anyway, it struck me: as Scrum grows in popularity, are we institutionalizing it? more »

In 1996 a friend of mine asked me what my New Years Resolution was. I didn’t have one, so I thought one up on the spot, “My resolution is to quit making New Years Resolutions.” I have stuck with it for fourteen years now.

The idea behind New Years Resolutions is great. We should reflect on the year that has past, think about what we would like to do differently in the future, and resolve to make that change. That is a great thing to do. The problem is that when we only do it once a year we tend to fail. Then, we wait a whole year to reflect and do it again.

So, this year resolve to stop making New Years Resolutions. Instead, reflect on your behavior soon, and make changes right then. Change every month, every week, every hour. Soon you will get good at changing and those promises you make to yourself will come true more often than not.

That is the lesson of Agile: self-reflection is hard, and we kind of suck at it. Change is hard, and we kind of suck at it. If we do it more often then we will have to do less each time and it won’t be so hard. If we do it more often then we will get good at it and it won’t suck so much.