BigVisible Blog

George pointed out that sometimes we have to rebuild systems. Sometimes they just aren’t that good. Sometimes the technology we are using is out-of-date and no longer supported. Sometimes we need to add functionality, and the design is just so bad that we can’t figure out how to add it.

My previous post about re-engineering did not discuss what to do in these circumstances. I would recommend the following:

1. Think really hard about it: is there a value proposition here? Or does the existing product do what we need it’s just not that great? Think about it again.
2. If you are certain that there is value then start a new project to replace the existing product.
3. This is the most important step. Forget everything about the existing product. Eliminate it from your brain.
4. Develop the new product in the Agile way. Find out what the customer wants to do and build software that enables them to do it. Do it incrementally and get as much feedback as you can.
5. Always, always challenge your assumptions. It is tempting to assume that the product has to work like the old one, or that you should do something that you wanted to do in the old one but never did. Forget it. Build what the customer needs.

It is worth repeating (again) that this is not a situation to enter lightly. You already have a product that is designed to meet this need. There is a real good chance you are throwing good money after bad. However, if you are going to do it, do it right — forget what you think you know. Start from the beginning. Learn at every turn.

Agile software development is really a product development strategy. Scrum has its origins in product development.  If you read the Agile Manifesto there are all sorts of hints that we are talking about developing software as a product that is valuable to some known customer.  That is product development.

Re-engineering is the activity of changing (hopefully to improve) the structure and design of software. It is distinct from refactoring which is a specific technique. Refactoring can be used to re-engineer, but it can also be used to incrementally improve the design as we add new capabilities. Re-engineering does not have to follow the contract of refactoring (small incremental design changes that preserve observable behavior.) In fact, re-engineering is often done to improve a system by changing its behavior, but not necessarily its user functionality.

If the last sentence in the preceding paragraph doesn’t scare you then I have not yet done the job I intend to do.

Re-engineering is not product development, because it is not primarily concerned with creating useful functionality for customers. Re-engineering can be customer driven, but it tends to be somewhat indirectly customer driven. It tends to be concerned with things that are traditionally considered architectural properties: performance, scalability, security, quality, usability, consistency, etc.  The customer might be saying, “This app does what I need, but it is too slow.” Something like that may drive us to re-engineer.

Re-engineering is highly problematic from an Agile point of view. We are not producing functionality. We are not working from user facing stories. There may be business value to what we are doing (such as improving the performance of our application) but it is not directly measurable business value in most cases.

Re-engineering is also too often driven by technical concerns within the team (or its management). We didn’t build it as well as we wish we had. We have learned new things now, and we want to incorporate them.

Learning is the key. Agile is all about learning. The feedback mechanisms in Agile processes are designed to help us learn as we go. The intent is that we will apply this learning immediately rather than waiting until some future (“post mortem”) time to re-evaluate.

For a functional Agile team these learning mechanisms exist at multiple (somewhat redundant) levels: TDD has tiny learning cycles that are as small as a few seconds to a few minutes. Each time a test passes or fails we have learned something. Continuous Integration is a learning cycle (Especially if we have an Informative Build.) When the build passes or fails we are meant to learn something. Stories, in the form of Card-Conversation-Confirmation are learning cycles. Releases, and the consequent customer feedback, are learning cycles. Each time we do these things we learn, and each time we learn we are meant to immediately attempt to improve.

Re-engineering is a poor business proposition. We are spending additional money on a product that is already functional. The return on that investment is hard to anticipate. We can tell the customers that we have improved the product in some ways that they may or may not be able to appreciate (or even notice.) Therefore, the risk is very high, and the best case outcome is the the product does a slightly better job doing what it already does.

Agile is not compatible with re-engineering. Agile is an alternative to re-engineering. Agile expects us to refactor our design and to incorporate our changing understanding (i.e. learning) as we develop our product, not after the fact.

Uncle Bob Martin says programmer certification is bad. He supposes that if you start giving out certifications to people that someone will assume those certifications mean something. They will start hiring certified people preferentially, or stop hiring non-certified people entirely. This would be bad, because certification is almost certainly a poor measure of whether you should hire someone. Uncle Bob is right.

Ron Jeffries suggests that the Scrum Alliance ought to just drop the word certification. He asserts that sending people to classes is useful because they get exposed to ideas that they may not be adequately exposed to otherwise. However, going to a class should not be called “certification,” because it is imprecise and alienates parts of the community it means to serve. Ron is right.

Tobias Mayer has complained that the new Certified Scrum Developer course is no better than the Certified Scrum Master course since both are merely a couple days of training during which a human being can only retain so much. Tobias is wrong for a number of reasons. First of all, he’s not really in a position to judge something he’s never seen or done. He is also wrong because the CSD class has the one thing that all of the official Scrum Alliance classes have lacked to date – relevance to software development.

Attendees of a CSD class have to write software. This is supposed to have something to do with Agile, and the Manifesto says we value working software. So, maybe we should write some? Tobias is right to say that this isn’t enough, but it is the very first step in the right direction… ever.

I believe that it is absolutely valuable to attend a CSM or CSD course. The CSM course ought to be called, “Intro to Scrum: Do Not Try This At Home.” The CSD course ought to be called, “Intro to Extreme Programming: No Really, Someone Might Get Hurt.” The two of them together are barely a starting point. They are a little bit better than reading a book and trying to get someone to pay you to attempt what it says (The way many of us learned, including myself.) They still aren’t enough by themselves.

To be able to do this stuff well, I think you need at least the following things:

1. A commitment to do your very best and always strive to do a bit better each time.
2. Some experience attempting these practices in a setting where mistakes don’t matter: a user group, a class, doing katas on your own or with friends or coworkers.
3. Access to people who have gone before you so that you can listen and learn from their experiences, and a willingness to do so.
4. Plenty of time for these ideas to ferment and turn into something useful.

I posted this to the LAJUG mailing list, then later to the Extreme Programming Yahoo! group. Some folks really liked it, so I’m putting it up here for posterity.

The Java Programmer’s Rules of Design are, in order of priority:

1. Should not have any tests, because I wrote it so it works.
2. Should clearly express how clever I am.
3. Good idioms should be repeated often. Better idioms oftener.
4. Should contain as many classes and methods as I might possibly need
   at some point in the future.

(Read here if you don’t get the joke.)

P.S. If you are a Java programmer and this makes you sad just substitute your least favorite of: C#, C++, or Perl. It works for them too…

A few of you may know that the first ever Certified Scrum Developer course taught by Ron Jeffries and Chet Hendrickson was held the first week of May in Cleveland, OH. A couple of those may also know that I attended along with a handful of well respected coaches (George Dinwiddie, John Kern, Jeff “Cheezy” Morgan, Paul Nelson, and Dave Nicolette) and some other Agile developers from all around. It was an interesting class and I think an exciting move forward for our community.

There is a lot of confusion and opinion around the idea of developer certification, much of it justified. I think that Ron and Chet have effectively taken the high road on this. On one hand, they have bowed to its inevitability and decided to become the first to attempt to do it well in the context of Agile and Scrum. On the other hand, they have done their best to make sure that the course is uncompromising in showing what it is really like to work on an Scrum team following a disciplined engineering approach.

In this last regard I feel that they have been quite successful. The course was, in many ways, identical to many real projects that I have participated in except for the absurdly abbreviated schedule (Necessary to fit the experience into just three days.) It was even effective in presenting some of the usual problems: set up problems, communications problems, arriving at shared expectations, etc. What was interesting (and enlightening, and mildly entertaining…) was that even this highly experienced group of attendees couldn’t avoid some well known problems including ones that we coach teams every day to avoid.

Too many cooks spoil the broth, or so the old adage goes. It seems that maybe too many coaches spoil the product as well. One of the interesting and unforeseen consequences of having so many highly experienced people work together is that we didn’t appear to be as effective as we might have expected.

There are a number of likely reasons for that. For one thing, I think we all wanted to give due respect to our peers, who we knew by reputation but had never worked with before. For many of us, myself included, it would have been more usual to take a strong, principled stance on certain issues, but instead we may have tried too hard to just get along. I think there was also a tendency for the coaches to coach instead of just getting stories done. In a real project these sorts of team dynamics would work themselves out over time, but in the short time allotted we had only begun to gel as a team.

Overall, it was a great experience. I met some cool people, and had a lot of fun slinging Java code with them using TDD, pair programming, and continuous integration through and through. I hope that everyone gets the opportunity to experience this course and find out what the whole XP/”engineering practices” hoopla is all about. Hopefully, BigVisible will soon be involved in making that happen.

It has been a while since I wrote the first part of this post. Sorry for the delay.

In the first part of my post about essential complexity I said that there would be a second part where I explained the approach to managing complexity on Agile teams. However, I also gave the answer in the first part. The answer is Test-Driven Development.

The diagram below shows the usual way that complexity is managed on software projects:

When we start building the solution we have certain ideas about what we are going to need to do and how we are going to solve upcoming problems. We build those ideas into the approach to our solution, but along the way we discover new problems and assumptions. So, we build those onto the solution we started with. By the time that we approach our first release we have created something that contains a certain amount of accidental complexity.

Our solution is represented by the black arrow in the diagram. The essential complexity is represented by the gray arrow. In order to make our solution perform well and respond to change we attempt to drive the complexity down through refactoring. Unfortunately, at this point the solution already contains too much complexity to refactor easily. This is the familiar situation of “legacy code.”

TDD recommends something radically different as represented by the following diagram:

We create a solution that is naively, terribly, overly simple. We know that this solution won’t solve all of our problems, but it solves the problem that we have immediately posed to it (As represented by the tests.) In order to arrive at a solution that contains our essential complexity we continue to add more and more tests that ask further questions of our system. Then we work diligently to ensure that the system is no more complex than necessary to continue to pass these tests. We use refactoring right from the beginning to achieve this.

Working in this way we are able to drive from a system that is too simple to one that is just complex enough. The real beauty comes from the realization that we can often get considerable business value from incomplete solutions. That is to say that our customers benefit when we deliver solutions that do not manage to tackle the essential complexity of the problem posed to us but merely a useful subset of it. Further, our overly simple solution, which has value to our customers, is very lightweight and easy to change. So, we can add features to it at a blistering rate, and continue to learn and get feedback.

There is another advantage to this approach. Our system is always tested, and therefore testable. Those tests provide some assurance that the features we have implemented still work. So, we are always able to make changes safely and quickly. The only bugs that such a system can produce are the result of tests that we forgot to write because we didn’t think of them. However, it is trivially simple to add those tests, and generally quite easy to add the minimum functionality needed to make them pass.

Wait a minute… that sounds just like what we said we do when we add a new feature. In fact, it is. Discovering bugs is just discovering essential complexities that we have yet to address adequately. If we have a lot of complexity to manage when we do that then it is difficult, but if the majority of our mistakes are a result of a solution that is too naively simple then it is almost trivially easy. This is the power of TDD.

[Update: I changed the size of the images so that they don't bork the whole page flow.]

This is an important topic, and I intend to approach it un-gently. Apologies in advance…

Software Architecture, at its worst, is the art of creating solutions looking for problems. Unfortunately, it has the ability to achieve its worst if we are not diligent.

There is not, has never been, and will never be a role called “Architect” on Scrum or XP teams. Scrum says that Product Owners own problems, “the what”, and Teams own solutions to those problems, “the how”. The Whole Team needs to decide how to best solve the problems it faces with the simplest solution possible in a short time frame. Therefore, the Whole Team owns the architecture of the solution.

Recently, at a client, it was pointed out that if teams are allowed to solve problems however they see fit, and if multiple teams solve the same problem different ways, then no one will know what the best solution is. Chaos will ensue.

Interesting problem. Let me take a stab at it:

1. I don’t have any problem with chaos per se. It might even be good for you.
2. If the organization has multiple teams that are capable of solving the same problem, over and over, in simple effective ways, without learning anything new, then we may have a problem. It’s a problem I would love to have.
3. Given that we mostly don’t have this problem (Not anywhere that I have looked) perhaps we should try to foster teams that can create simple solutions, learning as they go, and then solve that problem when we get to it.

Reuse is mostly a red herring. There is a distinct lack of evidence that we can design for reuse and create a solution that is sufficiently simple to be flexible and responsive to change. Since Agile teams value the latter, it may not be in their best interest to attempt the former. Instead, what we tend to do is to build solutions that are more complex than the problem at hand requires (Remember essential complexity?) That complexity is like a weight we have to carry with us everywhere we go. It effects the nimbleness with which we adapt to change.

When we “architect” we bring in that accidental complexity in the name of things like reuse, performance, scalability, testability, security, etc. Not that those goals aren’t worthwhile, but they are things which are best seen in the context of what our users need to do rather than as ends to themselves. Generally, from the user’s perspective, it needs to work and do something useful first and foremost.

This is where I go back to the idea of essential complexity: first build a solution that is too simple to work. That is zero architecture. Then build that solution up incrementally, massaging and caring for the design along the way, until you have something that solves the problems you set out to solve. That is evolutionary architecture. Anything bigger or more upfront than that is probably creating solutions looking for problems.