This way of thinking and approach to change may drastically limit the success of your journey to becoming an agile organization.

Some indications that your organization treats change as a temporary state to be controlled, and may need help moving to a more appropriate theory of change for today’s complex environment:

• Establishing a standards board to cull “best practices” from a cherry-picked pilot team and/or from research to then “roll out” a new ["standard" agile] process to other teams (“surely only the pilot team was able to do something effective without a standard!”)
• Preventing teams from improving until someone has determined the “best practices” that they can then implement (e.g. “you can’t introduce that practice until the centralized group has determined the best process™ and best tool™ for everyone to use!”)
• Taking experiences of success from implementing a practice or improvement in one context (organization, project, situation) and dictating their use as standards in others contexts (“<practice x> worked with <SuperPilotTeam>, other teams must use <practice x> too!”)
• Measures of transition success based on “adoption percentages”

A Reality Check for Most Organizations Considering (or in the middle of ) Transitions to Agile, Lean, etc.

If you can relate to this, the ship that is your agile initiative may be sinking.  Over time, the initial excitement generated by the introduction of agile principles may be crushed by your management and organization as they ignore the learning process your teams have gone through, and carve out standards to be applied to everyone.

Basically, you may have the appearance of agile, minus the agile part – the practices without the principle; the motions without the capability.

This is one of several causes of the phenomenon known as CrAgile (“Crappy Agile,” a.k.a. “Scrummerfall,” “Scrum-But,” etc.) – wherein an initial exciting introduction of new ways gradually reverts to old ways.

The Learning Journey

Standards, when used as described above, don’t result in success of complex endeavors like product development efforts.  They result in begrudging compliance and, if you are lucky, successful following of steps as opposed to achievement of outcomes.  From years of coaching, consulting, and being a member of many successful and unsuccessful product development and change efforts, I have realized a few things about learning new ways, such as:

• Each person and team proceeds on its own “learning journey” when trying new things (see diagram). The real learning is in understanding what things work (and don’t) in different situations, reflecting on these experiences to understand why they did or didn’t work, modifying and improving them to work, and continuing to expand their awareness to continuously determine better ways.
• Standards and best practices, used wrongly, are an attempt to short cut this learning process.  Ironically, the short cut only cuts out the “learning” part, and results in blind adherence to practice, and rarely greater capability.

This diagram shows the learning journey of a team.  Over time, the team tries new practices and techniques.  They learn how things work, and how to customize practices and interactions in ways that work for them – in their environment, for their challenges.  The practices alone were not the reason for growing in capability, but rather trying the practices and learning about when and how to apply them.  The common management mistake is to assume that taking the practices that a high-performing team is using, and copying them to other teams, will result in the same improvement.

The Only “Best” Practice – Continuous Learning and Improvement

“There are no best practices, only better practices”

I don’t know where I first heard/read this quote, or who it came from (let me know and I’ll update this page- my searches yield many writings). What I do know is that the pursuit of “best practice” has a number of subtle problems:

1. It implies that once these practices are found, improvement is done. This is consistent with the “unfreeze-change-refreeze” model of change associated with Kurt Lewin in the early 1900′s. This older model of change is often criticized for being too inflexible to deal with the overwhelming amount of change and disruption we face today.
2. It mistakenly assumes that effectiveness of practices is context-independent (i.e. what works well in/for one situation, person, team, group, environment will work as well in other situations, persons, teams, groups, environments). Consider how much can be different about situations, people, teams, groups, and environments and it becomes easy to imagine how a “best” practice in one place may not even be a useful practice in another.
3. It assumes that change is not constant.  More and more articles, books, and experience show that change is constant, and that organizations that will survive in the future must develop the capacity for continuous adaptation.

A Better Use of Standards and Best Practices

Reconsider how best practices are used in your organization.  Learn from lean thinking a different approach to improvement and standards.  

1. Change your focus on standards from that of arriving at “best practice” to constantly finding “better practice”.
2. Shift the purpose of centralized groups from centers of governance to centers of enablement – helping to encourage teams to visit each other to learn about successful approaches
3. Encourage the building of communities of practice wherein people from different teams and group can share experiences and learning.

Lean “Yokotenkai”

In the lean world, the concept of yoko tenkai (which I’ve read is poorly translated as something like “beside/to the side, expansion” – shortened to “yokoten”) is a process of sharing best practices from one part of the organization to others, and (as importantly) improving or expanding them constantly to meet the needs of the new context. There are two aspects of yokoten- both are necessary, and neither is sufficient on its own.

1. The first is the “copying” of best practice to the side. This involves learning about discovered “best practices” from other parts of the organization.  A subtle (but critical) nuance is that these practices are not designed and dictated by some governing body- instead, the process of “copying” involves people sharing learning and observing practices in other groups as they are being used, in context.  It is a sideways proliferation of learned practices across the organization, fueled by observation of these practices where they are working.
2. The second part is “expansion.” Practices are not to be blindly and arbitrarily applied to other parts of the organization.  Instead, the practices are modified and improved to suit the needs of each group and context.  The result is continuous improvement, fueling more sharing.  This results in continuous “better” practices.

When people from one team observe practices in use in another team, they are taking these practices, bringing them to their team, and will more likely customize these to work best on their team and “make it theirs,” with the goal of improvement.  Alternatively, when an authority tells a team to implement a practice, will do what they are told, with the goal of following the practice.

What can you do?

Here are some ways you can get started, today:

• Discuss the role of standards in your organization.  Discuss yokoten and what it means to your group.  Research the Toyota Production System and other lean implementations.
• If you are on a team, try to embody yokoten yourself!
  • reach out to other teams to learn how they’ve been successful
  • bring back learnings to your team and improve/modify them to suit your context
  • share any improvements you make with those who helped you
• If your organization is embarking on a large transformation in pursuit of organizational agility, and seems to be run in a manner similar to that discussed in the beginning of this article, seek expert guidance.

The following things may take a long time to instill in your organization – as they may need cultural change to be truly accepted:

Start shifting the role of centralized groups from designers and declarers of standards to enablers of yokoten

• Ensure teams get the help they need to be successful
• Be aware of learnings in various teams
• Encourage teams to visit other teams that have been successful, and bring back learnings to their teams to improve

Indeed, this binary obsession – that you must pick one specific approach and apply it to all dimensions of a project – is entirely unproductive. In fact, it is usually not the reality in an Agile or traditional project, but rather a limit of our own thought process. The challenge being that if we find merit in one approach, the mind can take an absolutist value. Agile is useful for dealing with product visions, therefore we should apply Agile principles to every aspect of our project.

The test of a first-rate intelligence is the ability to hold two opposed ideas in the mind at the same time, and still retain the ability to function.
F. Scott Fitzgerald, “The Crack-Up” (1936)

Fitzgerald accurately describes the challenge facing us when organizing and executing projects; we must hold two opposing ideas within our head and continue to function. These two opposing ideas well well described by Kevin Kelly when he coined the terms clockware and swarmware as the two different approaches to managing work. They have since been embraced by many complexity theorists and I present them here as my own attempt to hold two opposing ideas and continue to function and show how one builds upon the other, and to argue that a project requires one or the other will inevitably lead to failure.

Clockware

This is the perspective best described as deterministic or Newtonian in nature. Alluding the the mechanics of a clock, situations within this domain may be very complicated, but they are ultimately predictable and manageable with sufficient analysis and calculation. This perspective is widespread throughout western thought from the view of factory work expounded by Frederick Taylor’s Scientific Management to the modern educational models which lean heavily upon standardized testing and closed problem solving. When applied to modern management and project work, we see deep analysis, detailed plans and an overall effort to control the unfolding process. This is where most Scrum practitioners and Agilists become critical of this mindset, showing how a heavy analysis on planning leads to brittle projects that spend more time trying to adhere to a plan than ensuring they are getting the best outcomes possible.

Swarmware

If clockware assumes a static, closed system, then swarmware is focused on solving problems in a diametrically opposite environment. Swarmware frames problems as infinitely complex and changing in response to numerous factors including the actions one undertakes to manage the system. Based on this perspective, swarmware mandates a highly adaptive approach focused on experimentation, feedback and adaptation. Scrum would be an excellent example of a management approach that embraces this mindset. Frameworks like this are focused on emergent patterns and self organization over centralized planning.

One Building Upon the Other

The challenge in elaborating these two options as distinct choices is that it implies you must use one or the other, that there is some sort of active decision about which approach you think is better. Interestingly, some of the most powerful demonstrations of swarmware are conducted by organisms completely incapable of the simplest decision making, even if it appears they contain a significant level of intelligence. I recall at my old apartment a six month battle I waged against a colony of ants. Had I not known otherwise, I would have sworn there was a plotting queen ant somewhere telling her minions to avoid the places I put poison and to continue searching other cabinets as I tried numerous attempts to move food around and set traps to counter their offensive campaign into my kitchen. [Note of personal disclosure, I am an avid fan of National Geographic and museums of science. If you didn't know this about me already, you will by the end of this post]

If we consider the ants who plagued my apartment kitchen and terrorized my cat, they were clearly demonstrating complex, adaptive behavior, or as we’re calling it: swarmware. They first found a small spill of olive oil on the floor, and that got them excited. We cleaned up the spill, but it was too late. They found access to a few other spare scraps low to the ground and before long they were climbing up on counter tops and getting in cabinets. For creatures with a brain about 1/40,000th the size of a human, they were displaying some pretty creative solutions to find my food and overcome my attempts eliminate them. However, if you look closer at the behavior they were using, it was incredibly simple. We can clearly define the rules that would drive a foraging ant. In fact, we could almost write it as a series of strict rules. Let’s take a look at foraging behavior where ants use pheromones – scents they emit to lay navigation trails, communicate with other ants, and do a number of other activities – to enlist more ants in the efficient collection of newly discovered food sources.

1. Follow the strongest pheromone trail you can find until it comes to food.
2. If you can’t find a pheromone trail, begin to explore randomly until you find food
3. When you find food, follow the pheromone trail back to the nest
4. Repeat

My apologies to any biologists who may stumble upon this post – I’m sure I am simplifying a bit – but that is basically how it works. The ants follow a series of prescriptive rules without any judgement. The ants don’t look around, see that it is raining and adjust their strategy. Simply by blindly these 4 rules, they can get quite effective behavior. Here is a demonstration of how diligent application of a deterministic model – let’s call it clockware going forward – can lead to some highly complex and adaptive behaviors.

Figure 1 - Ants begin foraging

Our thought experiment follows the adventures of three ants. Assume it’s the beginning of the day, so no ants have laid down pheromone trails. Thus, they all pass through step #1 and move to step #2, begin foraging looking for food. Each ant heads out in a random direction. Two ants, after searching for some time, manage to find left over pizza. This triggers step #3, they collect food and follow the pheromone path they laid back to the hive. Our poor third ant doesn’t find food, so step #3 doesn’t get triggered for her – worker ants are all females, infact the only males in an ant colony are used for mating and they only live a few weeks, infer from that whatever you would like.

After one hour of foraging, two of the ants have made multiple round trips to the food, strengthening the scent of their trails. Now, four more ants come along to help forage. They start with step #1, look for a strong scent and follow the trail. The ant who wandered off an hour ago and hasn’t returned left the weakest trail – it was only laid by one single way trip – so the ants ignore it. The other two trails are 4x and 6x stronger as they have had multiple round trips laid by a single ant. Let us assume 3 ants follow the middle trail and 1 follows the trail to the right (nobody said ants were terribly smart).

Figure 3 - Ants after 2 hours of foraging

After a second hour, it has become very clear that one pheromone path is significantly stronger than the others. The trail on the left has never been walked by another ant – her fellow ants are  probably fearing that she may never be coming back at this point in time. The other two trails have been reinforced by ants traveling back and forth on them, but the middle one is 33% shorter than the far right trail, thus the same number of ants on that trail still make more round trips, thereby making the scent strong, thereby attracting more ants. This feedback loop continues and as more ants show up, by following the 4 rules they would self direct to shortest route leading to the food. I will offer my disclaimer again that I am oversimplifying; ants have demonstrated incredibly complex behavior including visual navigation, training younger ants, and understanding events tied to the time of day. However, the point remains that such a simple set of rules, strictly enforced, can lead to surprisingly complex and dynamic behavior. In the proper context, clockware can be used to empower adaptive swarmware behavior (Weston, 2001). (In case you are more interested in Ant behavior, I would refer you to http://blog.wildaboutants.com/)

Thus we see an elegant demonstration where the ants operate using simple, clockware-like, rules and the absence of a central planning authority allows these simple rules, when combined with feedback, to lead to some pretty adaptive behaviors. I should reinforce that the important characteristic of this sophisticated model is that the clockware was very simple in nature, the ants don’t apply rules about where to go searching for food, nor do they follow some centralized authority or plan. The clockware they use is a simple set of rules for responding to stimuli. The complexity comes in the number of ants applying these rules, the small randomization inherent in their activity, and the feedback where ants attract more when they find food that can be effectively retrieved. Of course, in modern projects, we are dealing with significantly more sophisticated actors and problems, so I would not venture to argue that team members on a software development team could operate with such a simple set of rules. However, the concept still applies: simple rules for small behaviors undertook by a number of people can enable highly complex and adaptive systems. Let me offer some examples.

Standardization and Innovation in Manufacturing

This concept has been realized and exploited by many companies under different names. Toyota offers a powerful example. Most people are very familiar with the innovative concepts Toyota has developed allowing them to implement what they call “operational excellence as a strategic weapon” (Liker, 2003). The company averages over 50 suggestions per employee per year, and implements well over 90% of the suggestions. This level of innovation and adaptive change can only be effective when coupled with a high level of disciplined clockware. Toyota is methodical about defining units of standard work and ensuring that people then work according to those standards. This may sounds like some traditional, heavily bureacratic organizations, but there are a couple key distinctions. Standards are created for the purpose of improving them, they are created by the people who do the work, and those people are empowered to change the process over time as they find ways to improve it. Thus they implement a system not unlike the scientific method with a number of variables under control in order to experiment different options and see their impact. Only with this level of discipline, measurement, and control, can the organization be free to try so many innovations and see if they make a positive performance. Much like the ants, disciplined simple behavior allows for feedback to foster more complex behavior within the system.

Applications within Agile Projects

Within an adaptive framework like Scrum, certain rules are meant to be clockware. With this perspective, the sanctity of what people call canonical Scrum begins to make sense.  This is the same paradox illustrated with Toyota, in order to be Agile we execute certain simple behaviors in a deterministic way. The mechanics of something like the daily stand up are strict and limiting, but adherence to them helps people gain better understanding of what’s going on and then leads to more self organization, because everyone has heard what everyone else is doing. Teams hold their stand ups the same time every day so that people are more likely to remember and be able to attend, even if this can be inconvenient from time to time. Participants self-censor their commentary to answer the three questions (what did you do yesterday, what are you doing today, what are your impediments) in order to focus discussion and keep the meeting short, otherwise people won’t be able to get all the information they need in a timely basis.

The counter example to this is the Agile team that, in the spirit of Agile, tries to adapt on everything all the time. People feel conversation is important, so they abuse the rules of a stand up and it ends up taking an hour a day (this could cause this single meeting to consume over 10% of the project time). Teams want to be able to finish a feature in an iteration, so they extend the date a couple days in order to finish, leading to inconsistent measurements of throughput and a false sense of progress. Scrum Masters allow product owners to accept work that is not really done, leading to an accumulation of technical or other forms of debt that will extract a serious toll from the project later. These are all examples where the swarmware concept was applied incorrectly, we tried to be dynamic and flexible on the wrong dimensions, which undermined actual feedback or execution. Generally speaking, you can tell your are experiencing this challenge when you hear people around you state some egregiously awful behavior they are about do to and then say, “it’s okay, we’re AGILE!”

When dealing with how a team will operate, Agile Coaches and ScrumMasters need to ask themselves if something is Swarmware or Clockware. For those things that we agree are in the domain of complex adaptive systems, we will want to leverage things like emergent design, experimentation, and adaptation. This is the swarmware of your project and it probably involves things like the overall technical design, the product backlog and the overall release plan. We expect these to change and will be very careful to not add too much forward looking detail so as to make them brittle or extend our assumptions too far past reality.

On the other hand, there are things that we don’t want to keep spinning on. This would include things like the application of something like test driven development, coding standards, or even the basic rules of a framework like Scrum or Kanban. Iteration dates and agreed upon standards like a definition of done should be set and not changed on the fly.

Evolve Swarmware, Increment Clockware

In any adaptive project, we will most likely change both clockware and swarmware, but we go about changing it in different ways. When dealing with swarmware, we embrace uncertainty and may allow new versions to emerge. Developers will begin with a simple system and take advantage of emergent design to slowly grow a more sophisticated system. In the case of clockware, we don’t evolve so much as increment. When a team sets a definition of done to state that done work must be tested in their QA environment, they shouldn’t adjust that definition on the fly from iteration to iteration if the QA environment is unreliable. By definition that is not a standard. However, the team may decide that they need a new environment for the PO to accept work, and change the definition going forward. This is exactly like the Toyota works setting standards and then methodically improving upon them. In these cases those, certainty is important and the standard should be followed until it is changed.

Figure 4 - Clockware progressing to Swarmware

In the end, the situation is akin the one explored with ants, most of what we do should consist of complex swarmware built on top of a series of simple pieces of clockware. Let me offer an example of a team I worked with recently. The teams developers had to build a new application upon a specific platform with certain agreed upon coding standards. Code was built based on automated unit tests and subject to code scans for things like cyclomatic complexity and other best practices. They delivered work every two weeks with agreed upon review dates for stakeholders and work was only called done when it adhered to an agreed upon list of conditions. This is all clockware. These rules were clear, detailed, and unchanging within the iteration. Between iterations they may adjust their definitions, implement new policies and make other changes, but once established, the rules must be followed. On top of this the developer maintained a dynamic design document they updated as they completed each feature. The product owner had a list of features that were not yet started and she was free to change the order and add new things at any point in time. The team would change task assignments as work progressed along their board. Thus, these more high level activities evolved over time and allowed a greater level of fluidity than the more concrete standards, they were treated as swarmware.

So as you look at your project and get into discussions about how flexible you should be about something, or what level of discipline is needed, ask yourself if you think this is clockware or swarmware. Is it a simple fundamental activity which can probably be clearly defined? Is it fundamental to the value proposition of your product? Using a model like this, hopefully you can maintain within your mind that we must hold two concepts within our mind. Our projects ultimately decompose to a lot of clockware, relatively predictable work that can be standardized, measured and managed. However, these small pieces quickly add up to highly complex things that operate in a qualitatively different way. This is what we would call swarmware, and it requires an adaptive approach leveraging feedback and adaptation. If you neglect the clockware, you won’t have the foundation to get consistent feedback or execute consistently. If you neglect the swarmware, you won’t be able to deal with the essential complexity of your problem domain. Thus in the end, both are important, the challenge is in identifying where one ends and the other begins.

Why Commit?

While this sounds like a simple question, it’s probably worth taking a closer look at precisely what we mean by a sprint or iteration commitment and we should be using it. We should be courageous enough to take any component of our project, including Agile practices, and ask ourselves “why are we doing this”. In fact, many within the Agile field are asking just that about the sprint plan and commitment and pointing to things, like Kanban, that simply do away with the idea of making a plan for a discrete point of time. I can certainly see that the concept of a sprint plan and commitment offer us several concrete things of value

• It offers us an opportunity to try and pace ourselves – whether we meet a commitment or not will give us a good idea if we are indeed progressing at the rate we anticipated. While most Agile practices discourage tracking actual time on individual stories or tasks, this gives us the same thing, the ability to reflect on an estimate, and see if we met it or not. Quite simply, if we planned to deliver 4 use cases of functionality in 2 weeks, and we only delivered 2, that’s an important data point into understanding if our expectations and planning is realistic. I frequently find that most members of an Agile team actually aren’t terribly good at planning at first. Years of making estimates in isolation and never being able to validate them in a timely manner has caused any estimation skills to atrophy.  This frequent feedback is very useful to them to improve their estimates and get better at making consistent, reliable plans.
• It creates a sense of tension – when a team commits to do some amount of work based on their own analysis and planning, they will have a bigger stake in it than most any mandated goal from on high. Thus, we tap into a powerful source of intrinsic motivation. This is particularly good when we see a team meeting or exceeding goals.
• It builds trust & cooperation – Robert Axelrod in his exploration of the Evolution of Cooperation sought to identify patterns that would lead to more cooperation. One of the most powerful dynamics in encouraging cooperation was building trust over iterative rounds. The regular cadence of making and meeting commitments can be a powerful pattern for improving trust amongst project stakeholders and engendering cooperation.
• It focuses on goals – a commitment is made by a team as a whole and thus shared by everyone. This can be a powerful catalyst for getting people to step beyond their boundaries and do what’s good for the team as opposed to optimizing on their own specialty. If the requirements are completed for all the work of the sprint, the analyst may realize that they need to help testers in order to meet their commitment, this will be more important to them than writing additional specifications of no immediate value in order to simply keep busy.

Of course, these cut the other way as well. Missed commitments may lower trust. Unrealistic goals imposed from outside may cause teams to simply not care, or teams may even offer to “give it the old college try” in an effort to be accommodating, without realizing that they are setting themselves up for disappointing their customers. Teams may fear missing a commitment so much that they consistently under commit and thus sacrifice capacity in the name of always meeting a commitment.

What are We Committing For?

At the risk of sounding like a consultant and saying “it depends”, the way we approach our strategy for commitment varies based on your circumstances. Let me offer some examples to illustrate some of the challenges we may face and how commitments could be used.

• The undisciplined team: Agile requires a tremendous amount of discipline and many teams may struggle at first to operate in a consistently disciplined fashion, paying close attention to detail and seeing tasks through to completion. We may see this manifest itself in the team that is “basically done” with everything, but we keep seeing hang over tasks, or clean up work bleeding over into the next sprint. Or perhaps they are taking credit but there are bugs or other things left unattended to. In this case, a clear commitment to complete work, and follow through by the ScrumMaster or other voice of conscious within the team is a critical factor in helping the team hold themselves to a higher standard.
• The fearful team: sometimes teams are afraid that a single missed commitment will cause them to incur the wrath of numerous managers trying to understand “what went wrong”. Of course, this is a tell for a bigger problem, namely managers or other stakeholders obsessing over single data points when the trend is most important. The negative impact this can have upon a team is that they consistently under commit and are unable to improve for fear of not being able to meet expectations. In this type of case, I have seen teams set two goals. A conservative goal is communicated to stakeholders, managers and the rest of an organization. That is the number they will be held to. The team then sets a stretch goal used for their own internal purposes of trying to push themselves and see how much they can do. Mike Cohn has an interesting take on this where he points out that teams should distinguish between estimating and committing.
• The ambitous team: sometimes teams see the commitment as an opportunity to impress everyone. In these situations, commitment may sometimes be better described as “aspiration”. Of course, this causes other problems as people may not take it as seriously. The risk here is that in an attempt to get an incredible amount of work done, they over extend and end up delivering less. Good coaches know to help the team appreciate a sustainable pace.

Guidelines for Committing to a Sprint

The work that a team commits to should be their best approximation of what they can realistically deliver within a given time frame. It should include all of the agreed upon work to complete a given increment of functionality. It should also be a pace which the team can maintain indefinitely, which means no technical short cuts, no excessive overtime, and no neglecting necessary preparation for the successive sprint. The principle value of the commitment is to the team, so that they can measure against their best approximation of their rate of work. It may also be used by the product owner or other stakeholders to assess the trend of performance. With all this in mind, here are some general guidelines to keep in mind.

• It’s the trend that matters, not a single data point: people may at first obsess over a single sprint where a team misses its commitment or wildly over delivers. Be sure to remember that the trend tells you what’s going on, not one data point. Make sure people don’t get too excited or disappointed over one sprint. The point of frequent iterations is to lower the stakes and allow for some failure. However, if the trend points to a bad pattern like stagnant performance, declining velocity or wild gyrations in performance, then there is probably something you need to investigate to understand what’s going on.
• Distinguish between commitments that can be missed verse ones that can’t: Agile projects are designed to allow for failure early in order to improve long term performance. Thus, missing one sprint commitment is not a crisis, but missing a release deadline for a project that had a hard date very well may be. Make sure people can differentiate between commitments made for purposes of baselining and pushing the team from commitments that are critical to project success. While circumstances vary, I can tell you that most commitments would fall into the first category. Sometimes this may also involve creating dual commitments “we commit to the business they will have these features, and we are committing to ourselves to do our best in delivering these additional ones as well this sprint”.
• Make sure you have an opportunity to adjust up or down: sometimes it will become clear early on in a sprint that something is not going to happen or that things are going spectacularly well. In these cases, its best to acknowledge reality and make sure you have an option to bring in more work, or adjust your commitment based on the reality of the circumstances. Generally speaking, I would say that a mid-sprint adjustment should consist of removing committed stories from a sprint or adding additional ones to those already committed. You don’t want to go into a complete replanning exercises where you are both swapping some out and bringing others in. Also, if you do adjust the commitment, it should be a discrete thing that happens, as opposed to being a floating decision that is continually adjusted.
• Talk to the team about the implications of team commitment: remember, we’re doing this to learn and to build predictability. We want to make sure that people understand our goals and if we are finding ourselves managing commitments to other goals, it is better to surface those issues and confront them.

So ultimately we come to the conclusion that it’s not necessarily bad if in a particular sprint you are missing your commitment and seeing work flow over to the next sprint. However, if this is a consistent pattern then we need to figure out what’s going on. Even in these cases, I wouldn’t necessarily classify it as a bad thing, but rather a valuable tell to understand more about your project.

Thanks to everyone who participated in the survey and I look forward to hearing from more people in our community about their experiences using Agile techniques and practices.

Why Do We Run Too Many Projects?

My colleague Mike Dwyer refers to Scrum as a Silver Mirror. By this he means that, while it won’t solve all your problems like a mythical silver bullet, it will present you with a level of reflection such that you can identify and confront your problems. To that point, my perspective has been that organizations struggling to launch Agile projects because they keep trying to run too many projects are suffering from a larger issue.

• Lack of Agility
  You may find this counter-intuitive, but I would posit that many traditional organizations use broad project portfolios as a means of achieving agility. With the heavy ramp up costs of any given project, having multiple options to respond to change. We can ramp up or down our energy on any given project based on changing circumstances or priority. While it’s not terribly flexible, it does offer some level of adaptability. However, it comes at the price of running all these projects at the same time, which is a very steep tax when you start adding up the cost of multiple status reports, daily or weekly meetings, code branches, design documents and any other overhead required for running your projects.
• Lack of Impediment Removal
  I remember before I formally began my foray into Agile project management being frustrated with a manager about a policy of putting everyone on 2+ projects at any given point in time.  After some lengthy discussion, he finally explained to me that by putting my on multiple projects, even though it would extend the delivery date of both, it would ensure I was productive. For if one project became impeded, I could simply focus my energy on the other. This may be true, but it completely hides the fact that we have issues with a project. It also defer risk by redirecting effort to the less impeded area, creating a false sense of a rate of progress.
• Lack of Priority
  In some organizations, especially with many influential stakeholders, prioritization can be difficult. I have seen organizations where prioritization is implicitly thrust upon the development teams. The business basically asks for everything at the beginning of the year and then the technology partners respond when each item will deliver. Of course in these cases, the vice of no prioritization can be mixed with a desire to “show progress” on all projects with awful results. This is where we see more projects run than people so that some portfolio dashboard can show progress on all these different capabilities, but unfortunately it is really just an illusion.

I enumerate this list not to justify behavior, but to empathize with the conundrums facing many project managers. In fact, I believe that – in a strange way – the propensity to solve every problem with a project is a testament to the success of projects within large organizations. A generation of business people have seen careful project management get things done in chaotic and uncontrolled environments. Good project manager have leaped into the void of poor organizational impediment removal and prioritization and responded to these challenges by starting more projects. The only problem is that slowly the added cost of running these projects undermines our ability to do anything. This is a problem more projects can’t solve. As Albert Einstein famously observed, “Problems cannot be solved by the same level of thinking that created them.”

This level of thinking has reached it’s limit. We’ve gotten to the point where organizations deify projects to the detriment of teams and people. How many of you have seen organizations that will never stop a project, but they have not problem asking someone to work 25% or less on multiple projects across a portfolio? Or companies that talk about the value of teams, but keep sloshing people – probably referred to as “resources” – between projects on a regular basis? Indeed when we see this situations, where people feel totally at the mercy of the projects to which they are assigned, is it any doubt why people don’t self organize into hyper-productive teams?

Kill Your Projects, Get More Work Done

Several of my coworkers have toyed with the concept of “killing projects”. This would be the natural progression of Derek’s earlier point about aligning projects to teams. Its a great idea, but in many organizations, we may never be able to realistically say no to several projects in order to focus on a few. There may be numerous stakeholders with very real concerns that demand action immediately. Or perhaps we are simply within the bowels of a large organization and we are presently unable to influence our stakeholders to the point where they would pick one project over another. In the old days, we would slice apart people to make whole projects. Today, I’m proposing that we take the knife and apply it to projects instead. Rather than try to run multiple projects.

By leveraging a single product owner or product team to triage and break apart the work, we can feed what would have been multiple projects to a single team. This frees the team to figure out the best way to balance the work. If they are operating according the practices of producing production ready code every iteration, then they will also be able to deliver value to all stakeholders, something that would go a long way in those low trust environments where teams needs to show progress across all fronts. The other benefit is that by putting all requirements through a single funnel, they bring the different stakeholders together to discuss what’s ultimately most important.

They don’t need to work together. I’ve seen teams allocate the capacity of there throughput based on funding levels. For example, imagine that three projects were approved: Project A funded for $250,000, Project B for $350,000, and Project C for $400,000. This would lead to a single team funded at $1,000,000. We could easily allocate the team’s velocity by funding levels, thus each team would get 25%, 35% and 45% of the points delivered for each sprint or throughput on a kanban system. Either way, stakeholders would be able to get what they want based on their funding, but hopefully putting them together would lead to the model described by David Anderson (2010) where people would move through phases of first selecting work based on their allocation, then moving to some sort of democratic model and eventually moving to a model where they dialog about what’s most important for the organization next. This model could even be scaled to multiple teams, where they would pool requirements across different projects and figure out what they can build for the next release. These joint planning sessions are usually quite energetic with multiple product owners, teams, and stakeholders operating in breakouts within a large room to trade stories, make prioritization decisions and plan the work.

Ultimately, we are proposing a model to help you confront the issues of your organization. In order to be successful with any type of project portfolio strategy, we will need to be couragous enough to look into that silver mirror and address our shortcomings. It is the fact that the multiple project strategy hides these issues that is most dangerous. Not only does it offer little in the means of helping us address impediments or answer prioritization questions, it conspires with us to brush all of those things under the carpet and simply look at the problem as one of planning and organization. Those are safer topics to be sure, but we do ourselves and our organization a disservice by not confronting the real challenges.

They were ripped that such an important game as American Football was led by an obviously inferior individual.  The one person on the team with a Quarter of a Back.  Insane! they cried, questioning the competency of our species – or at least our country – to be nominated for membership in the great intergalatic sports community.

What was  the problem with us, they demanded, to have a person with a full back or even one with a half back on the field and to still insist on fololowing the person with the least capability to remain in a vertical position.    It was not just their concern either. They let it slip that our solar system was on the short list to be converted into a rest area for the new hyper way.

I felt their minds had not been quite fried enough, so I told them about a software team being led by a ScrumMaster who  had only two days of training.  I shouldn’t have done that I guess, the gelatin-like masses where the voices seemed to come from went solid, turned ocre and started to emit a slow whiny sigh.

I took pity and explained to them that the name ScrumMaster was just like the name quarterback -  two great examples of the stupid way we earthlings label important roles in our organizations.  I went on to explain that these names were the names of positions people had, and did not reflect their competency or limitations in doing their job.  I then told them of other names in sports like gully foot and shortstop.  In addition I mentioned Project Manager and Assistant associate deputy to the second vice president for paper disposal.

This seemed to get them very excited and confused, then very happy.  So happy they forgot to take me with them and put probes in me.

It turns out the combination of the names we give roles and how we make such a big deal of some stupid names and say nothing about others will be the next season’s big hit. These two are setting up galaxy wide viewing licenses of the biggest show in all of time and space. A reality show that will document that Homer Simpson (a huge name in the Galaxy) is the smartest person on earth.

This is Bob

He’s a really good developer. Everyone wants Bob on their teams because he knows the company’s systems inside and out. In fact, he is the sole SME for several key components and is an indispensable resource. Unfortunately, because of this Bob is frequently pulled to help out with support or other projects. As a committed company man, Bob always wants to help out and won’t leave other people stranded. Besides, he’s such a bright person that he can take some time to help other projects and still be effective. So now Bob is joining an Agile project, let’s say he’s supposed to be split 50 / 50 between a new Agile project and a support release to do some bug fixing on a legacy system.

Betty is an “enlightened” project manager

Betty is managing the Agile project that Bob is on and she is very happy to demonstrate the power of self-organizing teams. She read a book on servant leadership and even is a Certified ScrumMaster class. She knows that commitments individuals make are the most important and she’s looking to trust the team to set its own pace. After all, we’re all professionals and are working as hard as we can. She’s not going to stare over Bob’s shoulder to make sure she’s getting her 50% of his time, that’s not what a good project manager should do.

Beatrice is a little more cynical

Beatrice is the project manager for the support project. She has worked for the company for many years and acquired some hard earned lessons about motivation and follow through. As such, she is pretty confident that the best way to get things done is to make sure you are keeping pressure on the developers. If they say 10 days, she’s going push them to do it in 7 and will probably ask around day 5 if they are done already. She has an aggressive schedule and has been put on this project by management specifically because her projects always stand out amongst the pack for getting things done.

What does Bob do?

Imagine Bob’s position where he is ostensibly assigned 50 / 50 between two projects. How is that to be enforced? Is it on a per day basis? Is it purely on time spent? After he meets his 4 hours, is he expected to simply go on to the other project? This is the challenge of trying to treat a human being in a complex situation like a machine. In reality, my experience is that people will try to balance a split like this by trying to meet commitments and responding to inquiries and requests, and therein lies the problem. Let’s imagine Bob’s day where he has 4 hours of work to do for both projects. Betty will ask him what’s going on during the stand up, but beyond that she probably won’t bother him to make sure he’s working on her project. Beatrice on the other hand will most likely drop by 2-3 times this day alone, especially if she’s expecting something. Is it possible that this pressure could cause Bob to work just a little bit longer on Beatrice’s project? At the very least, he would probably spend the first half of the day working on Beatrice’s project, that way he would have something to say if she cornered him demanding a status? Let’s imagine for a moment that the net impact is that Bob works one extra half an hour on Beatrice’s project over Betty’s Agile project. A very small sum, but let’s see what happens.

Beatrice Gets 60% More

Bear with me on some math. Let’s assume Bob works an 8 hour day. One hour is lost to overhead and administrative details. That leaves 3.5 hours for each project. But, as we discussed Beatrice hounds Bob, so he is quietly giving her an extra half an hour each day. This means that the distribution is actually 4 hours (non-Agile project) 3 hours (Agile project). But it is probably worse than that. Being micromanaged takes time, so Bob is probably losing another half an hour to calls from Beatrice or responding to emails. Of course, this time is probably also going to come out of Betty’s project. After all, Betty is so understanding. Thus, when you ad this all up, the traditional project gets 1.6 times more of Bob’s availability than the Agile one. This becomes an immediate handicap for the Agile project. Also remember that Bob is a critical resource, so losing his time may actually impact the overall project significantly if he’s the go to person to answer questions. In fact, the more critical a resource, the more likely they are to get into this situation.

What to do?

Without an intervention by someone, Betty’s project is now under serious duress. The key point to remember is that with projects, everything is relative. At the end of the year, people will look at the relative performance of each person and that’s how they will decide who to reward and who not to. From this point of view, it appears that Beatrice achieved a much better relative outcome.

Of course, we have no idea if her project was more important or it merited the extra attention. That was an implicit prioritization done by people trying to optimize individual project outcomes. This is where I have massive amounts of sympathy for the Beatrices of the world, because it is a prisoner’s dilemma. If we both abuse our resources, then we both end up paying a (small) price in productivity. However, we’re evaluated relative to one another, so that’s not a huge problem. However, if I don’t harass people and you do, there’s a good chance you will get more of their time and in turn get better relative project outcomes. Here we see a system that has been created to basically force project managers to distrust, push and harass their own team.

So what can you do? The best solution is run less projects. I am always amazed at just how much more we value project than human beings in modern corporations. We can split a person amongst 3, 4 ,5 projects, but we would never dream of putting a project on hold! Indeed it seems if we pursue a strategy of maximizing the number of projects in flight, we will invariably pay a price in increase overhead, work in process and conflict for critical people & resources. Well, if you can’t take on that challenge, here are some strategies to pursue.

• Exploit your team room: Team rooms are great for getting people to work together, they are also a safe haven and protection from distractions. If Beaty had Bob in a team room working with the team, it would be much harder to pull him away and the momentum would be for him to focus on this project. Perhaps she can’t get that every day, but in the case of a 50/50 assignment, perhaps every other day would at least be a fair request. Of course, this doesn’t really solve the problem of understanding which project is a priority, it just helps stack the deck for your team.
• Use the ScrumMaster or PM as a buffer: I have seen some teams literally force all external stakeholders to go through the PM or SM in order to request anything that’s not part of the sprint commitment. This may be harder in the case of a shared resource. However, in our hypothetical situation, the conflict is really between Betty and Beatrice’s projects, forcing the discussion up to a higher level at least shows the conflict on a level where it can potentially be solved, and it takes Bob out of the line of fire.
• Manage Assignments on a Sprint by Sprint Basis: depending on the nature of your work, I have seen teams manage their own commitment on a sprint by sprint basis. This is especially effective with technical SMEs around inter-related platforms. For some work, these people were not needed at all, and they were only lightly involved for those sprints. In other situations, the team had explicitly planned to work in one of those platforms and during those sprints, the SMEs were full fledged members of the team. Thus the team would plan ahead and telegraph to those specialists when they would need their help. This does require that the other groups leaning on these experts are able to handle these big swings in demand.
• Cross Train Resources: the situation of conflicted people between multiple projects is usually related to a constrained resource, where everyone needs some work on the XYZ widget, and there’s only one person who can do it. In this case, you may want to view that person not as a doer, but rather a mentor & trainer. Their goal is to help get other people up to speed. That way, when they are pulled to work on something else, there is not as dramatic an impact upon the team. This will ameliorate the split resource problem longer term, as you get a team of individuals with a broader range of skills such that you don’t really need Bob specifically and may be okay letting him go to another project in exchange for someone else full-time.
• Reconsider Project Boundaries: I have seen situations where the only thing that logically separated two “projects” was some cost account and a status report. The work was done by the same people, the business stakeholders were identical, and both existed in the exact same code base. Perversely, this approach not only introduced personnel conflict, but it also doubled the number of meetings, and introduced configuration management and dependency challenges. In one case I witnessed a savvy ScrumMaster and Product Owner combine both initiatives. With a little plugging through finance spreadsheets and expectation management, they were able to manage the budgeting and cost accounting for two projects. However, the team’s experience was one single project with one backlog, one code base, one product owner and one sprint schedule. This was such a success they continued this model, rolling several 3-month “projects” into this team such that it was able to run non-stop for 2 years and counting. Predictions are great, because they have an established velocity that can be perfectly applied to new work. They can also be more responsive, as there is no start up cost to begin the next “project”, the team is already normed and performing.

It’s All In How You Frame It

Regardless of the tactical solution, properly framing this type of situation is critical. The issue is not that Bob can’t do the work, or even that he has a problem. The issue is that these two projects have competing demands and we don’t have a good way to resolve that. If we fall into a situation of blame, then Bob may be motivated to suck it up and just work extra hours. This is the worst of both worlds, as it rewards the micromanaging Beatrice. She gets her extra hours, and it extracts a toll from poor Bob. In fact, it probably reinforces the behavior such that Beatrice may do it even more. So why wouldn’t Bob bring this up? Well, the most obvious time for him to speak up would be during the stand up, but what if he doesn’t feel safe there. What if the stand up is a high pressure event where people are making commitments and Bob doesn’t want to feel like he’s letting people down? So he says he’s okay and just taking a little longer than expected.

The fact of the matter is that when we run lots of projects, we will invariably run into conflicts. I recall one company that had about 30 technical people running IT projects and had over 100 projects. Conflict was a certainty. In traditional projects, we frequently confront this by hiring aggressive and assertive project managers. Unchallenged this model will starve Agile projects of needed expertise, cause nasty conflicts over people and can fundamentally undermine a team’s ability to self-organize.

Within the large organization, I can appreciate the desire. Indeed, there certainly are some benefits around transparency, consistency and institutional knowledge to be had when we get everyone to agree to a standard. This is a careful balancing act, as the benefits may prove elusive and the risk is very real. First with transparency, while working code is the best indicator of progress, as teams grow into the hundreds, that can become too much “product” for an executive to effectively review and understand. This closely dovetails with consistency, where teams need to have common understandings of how to exchange stories, what “done” really means, and even what technical standards they will follow.

Technical standards can be quite pernicious, as enterprise licensing agreements, technology stacks and other architectural decisions can place very real requirements on how teams operate. We do like to think of each team as a trailblazer, but at some point, if you have three web development teams, one chooses ASP.NET, one chooses Java and the third chooses PHP, it doesn’t take an expert to see there is wasted energy. Lastly we come to the idea of institutional knowledge. As a consultant I can certainly appreciate that companies have a desire to “do it on their own” and apply hard earned lessons in one project to future ones.

It is not that these are invalid desires or that some benefit can’t be reached from standardizing, or as I’ve been calling it “institutionalizing” a Scrum implementation. Rather, it’sthat we need to understand the cost of each decision we make and weight it against the benefit. I think back to one of my earliest experiences in project management. An aspiring PMP, I was kicking off a project and I took out my PMBOK to find out what tools I could use to charter and initiate a project. From analyzing stakeholders to managing risk, I’m pretty sure if there was a checklist or template, I used it. At the time I didn’t understand exactly how each one applied, so I compensated by using every single one. Surely, this is not Agile, nor how one would advocate something like the PMBOK be used. I took a list of possible tools and considered them a mandatory set of things to do. Just like that, a supporting resource became a requirement and my project was about following lists rather than delivering value.

I see a similar challenge if an organization begins to offer too many guidelines. So what is the right limit for acceptance criteria in a story? Well, in my case, I took a 3×5 note card and figured out how many sentences I could fit on one side. This brought me to the recommendation of 7, but does it mean if I only have 2 that my story is wrong? Of course not. A better measure would if it clearly articulates what the story needs to do, is agreed upon by the team and they can commit to do it in a sprint. Then again, even user stories are not explicitly part of Scrum, so are we now mandating that all project requirements be formatted in this construct? Just because something makes sense in one domain, doesn’t mean it should be applied elsewhere.

Indeed, using a framework for empirical feedback like Scrum provides one with infinite ability to inspect & adapt. Of course, a common challenge with standardization is that, we have two groups of people: one group who is defining the standard and a second group who is trying to use it. Thus we have one group using the process and getting feedback, but a different team is supposed to make adaptations to the process, but is not experiencing that feedback directly. This is the paradox I alluded to before. In order to get a large organization to use a highly flexible & adaptive process, we must take away some of that flexibility by taking decisions out of the hands of individual teams and delegating them to centralized committees. When this is done, does the value proposition remain intact? How do we go about confronting making consistent & transparent Scrum throughout a very large organization without fundamentally undermining the nature of this framework?

I know many people have talked about this topic and I will certainly have more to say about it, but for now I think the key is to figure out how to standardize on the principles and keep maximum flexibility, but I can see this blog post is already too long so let me leave it with that until I continue this thought later. I’d love to hear your thoughts in the meantime.

Are you considering a coach?

If you aren’t, are your organization and projects at risk?

Fire! Ready, Aim…
Many organizations, thinking that they can’t afford or don’t want to invest in coaching or training, read a book and some articles, and tell their teams that they are now doing Agile.

I have never seen a team get great benefits from Agile in this way.  When I am coaching, and I hear other teams that aren’t being coached say “we’re doing Agile,” I raise an eyebrow (in my mind anyway), and find out if I can spend some time with them to see what they are doing.  Without fail, these teams are doing “Scrum but,” “CrAgile,” “Scrummerfall,” or some other thing that only resembles an Agile method minimal ways.   There are many articles on these topics.

How might coaches be engaged?
I’ve heard that some people use the analogy of a Soccer Team to make the point about what a coach is and how coaches might be engaged:

Wing It
Consider a soccer team.  You could read about soccer in various books and other references, and then attempt to play.  Chances are, though you may learn some of the basic rules, your team will not perform that well without great luck.

The Clinic
You could have this team go through a 1-week soccer clinic to improve their abilities.  They will probably learn some new tricks, maybe a bit of strategy if you’re lucky.  But rarely will such a brief improvement effort result in drastic, long-term improvement as a team.

The Ramp-Up and Check-In
Now consider the same team if they had involvement from the expert who held the clinic for 3-4 months.  The coach could provide the same techniques and training, and apply them to real situations as the teams go through them.  This is FAR more powerful learning – it is contextual, it is about the team and its real challenges.  They can follow guidance as they are working and incorporate it into the way they do things.  The coach can also keep an eye on things that are nearly impossible to observe in the clinic- team dynamics, organizational obstacles, and more – and help any time they find a need.  The team’s game can really improve.  If you have a good coach, the team actually may even get to the point where it is able to improve on its own- perhaps the team members have watched the coach and adopted his/her techniques to observe and question and find improvements.  After a few months of working together, the coach can scale down her/his involvement, perhaps to the point where she/he is called in as needed and to perform periodic check-ins and assessments.

Though this is a far more powerful model, it may not sufficient – for all large/complex/business-critical projects that cost lots of money.  These projects and programs are the “pro” league of the project portfolio, and any opportunity to mitigate risk should be considered.  There may be argument that it is not even sufficient for smaller projects and programs, because much of the value a coach could add happens real-time, as the project is going through its iterations, as challenges arise (unexpectedly).  Depending on how infrequently your teams have help available, they may not be getting the help when they need it most.

The Embedded (the “pro” league of product development?)
Sports teams expected to perform at any professional level follow a different model of coaching – they have coaches and experts that stay around all the time.  The level at which they are expected to perform  makes the cost of great coaches and trainers a good investment.  The risk of not having a coach far outweighs the cost.  How does this compare to your projects?

Do you have a project on which you are spending  millions each year?   That sounds like a risky endeavor, considering project success rates over time (See any popular project success rates research etc).  Having a coach on board or accessible at all times can help your team deal with the infinite number of challenges that it may run in to.   Are you an executive that has “shelved” a multi-year, multi-million dollar project?  This is about you.

The bulk of the coaching value-add is probably not in specific things like Agile practices and techniques, but in other, less concrete things – like dealing with situations that aren’t covered in the books, maintaining focus despite difficult situations, mentoring leaders in the team, facilitating brainstorming, guiding team members in problem analysis, and helping to identify goals for continuous improvement.  If your coach is effective, teams will make measurable improvements every iteration- much more consistently than without one.

Effective coaches are rare, and they don’t come cheap (if you find one that does, start asking around for references ).  But they are a force multiplier, and a massive risk mitigation technique.  The cost of this level of risk mitigation pales in comparison to the benefits  – in continuous team improvement, in mentoring of future leaders, and in the pursuit of organizational agility.

An example…(We have many…)
You may be doing a great job of allowing your teams to follow the guidance they’ve been given and execute Agile very well.  Great job!   Product owner (PO) of project X, one of the highest-budgeted projects in your organization, realizes that a feature set that was originally deemed extremely important has been exposed as a nice-to-have,  or maybe not really necessary at all (a very common situation on well-executed Agile projects).  What should the PO do?

Clearly, the PO should talk to the stakeholders and let them know that we could save $400k on the development of this feature set that we would have otherwise spent.  Is it that clear?  Is YOUR organization ready to handle this situation?  Would the project be deemed a success and the fact that it was ended early be treated as a win, or would the message be that it was ‘canceled’?  What would happen to the project team if they were done early?   Would your organization be able to get this high-performing team a new project that actually has critical importance, or would they be disbanded?  Would your organization be able to re-allocate those funds to the next most critical endeavor?  In many organizations I know of, there are many reasons why a PO in such a position might not choose to terminate the project early (would it be uncertain to the PO what they would move on to?).  These are organizations that have not taken Agile and Lean to the enterprise level.

A coach provides an objective, guiding voice.  Any coach worth his or her salt would help the PO and stakeholders realize the opportunity and the reasons why the opportunity might be tough to take advantage of.  They could then help the right decision to be made, and help the organization improve so that it will be better able to handle this situation in the future – by exposing this “organizational obstacle” to agility and helping the organization resolve it.  If there were a coach in the aforementioned situation, would that have saved the organization $400k in poorly-spent development costs and earned them $___ in benefits from the more important efforts that those funds could be devoted to (which otherwise would be opportunity costs)?

There are many things to consider when you are deciding about hiring a coach.  It’s not all about Agile, training, and practices.  It is about success and risk management, and about prevention of the common phenomenon of less-than-sucessful Agile implementations.

When you live where we do, working with huge organizations spending enormous sums of money so that hundreds and even thousands of people can build, support, train, maintain, and even (gasp) plan how to handle the information needs of millions of people as they; trade at a local, regional, national, and international level concurrently; seek, provide, research, pay for healthcare;  protect/defend their person, family, home, community, nation, world from threats of others or their own environmental actions; and even download upload whatever they find entertaining in this world; when you live on this side of hill, architecture is at the core of being Agile.  And for the most part, we suck at making it great.  Why is that?

Perhaps it has something to do with what architects hear us say and what they take away with them.  I recently read a great blog Lean & Agile: Are they worth the effort? and made these comments.

It is startling to read such a well read and concise synopsis of what many of my colleagues in the Agile community get wrong in communicating what Agile is.
First Agile is more than just delivering the most valuable parts of what the customer wants.  The bit missing is the important one. – it works as near to perfectly as the team can make it.  Why do I harp (some say nag) on this.  Well, if the most valuable item the product owner has cannot be delivered, say for example a startrek replicator machine capable of making $1,000.00 bills in endless supply.  Then we cannot deliver it.  Sorry, not going to happen.  What else do you want?
So what is Agile, it is a way to deliver the most viable value at any given time and in saying this we begin talking about iterative development where we allow for the rapid delivery of emerging solutions.  You know the ones that are not quite right each time the product people look at them – the ones needing ‘one more thing’.  Agile is great because it delivers what was wanted in a short period of time and then ‘adapts and inspects’ to create the next version of “AH HA’s” the product people come up with.  Great stuff.
Agile also does incremental well.  That is the one that you do when the product is nailed down and you know what the end looks like for the most part.  now we can deeply dive in to what and how we repeatedly create the same product or process.

This does sound like lean but as you so quickly caught on, lean as in manfacturing is not what we do in software.  We do not build the same exact source code and ship it over and over again.  We have that bit down in release.  What the Leanies are talking about is ‘lean principles’ which IMO are basic good engineering practices that aided the allies in winning world war II.  Check out TWI and see what I am talking about.  TWI or Job Instruction Trainng is how to teach unskilled people processes that will permit them to produce highly technical product in a very short time.  If more of the Leanies had time in manufacturing line management they would know this. but that trade and proffession, like phrenology, is a thing of the past.

So what does Agile need to adapt and inspect on?

First and foremost we have not allowed architecture to emerge as a valued member of the Agile team.  There are probably a host of really good rationales for this and I am pretty certain that a therapist would have a field day with all the angst, anger, and other inhibiting behaviors that fostered this feeling and continue to fuel the issues.  Folks it is time to get pragmatic and move on.

There is one reason that I have found that makes this impossible and that is the how today’s Agile Architect defines themselves through their work patterns with each other and with Agile teams.

Agile is a team sport and we treat architects in general and Agile Architects in particular as individual contributors that sit on the periphery of the Agile Team.  Shame on us for being that conventional in our thinking.  Shame on Agile architects for not moving hard enough to show the lack of relevance that attitude has when integrating Agile into large projects and programs.  that word is something we need to use around architecture for a couple of good reasons.  The most important is that it really works well when it is done and it also really changes Agile biases because it rocks the boat into hyper gear.

In fact, this is probably the insidious challenge with things like RUP or PMBOK, which advocate selectively choosing the tools and artifacts you need for a specific project. While people are free to opt to exclude certain deliverables, in an unhealthy organization, one does so at their own peril. Any gap or inconsistency is likely to be pointed out in a postmortem as the ultimate mistake that doomed a project. Scope had to change? Clearly you should have done both a business requirements and then a system requirements document. Project schedule slipped because too many bugs were found late in the game? Looks like you really should have made a formal QA strategy document and a full traceability matrix mapping your requirements to test cases.

Ironically, this very dynamic can play against Agile. I was facilitating a discussion about Agile and one of the participants earnestly explained to me that they needed to write detailed technical specifications because so much of their work was sent offshore to the lowest bidder, where turnover was so rampant they were working with different people every 3 months. The idea that they should improve their off shoring strategy did not seem to be worth considering. I wish the team luck, but I wouldn’t be surprised if several months from now, they found themselves looking over a steaming pile of project wreckage and saying, “well, that’s why we shouldn’t use Agile and need to put together detailed specifications before we ever start coding!”

This draw is particularly relevant to me, as I would identify myself as a reformed process-aholic. When I was a newly minted PMP in my first project management role, there wasn’t an optional template, process or document that I avoided. Project charters, team orientation manuals, risk spreadsheets, progress dashboards, business requirements, testing strategy documents, data dictionaries, you name it my project had it. Nobody would say that I wasn’t managing my project well or that I was neglecting something. Of course it means that I authored a lot of documentation nobody read, and it wasn’t clear what the team had truly agreed on – were testing guidelines in the unit test strategy, the project plan, or the system test strategy – and incidentally I was so busy filling out documents that I wasn’t really around to help the project team. I do seem to recall complaining that nobody followed the beautiful plans I had put together. What I didn’t appreciate is the idea that everything you do in a project has a cost. Now many things don’t cost much, and are well worth the investment, but process isn’t free. Every minute we spend documenting something is a minute not spent collaborating. While documents are useful for historical context, they can also become inaccurate over time, or continue to cost more if they must perpetually be maintained. What is the cost of the fact that now, when people have a question, they go to the document instead of asking the subject matter expert, who would be able to explain much faster and effectively?

Interestingly enough, it seems many people, especially in large bureaucratic organizations realize that is these unwieldy processes that can ultimately consume so much time and energy they are effectively undermining the project. Perhaps its human nature that those of us closest to managing a process are at risk of becoming consumed by it and perceiving all problems as challenges to be solved with more process.