In Agile, the Practitioners Own the Process

Dear Junior

The term "process management" might be one of the driest around in system development, but in the shift to agile, the term hides a secret both surprising and delightful. 

Demystifying "process", is nothing but our daily habits, what we do when creating systems. Examples of process include how we decide when we commit stuff, how we manage our requirements, whether we do pair programming, when to do code reviews, how we decide something is finished , etc. It is really about our every-day habits of life when working.

Put that way "process management" becomes how we decide what habits we should have, or how we change them, like if we should introduce code review, or take them away, if we should be more rigorous on "no failing tests" when committing, etc.

Put that way "process management" is simply how we get better at development over time.

Process Management Traditional Style

In traditional process management each organisation has some kind of authority that decides what process we should follow. It might come in the form of a "Chief Process Officer" or as a "Project Office", but the point is that somewhere we decide the process. The groups that do the daily work of creating and maintaining systems are then to follow this process. These groups can be projects, maintenance groups or have some other organisational structure. For simplicity let us call them "practitioners" for short. So, project office decide the process, practitioners follow the process.

In many cases "project office decide, practitioners follow" is an illusion. There actually exist (take my word for it) project offices that write a process behind locked doors and then hand out a binder to the rest of the org, while believing the process will be followed. Truth is that those documents will never be read outside the project office, and will obviously not be followed.  

As an example, the project office might have decided that all code should be reviewed, and then fully believe that so is done — because it is in the process. The truth might be that no code is ever reviewed at all and the practitioners might not even know code review is part of the formal process. This "alienated project office" is of course a process management anti-pattern.

A good process office traditional style will of course behave differently. It will stay in touch with the practitioners and check whether the process is followed. If it is not, they will question why it is not followed, and explain the rationale of why they process look the way it does. It will also gather feedback and change the process to better fit  changed of discovered needs. 

However, even if the project offices adapt the process upon request, it is still in charge. If a practitioner what to change the process (e g "continuous, automated deployment to test environment"), she will suggest that change to the project office that will evaluate it. If the project office thinks it is a good idea, they will change the process, and the practitioner can start following the improved process. However, the practitioner cannot single-handed change the process just because she thought it was a "nice idea, worthy to give a try".

Process Management Agile Style

Agile development on the other hand relies heavily on the idea of continuous improvement. The basic idea is that that the process you have today is just the best you have come up with this far. Further more, the challenge of today is to find out what you should do different tomorrow. Agile development also honours doing experiments — if you get an interesting idea, then try it out for some time. At next retrospective, you evaluate the change and keep the it if it proved beneficial — or throw it away and try something else. 

Experimentation is not only a right in agile, it can also be claimed to be a responsibility. Taking it to an extreme, it can even be claimed that if you have not changed your process in some significant way the last year, you are probably no longer agile. It is probably a sign you no longer do experimentation coming up with no ideas to try. That you have found "the perfect process" is just to improbable to be likely. 

Here comes the mind-shift: for the practitioners to do this experimentation, they should have a mandate to do these continuous changes to the process. In other words, they should be empowered to change the process (ever so slightly) at their own will.They should not need to ask project office for permission. We want to keep the threshold for new ideas as low as possible. 

Rephrased, the practitioners are actually unilaterally allowed to change the process. They now own the process they follow themselves, it is no longer owned by the project office.

Project Office Still Valuable

This does not mean that the project office becomes obsolete. Quite the opposite, it might even become more important. When all practitioners can form their process, it becomes even more interesting to have some group that pays attention to all these changes, gather the changes that where successful, and can promote cross learning between practitioners of different groups. 

A team might for example struggles with that some pieces of the code is only understood by one single developer. Independently another team has solved a similar situation by using pair programming whenever such code is touched. The project office might then recognise the situation and suggest that the struggling team try that method to see if it will help them.

At this point of time the Project Office might change its name to enforce their new role. Some have re-branded themselves as the Agile Office, which I think is nice turn. Mike Cohn has reported that at one of his clients they called this group the "Agile Enablement Team" to emphasise that their goal is to enable other to become more agile.

What about Standard Process 

In agile process management you can still have a standard process. The difference is that the standard process will not be pre-defined, but rather emerge. The project office will gather ideas that have worked at some team, e g "pair program on baldy understood code",  and suggested it to some other team. If it worked at the second team as well it will promote the idea a little more active, and if it fits the organisation as a whole, it will soon enough be a standard among most or all teams. At this point "pair program on badly understood code" has become part of a de facto standard.

Over time, more a more practices will emerge and spread to all teams and a standard process will be formed. But, it will only do so if those practices actually work well throughout the organisation. And, it will not be cut in stone, but open to be challenged and evolved in the same dynamical manner. 

So, the standard process will not become a standard process because someone say so, instead it will become the de facto standard process because a majority of the teams have tried and found the same set of ideas valuable and fit within the organisation. Here the project office plays a key role in facilitating and guiding this evolution so that the good ideas are spread and that the practises have a chance to converge instead of diverging.

The Shift

In traditional process management, the process is owned by the process office and followed by the practitioners.

In agile process management, the process is owned by the practitioners and facilitated by the process office.

Small mind shift in theory, huge difference in practice.

Yours

   Dan

P.s. The situation of the Project Office in an agile environment resembles very much the situation of the architect who is unable to enforce architecture, but instead must work with the practitioners to explain and evolve it.

Refactoring as Time Machine

Dear Junior

If I had a time machine, I would not need to worry very much on how to invest my retirement pension plan. I could just pick any random fund, and if that one turns bad, I could just travel back in time and pick one that will succeed instead.

Now, who said that time travel should be free? If nothing else, the power to run it should send your electricity bill to unusual heights. So each time travel will certainly come with a fee. However, if it enables me to "have made" a better choice of pension plan thirty years ago, it would probably be worth it.

The Existential Angst of Making Decisions

How wonderful would that be, to not have to spend effort and energy on finding what hopefully is the best, and then have to worry about having made the wrong decision.

Unfortunately, I have no time machine. And chances are I will not have one either when time comes for me to retire, about thirty years from now.

So, boring and uninteresting it might be, I have to spend time and energy on investigating different choices to make a good one. And live with the angst.

In System Development We also make Decisions

When developing software, I constantly have to make a lot of decisions, small or big. It might be just what to name a method, or it might be how to split the system in tiers; still decisions, decisions, decisions. And, all those choices affect how easy the system will be to work with in the future - picking the wrong choice, and I will have a hard time.

Sometimes I find that I have earlier made a design- or architecture decision that "seemed like a good idea at the moment", but now feels like crap. Perhaps I should have spent a little more time and effort thinking over all possible futures?

In System Development We Have a Time Machine

To my delight, I have found that in system development I have the time machine I lack in financial investments. Often some not-too-complicated refactorings can "take me back in time" and change that template method solution (that does not suite me well any longer) to a shiny state and strategy pattern (that does suit my current needs).

Of course, the refactoring time travel is not for free - there is a fee to be paid in the work I have to do and the testing to ensure it went well. However, that fee is much lower than paying the price of continue working with a code-base that is not well aligned with my best-idea-at-the-moment.

Obviously, I still have to spend enough time and energy to find a decent solution when making a choice. The difference is that I do not have to spend the enormous effort to cover all options. The possibility of making future refactorings simply shrews the economics of making a decision. Refactorings make it possible to settle for a good solution now, keep the speed up, and pay the small fee for changing your mind in the future. And I can skip the "angst" part.

In that way refactoring is a wonderful little time machine.

Yours

  Dan


Ps The ability to do refactorings in the future makes it possible to keep options open longer, thus deferring decision-commitment. This is what I see as the central message of the Real Options-movement.

Complexity Drives Effort

Dear Junior

There has been a recent discussion on story point estimation (or sizing) that is so common in Scrum. Especially the discussion has been about how sizes should relate to complexity  and to effort needed to complete story. In a blog post Mike Cohn made a good example between effort and complexity by comparing "licking a thousand stamps" on one side, and "performing a simple brain surgery" on the other side. These where assumed to take approximately the same effort, but had drastically different complexity. Point was that they should have same amount of story points.

It is Effort that Makes a Difference

I think there is a lot of merit to the point Mike makes that it is amount of effort that is the usable parameter - we need it to make rough cost estimates and to predict what  functionality can plausibly be released when. Complexity is not usable in the same way.

However, even if "effort" is what we primary care about, to me complexity plays a very central role - more than I have seen mentioned in the discussion. To clarify I would like to dive into the thousand-stamps-example.

Story is Effect, not Task

To start with, the team should not get a story saying: "lick a thousand stamps"; because that prescribes a specific solution. A good story should only point out the desirable effect, and leave to the team to pick a good solution. I guess the story could instead have read: "ensure a thousand stamps get licked", or something along that line. The difference is subtle but important. In the first case, the team is given a task to perform. In the second case, the team is given a problem to be solved. 

Bulk Driving Effort

If you are to lick ten or a hundred stamps, you will certainly just do it. If you are to lick a thousand you will get mighty bored and tired - with ten seconds per stamp that amounts to three hours of work. However, it is still manageble, so you just get onto it. So here the effort will be driven by the bulk of the job - more stamps, more effort.

However, if you are to lick ten thousand stamps, or a million, then you are facing another magnitude of effort. To lick a million stamps would take 10 000 000 seconds, which is 70 weeks of full-time work - the better part of two years. Facing that task I would probably start building a stamp-licking machine.

Interesting enough this changes the game. The effort of building such a machine is independent of the amount of stamps to be licked. It is roughly as hard to build such a machine whether it will lick thousand or ten million stamps. So, bulk does no longer drive the effort.

Complexity Driving Effort

So, how hard is it to build a stamp-licking machine. Well, the actions involved is pretty straight forward, there are only one set of maneuvers, and no decisions to make - so, not-too-hard is a fair guess. 

And here comes the other half of the interesting change of the game. The effort is decided by how hard it is to build such a machine - i e the complexity of the task. So, bulk does not drive effort any longer, instead it is the complexity of the task that drives the effort.

Software Development Teams do not do Bulk

Going back to software domain, the equivalent of licking lots of stamps (10 seconds each) is probably to send a pre-formatted mail to a long list of email addresses found in an excel sheet ("new mail", CnP mail content from template, CnP email from next line in sheet, "send").

A team of developers given the story "send mail to top thousand customers" would certainly not open Outlook and start at top of list. Instead they would write some script that generates a mail to mailer daemon given the to-addr - effort 15 min. Then they would write another script that loops over a text-file with email addresses and sends same mail to all of them using first script - another 15 min. Last, they would export the mail addresses from the excel sheet to a file, feed it to the looping script, and off we go - another 10 min. OK, let us allow for extensive testing and coffee break but we are still around two hours.

Complexity Drives Effort

The important point here is not whether the script-solution would be faster or not. The important point is that the effort needed is not determined by the size of the batch. The effort needed is determined by how hard the problem is to solve - how much time must be spent discussing design, how tricky it is to get right in coding, how much testing is needed to feel confidence in the solution. In other words, its complexity.

The effort which we need to put in to solve a problem in software development is very much a function of the complexity of the problem. The complexity can be essential to the problem, i e if we are working in a knowledge-intensive domain. This is of course the call for Domain-Driven Design. The complexity can also be accidental, e g if we have an over-complex architecture or a codebase in bad shape. This is of course the call for software craftsmanship.

Disregarding the reason, overcoming said complexity will be what decides whether we need to spend a lot of effort or not. 

Sizing, or estimating, in story points should definitely reflect the amount of effort we think we need to put in. However, in most cases it is the underlying complexity - essential or accidental - that drives that effort.

Yours

   Dan

Ps On of the reasons why complexity of problem is so central is that software development to a large degree is a knowledge journey.

pps It is easy to be fooled into discussing what "story points really are", but from a pragmatic view that is uninteresting, unless it makes a difference. What matters is "what story points are useful for", i e the point of story points.