Agile Projects should have Effect Targets

Dear Junior

Measuring projects and setting targets for them is a tricky business. And, it does not get easier when agile projects enter the scene. This is not really because agile projects are strange per se, but because they are different from non-agile projects.

Setting targets and measuring projects is also an important business. I have seen several agile initiatives fail. Most of them have failed because they did not succeed in setting goals for projects, and monitor their progress. And if you cannot do that, you quickly lose the confidence and support from upper management. Agile initiative terminated, or left to dwindle away. End of story.

However, it need not to be so. Agile projects can have measurable targets, and they can be monitored - you just have to do it right.

There are some bad news and some good news.

The Well-Established Chaos Rod for Measuring Success

The bad news is that agile projects cannot be measured using the de-facto established standard rod for project success.

The standard rod for measuring projects is "on-time and on-budget, with all features and functions as initially specified", as used by e g Standish Group in their much-to-cited Chaos Reports. Let us call this the "Chaos rod". Most organisations use some version of the Chaos rod for measuring their projects.

As we have discussed earlier it is pretty obvious that "all features implemented" is not a good way to measure "project success" - not for any project. Nevertheless, this is the standard rod.

Damned if you do, damned if you don't

Of course agile projects will fail if measured using the Chaos rod. The reason is simple - agile projects does not manage to keep their hands away from fiddling with the original specification. Agile projects remove stuff, change stuff, add stuff - agile projects are in a constant state of scope-creep.

This is no coincidence - it is how agile projects are designed.

Think of it this way: If we learn something during the run of the project - shall we let that insight effect the plan of what we intended to do? Or shall we ignore that insight, sticking to the original plan even though inferior? Of course we will adapt! Anything else would be ridiculous.

An agile project anticipates that such insights will emerge, so its processes are designed to harness and leverage upon those insights: demos, retrospectives, re-prioritisation of product backlog etc. These practises are all aimed at constantly refine and redefine the scope. But then, we have derived from the original specification "all features and functions as initially specified".

Thus, any agile project longer than a sprint will fail - by definition. That is, if you use the Chaos rod for measuring success.

To put it bluntly, if you measure an agile project using the Chaos rod, it will fail. Either the agile process will fail, or the project as defined will fail.

Either, the project adapts to its measuring rod and blindly follows the "functions as specified", throwing whatever insight gained aside. Then, "project" will succeed, but "agile" will fail.

Or,  the project will work according to its agile-minded processes, and then certainly the result of the project will not be "all features and functions as initially specified". Then, "agile" will succeed, but "project" will fail. I e project will fail as measured by Chaos rod.

Damned if you do, damned if you don't.

A New (or Old) Hope

The good news is that the Chaos rod is not the only way to measure projects. As we have discussed earlier, there has always been two ways to measure projects: feature list (Chaos rod) or measuring business effect, what we can call the Effect rod.

Let us take our earlier example with an on-line book store. They had an idea that recommendations of the type others-have-bought would increase their sales, so they set of some money for that projects.

Using the Chaos rod they would have created a feature list around others-have-bought recommendations. The project would later be evaluated on how well it implemented the list. But let us look at a better idea.

Using the Effect rod they might set a target that customers will by 0.35 more books per checkout on average. The Effect rod is used to state the value of a project. These 0.35 books can probably be converted to money, that makes the project worth the effort.

The project might start out with an idea that others-have-bought recommendations would cut the cake. After the first small stories, deployed to production and put into hands of customers, the team learns that some kind of category would be helpful. To facilitate this they implement a simplified "search similar" as one of their features.

Suddenly the number of books in the checkout carts raise to a level 0.4 above the pre-project baseline. And the level sustains, it was not just random noise - the change is statistically significant.

The project has fulfilled its target according to the Effect rod, and is declared a success.

Measuring using the Chaos rod "on time, budget, and specification" - the project would have been deemed a failure, because it did not deliver the initially specified others-have-bought.

So, agile projects can be measured. You just have to use a measurement that is well suited. And, to be honest - which is better anyway.

A sad side-note is that I know of several projects which have worked in an agile manner, and delivered enormous business benefit - but in the project report the project lead has had to apologise repeatedly for not meeting the project target as defined in the project specification - a feature list.

Granted, to measure projects on business effect is not a new idea in any way at all. The idea was certainly there before the Agile Manifesto was written around the turn-of-millennia. What is new is that this way of measuring is essential to provide rigour to agile projects.

For Agile to succeed at larger scale, we certainly need lots practises around agile-minded processes. Measuring agile projects on effects is certainly one of them.

The way to measure agile projects is to set targets for business effect.

Yours

   Dan

PS Interesting enough, there is a sub-category "agile projects" in the later Chaos Reports, but the rate of success is not 0%, it is around 40%. I wonder what is going on here.

PPS Within the agile community, the practice of projects is debated - at least in the sense of the common description "temporary organisation with limited time, resources, and ambition". So, it would probably be better to use some other term, e g talking about "initiatives". However, for convenience, I have kept the often-used and familiar term "project". Check out the twitter hashtag #NoProjects.

All Features Implemented does not equal Project Success

Dear Junior

Setting goals for agile projects is trickier that setting goals for waterfall projects. As I mentioned in an earlier letter, for waterfall projects it is possible to set the goal in the form of a feature list to be delivered. This approach has several drawbacks - it does not guide the thousands of micro-decisions that are made, it gives no sense of purpose to support the drive or inner motivation, and it gives no guidance for evaluating whether the money, time, and effort was well spent.

However, although all these problems, and although the approach is not advisable - it is still possible. The project management mindset and tools for waterfall projects are applicable for reporting project status or following up visavi a feature list. It is also possible to evaluate success by checking whether everything on the feature list is implemented.

Well, I still think it would be better to evaluate whether the project created some value.

Now, ridiculous as this seems it is still industry standard.

The CHAOS report 1995 from Standish Group might be one of the most cited reports in system development. According to its statistics only 16% of software projects succeed. In the 2012 report this has been updated to 39%. And they collect data from tens of thousands of projects so the figures should be pretty reliable. 

Now, this seems scary and low, but only until you check out their definition of success.

"Resolution Type 1, or project success: The project is completed on-time and on-budget, with all features and functions as initially specified."

OK. Not a word about actually getting value for the effort. Remember the on-line bookstore where they wanted others-have-bought recommendations to increase the number of books customers. Now, imagine they implemented the others-have-bought recommendations, but the customers did not buy more books anyway. Is this project a success?

Well, you have spent lots of money building something, but made no money from it. To me it sounds like money, time, and effort down the drain - not as a success.

To Standish Group, it is considered a success.

On the flip side, imagine the project realising that a simplified "search similar" would increase the number of books each customer buys. Imagine further that this feature would be much easier to implement. So, the project decides to implement that feature instead. And the number of books sold per customer increases 0.4 on average.

Spending less money, time, and effort on something else than originally envisioned, but still getting the benefit - that sounds like success to me.

To Standish Group, it is considered a failure.

Thus, the figure 16% or 39% actually says nothing at all about the state of software projects.

So, Standish Group is probably filled with smart people. Why did they not evaluate software projects according to some meaningful metric instead? Most probably "generated business value as specified upon funding" would be more interesting. 

My guess is that too few projects actually stated an envisioned business effect, so analysing those projects would give so little data that it would not be possible to make any significant conclusions.

So, instead of measuring something that would actually matter ("fulfilled envisioned business effect") they just measured something that could be measured.

Now, from the second example, where the team implemented another feature than originally envisioned, it is also clear that this approach is an absolutely worthless way of evaluating agile projects.  

Yours

   Dan

PS The way to measure agile projects is to set a target for a business effect.

PS The CHAOS 1995 report has been republished openly available for academic purposes. It can be found at http://www.projectsmart.co.uk/docs/chaos-report.pdf. The 2013 version can be found at http://www.versionone.com/assets/img/files/CHAOSManifesto2013.pdf

Project Goals using Effects or Feature List

Dear Junior

There has always been two ways to set goals for a project, either you can define the goal of the project to be a list of feature to be implemented, or you can define some effect you want to see. Both of these have been around for a long time, and effect goals have always been superior. 

Setting goals as a feature list is actually pretty simple - you simply state "these features I want to see before we consider this project closed". An online bookstore could say that it wants search-by-author, categories, and others-have-bought recommendations.

Following up during the project will simply consists of checking how far on this check-list the development team has progressed.

Effect goals are a little bit harder to set. You have to figure out and express why you want to have work done, what purpose it fulfils, in what way it makes your world better. Said bookstore must then express that it hopes people each customer will by 0.35 more books per checkout on average, or that it will increase its customer base with 10 000 new customers.

Following up during the project using this approach will consist of measuring the business parameters and watch the values change. A tricky part is that the full impact of the project might not be seen until the development work is "finished" (whatever that means), or even not until some time thereafter.

To put things a little bit into context, it is not the "bookstore" that starts a project - it is always a person involved, in this case probably the product owner of the online store. And, this is the person who should explain why she is about to spend a lot of peoples' time and a lot of the organisation's money.

I have always found that measuring success on effect, or impact, is the more intellectually honourable approach. To be frank, setting goals as a feature list does really say "this much work I want to have done" or rephrased "this much money I want to spend", nothing about what value it should result in. And if you want to have a group of people to work to bring your ideas to come real, the least you can do is to explain to them the value you think it will bring. Setting goals on effect is really about that - bringing purpose to the work.

Now, these two ways of setting goals and measuring success have a fundamental impact on how to manage projects in an agile manner, but that will have to wait until a later letter.

Yours

   Dan

PS An agile project cannot be feasibly measured using feature lists; agile projects should be measured by setting a target for a business effect.

Two Observations on Overtime in Traditional Projects

Dear Junior
I have seen quite a few classically managed projects, and through friends and collegues I have been in contact with even more of them. I would just like to make two observation on the practice of working
overtime in different phases of such projects.

* It is very uncommon, actually I have neither experienced nor heard of, that project management orders overtime for requirements analysists so that they will deliver the full and finished requirement documentation on a specified date.

* It is very common, actually I have experienced it several times myself and heard of it numerous times, that project management orders overtime for programmers so that they will deliver the full and finished implementation on a specified date.

From these two observations it is probably possible to deduct lots of interesting conclusions. I will refrain from doing so here and leave it up to you as an interesting mind-game.

Yours

Dan

In Agile, the Practitioners Own the Process

Dear Junior

The term "process management" might be one of the driest around in system development, but in the shift to agile, the term hides a secret both surprising and delightful. 

Demystifying "process", is nothing but our daily habits, what we do when creating systems. Examples of process include how we decide when we commit stuff, how we manage our requirements, whether we do pair programming, when to do code reviews, how we decide something is finished , etc. It is really about our every-day habits of life when working.

Put that way "process management" becomes how we decide what habits we should have, or how we change them, like if we should introduce code review, or take them away, if we should be more rigorous on "no failing tests" when committing, etc.

Put that way "process management" is simply how we get better at development over time.

Process Management Traditional Style

In traditional process management each organisation has some kind of authority that decides what process we should follow. It might come in the form of a "Chief Process Officer" or as a "Project Office", but the point is that somewhere we decide the process. The groups that do the daily work of creating and maintaining systems are then to follow this process. These groups can be projects, maintenance groups or have some other organisational structure. For simplicity let us call them "practitioners" for short. So, project office decide the process, practitioners follow the process.

In many cases "project office decide, practitioners follow" is an illusion. There actually exist (take my word for it) project offices that write a process behind locked doors and then hand out a binder to the rest of the org, while believing the process will be followed. Truth is that those documents will never be read outside the project office, and will obviously not be followed.  

As an example, the project office might have decided that all code should be reviewed, and then fully believe that so is done — because it is in the process. The truth might be that no code is ever reviewed at all and the practitioners might not even know code review is part of the formal process. This "alienated project office" is of course a process management anti-pattern.

A good process office traditional style will of course behave differently. It will stay in touch with the practitioners and check whether the process is followed. If it is not, they will question why it is not followed, and explain the rationale of why they process look the way it does. It will also gather feedback and change the process to better fit  changed of discovered needs. 

However, even if the project offices adapt the process upon request, it is still in charge. If a practitioner what to change the process (e g "continuous, automated deployment to test environment"), she will suggest that change to the project office that will evaluate it. If the project office thinks it is a good idea, they will change the process, and the practitioner can start following the improved process. However, the practitioner cannot single-handed change the process just because she thought it was a "nice idea, worthy to give a try".

Process Management Agile Style

Agile development on the other hand relies heavily on the idea of continuous improvement. The basic idea is that that the process you have today is just the best you have come up with this far. Further more, the challenge of today is to find out what you should do different tomorrow. Agile development also honours doing experiments — if you get an interesting idea, then try it out for some time. At next retrospective, you evaluate the change and keep the it if it proved beneficial — or throw it away and try something else. 

Experimentation is not only a right in agile, it can also be claimed to be a responsibility. Taking it to an extreme, it can even be claimed that if you have not changed your process in some significant way the last year, you are probably no longer agile. It is probably a sign you no longer do experimentation coming up with no ideas to try. That you have found "the perfect process" is just to improbable to be likely. 

Here comes the mind-shift: for the practitioners to do this experimentation, they should have a mandate to do these continuous changes to the process. In other words, they should be empowered to change the process (ever so slightly) at their own will.They should not need to ask project office for permission. We want to keep the threshold for new ideas as low as possible. 

Rephrased, the practitioners are actually unilaterally allowed to change the process. They now own the process they follow themselves, it is no longer owned by the project office.

Project Office Still Valuable

This does not mean that the project office becomes obsolete. Quite the opposite, it might even become more important. When all practitioners can form their process, it becomes even more interesting to have some group that pays attention to all these changes, gather the changes that where successful, and can promote cross learning between practitioners of different groups. 

A team might for example struggles with that some pieces of the code is only understood by one single developer. Independently another team has solved a similar situation by using pair programming whenever such code is touched. The project office might then recognise the situation and suggest that the struggling team try that method to see if it will help them.

At this point of time the Project Office might change its name to enforce their new role. Some have re-branded themselves as the Agile Office, which I think is nice turn. Mike Cohn has reported that at one of his clients they called this group the "Agile Enablement Team" to emphasise that their goal is to enable other to become more agile.

What about Standard Process 

In agile process management you can still have a standard process. The difference is that the standard process will not be pre-defined, but rather emerge. The project office will gather ideas that have worked at some team, e g "pair program on baldy understood code",  and suggested it to some other team. If it worked at the second team as well it will promote the idea a little more active, and if it fits the organisation as a whole, it will soon enough be a standard among most or all teams. At this point "pair program on badly understood code" has become part of a de facto standard.

Over time, more a more practices will emerge and spread to all teams and a standard process will be formed. But, it will only do so if those practices actually work well throughout the organisation. And, it will not be cut in stone, but open to be challenged and evolved in the same dynamical manner. 

So, the standard process will not become a standard process because someone say so, instead it will become the de facto standard process because a majority of the teams have tried and found the same set of ideas valuable and fit within the organisation. Here the project office plays a key role in facilitating and guiding this evolution so that the good ideas are spread and that the practises have a chance to converge instead of diverging.

The Shift

In traditional process management, the process is owned by the process office and followed by the practitioners.

In agile process management, the process is owned by the practitioners and facilitated by the process office.

Small mind shift in theory, huge difference in practice.

Yours

   Dan

P.s. The situation of the Project Office in an agile environment resembles very much the situation of the architect who is unable to enforce architecture, but instead must work with the practitioners to explain and evolve it.

Agile Product Owner - the New Project Manager Role

Dear Junior

From time to time I hear claims that: "with agile/Scrum, there will be no need for project managers any longer - they will have to find some other job". This seems strange to me, I get what they are getting at, but the analysis is very incomplete.

To start with, a role cannot go and find some other job - role is a job description, not a person. A person can go find "some other job". So, I guess that they mean that all people currently employed as project managers will have to go find themselves something else to do, because "Agile" has come along making them as obsolete as the advent of cars made company stable-hands no longer needed, simply because their skill sets where no longer needed.

Interpreted that way, I can see what they mean. If you have a project manager, who's major part of the workday is to "administer resources" by assigning tasks to programmers and controlling the result - then than those tasks will be absorbed into the Scrum team (using Scrum as an example) and their self-organisation and commitment-driven collaborative work under the lead of a scrum master. And, if the rest of the workday is filled with planning and fiddling with requirements - then those tasks will be absorbed by the engaged, involved product owner (still using scrum as an example). What is left in-between is the clerk accounting tasks like compiling time reports and other stuff - and those things are not considered necessary any longer. Thus, what are left are unemployment, retraining, and disappearance from the scene.

Throughout my years in software development I have met more than a few project managers. Some of them have done a good job and I have gained respect for their skills and efforts, some not. For those, I have not been impressed by, the above analysis hold - I think software development as a whole, and the poor teams they managed specifically, are and would be better off without them. However - the good project managers I have met have made a difference through their skills and efforts - chasing them on the door for having "project manager" on their business card would be a serious mistake.

Where do the skills of these good project managers fit in the "Brand New Agile World of Scrum"?

The naïve answer is often "they become scrum masters". However, when I look at the tasks and skills needed for being a scrum master, and at the skills of the good project managers I have met, I do not see a very good match. Well, "good leader" is a match, but then it stops. The scrum master leads the work in a very tight loop, facilitating the synchronisation of work between programmers, keeps the work focused on the next "smallest step" etc. If I would appoint someone as scrum master, I would not look among the project managers, I would look among the programmers, looking for the informal leader, the one trusted and respected by the rest of the team; perhaps the person already have a title like "lead programmer" or similar. Given complete freedom, I would probably go even one step further: I would let the team itself elect the scrum master, with some mandate period and some mechanism for re-elections.

If not scrum master, then what to do with our excellent project manager? When thinking about where I have really seen them making a difference, there are a few skills that pops into my head. First, they have been leaders with a vision. Even tough they have had some kind of steering board or committee that supposedly has given directions, it is the project manager that has embodied the vision and purpose of the project, and done the job of "walking the floor", being involved and keeping people engaged. Secondly, they have had a really good grasp of requirements and prioritisation thereof. They have understood to the minute detail what the system should do and why, they have been available to answer questions and straighten question-marks when needed. And, thirdly, they have taken responsibility, making decisions when needed - even when putting them self at risk by not waiting for the next steering committee meeting to pose the question, realising some decision is needed right now.

So, where do these skills come best to use in e g Scrum? It should be pretty obvious: they should be product owners!

But, the product owner, should that not be the Manager of Web Sales, or some CXO? For crying out load: NO! No CXO I have met have the time to do a good project owner job. They will not be willing to spend at least an hour a day in the programmers' workspace to answer questions, they will not be at the daily standups, and they will not come to the retrospectives. The former project manager will do all this. And will most probably talk with the Mgr of Web Sales and the CXO, as they are important stakeholders (former members of steering committee), to understand the interests at stake in the system, where after she will make an informed decision, now also having the formal mandate to do what she has always done.

I wish I could give one piece of advice to the excellent project managers I have met. I that case it would be this: do not get fooled to think you should become a team lead or a scrum master; it is a waste of your skills. Your role in the brand new world of agile is the one of a product owner - do what you have been doing so well, now with full empowerment. We all need you there.

Yours

Dan

ps Check out my fellow Swede Anna Forss' essay on "Confessions of a Serial Product Owner". She is an excellent example of a project manager turned product owner. She is also one that shares my interest in the combination of domain driven design and product ownership.

pps Some of the skills for priority of requirements/stories I have seen at good project managers that would make excellent product owners are spending a lot of focus on keeping things upcoming really crisp, and let the rest of the list be a little bit more informal - but never sloppy.

ppps There are many ways to describe a good product owner. One is through the four characteristics engaged, involved, decisive, and empowered.
ppps To give credit where credit is due, the fragments of this idea had been lingering around in the back of my head for some time, but it was not until my friend and colleague John Wilander said it out aloud that I got the push to phrase it properly.

The "Resource" Abstraction - Discarding Information

Dear Junior

I constantly become surprised of the use of the abstraction ”resource”, and have actually not really understood how it has become so popular. Abstractions are used to discard irrelevant details to be able to focus on what is important, e g when we write a class Customer and actively omit the shoe size or waist length (unless we are in the shoemaker or tailing domain). However, the way “resource” is used by a lot of project managers does not fit that description.

Everyone that knows anything about system development does also know that one java programmer is not the same as another java programmer. The difference in skills might make Alice, who happen to know a lot about databases, totally impossible to replace with Bob, who is an expert on unit testing, but know little about databases. Not to talk about that Cedric and Dawn just become a killer team when working together.

OK, I understand that if you are the VP of Ericsson’s mobile device division, it might be interesting to talk about groups of a few hundred developers (resources), because at that level statistics will level out stochastical differences between the exact composition of each team.

That said, most development groups are not a few hundred people. They are five, ten, twenty, or fifty individuals.

If we are to make dinner at home I do not look around and say “we have three resources”, “someone need to chop vegetables”, and “allocate 'vegetable chopping' to resource number two”, because that might end up with a sharp knife in the hands of my two-year-old son. Even if we are all my siblings, parents and in-laws (which might add up to twenty people), not even then does in make sense to “deidentify” them to resources – because some of them are good at making sauce and some are crap at that trade.

So, when planning the department’s job for the coming quarter or year, why not say: Lisa, Per, Ashok, and Yasmine will work on CogStat; John, Hassan, and Helga on FormFeed, but Helga will also help out with persistence in CogStat; finally Jari and Dimitri will prototype CaseSmart and also help out with CM problems. Probably the people involved will come with suggestions, improvements and problems; plus that they feel comfortable in having a personal plan. Saying: Cogstat 4.5 resource, FormFeed 3.75 resource, and CaseSmart 0.75 fulltime resource will only cause confusion and not help creating an efficient work place.

When it comes to the bottom line it is really simple: Using the information that people are different (in relevant ways) must logically lead to a more optimal solution (stable release plans, productive teams, happy members), than discarding that information.

In the end we all know that individuals and interactions make more of a difference than processes and tools.

Yours

Dan

ps This is an issue where I think agile is different