What is Snow?

Dear Junior

On the morning after a night of snowfall, I took a morning walk to kindergarten with my two sons. The older of them (four years old, but soon five) asked:

"Why does snow glitter?"

Good question and I described how snow consist of snow flakes that are flat, and that it is the top-most layer of flakes that glitter - in the same way as the glitter palettes that you can glue onto paper. We discussed it a while, and he seemed to think that it made sense.

After some moments of silence came the next question:

"Why is snow soft and fluffy?"

Good question, and I described how the flakes fall on top of each other in loose layers. We discussed it a while, and he seemed to think that it made sense.

To no big surprise there came yet another question:

"Why are snow balls hard?"

Good question again, and I described how the loosely layered flakes are squeezed together and get stuck in each other - that is why the snowball does not fall apart. We discussed it a while, and he seemed to think that it made sense.

Now, is snow *really* snowflakes? Nahhh, snow is water that sticks together through covalent bindings that behaves in a certain way when the temperature drops so much that the kinetic energy of the molecules is so low that it make them bind to each other in certain geometric patterns.

But, you do not expect me to explain that to a four year old kid, I guess …

The snow flake metaphor is a model. It abstracts away a lot of really complex stuff to give us a way of explaining a limited set of phenomenons. Put another way, it answers some interesting questions.

As all models it has its limits. For example, had my son asked "How come that when snow melts, it become water, but when it freezes again it does not become snow, but ice?". Well, then the snow-flake model would not have helped much. It is just a model. It answers some questions in a way that makes sense, but not all questions.

However, the same goes for the "covalent binding" explanation. It is also just a model. It is not more or less "true" that the snowflake model. It is just capable of explaining another range of phenomena in a way that makes sense.

They are both useful models, and that is where they get their value. The interesting question is not whether the model is "true" or not, but whether the model is useful.

The "snow flake" model enables a father to give sensible answers to some questions a four year old boy might ask on the way to kindergarten on the morning after a night of soft snowfall.

From my point of view, that is a good model.

Yours 


Dan

Fix the Problem, not the Bug

Dear Junior

It is not always "fixing the bug" is the right thing to do. Well, it depends on what you mean with the word "bug" I guess. But, to often "fix the bug" means "make a local change to make the bad behaviour go away". Instead we should search for the cause of the trouble, the problem itself - and fix that.

Say you have an application where you handle train departures. And somewhere you are to show a result of some search - a result that comes packed as a ResultInfo object containing timestamp and matching departures.


Simple, Clear, Readable Code

So there might be a class for listing results looking something like:

/** Lists train departures from searches. */

public class DepartureLister {

  private PrintWriter resultwriter;

  /** Render search result */

  void showlist(ResultInfo resultinfo) {

    resultwriter.println("resulttime: " + resultinfo.timestamp);

    for (Departure departure : resultinfo.matching) {

      resultwriter.println(departure.trainNr);

  }

  resultwriter.println("matches: " + resultinfo.matching.size());

  }

}

For simplicity, let us say that the classes ResultInfo and Departure are the most simple possible.

/** Search result with matching departures */

class ResultInfo {

  String timestamp;

  Collection matching;

}

/** Train departure identified by its train number */

class Departure {

  String trainNr;

}

OK, pretty obvious what the code does as well as its purpose.

Finding a Bug

Now imagine that this code at some occasion barfs with

java.lang.NullPointerException

at client.DepartureLister.showlist(DepartureLister.java)

on the line

for (Departure departure : resultinfo.matching)

What is going on? From a technical standpoint it is pretty obvious that this code is wrong as it did not take into account that the list "resultinfo.matching" could be null.

Fixing the Bug

So fixing the bug we change the code by wrapping the for-loop with a null-guarding if.

if (resultinfo.matching != null) {

  for (Departure departure : resultinfo.matching) {

    resultwriter.println(departure.trainNr);

  }

}

And we also need to do something about the result-size printout.

resultwriter.println("matches: " +

  (resultinfo.matching != null ? 

    resultinfo.matching.size() : 0))

Well, it works, but I would not say that the code is "pretty obvious" any longer. Even if we refactor it to make it neater, there will still be the null handling hanging around messing up things. Code should not look like this. If it is easy to express "list the matching departures", then that message should be easy to read from the code. It was easy to read it, it is no longer.


Furthermore, any use of ResultInfo throughout the code will need these kinds of guards and switches. Apart from messing up the code in all those places, it is also a blatant violation of the DRY principle, stating that each "idea about the code" should only be expressed "once within the code".


Obviously "fix the bug" was not the right thing to do.

Find the Problem

Let us see what would happen if we try to "fix the problem" instead. To start with, let us formulate the trouble, but from a conceptual standpoint - what the code tries to achieve, not a technical - what the code does. The technical standpoint was "resultinfo.matching can be null". Rephrased from a purpose-perspective that would be "situations with no matching departures".

Now it becomes blatantly obvious that the natural counter-question is "Why on earth is that not represented as an empty list?". So we dig into the departure-matching code and find the search where you call a legacy backend system.

/** Searches for departure according to matching some criteria */

public class DepartureMatcher {

  private Legacy legacy = new Legacy();

  public ResultInfo searchMatching(String[] matchcriteria) {

    ResultInfo resultInfo = new ResultInfo();

    resultInfo.timestamp = new Date();

    int[] trainnrs = legacy.find(matchcriteria);

    if (trainnrs[0] == 0) {// special signal no match

      resultInfo.matching = null;

      return resultInfo;

    }

    resultInfo.matching = new ArrayList<Departure>();

    for (int trainnr : trainnrs) {

       resultInfo.matching.add(new Departure(trainnr));

    }

    return resultInfo;

  }

}

Okey, here we can see the root of the problem - why "empty list" is handled as a special case and signaled with null. One can almost see how the if-clause in the middle have been "cranked in" into the middle of the searchMatching when some programmer found out about the non-intuitive behaviour of the backend system to represent "no match" with "[0]".

Fix the Problem


Now, fixing the underlying problem is pretty easy. We change the "no match" clause

if (trainnrs[0] == 0) {// special signal no match

  resultInfo.matching = null;

  return resultInfo;

}

and replace it with

if (trainnrs[0] == 0) {// special signal no match

  resultInfo.matching = Collections.emptyList();

  return resultInfo;

}

Now the two cases "empty list of matches" and "non-empty list of matches" behave analogous. This leaves room for some refactoring, replacing the messy method with:

public ResultInfo searchMatching(String[] matchcriteria) {

  int[] legacyreturnlist = legacy.find(matchcriteria);

  int[] trainnrs = legacyreturnlist[0] == 0 // no match

                   ? new int[0]

                   : legacyreturnlist;

  List matching = new ArrayList();

  for (int trainnr : trainnrs) {

    matching.add(new Departure(trainnr));

  }

  return new ResultInfo(new Date(), matching);

}

Here it is more clear what each part of the method is doing. Of course, to make this into clean code, some more refactorings are justified - especially extracting each functional part (of a few lines) to methods with names of their own. But, we can be proud over what we have done this far.

Here is time to take a small break and ponder what we have done. We have made a piece of the code behave a little bit clearer, yes. We have removed a weird special case (null list), yes. But we have also changed the API of a public method - and that should not be taken lightly.

Can we be sure that we have not broken anything else? Do we not risk introducing an unknown heap of new bugs? And if so, how can that be better than changing the code in one place (the client where we originally had the bug reported)?

I would argue, that even if we introduce new bugs, we have still fixed a problem. The key difference is that those bugs will be code locales that rely on a non-intuitive API - so those bugs will be easy to fix. In other words: when moving to a clearer way to formulate the code, you should not be too afraid for "causing bugs". As long as the change you make solves a problem and leaves the code in better shape - then subsequent bugs are easier to fix.

Fast Fail


Of course we should do what we can to aid in finding those bugs. In this case this can be done by changing the contract for the ResultInfo. Up until now it have been legal to have a null-list of matches. Now we want empty match to be represented by empty matching-list. So let us enforce this:

ResultInfo(Date timestamp, Collection matching) {

if (matching == null)

  throw new NullPointerException( 

                "Matching list not allowed to be null " + timestamp);

  this.timestamp = timestamp;

  this.matching = matching;

}

Now there will probably be a pile of test-cases that will fail with NullPointerException and hopefully pointing out places in the code to change to adhere to the new and clearer API. 

Sum it up

Trying to fix the bug where it surfaced just led to messy code in a lot of places. That was because the problem was not really there. Not a strange thing: effect is not cause, symptom is not illness - we all know it. The real problem was "empty list" represented by null.


Instead, finding the root of the trouble - the problem itself - we could solve the reason for the unclear code. Let "empty list" be represented by an empty list, from the source. Now, the client code could stay in its original lucent form, and we could also make some refactorings of the search code to make it easier to grasp. 


On top we added some enforcement of this new policy - the ResultInfo simply refuses to accept the mission to carry "null-lists". Now all clients can rest assured they will always get a proper list of Departures.


Al in all, the code becomes clearer.


Exceptions might Apply


Of course there are points of time you do not want to do this, like a few days before a release. However, as a general rule I prefer to fix the underlying problem in the code, than to fix the bug where the problem manifest itself.

Yours

   Dan

Presenting on Ubiquitous Language at Jfokus 2011

Dear Junior

I have gotten the delighting message from the Jfokus committee that they want me to speak at the Jfokus 2011 conference in Stockholm in February. I had proposed a presentation on the Domain Driven Design concept of Ubiquitous Language which they have accepted. 

It seems to me that the ubiquitous language is one of the hardest things to get right. When I look at my own experience it is a part where there are a lot of small distinction, distinctions that are subtle but make a huge difference.

Just take an example of how you can express the same thing in different ways that seem synonymous, but actually make a huge difference.

Compare

"An account has a phone number. The account can change phone numbers."

with

"An account has a phone number. The phone number can be changed."

The difference might be subtle, but in a precise language the distinction is important. In the first case, the account have still the same (conceptual) phone number, which has been edited. The account and the phone number are both entities. In the second case, the account have changed its phone number to another phone number. In code the two cases would be

1. account.number.changeto(numbersequence)

2. account.number = new PhoneNumber(numbersequence)

So, what difference does it makes?

Well, somewhere else we might require that a new account might have the same number as an old account. Fair enough. But what happens if the old account gets a new number? Does the new account change its number as well?

Have we taken the pain to actually create a strict language that is understood in common, then we have the answer in the language. If we do not have that strict language, chances are high that we arbitrarily select one of them that "feels right at the moment". However, next time a similar situation arises there is no guarantee that what we pick arbitrarily that time is logically consistent with the first pick. Over time we risk getting a system that behaves subtly, or obviously, inconsistent with itself.

Of course, this is just one example. In the presentation, what I want to cover are practical aspects of to make a shared language to become useful - and what it takes to become useful. E g it has to be both rich enough to express the complexity of the problem, but it must still be precise enough so we can use it as a base to build a system.

Other aspects I want to cover is how such a language can grow over time (not a Big Upfront Design), problems and traps when introducing a new term, and the challenges of "mixed language environment" such as when there are both Swedish and English around.

Of course you recognize several of these subjects as we have discussed these earlier. However, I hope to put them together in a concise and structured format - and want to roll in some new ideas. I am sure there will be plenty of reasons to return to these areas in subsequent letter.

Yours

   Dan

4 Points of Story Points

Dear Junior

For quite some time there have been a debate on "what story points really are". To some extent there have been insightful discussions about important things to consider when sizing stories. However, the "really are" part of the debate just leaves me tired. 

To discuss the true nature of story points is pretty pointless: they are a construct, created by us, and can be given any arbitrary meaning. It does not matter. What matters is whether that construct is useful for a purpose. This is the core of pragmatism.

In other words, the relevant question is not "what story points are", but "what story points are useful for". Please excuse the pun, but rephrased "What is the point of story points?"

If I try to pry things apart I can distinguish four different situations where I have found story points useful. Basically they cover the questions "Is it small enough?", "What does it cost?", "Which should we do first?", and "When can I have it?".

Small Enough

One of the cardinal faults a team can do is to start working on a story that is way to big. If they do, they will surely fail to finish it within the sprint, and having a demo with nothing to show is pretty depressing. Having a lot of half-finished work also tie their hands of what they can do next sprint. No good.

A mature team might have developed a gut feeling of how big a bite they can take and still eat it. The stories considered too big are simply sent back to product owner for delimitation or splitting.

A less mature team might see that bites have different sizes, but have not yet the insight of how big a bite they can take without choking. 

Here story points can come in useful. The team can size the stories as 13, 20, 3, 8 etc. But they need not to know their limits. Instead we can observe the velocity over a few sprints and see what "small enough" means. For example, if the team have a velocity of 18, I would advice to set the limit to 9 (half velocity). So, the stories sized 3 and 8 are small enough, and those of 13 and 20 will need some more pre-sprint work to make them manageable.

It can be handy to reserve the top of backlog for stories that are small enough, a "backlog shortlist" of ready-to-develop stories. Or, if you prefer the kanban style to have one stage "delimit and split" followed by a "ready-to-develop" queue before pulling them into active development.

Cost

Whether we like it or not, the question about money always come up. It can be hard to just look at a requested feature and say how much it would cost to develop it. In a lot of development efforts the dominating cost is the cost of labour. Either the work will tax the amount of available work by the employees, or there will be contractor bills to be paid.

Here story points can come in useful. Someone probably knows what the team costs per week, or can calculate it. If you have some historical velocity data you can make a rough estimate of what each point costs. So, if you know roughly the size of the story, you can calculate the cost.

Say that your team costs EUR 27 000 a week, you run two weeks sprint, and your sprint-velocity hoover around 18. Then each point costs around EUR 3 000. So, a story of size 40 will cost roughly EUR 120 000.

Well, not all of the time will be "pure development work". There will be meetings, phone calls, administration (filling out time reports) etc. But, that does not matter. Pulling through a story of size 40 will take roughly two sprints and during that time the team will cost that amount of money.

Of course, in practice you will rather want to give an interval than a precise number. The velocity might be within 17-19 (with 90% confidence) and a "40" story can be anything from 21 to 40. So the cost will rather be in the range EUR 60 000 - 130 000. Still, it will be a figure good enough for business to decide if it is remotely interesting to proceed or not.

Priorities

A misconception among business side "customers" is that they should set priorities on business value. Well, but "economics" is really about alternatives - the cost of a bar of chocolate is that you cannot get two lollipops. In the same way, for the product owner to make a wise balance between what different stakeholders want, she must be able to compare their cost.

Interesting enough, to make priorities, we do not need to know the absolute cost of each story. It is enough to know their relative cost.

Here story points can come in useful. If feature A is size 20, feature B is 8, and feature C is 13, then we know that we can swap out feature A from a release and switch in feature B and C instead. And in doing so we do not need to care about the details of how much money it is about - all we need is aid in choosing.

Planning

Business need to look ahead to synchronize different parts of their work. E g there are some benefits in having a marketing campaign at the same time as you release some new feature of your software. But, waiting for the software to be complete before ordering the marketing will obviously make you loose market-time. 

Here story points come in useful. If you observe the velocity of the team over some time you can apply some not-too-advanced statistics to predict how much work will be completed at some future point of time. Of course, each such prediction will have a probability to fail, and the surer you want to be, the lower you must set the prediction.

For example, if you have observed the velocities 36, 28, 36, 38, 24, 35, 32, 35 and you have five sprints to go, you can calculate the average and predict the team will finish 165 points. However, that prediction is just a prediction - the real result will be as likely to be higher as it is to be lower. In other words, your prediction has a confidentiality of 50 % - or a 50 % risk of failing. 

If you want to make a safer prediction, say taking just a 5% risk of failing, you can calculate an interval with 95% confidentiality. In this case it will be the interval 144-186. Now you can mark the backlog, colouring all stories up to 144 as green (very likely to be delivered), those from 144-186 as yellow (totally "depends-on") and those from 186 and up as red (very unlikely to be delivered).

In Summary

It is very hard to talk about the "true nature" of story points. They are abstractions that say something about development work. And, the question of "what they really are" is not a very interesting one. Working with story points is a model, and a model should not be evaluated on "how true" it is - but on how useful it is.

Story points might be useful for a team to decide whether a story is small enough to fit in a sprint, of if it should be pushed back to the product owner - if they cannot do it by gut feeling. 

Story points might be useful in assessing the cost of developing a story - if it is questionable whether its value justify the cost. 

Story points might be useful for setting priorities - if it is difficult balancing the stakeholder interests. 

Story points might be useful for planning at release level - if the organisation need to synch the work of different departments or groups

Apart from these four points, story points might be useful in doing other useful things - if it helps the organisation to act more wisely.

It might well be that in your particular setting, none of the "ifs" apply - and in that case story points are not useful to you. And if they are not useful, they steal time and attention from other things that would serve you better - i e they are waste that should be discarded.

In the end "what story points really are" is not as interesting as "what is the point of story points".

Yours

   Dan


ps Story points being helpful for planning is of course key to why release planning works.


pps If story points help in planning it is interesting to see what factors drive high story points. For planning, it is the amount of effort that differs, but that is to a large extent driven by the complexity.


ppps Unless you are really good at making statistical computation, it helps to have a spreadsheet to aid in the planning.


pppps One way to ensure team does not embark upon developing something "too big" is to make a backlog shortlist of the top part of the product backlog.

Good Product Owner Wrapup

Dear Junior

I would like to wrap up our discussion about the good product owner and the characteristics thereof. Of course there are many important attributes, and there are even more ways to select words for them. However, I think a lot of them are caught by the two-by-two containing "engaged, involved, decisive, and empowered".

                  personal   organisational

social:       engaged    involved

formal:     decisive    empowered

It does not tell everything about a good product owner. It is just a model, meant to capture some important points in a concise way. So, let's just have a walk around this picture.

Starting with engaged we have a product owner that cares, who thinks the product and the system is important. This product owner has an inner drive to do things for the benefit of the product. So, it is a personal attribute. Also it is not so much about mandate and formalities, it is more about interacting with people, so "engaged" is set in a social setting.

Moving over to right we stay in the social setting, but transfer to the organisational attribute of being involved (from Latin "in the turning"; today we would probably say:"in the loop"). We are still talking about "interacting with people", but now it is not the person's inner drive. Instead it is the organisation "letting" the PO being involved. Rephrased, the organisation thinks that being product owner is the most important thing on this persons job description - there is nowhere else this person rather should be.

Moving down we stay on organisational attributes, but switch to talk about formalities. Being empowered is all about having the organisational mandate to take decisions that will stand. Of course the product owner can change her mind, and re-decide something else later. However, that is the decision of the product owner herself - not something the rest of the organisation can force upon her. The PO is a little bit like the "local CEO" of the product.

Moving to the left we stay on formalities, but move back to personal attributes. Being decisive is to have the personal drive to actually make decisions when they are needed. Of cause even a decisive person can change her mind, but the good product owner will take decisions with respect to the team, not throwing them "of balance" by completely changing direction. Still it is being decisive in the moment needed that keeps progress going.

Moving up we stay on personal attributes, but cross the border of formalities back to social settings and we end up at engaged where we started.

Now, this model is nothing but a model, and thus it is limited in its power and limited in its reach. In other words: it misses a lot of important points. It is also a product of annotating specific words specific meanings, that are by no mean ubiquitous.

Other words

Steven Hale pointed out that the distinction between "engaged" and "involved" is unclear (to say the least). He has suggested "interested" and "active" as a more distinct pair for the social attributes. This also makes the acronym IDEA possible  - which is neat. However, in defense of "engaged/involved" he also notes that there was a girl with whom he got involved, later engaged, and now they are married.

Important things missed

There are also important things this model leaves out. One of the things Joakim Holm pointed out is that fails to mention the "burning vision" the product owner should uphold. I could not agree more - that is one of the most of important things a product owner should establish, nurture, and communicate (I am keen to say "radiate"). And that dimension is not caught in this discussion on "characteristics".

In the end - this two-by-two is just a model. Being a such some important things fall outside, for the simplicity of expressing many important things concisely. To me, this model capture a lot of important attributes.

The way to judge a model is not whether it is "true, or not". It is whether it is useful in important circumstances. Personally  have used this model a lot of times when trying to figure out what is wrong in situations that does not work well. It has also deepened my understanding of agile and how it relies on humans, as individuals and their interactions, rather than setting and trusting formal structures.

I hope it will come useful to you.

Yours

   Dan


ps So, who will become a good product owner? Well, for example, I have met some really good project managers that I think would become excellent product owners.