Dear Junior - Letters to a Junior Programmer

Akka 1.2 Made it Simple to Get Started

2011-11-15T22:54:00.001+01:00

Dear Junior

The release of Akka 1.2 is about a month old, but it contains a change that is makes it much easer to try out Akka. It is not some revolutionary new feature, quite the contrary. They have simplified the package structure of Akka so much that the core Akka actor package now has no external dependencies. Correct: no, none, zip external dependencies - stands for itself.

Earlier, if you wanted to try out Akka, you had the problem that Akka relied on a ton of other open source libraries. So, if you just wanted to try out to write a simple actor, you still had to download a long list of other jars. Well, that can be handled by maven, or by the wonderful little tool "sbt" (Simple Build Tool), but still that is a threshold to get over before getting started. And that threshold has noting to do with "understanding actors".

In Akka 1.2 they have done a wonderful job of splitting out dependencies and managed to refine to "core actor" parts to be self-contained, i e it does not require any other libraries in place to work.

To get started writing your first Akka actor, simply download Akka 1.2 and put "akka-actor-1.2.jar" on your classpath. Ready to start hakking.

An Example

In InteillJ, I create a new project "akka12" in a library with the same name:


dajob05:akka12 danbj$ pwd
/Users/danbj/var/tmp/akka12
dajob05:akka12 danbj$ ls -l
total 8
-rw-r--r--  1 danbj  staff  796 15 Nov 15:55 akka12.iml
drwxr-xr-x  2 danbj  staff   68 15 Nov 15:55 src

Well, I happen to use IntelliJ, but any other modern IDE would work as well.

Now, I create a "lib" directory for third party libraries (i e Akka), download akka and put the jar in "lib"


mkdir lib
curl http://akka.io/downloads/akka-actors-1.2.zip >akka-1.2.zip
unzip akka-1.2.zip akka-actors-1.2/lib/akka/akka-actor-1.2.jar
mv akka-actors-1.2/lib/akka/akka-actor-1.2.jar lib/

Now the self-contained Akka jar is in the lib directory.


dajob05:akka12 danbj$ ls lib
akka-actor-1.2.jar

I add the jar as a project library in my IntelliJ project, and now I can create an actor class together with a small script that sends a message to that actor.


import akka.actor.Actor

object helloworld extends App {
  val worlder = Actor.actorOf[Worlder]
  worlder.start()
  worlder ! "hello"
}

class Worlder extends Actor {
  def receive = {
    case msg => println(msg + " world") 
  }
}

Running it (from inside the IDE) gives the expected


hello world

I just love what the Typesafe crowd have done to make it so easy to get started.

Happy hakking

Yours

Dan

Scala Actors and Estragon's Bad Memory

2011-10-13T09:26:00.001+02:00

Dear Junior

The event-based actor model in Scala is obviously so much more memory-efficient than the thread-based, so there must be a catch somewhere. There sure is, the price we pay is that the code does not always behave in the way we intuitively expect. And unfortunately the coding model does not shield us completely from those situations.

The reason I think this is important is that I want to walk the road towards Akka. Once we get there it will be obvious why Akka’s design give us both an efficient runtime model and a carefully guiding programming model.

Unintuitive Effects of Event-Based Model

Let us return to the programming model of event-based actors in Scala standard library, and the things that might baffle us.

One of the things that might happen is that inside one instance of actor, consecutive lines of code might be executed by different threads. That is not what we expect.

Let us pick apart an event-based actor to see what is going on, and the same with a thread-based actor – just for reference.

Dissecting Thread-Based Vladimir

To see how that can happen, let us start with a simple thread-based actor – our dear friend Vladimir. In a small script, we create an actor, start it and send it the message (Godot) that it is waiting for. OK, in the play Waiting for Godot, the point is that Godot never shows up, but let us be nice to poor Vladimir.

object vladimirwaitandrun extends Application {

def T() = "T" + Thread.currentThread().getId

println("Creating actor " + T)

val vlad = new Vladimir(42)

println("Starting actor " + T)

vlad.start()

println("Sending Godot message " + T)

vlad ! Godot

println("Finished script " + T())

}

The Godot-waiting actor Vladimir is implemented as a subclass to “actors.Actor” of the Scala standard library.

class Vladimir(id : Int) extends Actor {

def T = "T" + Thread.currentThread.getId

def name: String = "Vladimir" + id

def act() {

println(name + " is waiting " + T)

receive {

case Godot =>

println(name + " saw Godot arrive! " + T)

}

All set up – let us run the script.

danbj$ scala godot.vladimirwaitandrun

Creating actor T1

Starting actor T1

Vladimir42 is waiting T11

Sending Godot message T1

Finished script T1

Vladimir42 saw Godot arrive! T11

First we note that the script and the actor are run by different threads: T1 and T11 respectively – no surprise. We also note that both lines of Vladimir where run by the same thread. No surprise that either, the thread-based model gives each actor a thread and when the actor pauses (waiting for Godot), the thread pauses.

But let us walk through this just to see the difference when we come to the event-based model.

An actor of the class Vladimir is created and started – its “act” method start executing in a thread of its own (T11) starting with a print.
The actor executes the method “receive” (still in thread 11). The call to receive takes an argument, which is the so-called “message handler” – a code-block to be used when matching the incoming message. To be precise, the argument is an anonymously defined function, which is passed as the argument to “receive”.
As there is no message in the actor’s mailbox, “receive” puts the thread (T11) to sleep.
Message “Godot” is dropped into Vladimir’s mailbox (by T1). Waiting thread T11 is notified. Thereafter the script finishes.
T11 wakes up and starts applying the message-handler to the arrived message. There is a match in the (only) case-clause and T11 prints “Vladimir42 saw Godot arrive! T11”
Execution of message handler is finished. As there is no more code in “act” the thread returns.

OK, there are more subtleties going on, but this will do for now.

Dissecting Event-Based Estragon

Now let us look at the same thing happening with an event-based actor: our dear friend Estragon.

object estragonwaitandrun extends Application {

def T() = "T" + Thread.currentThread().getId

println("Creating actor " + T)

val estragon = new Estragon(42)

println("Starting actor " + T)

estragon.start()

println("Sending Godot message " + T)

estragon ! Godot

println("Finished script " + T)

}

The Godot-waiting actor Estragon is also implemented as a subclass to “actors.Actor”, and the only difference is that it uses another method for awaiting a message in the mailbox. Instead of “receive” it uses “react”.

class Estragon(id : Int) extends Actor {

def T = "T" + Thread.currentThread.getId

def name: String = "Estragon" + id

def act() {

println(name + " is waiting " + T)

react {

case Godot =>

println(name + " saw Godot arrive! " + T)

}

All set up. Let us run this.

danbj$ scala godot.estragonwaitandrun

Creating actor T1

Starting actor T1

Estragon42 is waiting T10

Sending Godot message T1

Finished script T1

Estragon42 saw Godot arrive! T13

Now, let us walk through this run and see what actually happened.

An actor of the class Estragon is created and started. A thread (T10) is taken from the thread pool and connected to the actor. The thread start running the “act” method, which starts with a print.
The actor executes the method “react” (still in T10). The call to react takes an argument, which is the so-called “message handler” – a code-block to be used when matching the incoming message. To be precise, the argument is an anonymously defined function, which is passed as the argument to “react”.
As there is no message in the actor’s mailbox, “react” disconnects the thread from the actor. The message-handler is registered at the event handler. The thread goes back to the thread pool.
Message “Godot” is dropped into Estragon’s mailbox from the script (by T1). The event-handler is notified. The script thereafter finishes.
The event-handler picks an available thread (T13) from the thread pool and connects it with the actor. The thread is given the message-handler to run and starts applying the message-handler to the arrived message. There is a match in the case-clause and T13 prints “Estragon42 saw Godot …”
Execution of message handler is finished. Thread is disconnected and goes back to pool.

Note that there are two different threads (T10 and T13) involved in executing the actor Estragon42. Having a pool of threads does not only mean that one thread can serve many actors (thus conserving resources). It does also mean that different threads might be involved in running the same actor during its lifespan.

To be honest, I must admit that I did run the script a few times before I got a run with different threads in “is waiting” and “saw Godot arrive”. The allocation of threads is non-deterministic from the point of view of the program, and the first few runs happened to reuse same thread in both phases – but that would not serve to make my point.

The Difference in Short

Let us focus on the core difference between these two examples; first thread-based Vladimir.

println(name + " is waiting " + T)

receive {

case Godot =>

println(name + " saw Godot arrive! " + T)

}

Here the same thread (T10, marked as green), execute the consecutive lines. It executes the println and the receive, makes a small pause, and finish with the matching case clause and the last println.

This is execution of code as we learned in Programming 101.

Now, time for event-based Estragon

println(name + " is waiting " + T)

react {

case Godot =>

println(name + " saw Godot arrive! " + T)

}

Here two threads are involved. First one (marked blue) execute println and react, ending with registering the message-handler. Then at a later point of time some other thread (marked purple) executes the case-clause and the last println.

The result is that consecutive lines of code inside the same actor object is executed by different threads.

Not what we learned in Programming 101.

Does it Matter?

OK, but does it matter? Unfortunately there are situations where this makes a difference for us as programmers. For example, many security frameworks take user credentials of the authenticated user and stuff it into thread locals. In that way the credentials need not to be passed around explicitly but can be fetched from the thread when needed. Alas, that does not work if the code “suddenly changes horses midrace”.

There are also other situations where the code behaves in unintuitive ways due to this “thread switching” and where the programming model does not shield us programmers from strange effects and risk of making errors. More on that later.

Estragon’s Bad Memory

On a side not the difference in execution model can also explain one aspect of the play Waiting for Godot. In the play, Estragon suffers from a severe memory condition. When the Act II starts on the morning of the second day, it seems like Estragon has no recollection at all from what happened in Act I the previous day. This is enormously frustrating for his companion-in-waiting Vladimir, who clearly remembers how they waited in vain for Godot to show up.

Now given the knowledge about threading it is obvious why Vladimir remembers the previous day. Thread-based Vladimir is in Act II connected to the same thread of execution as in Act I – so the events in Act I happened to “the same memory-line” as Act II. Event-based Estragon on the other hand is not necessarily connected to the same thread in Act II as he was in Act I, so Estragon in Act II is not “the same memory-line” as Estragon in Act I. In a way Estragon experience the same situations as a person with multiple-personality disorder. Even if it is the same Estragon-body (object, actor), it is not the same Estragon-mind (thread) from time to time.

That might be an explanation for Estragon’s bad memory. It would be interesting to diagnose it more closely, like if we could probe his mind for what is going on from time to time.

Yours

Dan

Public final and data encapsulation

2011-09-16T08:24:00.000+02:00

Dear Junior

We were discussingimmutable value objects and using "public final" data-fields fortheir representation. In that discussion I started off mixing up immutabilityand encapsulation. Now that we have covered immutability, let us return toencapsulation.

Obviously, declaring afield as public will break data encapsulation. You lose your freedom to changedata representation without having to bother the clients.

Let us have a look at aname-class with some actual usage.

public class Name {

public final String fullname;

public Name(Stringfullname) {

if(!fullname.matches("[a-zA-Z\\ ]+"))

throw new IllegalArgumentException();

this.fullname = fullname;

}

public String[] names() {

return fullname.split(" ");

}

The public API of thisclass consists of three parts: the construction of a name from a string, theattribute “fullname” (accessible through property data field), and theattribute “names (accessible through accessor method).

An example of whatclient code looks like we can find in the tests.

public class NameTest {

private final Stringdanbjson = "Dan Bergh Johnsson";

@Test

public voidshouldHaveFullNameAsAttribute() {

Assert.assertEquals(danbjson, new Name(danbjson).fullname);

}

@Test(expected =IllegalArgumentException.class)

public voidshouldNotAllowWeiredCharsInName() {

newName("#€%&/");

}

@Test

public voidshouldSplitIntoNamesAtSpaces() {

Assert.assertEquals(
new String[] {"Dan", "Bergh","Johnsson"},

new Name(danbjson).names());

}

Imaging that we want tochange the internal data representation to using a char-array instead. Doing sowill break all the clients as they rely on having that public data-field“fullname”. Thus, it is a bad design. Or?

I would argue that itis still a good design. Using modern tools it is easy to add the encapsulationwhen needed. Applying "Encapsulate Field" in e g IntelliJ yields.

public class Name {

private final Stringfullname;

public Name(Stringfullname) {

if(!fullname.matches("[a-zA-Z\\ ]+"))

throw new IllegalArgumentException();

this.fullname = fullname;

}

public String[] names() {

return fullname.split(" ");

}

public String fullname() {

return fullname;

}

And of course theclient code has been changed accordingly.

@Test

public voidshouldHaveFullNameAsAttribute() {

Assert.assertEquals(danbjson, new Name(danbjson).fullname());

}

Now, I could havechosen "getFullname" as the method name for the new method. However,I have always found that naming convention a little bit awkward, the propertyis “full name” and adding a boilerplate “get” does not add any value in myopinion. By the way, JavaBeans is just one naming convention in Java, thenaming convention for CORBA predates JavaBeans. In the CORBA convention if youhave a property “fullname” of type String, then the way to access it was tocall a method “String fullname()” and the way to change the property was tocall a method “void fullname(String)”.So the convention I use is not new to Java at all.

In some languages thereis no distinction in syntax between accessing a field and calling a no-argmethod. Had the code been written in Eiffel or Scala, the syntax would havebeen the same and there would be no change to the client code at all.

Now that we haveencapsulated the usage via “fullname()” we can change the internalrepresentation to a char array.

public class Name {

private final char[] chars;

public Name(String name) {

if(!name.matches("[a-zA-Z\\ ]+"))

throw newIllegalArgumentException();

this.chars = name.toCharArray();

}

public String[] names() {

return new String(chars).split(" ");

}

public String fullname() {

return new String(chars);

}

Just for reference, inScala the corresponding change would start with this "final public"representation – here represented by the keyword “val”.

class Name(valfullname: String)

{

if(!fullname.matches("[a-zA-Z\\]+")) throw new IllegalArgumentException();

def names = fullname.split(' ')

}

The change would takeus to the slightly more verbose char array representation. Here we have no“val” in external class declaration, but a private val-field hidden inside the class-blockinstead.

class Name(name:String)

{

if(!name.matches("[a-zA-Z\\]+")) throw new IllegalArgumentException();

private val chars = name.toCharArray

def fullname = chars.toString

def names = fullname.split(' ')

}

In conclusion: As thepublic final field is accessible by the clients, it is also a part of the APIfor Name: This breaks data encapsulation. If you want to change datarepresentation, you will need to create an accessor method to which you directthe client.

In Scala or Eiffel thisis a no-issue, as the new accessor could transparently have the same name asthe old datafield ("fullname") and be accessed with exactly the samesyntax ("name.fullname"). However, in Java you have to include anempty pair of parenthesis to call the accessor method - thus the client codemust be changed.

Now, I consider this asmall issue, as there is excellent refactoring support in modern IDEs that eliminatethe change to a fully automated four-click, one-minute exercise. Thus, there isno point in designing for that change up front.

So, even if we *do*break data encapsulation, I would say that it is not much of a problem.

Now, there are otherkinds of encapsulations that are more interesting than data encapsulation, butthat analysis will have to wait for some other letter.

Yours

Dan

Public final is also Immutable, Again

2011-09-12T16:26:00.000+02:00

Dear Junior

I must apologise as in my last letter I mixed two separate discussions into one: one about immutability, and one about encapsulations. What I was after was immutability, even though encapsulation is also interesting.

To make thinks clear, let us return to the Name class for representing value objects for a name e g "Dan Bergh Johnsson". To focus on immutability, let us simplify it.

public class Name {
    public final String fullname;

    public Name(String fullname) {
        this.fullname = fullname;
    }
}

Now, if I create an object of this class, then no client can mutate the state of that object. This is because

The datafield is final so the client cannot make the reference point to some other String
Strings are immutable, so the referred object cannot be changed

Just for reference. I mentioned that Scala has a very elegant way of defining "public immutable datafield properties". The corresponding Scala class would be a one-liner.

class Name(val fullname: String)

Elegant, isn't it? "Name is a class with the attribute value 'fullname' of type String". Scala promotes the use of immutable constructs by making it simple to declare them.

Now, as my friend Tommy Malmström pointed out to me, I should also make my class final. This is a very valid and interesting point, so let me dig into.

The problem in this case is not the objects we construct ourselves, but objects that are sent to us. We should rightfully assume that such objects are also immutable.However, someone could wittingly and deviously create a mutable subclass of Name.

Back in Java land such a subclass could for example overrride toString

package names;

class EvilName extends Name {
    public EvilName(String fullname) {
        super(fullname);
    }

    public String toString() {
        return "Voldemort";
    }

    public static void main(String[] args) {
        Name name = new EvilName("Harry Potter");
        System.out.println(name);
        System.out.println(name.fullname);
    }
}

danbj$ java names.EvilName
Voldemort
Harry Potter

You see how confusing it could be when toString is used to render the name object "Harry Potter".

So, forbidding such overrides by declaring the Name class as final is really a good idea.

On a side note, this is the reason why String is declared final. The String class is for example used to represent class and package information when loading code dynamically over the network - so imagine the consequences had it been possible to make a mutable phoney String.

Well, that was about immutability.

On the issue of encapsulation it can be argued whether "public final String fullname" breaks encapsulation or not - and that discussion also depends on what encapsulation we mean. Any way, that is a separate discussion.

Yours

Dan

"public final" is also Immutable

2011-09-09T16:35:00.000+02:00

Dear Junior

Immutable value objects are one of my favourite programming idioms. I really like how they aid and ease the burden of the rest of the code by taking care of small pieces of complexity. When they are based on the concepts of the domain they become yet another magnitude more valuable.

Implementation-wise they are most often a primitive type wrapped up in a protecting box. So it is pretty natural that a "name" is stored with a String containing the full name. Wrapped together in the box is probably as well some complexity, like the validation of the name and some interpretations of the data - in this case the logic to split a full name into its parts. At the end of the day, it is not so interesting to encapsulate the data as such - it is encapsulating the interpretation of the data that is crucial.In Java such a name class would look like this.

public class Name {
    public final String fullname;

    public Name(String fullname) {
        if(!fullname.matches("[a-zA-Z\\ ]+"))
            throw new IllegalArgumentException();
        this.fullname = fullname;
    }

    public String[] names() {
        return fullname.split(" ");
    }
}

Now, one thing worth noting is the datafield "fullname". It plays double roles both as data storage and as an attribute. Should we not have a private field and an accessor method instead?

Well, the integrity of the object is still guaranteed as

the datafield is final so the referred object cannot be exchanged
the referred object (String) is immutable so the referred object cannot be changed

So, yes, people can get to the field from the outside, but they cannot break anything.Of course there is the question that if you change the data representation, then you will break the clients.

However, that is no big deal. If the situation should arise, we can apply the refactoring "encapsulate field" to introduce a new method "String fullname()" and replace every access to the field with a call to that method instead. Checking the entire codebase for accesses to "fullname" might be a large task. But, guess what, using a modern IDE there will be a menu item in the "Refactoring" menu that will do exactly that - fully automated.

The alternative would be to have that code in from the start. However, I cannot see that there is a point in paying the overhead of more lines of code in the meantime.

Making a field public does not break encapsulation. The important encapsulation is the interpretation and constraints of the data that is found in the constructor validation and the logic of the method "names()".

By the way: in Scala you would not be able to see the difference between a public field and a method with the same name. Nice.

Yours
Dan

Scala Actors are Just Code - No Magic

2011-08-07T00:34:00.000+02:00

Dear Junior

First time I tried out actors in Scala I though: "OK, there is a fair amount of magic going on here". I have later realised that there is actually no magic at all involved, but I would like to share with you my misconception and how it cleared out.

I am talking about syntax like

vlad ! Godot

and like

receive {
    case Godot => ...
}

The good part is that once I had my misconceptions cleared out, it was much easier to understand some weird parts of the Scala standard actors, and the path to understanding the design of Akka became much clearer.

What I Mean with Magic

Let me for a brief moment clarify what I mean with "magic" in this context. A program consists of two parts - those I can build myself, and those where magic occurs. And when there is magic, there are often some magic words involved.

Take for example the thread model of Java (and the JVM) where objects can be "locked". Down in the runtime, each object has a lock associated with it and this lock can be "obtained" using the magic formula

synchronized (objectToLock) { … }

Here “synchronized” is a special construct to handle those locks. There is no way for me to create a lock with the same functionality through "ordinary programming". I have to wave my wand and utter that precise magic phrase - then I conjure the object-locks down in the runtime to do my bidding.

In language design these constructions are sometimes referred to as "special forms" because they often have special syntax.

Another example of special forms is the for construct in Scala

for(vlad <- vladimirs) { vlad.start() }

There is no way for me as a programmer to build a construct that behaves the same way. The special syntax is built into the language, and reserved for some special behaviour decided by the language designers. For me as a programmer – some magic occurs. I can understand its effect, but not reproduce how it works.

The actor syntax in Scala definitely looked like this kind of magic to me.

The Stage

Let me start with what confused me. To simplify the discussion slightly, there are two places where there seem to be "magic" involved about actors: on the outside, and on the inside.

Allow me to reuse my example of the actor "Vladimir" with inspiration from Waiting for Godot.

On the "outside" of the actors we have the way one actor sends a message to another actor. A really simplified example is a script that creates an actor and then sends a message to it.

object vladimirwaitandrun extends Application {
    val vlad = new Vladimir(42)
    vlad.start()
    vlad ! Godot
}

In this example we assume that the class "Vladimir" is defined as an actors, and that "Godot" is a simple message type (defined as a case class). Now what looks very "magic" to me is the line where the message is sent.

vlad ! Godot

This does not look like ordinary Scala syntax to me. It looks like one of those special forms with its associated magic going on.

We will later see that I was mistaken here.

On the "inside" we have how the actor reacts on messages that are passed to it. In Scala standard actors this happens inside the "act" method of a class that extends "actors.Actor"

class Vladimir (id : Int) extends actors.Actor {

    def name: String = {
        "Vladimir" + id
    }

    def act() = {
        println(name + " is waiting ")
        receive {
            case Godot =>
            println(name + " saw Godot arrive! ")
        }
    println(name + "'s wait is over ")
    }
}

In this code the "receive-block" matches out arrived messages and run the appropriate code for the message.

receive {
    case Godot =>
        println(name + " saw Godot arrive! ")
}

Well, this is the thread-based model, there is also the event-based model where "receive" is replaced with "react" and does something very similar. The difference is beside the point for this discussion, but this discussion is crucial to understand the difference and its subtleties.

To me, this receive-block looks like a "special form" that invokes some magic. If anything, it reminds me of the synchronized-block mentioned earlier.

We will later see that I was mistaken here as well.

To make the setting of the stage complete, let us run the script that creates an actor and sends a message to it, a message upon which the actor reacts.

danbj$ scala godot.vladimirwaitandrun
Vladimir42 is waiting
Vladimir42 saw Godot arrive! 
Vladimir42's wait is over

My Erlang Heritage

One way to explain my misconception is of course my preconceived notion based on my previous experience. When I first learned actor programming some fifteen years ago, I did so using Erlang.

In Erlang actors are called "processes" and they interact by dropping messages to each other. The receiving process reacts on the message and does something, very often to send other messages to other processes. The syntax for this message passing and handling is very tightly woven into the heart of the language.

The syntax for dropping a message to another process looks like

NetlistenProcess ! terminateSignal

and the syntax for receiving and reacting upon an incoming message looks like

receive
    onhook ->
        disconnect(),
        idle();
end

Looks familiar, does it not?

Now, in Erlang both the exclamation mark (!) and "receive" are special forms in the language - syntax with a special meaning that unleashes some magic going on deep down in the runtime.

Seems not to far-fetched to assume same thing was going on in Scala.

However, I was mistaken. In Scala actors are just plain old objects with perfectly normal method calls. It is not even special syntax involved.

No Magic on the Outside

Now let us pick apart the "special syntax" for passing messages to see that there is actually no magic at all involved. We take a fresh look at the message-passing script.

object vladimirwaitandrun extends Application {
    val vlad = new Vladimir(42)
    vlad.start()
    vlad ! Godot
}

We start with noting that Vladimir is in fact an object. It is an object that is instantiated from the class Vladimir and that object is referred to with the reference "vlad".

val vlad = new Vladimir(42)

Nothing magic here really.

What makes "vlad" an actor is that the class Vladimir is a subclass of Scala actors.Actor.

class Vladimir(id : Int) extends actors.Actor {

This means that all the methods that are defined in actors.Actor are inherited to Vladimir, and thus are available to call using the reference "vlad". And one of those methods are named "!" (pronounced “bang”).

As you are as curious as I am, we can even have a look at the definition of that method. We find it in ReplyReactor which is a trait that Actor makes use of.

override def !(msg: Any) {
    send(msg, …)
}

So, “bang” is just a method, which we call on the actor to receive the message. We now realise that the strange syntax is just one way to write an ordinary method call.

vlad ! Godot

To make it totally obvious that this is just a method call, we can use the alternative syntax – the one that looks a little bit more familiar to people used to the syntax of C++/Java/C#.

vlad.!(Godot)

Oh, yes, that means exactly the same thing. It compiles and runs the same way.

danbj$ scala godot.vladimirwaitandrun
Vladimir42 is waiting
Vladimir42 saw Godot arrive! 
Vladimir42's wait is over

Now, I do agree that the exclamation mark feels like a weird method name. Now, remember that in ReplyReactor everything method “bang” did was to delegate to method “send”.

override def !(msg: Any) {
    send(msg, …)
}

To make things more “object-readable” we can inline the body of “bang” and use “send” instead. Only difference is that “send” takes another argument, but for the purpose of this discussion that does not make any difference – we can send in null there in this case.

The “send” method in its turn is just an ordinary method. It takes the argument (the message) and stuffs it into the mailbox – which is just some kind of list. There the message will sit waiting for processing, but that is not anything that concerns the “send” method. In other words – it is a very ordinary method.

There is no need to dig into details, but there is no magic going on. Now we can change the script to use the alternative syntax and "send" as method.

object vladimirwaitandrun extends Application {
    val vlad = new Vladimir(42)
    vlad.start()
    vlad.send(Godot, null);
}

Now, this looks like plain use of an object, does it not? No magic, just a method call.

No Magic on the Inside Either

Now, on the inside of the actor we have the magic "act" method, which contains

receive {
    case Godot =>
        println(name + " saw Godot arrive! " + threadid)
}

Now, surely this must be some "special form" syntax with associated magic?

To start with, there is nothing magic with the method "act". It is just a choice of name for the code where the actor has its code for "this is what you do". In a way, it is just like the "run" method in the Java interface "Runnable" - just a way to point out what code to run in a separate thread when the actor is started.

When it comes to the recieve-block it turns out to be no magic there either. In fact, "receive" is just a utility-method in the framework class "actors.Actor". Remember that we created the actor by letting the class Vladimir subclass "actors.Actor".

class Vladimir(id : Int) extends actors.Actor {

Thus our class Vladimir contains all the methods that are in Actor. And one of those methods is "receive". In the code-block it looks like "receive" is a keyword, but it is just a call to the method "receive" defined in the super-class. We can make this a little bit clearer by writing that explicitly.

this.receive {
    case Godot =>
        println(name + " saw Godot arrive! ")
}

Now the code-block with the case clause seems to be hanging into thin air. But it does not. According to the syntax of Scala it is the argument for the method. We can make that clearer by putting the argument inside parentheses.

this.receive ({
    case Godot =>
        println(name + " saw Godot arrive! ")
})

Still looks a little bit weird?

Well, remember that in Scala a function (like "sqr") is also a value. And, as such, it can be passed as an argument to a method (like the list-method "map" for example). Now, the code-block we see is just a function defined "in-line". We can make things a little bit less convoluted by giving that function a name through a "val"-declaration.

val messagehandler: PartialFunction[Any, Unit] = {
    case Godot =>
        println(name + " saw Godot arrive! ")
}
this.receive (messagehandler)

I agree that the type of “messagehandler” becomes pretty weird – those partial functions are not always trivial to wrap your head around. But the important point is that the “messagehandler” is just a Scala value.

Let us revise the last line to have a look at what has happened to the “receive-block” that looked like a special form.

this.receive (messagehandler)

Viewed this way we see that "recieve" is just a method of the actor class. That method scans the mailbox for messages and does something when it finds a match. Now, to do this, the receive method must be told what to match and what to do about matches - an that is exactly what the (partial) method "messagehandler" does.

So the “special form” suspect seems to be just an ordinary method call. Just to ensure there is no magic, let us sketch if we could implement it ourselves.

Looping through the mailbox (a list) and searching for matches does not sound to complicated. Basically we just try to apply the message-handler and let it do its job. Of course there might be tricky corners, but there does not need to be any magic involved.

The only remaining magic is what "receive" does when there is no match. In that case it waits until there arrives a message, and then it tries to match the message-handler onto the newly arrived message.

Well, we can build even that part ourselves. If the mailbox is empty (or there are no matches) we can suspend the thread in a wait-state where we poll the mailbox at regular intervals. Even better, we can use the "notify" mechanism in the JVM where "receive" can call "mailbox.wait()" using the Java API. To wake up the thread we can let "send" (i e method "!") contain a "mailbox.notify()" so that the actor will resume its thread and rescan the mailbox.

And, basically, this is what the "thread-based actors" do. It is just someone else that has written the code for us.

No Magic at All

The syntax for sending messages ("!") and for receiving and reacting upon them "recieve {…}" looked like a lot of magic. However, it turned out that they where just ordinary methods defined on an ordinary class - the class "actors.Actor".

This is what is meant when people say that "Scala actors is not a language feature, it is a library". In other words: Scala actors are just ordinary code. The only thing that is special with that code is that it is pre-packaged together with the Scala download. However, it has no special status over any code written by you and me.

This design is actually very lucky for Scala. It has opened up for alternative actor frameworks to be written. The most prominent is Akka (champed by Jonas Bonér), and it has become so successful that the plan is to replace the standard actor library with Akka instead.

Of course, Akka is not magical either. It is just code which is very well designed and written. But in principle it is not different from code that could have been written by you and me. No magic involved.

Yours

Dan

How Heavy is Estragon - Event-Based Scala Actor

2011-05-27T13:20:00.000+02:00

Dear Junior

As we have seen thread-based actors are quite an intuitive model, but is not very efficient in conserving resources. Its main drawback is that each actor needs a thread each, and each thread will take some memory - about 60-70kB.

Also, this amount of threads seem unnecessary as the thread is only used a potion of the time - so the threads could be pooled instead.

So, let us create the Estragon version - an actor that drowses off and takes a nap whenever he is not actively needed. Thus the thread can be used by some other actor that wants to be active at the moment.

class Estragon(id : Int) extends Actor {
  def threadid = {
    "T" + Thread.currentThread.getId
  }

  def name: String = {
    "Estragon" + id
  }

  def act() = {
    println(name + " is waiting " + threadid)
    react {
      case Godot =>
        println(name + " saw Godot arrive! " + threadid)
    }
    println(name + "'s wait is over " + threadid)
  }
}

In the code the difference from thread based Vladimir is really small - we use the method "react" instead of "receive". The rest of the API is the same: we subclass from Actor, we define an "act"-method. The only difference to get event-based actors (using a thread pool) instead of thread-based actors (having one thread each) is to switch from "receive" to "react".

Simple? Yes, until we get into details and subtleties later.

Now let's run the script for the play putting Estragon and Vladimir on stage and starting them.

object waitingforgodot extends Application {
  println("Setting the stage " +
    Thread.currentThread.getId)
  val estragon = new Estragon(1)
  val vladimir = new Vladimir(2)
  println("Starting the play")
  estragon.start
  vladimir.start
  println("Main script is over")
}


danbj$ scala godot.waitingforgodot
Setting the stage 1
Starting the play
Main script is over
Estragon1 is waiting T10
Vladimir2 is waiting T11
^C

Well not very exiting. Both Vladimir and Estragon start and enters into waiting-state, waiting for Godot. Only difference is their respective ways of waiting.

Vladimir keeps his thread, just putting it into a waiting state. Deep down under this is implemented through some "object.wait()". So whenever he gets out of his waiting state (when the message Godot finally arrive), the same thread can process the message-handler i e the receive-block.

Estragon on the other hand discards his thread when going into "react". As he has no immediate use of the thread, it is given back to the pool to serve some other actor that need to run. So whenever Godot arrives to Estragon the message-handler will be activated and run by some thread, not necessarily the same as earlier.

Nevertheless, the point is that when Estragon enters "react", the thread is no longer occupied but be used by other actors. So even if we have a lot of actors we still can manage with just a few threads.

This becomes more obvious if we create lots of Estragons

object estragongalore extends Application {
  override def main(args: Array[String]) {
    val actors = Integer.parseInt(args(0));
    val ids = 0 until (actors)
    val estragons = ids map (id => new Estragon(id))
    println((actors) + " actors on stage")
    for(estr <- estragons) { estr.start() }
  }
}

scala godot.estragongalore 10
10 actors on stage
Estragon0 is waiting T10
Estragon1 is waiting T11
Estragon3 is waiting T12
Estragon2 is waiting T13
Estragon5 is waiting T10
Estragon7 is waiting T10
Estragon8 is waiting T10
Estragon9 is waiting T10
Estragon6 is waiting T11
Estragon4 is waiting T12
^C

Estragon 0, 1, 2, and 3 where started in separate threads. But, the next actor to start (Estragon5) could use thread T10 that had been used by Estragon0 and returned to the pool. So to create 10 actors we only needed four actor threads.That should conserve a lot of resources.Remembering that my poor laptop cringed when we put 2500 Vladimirs on stage? Let us see how many Estragon we can put on stage. It ought to be more as this model reuses the threads. What about 10 000 actors?

danbj$ scala godot.estragongalore 10000 
10000 actors on stage
Estragon3 is waiting T12
Estragon2 is waiting T13
Estragon0 is waiting T10
Estragon1 is waiting T11
Estragon4 is waiting T11
…
Estragon9998 is waiting T12
Estragon9935 is waiting T11
Estragon9989 is waiting T13
Estragon9956 is waiting T10
Estragon9999 is waiting T12
^C

10 000? No problem. Let us tenfold that.

danbj$ scala godot.estragongalore 100000 
100000 actors on stage
Estragon0 is waiting T10
Estragon3 is waiting T13
Estragon2 is waiting T12
…
Estragon99991 is waiting T10
Estragon99999 is waiting T11
Estragon99998 is waiting T12
Estragon99996 is waiting T13
^C

100 000 worked fine. What about a million actors on stage?

danbj$ scala godot.estragongalore 1000000 
1000000 actors on stage
Estragon1 is waiting T13
Estragon0 is waiting T10
Estragon3 is waiting T12
…
Estragon163350 is waiting T10
Estragon163351 is waiting T10
Estragon163352 is waiting T10
java.lang.OutOfMemoryError: Java heap space
 at scala.concurrent.forkjoin.LinkedTransferQueue.xfer(LinkedTransferQueue.java:187)
 at ...
 at scala.actors.Scheduler$.execute(Scheduler.scala:21)
 at scala.actors.Reactor$class.dostart(Reactor.scala:222)
 at ...
 at godot.Estragon.start(waitingforgodot.scala:42)
 at ...

At last we got an OutOfMemoryError. This time we did not get it when trying to start a new thread, but somewhere in the scheduler instead.It turns out that 900 000 actors is just short of what a 256M heap can handle.

danbj$ scala godot.estragongalore 900000
900000 actors on stage
Estragon0 is waiting T11
Estragon2 is waiting T10
…

danbj$ ps -m -O rss
  PID    RSS   TT  STAT      TIME COMMAND
18186 345524 s001  S+     0:37.07 /usr/bin/java -Xmx256M …

RSS is "real memory" in kB and we note again that the scala startup script restricts heap to 256M per default. Let us plot memory use for some different number of actors.

actors    RSS
     1    61516
  1000    62044
  2500    61460
 10000   71452
100000  101588
900000  345524

So memory consumption is roughly 60MB in startup and 3kB per actor. That is a lot better than 60kB per actor for thread-based.

That is pretty good. Now we can structure our systems using a lot of actors because the "overhead payload" of using an actor is not overwhelming. This is by the way an area where the framework Akka excels.

In short: switching from thread-based actors to event-based actors is not more complicated than changing from "receive" to "react". Not at a syntactic code level at least.

However, we also change execution model. We are no longer guaranteed that our actor is run by the same thread. In fact, the thread that registers the message-handler by running "react" might not be the same thread that later runs the message-handler itself. And that might give us a clue to Estragon's bad memory. But that will have to wait for another letter.

Yours

Dan

Vladimir Galore - Lots of Threaded Scala Actors Waiting for Godot

2011-05-18T10:14:00.054+02:00

Dear Junior

When we discussed the model for programming actors in Scala we saw that the thread-based model could be compared to the very actively waiting character Vladimir of Waiting for Godot. I also mentioned that the more drowsy, laid-back Estragon of the same play could be a good picture of the other model, the event-based.

However, before proceeding to the event-based actors, I think it enlightening to dive a bit more into the thread-based and see what the problem is.

The point of actor-based programming is that you want to mimic a "society of collaboration" where each actor performs a focused task. And you do not just want one actor per function (or type of task), you actually want one actor per task. So you want a lot of them.

This is similar to object orientation - you do not want one object per class (e g representing phone numbers), you want one object per instance of the class (one for each phone number). Actually, it can be claimed that the restricted actor-message model is closer to the original ideas of object orientation than the object-method model we have become accustomed to.

So we really want loads of actors. Let us see how many Vladimirs we can create before my poor laptop cringes.

Let us revise the code for Vladimir in his waiting for Godot. As we will create a lot of Vladimirs I have given each an id.

class Vladimir(id : Int) extends Actor {
  def threadid = {
    Thread.currentThread.getId
  }

  def name: String = {
    "Vladimir" + id
  }

  def act() = {
    println(name + " is waiting " + threadid)
    receive {
      case Godot =>
        println(name + " saw Godot arrive! " + threadid)
    }
    println(name + "'s wait is over " + threadid)
  }
}

case class Godot

Now we also need to create loads of them. Let us make a list of integers and turn each of them into an instance of Vladimir.

object vladimirgalore extends Application {
  override def main(args: Array[String]) {
    val actors = Integer.parseInt(args(0));
    val ids = 0 until actors // [0,1,2 ...]

// turn each int to an actor using the int as id
    val vladimirs = ids map (id => new Vladimir(id))
    println(actors + " actors on stage")
    for(vlad <- vladimirs) { vlad.start }
  }
}

scala godot.vladimirgalore 6
6 actors on stage
Vladimir2 is waiting 12
Vladimir0 is waiting 10
Vladimir1 is waiting 11
Vladimir3 is waiting 13
Vladimir4 is waiting 17
Vladimir5 is waiting 18
^C

Never mind the order of the output - actors are threads and are thus entitled to run scheduled in any order. What we see is six actors, each an instance of Vladimir, and each given a thread of its own. And every one of them are in a wait-state waiting for the message "Godot". Let us relieve one of them from its wait, just to see one completion. Let us send Vladimir4 the happy message of Godot's arrival.

object vladimirgalore extends Application {
  override def main(args: Array[String]) {
    val actors = Integer.parseInt(args(0));
    val ids = 0 until (actors)
    val vladimirs = ids map (id => new Vladimir(id))
    println(actors + " actors on stage")
    for(vlad <- vladimirs) { vlad.start }
    vladimirs(4) ! Godot
  }
}

danbj$ scala godot.vladimirgalore 6
6 actors on stage
Vladimir1 is waiting 11
Vladimir2 is waiting 12
Vladimir3 is waiting 13
Vladimir0 is waiting 10
Vladimir4 is waiting 17
Vladimir4 saw Godot arrive! 17
Vladimir4's wait is over 17
Vladimir5 is waiting 17
^C

Ahaa. Vladimir 4 was started with thread 17 - which used "receive" to register the message handler (the code block containing the "case"). It was also thread 17 that later executed the message handler, doing the pattern matching of the case and running the corresponding code for "case Godot". Finally it was thread 17 that continued the code after the handler-block. Same thread all the way - that is why they are called thread-based. They do not only behave as if they where a thread, they are actually implemented using the same thread all the time.

Accidentially, thread 17 managed to complete the act-method of Vladimir4 so that specific thread could be reused for Vladimir5. However, in all other cases, a new fresh thread was required.

Now let us put loads of actors on stage. For clarity we remove the release of Vladimir 4 so that every actor will be waiting and all threads be locked up.

Let us see if we can put 1000 Vladimirs on stage and set them acting.

object vladimirgalore extends Application {
  override def main(args: Array[String]) {
    val actors = Integer.parseInt(args(0));
    val ids = 0 until (actors)
    val vladimirs = ids map (id => new Vladimir(id))
    println(actors + " actors on stage")
    for(vlad <- vladimirs) { vlad.start }
    // vladimirs(4) ! Godot
  }
}

danbj$ scala godot.vladimirgalore 1000
1000 actors on stage
Vladimir0 is waiting 10
Vladimir3 is waiting 13
Vladimir2 is waiting 12
Vladimir1 is waiting 11
Vladimir4 is waiting 16
Vladimir5 is waiting 17
…
Vladimir115 is waiting 128
Vladimir116 is waiting 129
Vladimir117 is waiting 130
Vladimir118 is waiting 131
Vladimir119 is waiting 132
…
Vladimir251 is waiting 264
Vladimir252 is waiting 265
Vladimir253 is waiting 266
Vladimir254 is waiting 267
Vladimir255 is waiting 268
^C

So, the system hangs on Vladimir255 even though it has not yet started all the 1000 actors I asked for.

Hmm … "Vladimir0" to Vladimir255" - that is 256 actors that have been started before the system hangs. Such a number is hardly a coincidence … Here my colleague George Spalding came to the rescue by pointing out the relevant JVM properties, in this case "actors.maxPoolSize (default 256)". So, as I understand it Scala will not allow the JVM to allocate more than 256 threads for actor stuff. This means that when 256 Vladimirs had been started, then all 256 threads where sitting waiting for receiving Godot.

receive {
      case Godot =>
        println(name + " saw Godot arrive! " + threadid)
    }

And if the runtime-system refuse to allocate more threads, then no more actors will be started.

Let us run it again, with modified properties, increasing the maximum number of actor threads.

scala -Dactors.maxPoolSize=10000 godot.vladimirgalore 1000
1000 actors on stage
Vladimir0 is waiting 10
Vladimir3 is waiting 13
Vladimir2 is waiting 12
Vladimir1 is waiting 11
Vladimir4 is waiting 16
Vladimir5 is waiting 18
…
Vladimir997 is waiting 1010
Vladimir998 is waiting 1011
Vladimir999 is waiting 1012
^C

Ok now it worked… what about 2500?

danbj$ scala -Dactors.maxPoolSize=10000 godot.vladimirgalore 2500
2500 actors on stage
Vladimir0 is waiting 10
Vladimir3 is waiting 13
…
Vladimir2498 is waiting 2511
Vladimir2499 is waiting 2512
^C

Seems to work … and 5000?

danbj$ scala -Dactors.maxPoolSize=10000 godot.vladimirgalor 5000
...
Vladimir2538 is waiting 2552
Vladimir2539 is waiting 2553
godot.Vladimir@35a631cc: caught java.lang.OutOfMemoryError: unable to create new native thread
java.lang.OutOfMemoryError: unable to create new native thread
 at java.lang.Thread.start0(Native Method)
 at java.lang.Thread.start(Thread.java:658)
 at scala.concurrent.forkjoin.ForkJoinPool.createAndStartSpare(ForkJoinPool.java:1609)
 at ...
 at scala.actors.Scheduler$.managedBlock(Scheduler.scala:21)
 at scala.actors.Actor$class.receive(Actor.scala:512)
 at godot.Vladimir.receive(waitingforgodot.scala:38)
 at godot.Vladimir.act(waitingforgodot.scala:49)
 at ...
 at scala.actors.ReactorTask.run(ReactorTask.scala:36)
 at ...
 at scala.concurrent.forkjoin.ForkJoinWorkerThread.mainLoop(ForkJoinWorkerThread.java:340)
 at ...
Vladimir2540 is waiting 2553
^C

Nope it crashed.

But … systems always reveals interesting information when breaking down.

In this case it was the scala.actors.Scheduler that tried to start a new Thread in order to serve the Vladimir "receive" inside its act-method. And creating this "new native thread" was too much load for the JVM that crashed with OutOfMemoryError.

Let us run this once again, just below the limit where it crashes.

danbj$ scala -Dactors.maxPoolSize=10000 godot.vladimirgalor 2539

and have a look at process status

danbj$ ps -m -O rss
  PID    RSS   TT  STAT      TIME COMMAND
 8164 264280 s001  R+     0:06.80 /usr/bin/java -Xmx256M …

OK, "RSS" stands for "resident set state" and is basically "real memory". So, memory use is 264280 kB. Also note that the scala runtime has set the JVM "maximum heap size" (Xmx) to 256M. These two numbers are strikingly close to each other. My conclusion is that it seems to be the allocated heap that has filled up.

This does makes sense, a few thousand threads will need one allocated stack each, and that will eat the available space pretty quickly. Trying out with different number of actors give us some data on how much stack is needed per actor.

actors    RSS (in kB)
    1    62752
  500    95676
 1000   134132
 2500   239248

So, apart from that the system needs 62M just to start, every Vladimir (thread-based actor) seems to need just below 70k each. That is not very slim, not if we want to create truckloads of actors.

Obviously, the thread based actors have their advantage in being pretty easy to understand. But they definitely have their drawback in eating system resources by asking for one thread and a stack allocation each.

But, hey … in our benchmark all the threads where in wait-state. Could they not have been pooled in some way? We could have gotten away with just a few threads, and we would have been able to create many more actors!

Sure we could, and that is exactly what event-based actors do. Instead of being on their feet like Vladimir, we want them to drowse off and take a nap like Estragon. And in the mean-while the thread could be used by some other actor that has something in its inbox it wants to process. Unfortunately, they will also loose a little bit of their sense of time like him. But that will have to wait to another letter.

Yours

Dan

Scala Actor Waiting for Godot - Vladimir Thread Version

2011-05-16T10:20:00.005+02:00

Dear Junior

My collegue Roozbeh Maadani and I where curious about the different actor models of Scala so we decided to try them out. I have also had a distant interest in Akka for a while, an interest re-vigored by Jonas Bonér's presentation at Jfokus, and deepened by attending his mini-course on Akka at OP-KoKo. So, I have been toying around with Akka a little as well. I will get back to that.

Let us get started and create an application with some actors - in this case an reenactment of Waiting for Godot. The basic plot of Waiting for Godot is roughly two characters, Vladimir and Estragon, waiting for a person named Godot to show up - which he does not. So let us start with the setup.

object waitingforgodot extends Application {
  println("Setting the stage")
  val vladimir = new Vladimir
  val estragon = new Estragon
}
class Estragon extends Actor {
  // here we will later read Estragon's script
  def act() = null
}
class Vladimir extends Actor {
  def act() = null
}

In the application script we create two actor objects, one "Vladimir" and one "Estragon", defined by one class each. When started (which they are not yet), they will run their respective scripts which is defined in their act()-method. Agreeably neither of them have much of a script to follow yet.

Nevertheless, we can run the application, even though the result is not very thrilling:

danbj$ scala godot.waitingforgodot
Setting the stage

Time to give one of the actors some script to follow. Let us start with Vladimir.

class Vladimir extends Actor {
   def act() = {
      println("Vladimir is waiting")
      // ... stuff happen
      println("Vladimir's wait is over")
   }
}

Running the app still gives

danbj$ scala godot.waitingforgodot
Setting the stage

Obviously, the script for Vladimir is not executed. This is because in the Scala "actors.Actor" API the basic metaphor is that an actor is a thread (conceptually). It is not necessarily implemented that way, but the API keeps the mental model as close to thread as possible. Thus, we need to start it.

object waitingforgodot extends Application {
   println("Setting the stage")
   val vladimir = new Vladimir
   val estragon = new Estragon
   println("Starting the play")
   vladimir.start
   estragon.start
   println("Main script is over")
}

Now we see the actor Vladimir acting his part as well

danbj$ scala godot.waitingforgodot
Setting the stage
Starting the play
Main script is over
Vladimir is waiting
Vladimir's wait is over

Note that the Vladimir script is actually running even after the main script is over. This is explain by "vladimir.start" is actually starting a new thread which keeps running even after the original main-thread has finished.

This becomes even more obvious if we add some thread information to our printlns.

object waitingforgodot extends Application {
   println("Setting the stage " + Thread.currentThread.getId)
   val vladimir = new Vladimir
   val estragon = new Estragon
   ...
}

class Vladimir extends Actor {
   def act() = {
      println("Vladimir is waiting " + 
         Thread.currentThread.getId)
      println("Vladimir's wait is over " + 
         Thread.currentThread.getId)
   }
}

Which yields

danbj$ scala godot.waitingforgodot
Setting the stage 1
Starting the play
Main script is over
Vladimir is waiting 10
Vladimir's wait is over 10

Main script is run by thread 1 and character Vladimir is run by thread 10. Curiously this actually apply to the play itself. In the play Vladimir is the only one who have a clear sense of time: he remembers the past (which the other characters does not, at least not very well), and he has an idea of the future to come.

Now let's get over to the point of the play, which is waiting for a specific person named Godot. Vladimir's script is extended by a "wait for Godot to arrive". The actor representation of this is to send a message to the actor Vladimir, a message consisting of "Godot".

Vladimir on his part must wait for this message to arrive in his "inbox". He does so by using "receive" and a message handler. Using "recieve" makes the actor to check its mailbox and act upon the messages it finds. The "message handler" is simply a codeblock that tells what to do with messages.

If there is no message in the inbox when the actor gets to "recieve", then the actor blocks and waits for a message to arrive.

class Vladimir extends Actor {
   def threadid = {
      Thread.currentThread.getId
   }

   def act() = {
      println("Vladimir is waiting " + threadid)
      receive {
         case Godot =>
            println("Vladimir saw Godot arrive! " + threadid)
      }
      println("Vladimir's wait is over " + threadid)
   }
}

In this code "receive" causes the thread to be put in a waiting-state and watch the actor's "inbox". The "inbox" is the way to communicate with an actor. You should neither access its data, nor should you call methods on the object directly. Instead you send "messages" to it, which it will receive and react upon. I think of it like corresponding with mail.

So, whenever a message arrive, the actor will run its message handler to see what it should do about it. The message handler in this case is the code-block following "receive"

receive {
   case Godot =>
      println("Vladimir saw Godot arrive! " + threadid)
}

So, if there is a message that matches the value "Godot", then it will be handled by running the code

println("Vladimir saw Godot arrive! " + threadid)

Let us ponder this for a while. The actor support in Scala is a library, not a part of the system. So "receive" is not a reserved word in the language, it is the name of a method of the Actor class (which we subclass). So, what actually is happening is that we execute the receive-method and pass in a code-block as the argument, the message handler.

So another way to phrase it is that we use "receive" to register a message-handler with the actor runtime (i e the thread). As the message-handler is passed as a parameter it will sit on the top of the stack. And there the processing stops (the thread goes to sleep).

At a later point of time (when there is message in the inbox), the processing will be resumed (by the same thread). Now, the message-handler is still there on the top of the stack so we can run it, do the case-pattern-matching, select the "println(...)" and execute it. Thereafter, the thread pops out of the "recieve {...}" and continue with the rest of the program.

So, the entire "receive, wait for message, pattern-match, run message handler" was just a slow subroutine call.

Now over to the message.

We have one piece missing: who (or what) is Godot?

In this example Godot is a simple value. There is no need to make it an actor, because it does not play any active part in the program. Actually, in the play there is no character Godot on stage at any time - he simply never shows up. So casting the Godot part is easy - there is no actor needed.

Thus we can represent Godot with a value object defined by a case class

case class Godot

Yepp, that is all that is needed to define a message.

This defines a class Godot which contains exactly one object: Godot (a little bit like an enum would). As there is no mutable state, this object can safely be shared by anyone interested in Godot, in the same way as there only need to be one integer 4711 or one string "fubar".

The main advantage of case classes is that they can be used for pattern matching, which is necessary if we want to receive them as messages the way we did.

Now let us run this.

danbj$ scala godot.waitingforgodot
Setting the stage 1
Starting the play
Main script is over
Vladimir is waiting 10
^C

This time I had to kill the process as there was one thread that had not finished. Thread 10 was still active running the Vladimir actor and fully occupied waiting for Godot. I e it was hanging in "receive" but as it never got "Godot" it kept hanging there.

This is by the way also a parallel to the play. In the play Vladimir is very energetic, at least compared to Estragon who also waits for Godot. Vladimir is on his feet all the time during the play, whereas Estragon often sits down and even takes a nap. Obviously, Estragon's strategy conserves resources better, but let us not hold that against Vladimir right now.

In the play, Godot never show up, but let us be nice to poor Vladimir just for a moment and send him a Godot to his inbox.

object waitingforgodot extends Application {
   println("Setting the stage " + Thread.currentThread.getId)
   val vladimir = new Vladimir
   val estragon = new Estragon
   println("Starting the play")
   vladimir.start
   estragon.start
   println("Let Godot arrive")
   vladimir ! Godot
   println("Main script is over")
}

Here we use the "!" (pronounced "bang") syntax for putting a message into the inbox of the actor referred by "vladimir".

This will wake vladimir from its recieve-wait-state.

receive {
   case Godot =>
      println("Vladimir saw Godot arrive! " + threadid)
}

Now that actor is reactivated, it will pattern-match the "case" clauses (only one in this example) in the handler and run the corresponding code.

danbj$ scala godot.waitingforgodot
Setting the stage 1
Starting the play
Let Godot arrive
Vladimir is waiting 11
Main script is over
Vladimir saw Godot arrive! 11
Vladimir's wait is over 11

This time I did not need to kill the process as the Vladimir thread had a chance to finish its execution.

Note also that messages are sent asynchronously so there is no guarantee that message handling is done immediately after message is sent. On the contrary, things can happen in apparently random order. In this case, the Godot-message is sent before Vladimir had stated running, and the main script interleaves with the Vladimir script. Of course your code should not rely on any specific order of execution.

In this run some significant things happen from a message passing perspective

"Let Godot arrive" (by thread 1)- here a message is sent to (and put in) Vladimir's inbox
"Vladimir is waiting" (by thread 11) - Vladimir enters the "receive", pushes the message-handler on the stack, and goes into thread wait
"Vladimir saw Godot arrive!" (by thread 11) - the actor runtime (the same thread) realises that there is a message in the inbox; it wakes up and run through the pattern-matching; it finds that the value Godot matches the case Godot and run the corresponding code.

A drawback with this Vladimir design is that each actor needs to have its own thread. The thread will be in wait-state when there is no message processing going on, so no CPU will be wasted. However, the number of threads a system can handle is quite limited so dedicating threads to waiting actors is pretty wasteful. However, that is a letter in its own right.

In contrast to Vladimir who is on his feet the entire play we have the other character, Estragon. As I mentioned, Estragon is more laid back; he sits down and occasionally takes a nap. We will see that he is a pretty good picture of the other actor execution model, the event-based. But that will also need to be the subject of another letter. And, then we want to cover Akka as well ....

Nevertheless, the thread-based actor model that Vladimir represents is a pretty good starting point for understanding actors, message passing, and message-handling execution.

Yours

Dan

Getting Started with JUnit and TDD in Scala

2011-03-03T16:24:00.009+01:00

Dear Junior

As I like the flow of test driven development, it was natural to me to try TDD when starting using Scala. As Scala runs on the JVM and interoperates well with Java it should be easy to just use JUnit. However, it was not totally obvious how to do it, and looking around the net I found no "start from scratch, this is how to do it"-tutorial. Anyway, this is how I did it, starting from scratch.

A First Test - without JUnit

Let's start with writing a test for the now-classical Money-example. Using my IDE at hand (IntelliJ), I created a file 'MoneyTest.scala' and wrote the test checking that one swedish krona is one swedish krona.

class MoneyTest {

def should_equal_same_amount_same_currency = {

new Money("SEK", 1) == new Money("SEK", 1)

}

Let us pick this apart. This is a class (MoneyTest) that can be used to create instances of that class. Such an instance ("a MoneyTest") is in other words "an object that can test properties of Money". One of the properties it can test is that Money "should equal same amount same currency". Testing the property is done by running the method.

We have not yet got JUnit in the loop, but this is the anatomy of a unit test in place.

Get it to Compile

To get the test through the compiler we need a Money class as well, which will reside in Money.scala.

class Money(currency : String, amount : Int) {}

Yepp, that's it - class declaration and constructor rolled together.

Now we are in shape having our first unit test and enough implementation to satisfy the compiler. Time to run the test. If we want to we can run it on the interactive read-eval-print-loop (REPL).

scala> new MoneyTest().should_equal_same_amount_same_currency

res0: Boolean = false

Well, this is actually a failing unit test, giving us the permission to hack away.

Taking the Red Test to JUnit

However, I have come to be used to JUnit - I like the assert methods, I like how it integrates with my IDE, I like how it integrates with build servers. So, I would like to pimp up this pretty naked unit test to make use of JUnit.

First step was to add JUnit to the Scala project. In IntelliJ I created a new "External Library" named "junit" to which I added junit-4.4.jar - I happened to find it in ~/.m2/repository/junit/junit/4.4/junit-4.4.jar. Then I added the new external library to the project.

Next step is the code. As JUnit is now on the classpath, I can do the imports, annotations, and use the methods I am used to (albeit the import will be Scala-scented).

import org.junit.Test
import org.junit.Assert._
class MoneyTest {
  @Test
  def should_equal_same_amount_same_currency = {
   assertTrue(new Money("SEK", 1) == new Money("SEK", 1))
  }
}

Looks familiar, doesn't it? Now I can run it from within IntelliJ like any JUnit-test. Running it gives another familiar message:

java.lang.AssertionError:

at org.junit.Assert.fail(Assert.java:74)

at org.junit.Assert.assertTrue(Assert.java:37)

at org.junit.Assert.assertTrue(Assert.java:46)

at MoneyTest.should_equal_same_amount_same_currency(MoneyTest.scala:10)

...

Well, using assertTrue never gave helpful error messages. Let's do better.

Using JUnit

Let us change the assert and use assertEquals instead. This works fine as the mother-class in Scala (named AnyRef) is nothing but "java.lang.Object" so there will be an equals method around.

To be precise, there is actually a class above "AnyRef" named "Any" which include the immutable value-types in Scala that parallel the Java primitives. But that distinction does not matter here.

Changing the code to use assertEquals we get:

  @Test
  def should_equal_same_amount_same_currency = {
   assertEquals(new Money("SEK", 1), new Money("SEK", 1))
  }

This lands me with the error message:

java.lang.AssertionError: expected:<Money@348bdcd2> but was:<Money@4a4e79f1>
at ...
at MoneyTest.should_equal_same_amount_same_currency(MoneyTest.scala:10)

OK, two Money instances with different hashcodes are obviously not considered equal, even though their data are the same. This might be what we want for entities or domain events, but not for value objects such as Money.

However the error message looks kind of obscure due to the default toString. Let us add a custom toString to get readable messages in the future.

class Money(currency : String, amount : Int) {
override def toString() : String = currency + " " + amount
}

Running the tests again gives:

java.lang.AssertionError: expected: Money<SEK 1> but was: Money<SEK 1>

at ...

This is definitely more readable. Arguably it would be more confusing had we not remembered the last error message: they are different instances. So, the equality test that Scala gives us from the Scala top class AnyRef (which is the same as java.lang.Object on the JVM) does not cut it for value objects - exactly as in Java.

Fixing Equality

What to do? Of course we could get into Money and add an override def equals returning true. Then we could add a testcase to provoke a failing equals, forcing us to elaborate our equals method etc. However, in Scala there is a simpler solution at hand - declare Money as a "case class".

case class Money(currency : String, amount : Int) {
override def toString() : String = currency + " " + amount
}

Let us not get into the details, but apart from giving us well-behaving equals and hashcode, it actually makes sense for value objects to be case classes.

Green Test, Refactoring, and Ready to Start over again

The good part is that this change take us home to a green bar. There is not much refactoring to do, so now we have gone a full TDD cycle using Scala and JUnit. Time for next round.

Now we have gotten started

Yours

Dan

ps Once getting started, you might want to try out some actors in scala.

Conceptual Abstraction, and Technical

2011-02-02T21:22:00.000+01:00

Dear Junior

I think there is an important distinction between solving a technical problem and solving the conceptual problem. And this distinction has a profound effect on what the code will look like.

Let me take an example. Imagine that you have some class that keeps track of the score during a football game. It might look something like this.

class FootballGame() {

int home = 0;

int guest = 0;

void homeGoal() { home++; }

// void guestGoal()

// String score () , e g "4-3"

// etc

}

Now you want to add goal logging, that logs each goal to stdout. This might change homeGoal to

void homeGoal() {

home ++;

System.out.println("home goal");

}

Next reasonable change can be that you want the logging to be configurable so that you can log to a file instead of standard out.

This can be done from a technical perspective or a conceptual perspective. Let's have a look at each to see why I prefer doing it the conceptual route.

Technical abstraction

The ambition of technical abstraction is to make yourself independent of the particular implementation you have at the moment. The "how" is here "to log goals to standard out", the "what" is "to log goals somewhere". We want the solution less dependent on the "how".

We refactor homeGoal to use a configurable output stream instead of hard-coding. The code inside FootballGame might turn out something along

FootballGame(PrintStream out) {

this.out = out;

}

void homeGoal() {

home++;

out.println("home goal");

}

This is a perfectly acceptable solution from a technical perspective. It uses dependency injection, so we can easily configure it with stdout, a stream to a log file or sending the data remote through a socket. For testing we can use a PrintStream backed with a ByteArrayOutputStream or we can mock the PrintString when doing TDD. The unit test for driving that step might look like

@Test

public void shouldLogHomeGoal() {

ByteArrayOutputStream out = new ByteArrayOutputStream();

FootballGame game = new FootballGame(new PrintStream(out));

game.homeGoal();

Assert.assertEquals("home goal", out.toString());

}

(Arguably we also test some of the functionality of ByteArrayOutputStream - but as that is a standard API class I am pretty comfortable with that.)

Typically technical abstraction can be done by using the standard APIs given by the platform (OutputStream, String, List, File etc).

The weakness of this approach is that the code/solution/architecture gets riddled with technical constructs, and after a while it is hard to remember exactly what purpose that particular OutputStream was for in that particular place.

Conceptual abstraction

The ambition of the conceptual abstraction is also to make yourself independent of the particular implementation. However, we look at the situation a little bit different and thus set of in a slightly different directions. The "how" is here "to log goals to standard out", but the "what" becomes "to log goals".

We refactor homeGoal to use a GoalLogger.

FootballGame(GoalLogger goallogger) {

this.goallogger = goallogger;

}

void homeGoal() {

home ++;

goallogger.log("home goal");

}

interface GoalLogger {

void log(String msg);

}

This solution does also use dependency injection, it can also be configured to log to standard out, to a log file, or cross network. For TDD we will probably using mocking in the driving testcase - which is really easy as we have an explicit interface.

@Test

public void shouldLogHomeGoal() {

GameLogger gameLogger = Mockito.mock(GameLogger.class);

FootballGame game = new FootballGame(gameLogger);

game.homeGoal();

verify(gameLogger).log("home goal");

}

The difference is that we will need explicit classes. E g for stdout-logging we will use StdoutGoalLogger.

class StdoutGoalLogger implements GoalLogger {

void log(String s) { System.out.println(s); }

}

For other purposes we will use a FileGoalLogger or a NetworkGoalLogger.

Obviousness is the Difference

The main difference to me is that it is totally obvious what the intent of the GoalLogger is, as apart from the generic PrintStream in the example of technical abstraction.

Granted, there are more code in the conceptual abstraction than in the technical abstraction - but I would still prefer the former if it came to maintenance or further development.

But the real difference comes in the next step. We will probably notice that there are two calls that are repeated:

goallogger.log("home goal");

goallogger.log("guest goal");

Now that we have the GoalLogger abstraction we can let it evolve.

interface GoalLogger {

void homeGoal();

void guestGoal();

}

Note that we have also removed the now obsolete "void log(String msg)", and this is what makes this GoalLogger distinct from some or any general-purpose logger.

Had we done the technical abstraction of working with an PrintStream, it would not be as obvious what could be done next. What I have seen most of the times is some kind of util class emerging containing static "helper methods" like GoalUtil.logHomeGoal(PrintStream), helper methods that are sometimes used, sometimes not. This in turn leads to similar (or duplicated) code in several location and an increase in code line count.

In my experience the small increase in line count for the conceptual abstraction is far better than the "ripple" increase in line count that the technical abstraction cause over time.

The bottom line. Focusing on the conceptual problem at hand gives a better codebase in the long run than focusing on solving the technical problem does. It might give more code but it need not to take much more effort.

Yours

Dan

What is Snow?

2010-11-30T08:13:00.003+01:00

Dear Junior

On the morning after a night of snowfall, I took a morning walk to kindergarten with my two sons. The older of them (four years old, but soon five) asked:

"Why does snow glitter?"

Good question and I described how snow consist of snow flakes that are flat, and that it is the top-most layer of flakes that glitter - in the same way as the glitter palettes that you can glue onto paper. We discussed it a while, and he seemed to think that it made sense.

After some moments of silence came the next question:

"Why is snow soft and fluffy?"

Good question, and I described how the flakes fall on top of each other in loose layers. We discussed it a while, and he seemed to think that it made sense.

To no big surprise there came yet another question:

"Why are snow balls hard?"

Good question again, and I described how the loosely layered flakes are squeezed together and get stuck in each other - that is why the snowball does not fall apart. We discussed it a while, and he seemed to think that it made sense.

Now, is snow *really* snowflakes? Nahhh, snow is water that sticks together through covalent bindings that behaves in a certain way when the temperature drops so much that the kinetic energy of the molecules is so low that it make them bind to each other in certain geometric patterns.

But, you do not expect me to explain that to a four year old kid, I guess …

The snow flake metaphor is a model. It abstracts away a lot of really complex stuff to give us a way of explaining a limited set of phenomenons. Put another way, it answers some interesting questions.

As all models it has its limits. For example, had my son asked "How come that when snow melts, it become water, but when it freezes again it does not become snow, but ice?". Well, then the snow-flake model would not have helped much. It is just a model. It answers some questions in a way that makes sense, but not all questions.

However, the same goes for the "covalent binding" explanation. It is also just a model. It is not more or less "true" that the snowflake model. It is just capable of explaining another range of phenomena in a way that makes sense.

They are both useful models, and that is where they get their value. The interesting question is not whether the model is "true" or not, but whether the model is useful.

The "snow flake" model enables a father to give sensible answers to some questions a four year old boy might ask on the way to kindergarten on the morning after a night of soft snowfall.

From my point of view, that is a good model.

Yours

Dan

Fix the Problem, not the Bug

2010-11-23T07:56:00.001+01:00

Dear Junior

It is not always "fixing the bug" is the right thing to do. Well, it depends on what you mean with the word "bug" I guess. But, to often "fix the bug" means "make a local change to make the bad behaviour go away". Instead we should search for the cause of the trouble, the problem itself - and fix that.

Say you have an application where you handle train departures. And somewhere you are to show a result of some search - a result that comes packed as a ResultInfo object containing timestamp and matching departures.

Simple, Clear, Readable Code

So there might be a class for listing results looking something like:

/** Lists train departures from searches. */

public class DepartureLister {

private PrintWriter resultwriter;

/** Render search result */

void showlist(ResultInfo resultinfo) {

resultwriter.println("resulttime: " + resultinfo.timestamp);

for (Departure departure : resultinfo.matching) {

resultwriter.println(departure.trainNr);

}

resultwriter.println("matches: " + resultinfo.matching.size());

}

For simplicity, let us say that the classes ResultInfo and Departure are the most simple possible.

/** Search result with matching departures */

class ResultInfo {

String timestamp;

Collection matching;

}

/** Train departure identified by its train number */

class Departure {

String trainNr;

}

OK, pretty obvious what the code does as well as its purpose.

Finding a Bug

Now imagine that this code at some occasion barfs with

java.lang.NullPointerException

at client.DepartureLister.showlist(DepartureLister.java)

on the line

for (Departure departure : resultinfo.matching)

What is going on? From a technical standpoint it is pretty obvious that this code is wrong as it did not take into account that the list "resultinfo.matching" could be null.

Fixing the Bug

So fixing the bug we change the code by wrapping the for-loop with a null-guarding if.

if (resultinfo.matching != null) {

for (Departure departure : resultinfo.matching) {

resultwriter.println(departure.trainNr);

}

And we also need to do something about the result-size printout.

resultwriter.println("matches: " +

(resultinfo.matching != null ?

resultinfo.matching.size() : 0))

Well, it works, but I would not say that the code is "pretty obvious" any longer. Even if we refactor it to make it neater, there will still be the null handling hanging around messing up things. Code should not look like this. If it is easy to express "list the matching departures", then that message should be easy to read from the code. It was easy to read it, it is no longer.

Furthermore, any use of ResultInfo throughout the code will need these kinds of guards and switches. Apart from messing up the code in all those places, it is also a blatant violation of the DRY principle, stating that each "idea about the code" should only be expressed "once within the code".

Obviously "fix the bug" was not the right thing to do.

Find the Problem

Let us see what would happen if we try to "fix the problem" instead. To start with, let us formulate the trouble, but from a conceptual standpoint - what the code tries to achieve, not a technical - what the code does. The technical standpoint was "resultinfo.matching can be null". Rephrased from a purpose-perspective that would be "situations with no matching departures".

Now it becomes blatantly obvious that the natural counter-question is "Why on earth is that not represented as an empty list?". So we dig into the departure-matching code and find the search where you call a legacy backend system.

/** Searches for departure according to matching some criteria */

public class DepartureMatcher {

private Legacy legacy = new Legacy();

public ResultInfo searchMatching(String[] matchcriteria) {

ResultInfo resultInfo = new ResultInfo();

resultInfo.timestamp = new Date();

int[] trainnrs = legacy.find(matchcriteria);

if (trainnrs[0] == 0) {// special signal no match

resultInfo.matching = null;

return resultInfo;

}

resultInfo.matching = new ArrayList<Departure>();

for (int trainnr : trainnrs) {

resultInfo.matching.add(new Departure(trainnr));

}

return resultInfo;

}

Okey, here we can see the root of the problem - why "empty list" is handled as a special case and signaled with null. One can almost see how the if-clause in the middle have been "cranked in" into the middle of the searchMatching when some programmer found out about the non-intuitive behaviour of the backend system to represent "no match" with "[0]".

Fix the Problem

Now, fixing the underlying problem is pretty easy. We change the "no match" clause

if (trainnrs[0] == 0) {// special signal no match

resultInfo.matching = null;

return resultInfo;

}

and replace it with

if (trainnrs[0] == 0) {// special signal no match

resultInfo.matching = Collections.emptyList();

return resultInfo;

}

Now the two cases "empty list of matches" and "non-empty list of matches" behave analogous. This leaves room for some refactoring, replacing the messy method with:

public ResultInfo searchMatching(String[] matchcriteria) {

int[] legacyreturnlist = legacy.find(matchcriteria);

int[] trainnrs = legacyreturnlist[0] == 0 // no match

? new int[0]

: legacyreturnlist;

List matching = new ArrayList();

for (int trainnr : trainnrs) {

matching.add(new Departure(trainnr));

}

return new ResultInfo(new Date(), matching);

}

Here it is more clear what each part of the method is doing. Of course, to make this into clean code, some more refactorings are justified - especially extracting each functional part (of a few lines) to methods with names of their own. But, we can be proud over what we have done this far.

Here is time to take a small break and ponder what we have done. We have made a piece of the code behave a little bit clearer, yes. We have removed a weird special case (null list), yes. But we have also changed the API of a public method - and that should not be taken lightly.

Can we be sure that we have not broken anything else? Do we not risk introducing an unknown heap of new bugs? And if so, how can that be better than changing the code in one place (the client where we originally had the bug reported)?

I would argue, that even if we introduce new bugs, we have still fixed a problem. The key difference is that those bugs will be code locales that rely on a non-intuitive API - so those bugs will be easy to fix. In other words: when moving to a clearer way to formulate the code, you should not be too afraid for "causing bugs". As long as the change you make solves a problem and leaves the code in better shape - then subsequent bugs are easier to fix.

Fast Fail

Of course we should do what we can to aid in finding those bugs. In this case this can be done by changing the contract for the ResultInfo. Up until now it have been legal to have a null-list of matches. Now we want empty match to be represented by empty matching-list. So let us enforce this:

ResultInfo(Date timestamp, Collection matching) {

if (matching == null)

throw new NullPointerException(

"Matching list not allowed to be null " + timestamp);

this.timestamp = timestamp;

this.matching = matching;

}

Now there will probably be a pile of test-cases that will fail with NullPointerException and hopefully pointing out places in the code to change to adhere to the new and clearer API.

Sum it up

Trying to fix the bug where it surfaced just led to messy code in a lot of places. That was because the problem was not really there. Not a strange thing: effect is not cause, symptom is not illness - we all know it. The real problem was "empty list" represented by null.

Instead, finding the root of the trouble - the problem itself - we could solve the reason for the unclear code. Let "empty list" be represented by an empty list, from the source. Now, the client code could stay in its original lucent form, and we could also make some refactorings of the search code to make it easier to grasp.

On top we added some enforcement of this new policy - the ResultInfo simply refuses to accept the mission to carry "null-lists". Now all clients can rest assured they will always get a proper list of Departures.

Al in all, the code becomes clearer.

Exceptions might Apply

Of course there are points of time you do not want to do this, like a few days before a release. However, as a general rule I prefer to fix the underlying problem in the code, than to fix the bug where the problem manifest itself.

Yours

Dan

Presenting on Ubiquitous Language at Jfokus 2011

2010-11-18T07:39:00.001+01:00

Dear Junior

I have gotten the delighting message from the Jfokus committee that they want me to speak at the Jfokus 2011 conference in Stockholm in February. I had proposed a presentation on the Domain Driven Design concept of Ubiquitous Language which they have accepted.

It seems to me that the ubiquitous language is one of the hardest things to get right. When I look at my own experience it is a part where there are a lot of small distinction, distinctions that are subtle but make a huge difference.

Just take an example of how you can express the same thing in different ways that seem synonymous, but actually make a huge difference.

Compare

"An account has a phone number. The account can change phone numbers."

with

"An account has a phone number. The phone number can be changed."

The difference might be subtle, but in a precise language the distinction is important. In the first case, the account have still the same (conceptual) phone number, which has been edited. The account and the phone number are both entities. In the second case, the account have changed its phone number to another phone number. In code the two cases would be

1. account.number.changeto(numbersequence)

2. account.number = new PhoneNumber(numbersequence)

So, what difference does it makes?

Well, somewhere else we might require that a new account might have the same number as an old account. Fair enough. But what happens if the old account gets a new number? Does the new account change its number as well?

Have we taken the pain to actually create a strict language that is understood in common, then we have the answer in the language. If we do not have that strict language, chances are high that we arbitrarily select one of them that "feels right at the moment". However, next time a similar situation arises there is no guarantee that what we pick arbitrarily that time is logically consistent with the first pick. Over time we risk getting a system that behaves subtly, or obviously, inconsistent with itself.

Of course, this is just one example. In the presentation, what I want to cover are practical aspects of to make a shared language to become useful - and what it takes to become useful. E g it has to be both rich enough to express the complexity of the problem, but it must still be precise enough so we can use it as a base to build a system.

Other aspects I want to cover is how such a language can grow over time (not a Big Upfront Design), problems and traps when introducing a new term, and the challenges of "mixed language environment" such as when there are both Swedish and English around.

Of course you recognize several of these subjects as we have discussed these earlier. However, I hope to put them together in a concise and structured format - and want to roll in some new ideas. I am sure there will be plenty of reasons to return to these areas in subsequent letter.

Yours

Dan

4 Points of Story Points

2010-11-10T16:29:00.003+01:00

Dear Junior

For quite some time there have been a debate on "what story points really are". To some extent there have been insightful discussions about important things to consider when sizing stories. However, the "really are" part of the debate just leaves me tired.

To discuss the true nature of story points is pretty pointless: they are a construct, created by us, and can be given any arbitrary meaning. It does not matter. What matters is whether that construct is useful for a purpose. This is the core of pragmatism.

In other words, the relevant question is not "what story points are", but "what story points are useful for". Please excuse the pun, but rephrased "What is the point of story points?"

If I try to pry things apart I can distinguish four different situations where I have found story points useful. Basically they cover the questions "Is it small enough?", "What does it cost?", "Which should we do first?", and "When can I have it?".

Small Enough

One of the cardinal faults a team can do is to start working on a story that is way to big. If they do, they will surely fail to finish it within the sprint, and having a demo with nothing to show is pretty depressing. Having a lot of half-finished work also tie their hands of what they can do next sprint. No good.

A mature team might have developed a gut feeling of how big a bite they can take and still eat it. The stories considered too big are simply sent back to product owner for delimitation or splitting.

A less mature team might see that bites have different sizes, but have not yet the insight of how big a bite they can take without choking.

Here story points can come in useful. The team can size the stories as 13, 20, 3, 8 etc. But they need not to know their limits. Instead we can observe the velocity over a few sprints and see what "small enough" means. For example, if the team have a velocity of 18, I would advice to set the limit to 9 (half velocity). So, the stories sized 3 and 8 are small enough, and those of 13 and 20 will need some more pre-sprint work to make them manageable.

It can be handy to reserve the top of backlog for stories that are small enough, a "backlog shortlist" of ready-to-develop stories. Or, if you prefer the kanban style to have one stage "delimit and split" followed by a "ready-to-develop" queue before pulling them into active development.

Cost

Whether we like it or not, the question about money always come up. It can be hard to just look at a requested feature and say how much it would cost to develop it. In a lot of development efforts the dominating cost is the cost of labour. Either the work will tax the amount of available work by the employees, or there will be contractor bills to be paid.

Here story points can come in useful. Someone probably knows what the team costs per week, or can calculate it. If you have some historical velocity data you can make a rough estimate of what each point costs. So, if you know roughly the size of the story, you can calculate the cost.

Say that your team costs EUR 27 000 a week, you run two weeks sprint, and your sprint-velocity hoover around 18. Then each point costs around EUR 3 000. So, a story of size 40 will cost roughly EUR 120 000.

Well, not all of the time will be "pure development work". There will be meetings, phone calls, administration (filling out time reports) etc. But, that does not matter. Pulling through a story of size 40 will take roughly two sprints and during that time the team will cost that amount of money.

Of course, in practice you will rather want to give an interval than a precise number. The velocity might be within 17-19 (with 90% confidence) and a "40" story can be anything from 21 to 40. So the cost will rather be in the range EUR 60 000 - 130 000. Still, it will be a figure good enough for business to decide if it is remotely interesting to proceed or not.

Priorities

A misconception among business side "customers" is that they should set priorities on business value. Well, but "economics" is really about alternatives - the cost of a bar of chocolate is that you cannot get two lollipops. In the same way, for the product owner to make a wise balance between what different stakeholders want, she must be able to compare their cost.

Interesting enough, to make priorities, we do not need to know the absolute cost of each story. It is enough to know their relative cost.

Here story points can come in useful. If feature A is size 20, feature B is 8, and feature C is 13, then we know that we can swap out feature A from a release and switch in feature B and C instead. And in doing so we do not need to care about the details of how much money it is about - all we need is aid in choosing.

Planning

Business need to look ahead to synchronize different parts of their work. E g there are some benefits in having a marketing campaign at the same time as you release some new feature of your software. But, waiting for the software to be complete before ordering the marketing will obviously make you loose market-time.

Here story points come in useful. If you observe the velocity of the team over some time you can apply some not-too-advanced statistics to predict how much work will be completed at some future point of time. Of course, each such prediction will have a probability to fail, and the surer you want to be, the lower you must set the prediction.

For example, if you have observed the velocities 36, 28, 36, 38, 24, 35, 32, 35 and you have five sprints to go, you can calculate the average and predict the team will finish 165 points. However, that prediction is just a prediction - the real result will be as likely to be higher as it is to be lower. In other words, your prediction has a confidentiality of 50 % - or a 50 % risk of failing.

If you want to make a safer prediction, say taking just a 5% risk of failing, you can calculate an interval with 95% confidentiality. In this case it will be the interval 144-186. Now you can mark the backlog, colouring all stories up to 144 as green (very likely to be delivered), those from 144-186 as yellow (totally "depends-on") and those from 186 and up as red (very unlikely to be delivered).

In Summary

It is very hard to talk about the "true nature" of story points. They are abstractions that say something about development work. And, the question of "what they really are" is not a very interesting one. Working with story points is a model, and a model should not be evaluated on "how true" it is - but on how useful it is.

Story points might be useful for a team to decide whether a story is small enough to fit in a sprint, of if it should be pushed back to the product owner - if they cannot do it by gut feeling.

Story points might be useful in assessing the cost of developing a story - if it is questionable whether its value justify the cost.

Story points might be useful for setting priorities - if it is difficult balancing the stakeholder interests.

Story points might be useful for planning at release level - if the organisation need to synch the work of different departments or groups

Apart from these four points, story points might be useful in doing other useful things - if it helps the organisation to act more wisely.

It might well be that in your particular setting, none of the "ifs" apply - and in that case story points are not useful to you. And if they are not useful, they steal time and attention from other things that would serve you better - i e they are waste that should be discarded.

In the end "what story points really are" is not as interesting as "what is the point of story points".

Yours

Dan

ps Story points being helpful for planning is of course key to why release planning works.

pps If story points help in planning it is interesting to see what factors drive high story points. For planning, it is the amount of effort that differs, but that is to a large extent driven by the complexity.

ppps Unless you are really good at making statistical computation, it helps to have a spreadsheet to aid in the planning.

pppps One way to ensure team does not embark upon developing something "too big" is to make a backlog shortlist of the top part of the product backlog.

Good Product Owner Wrapup

2010-11-02T07:58:00.001+01:00

Dear Junior

I would like to wrap up our discussion about the good product owner and the characteristics thereof. Of course there are many important attributes, and there are even more ways to select words for them. However, I think a lot of them are caught by the two-by-two containing "engaged, involved, decisive, and empowered".

personal organisational

social: engaged involved

formal: decisive empowered

It does not tell everything about a good product owner. It is just a model, meant to capture some important points in a concise way. So, let's just have a walk around this picture.

Starting with engaged we have a product owner that cares, who thinks the product and the system is important. This product owner has an inner drive to do things for the benefit of the product. So, it is a personal attribute. Also it is not so much about mandate and formalities, it is more about interacting with people, so "engaged" is set in a social setting.

Moving over to right we stay in the social setting, but transfer to the organisational attribute of being involved (from Latin "in the turning"; today we would probably say:"in the loop"). We are still talking about "interacting with people", but now it is not the person's inner drive. Instead it is the organisation "letting" the PO being involved. Rephrased, the organisation thinks that being product owner is the most important thing on this persons job description - there is nowhere else this person rather should be.

Moving down we stay on organisational attributes, but switch to talk about formalities. Being empowered is all about having the organisational mandate to take decisions that will stand. Of course the product owner can change her mind, and re-decide something else later. However, that is the decision of the product owner herself - not something the rest of the organisation can force upon her. The PO is a little bit like the "local CEO" of the product.

Moving to the left we stay on formalities, but move back to personal attributes. Being decisive is to have the personal drive to actually make decisions when they are needed. Of cause even a decisive person can change her mind, but the good product owner will take decisions with respect to the team, not throwing them "of balance" by completely changing direction. Still it is being decisive in the moment needed that keeps progress going.

Moving up we stay on personal attributes, but cross the border of formalities back to social settings and we end up at engaged where we started.

Now, this model is nothing but a model, and thus it is limited in its power and limited in its reach. In other words: it misses a lot of important points. It is also a product of annotating specific words specific meanings, that are by no mean ubiquitous.

Other words

Steven Hale pointed out that the distinction between "engaged" and "involved" is unclear (to say the least). He has suggested "interested" and "active" as a more distinct pair for the social attributes. This also makes the acronym IDEA possible - which is neat. However, in defense of "engaged/involved" he also notes that there was a girl with whom he got involved, later engaged, and now they are married.

Important things missed

There are also important things this model leaves out. One of the things Joakim Holm pointed out is that fails to mention the "burning vision" the product owner should uphold. I could not agree more - that is one of the most of important things a product owner should establish, nurture, and communicate (I am keen to say "radiate"). And that dimension is not caught in this discussion on "characteristics".

In the end - this two-by-two is just a model. Being a such some important things fall outside, for the simplicity of expressing many important things concisely. To me, this model capture a lot of important attributes.

The way to judge a model is not whether it is "true, or not". It is whether it is useful in important circumstances. Personally have used this model a lot of times when trying to figure out what is wrong in situations that does not work well. It has also deepened my understanding of agile and how it relies on humans, as individuals and their interactions, rather than setting and trusting formal structures.

I hope it will come useful to you.

Yours

Dan

ps So, who will become a good product owner? Well, for example, I have met some really good project managers that I think would become excellent product owners.

Empowered Product Owner

2010-10-21T08:28:00.003+02:00

Dear Junior

Let us say that you have a product owner personally engaged, organisationally involved, and who is decisive enough to take decisions. For having a good product owner you only lack the fourth characteristics; you want the product owner to be empowered.

Engaged Involved

Decisive Empowered

Of course it does not help much having a product owner taking decisions if she does not have the authority to do so. The problem is that any decision the team has started working from can suddenly be revoked.

Sorry folks, I just talked to the Guy-Who-Decides-Stuff and you remember that delimitation we settled for yesterday …. well, to make a long story short, he thinks it is an over-simplification, so we cannot do that. I am really sorry if it messes up things for you …

Agile methods in general are built do handle change. But you want that change to come controlled — not on things in-sprint after you have spent considerable time and effort working on it. If the team gets contra-orders repeatedly during a sprint they start mistrusting the decisions and guess if they lose tempo!

This is really what I do not like about "Proxy Product Owner". Such a one can be engaged, involved, and decisive. However, they are not empowered - the "Real PO" is somewhere else and any decision might be overruled. The unfortunately result is often that the Proxy PO gets indecisive.

Proxy PO: Sound good, just let me check with Guy-Who-Decides-Stuff. I will know for sure by tomorrow.

Team: So, what do we do until tomorrow?

Point here is that the team loses focus and tempo.

To make things worse, the etymology of the word "proxy" unveils that "proxy without authority" is a misuse. The word is closely related to "procurator" which was the province manager in the Roman empire and who held administrative powers as agent of the emperor. No talking here about "have to check with the Emperor in Rome first".

So, what to do about it?

I told you about the example of the non-involved Head of Marketing. In that case, someone else did the day-to-day job. Taking care of the backlog etc was done by an assisting controller from economy department. And he had to constantly run around the building to chase down stakeholders to validate any decision he made. The poor bastard did not even have the title "Proxy PO".

The solution I advice is to empower the Proxy to become the Real Product Owner, and the Head of Marketing in this example will become one important stakeholder.

In some organisations this sounds strange: "Can you give authority to someone who is not the boss?" I would argue you can.

I think of the relationship between the product owner and the stakeholders a little bit like the relation between the CEO and the Board. The Board is not the "boss" of the CEO. All decisions the CEO makes, she makes based on her mandate. If the CEO makes consistently bad decisions, she will be replaced - but until that point of time she has her mandate. Of course the good CEO will carefully listen to what the Board thinks is important. But, the decisions will be her own.

In the same way, the stakeholders are not the boss of the product owner. All decisions the PO makes, she makes based on her mandate. If the PO makes consistently bad decisions, she will be replaced. Of course the good PO listens carefully to what the stakeholders think is important, and try to balance those interests. But, the decision will be her own.

In traditional organisations there is a glory around the word "owner" that tend to push the title "up the ladder". If it is a really important system, then it might make sense to have the Head of Marketing to be product owner. But then the system must also be important enough for that person to spend considerable time on its development. Otherwise, delegate both the task and the authority to someone who has the time.

This is actually nothing but an instance of the good advice on organisational structure: Strive for aligning task, competence, and mandate.

Sounds pretty obvious, but it is interesting to see how often these are not well-aligned in many organisations.

Being empowered is a characteristics in an organisational context, in the same way as "involved" is "allowed by the organisation to spend time with the team". However, it is not so much about the informal situations that "involved" is mostly about, it is more about when there are time to make decisions. In that respect it is more related to being decisive.

Engaged Involved

Decisive Empowered

If you are in an organisation that manage to hand you an empowered product owner, you have a good chance to keep things rolling.

Yours

Dan

Decisive Product Owner

2010-09-29T16:28:00.001+02:00

Dear Junior

In the list of characteristics of a good product owner, we have this far discussing those in informal situations - being engaged and involved. However, product ownership is also a lot about making decisions, and then I like the product owner to be decisive.

Engaged Involved

Decisive Empowered

System development is a lot about making decisions. Even when we write code (and perhaps especially then) we make hundreds and thousands of micro-decisions. Some of them have mainly technical implications (make this design one class or two), but some of them form enterprise logic (what fields should I use when computing this if-conditional).

Sometimes these decisions grow a little bit larger, and we want to ask someone about what we should do. For example we might be writing a batch-file reader and wonder how we should handle if someone sends in the same file twice. Perhaps we consider saving some kind of file ID and a content hash to detect that situation. This might take a day or so to implement so most probably we will bring that up during the daily standup.

That standup is when the product owner can step in and make a huge difference. Of course the PO needs to be there (involved) and needs to care (engaged), but if she is she might make a huge difference by saying:

"Hold that thought. To me that seems like a separate story (Avoid duplicate file read). It is good enough if you implement this without that feature. if same file is sent twice, and we read it twice - then so be it. I make it new story and bring it up with the stakeholders."

Viola, within one minute the product owner have saved the team one day of coding, just by being decisive.

Product owner need also be able to take decisions at point blanc, even when there might not be enough information.

Team member: "I wonder, what shall we do about social security numbers for people with protected identity"

Product owner: "I do not know whether we have a case for supporting protected SSNs - need to check that up. Ignore that feature for now, if necessary I will add that as a new feature later."

This is just a few minute conversation, yet it saves the team hours of work by not loosing tempo.

Being decisive rules out product owners that "just want to investigate a little bit further before saying anything definitive" - such behaviour leaves the team floundering on what to do, whether they can proceed on the assumption or if they should wait.

Being decisive is a characteristic in a personal context, as is engaged. However it is more about making decisions rather than informal situations - so in that respect it is more related to being empowered.

Engaged Involved

Decisive Empowered

Nevertheless, having a decisive product owner makes things so much easier to keep things rolling.

Yours

Dan

Involved Product Owner

2010-09-17T09:07:00.003+02:00

Dear Junior

Let us continue the discussion about four characteristics of a good product owner. Having an engaged product owner is very well, but it does not really help much if she is not involved.

Engaged Involved

Decisive Empowered

Being involved is basically spending time with the team. I want the product owner to show up at daily stand-ups, and to be available for questions and discussions. Without being involved, the engagement does not lead anywhere - like "that is a really interesting and important point; we really need to discuss this, but unfortunately I am extremely short of time - can we set up a meeting next week?". Few things stops up development more that not being able to put forward those important questions and have those discussions.

Being an involved product owner is much about discussions and other informal situations - much like being engaged also is. However, where "engaged" is an attribute of the person, I would say that "involved" is an attribute of the organisation: does the (rest of the) organisation let the product owner to be involved or does it expect her to be elsewhere?

Having the product owner being involved rules out product owners that have lots of other important stuff to do, and being product owner will have to take whatever time that "is left" (obviously seldom more that extremely little). Product ownership is nothing that can be handled "on the side, with left hand".

One striking example is the Head of Marketing, who was the product owner of a web sales system. However, that person had so much to do with campaigns and strategy planning he did not even have time to come to the bi-weekly demos.

It is not uncommon to hear this kind of situation "solved" by a Proxy Product Owner. The proxy will fulfill the "involved" part while the Real Product Owner is elsewhere doing more important stuff. I do not think this is a good solution.

The Proxy Product Owner does give the team someone to talk to and discuss with on a daily basis, the problem is that that person seldom hold authority to take decisions - and if they do those decisions often need to be ratified by the Real Product Owner. In the two-times-two-matrix, it is rather a problem about being empowered - so let me return to the Proxy Product Owner in a later letter.

Engaged Involved

Decisive Empowered

Being involved is an attribute that applies to informal situations, like being engaged. In contrast to "engaged" it is however not so much about the person per se, but rather about the surrounding organisation. In that respect "invloved" is more akin to "empowered", with the distinction that the latter is more about formal decisions.

Still, having the product owner involved in the daily work is crucial for success.

Yours

Dan

Engaged Product Owner

2010-09-15T21:31:00.004+02:00

Dear Junior

One of the things I appreciate with people I work with are that they are engaged, and a good product owner is no exception to this. Thus, in the list of four characteristics of a good product owner, I would like to start with that one.

Engaged Involved

Decisive Empowered

With "engaged" I basically mean that the product owner should care about the product that we are working on together. This manifests itself as the product owner wanting to participate.

For example, there might be a problem with some story that is under development - border cases might be unclear or we have unveiled an alternative flows that we did not foresee. In that case, the product owner should consider it important to discuss those to help the development move forward without stopping. Or, if there is a business concept a developer do not understand, the product owner should want to explain it.

Further than that, the product owner is not just reactive. The engagement is also proactive. As an example, there might be a story on the backlog that the development team has deemed too big or unclear to start developing. The product owner should then consider that important enough to work with stakeholders and developers to re-phrase that story in a way that makes it more feasible for development, e g by delimiting it or splitting it.

Being engaged rules out product owners that became product owner "because someone had to be" or - even worse - where ordered to take that position, perhaps against their wish. Such a person will have no particular interest in the system, how it is used, or how it should evolve.

So being engaged is mostly about informal situations - like being involved - not about formal decisions. And it is also an attribute in a personal context - like being decisive - not about what the surrounding organisation lets you.

Having an engaged product owner is crucial for sustained success in development.

Yours

Dan

Four Characteristics of a Good Product Owner

2010-09-13T12:41:00.014+02:00

Dear Junior

A good product owner is hard to find — but it is well worth the effort. Once a development team start becoming agile it does not take long before the quality of the product-ownership becomes one of the most significant factors for the productivity of the team. A good product owner works with the rest of the team to gain and uphold a high velocity. A bad product owner can slow the team down to close to a grinding halt.

I have seen several organisations that have attempted to establish a product owner role, where some attempts have succeeded and others have failed miserably. I have tried to find some way to characterise the product owners of these organisations where I have seen success. Finally I have boiled it down to four characteristics that are to some degree linked to each other.

The product owner I have come to appreciate can be summed up by being engaged, involved, decisive, and empowered.

Engaged Involved

Decisive Empowered

Of course, each word here have a multitude of interpretations so some explanation is needed to clarify what I mean. I will elaborate each of them in subsequent letters, but right now I just want to try to convey the larger picture to you.

Context - Person and Organisation

One thing I find interesting with this list of four characteristics is that it contains attributes of the person being product owner as well as the organisation surrounding that person. On one hand side, being engaged and decisive are personal attributes — they come from the "inside".

Engaged Involved

Decisive Empowered

On the other hand being involved and empowered are attributes given by the surrounding organisation, it can let you be involved or being empowered, or it can stop you from doing so — they come from the organisation environment.

Engaged Involved

Decisive Empowered

So, the verticals in the matrix is about the context — personal or organisational.

Situation - Informal or Formal

In a similar manner the attributes describe how the product owner acts in different kinds of situations. Being engaged and involved are about informal situations — day to day discussions, coming to daily stand-ups , etc.

Engaged Involved

Decisive Empowered

On the other hand we have decisive and empowered. These are attributes that talk about formal situations — when there is a decision to make. The decisions are small as well as large. A small decision can be to scope a story during sprint, a large decision to say: "let's release". Disregarding small or large, those decisions are important.

Engaged Involved

Decisive Empowered

So, the horizontals in the matrix is about situation — informal or formal.

I definitely want to dive into each of these characteristics, but that will wait for another day.

To sum up this observation: I think this cross-section of personal-vs-organisational context and informal-vs-formal situations neatly catches most of the really important parts of what a good product owner is like, and what behaviour it leads to.

Yours

Dan

Multicore, Power, and Performance

2010-09-02T22:09:00.000+02:00

Dear Junior

Next time your data centre upgrade, you will of course get better performance out of your deployed systems. However, as there are several kinds of performance it can be interesting to think about what will happen to response time and capacity. Chances are capacity will improve, but response time might decline. The reason is spelled multicore.

In ancient days of yore, long gone, processors improved by running on higher clock frequency. Higher frequency made them able to process more stuff per unit of time. New processor meant higher frequency and thus better response time. If high capacity was needed, it was achieved by the processor thread-switching between several requests serving them all within reasonable response time.

Those days are gone. Nowadays high-speed processors are no longer in demand. The reason for this is power consumption. The power-consumption of a processor is roughly proportional to the frequency, squared. So for thrice the speed, you pay nine times the power.

Slower Saves Power

There are of course several problems with high power consumption such as electricity bills and environmental issues. However, the driving force is cooling. Every single Joule that is fed into a processor in a data hall is transformed to heat. That heat needs to be transported out of the hall by the cooling system - and that is the limiting factor.

In other words — if you populate your data centre with high-speed processors you will hit cooling capacity at one point. However, if you use low-speed processors (frequency third of the high-speed), you can stuff in nine times more in the same data centre. So, nine times more, each giving a third of the MIPS compared to high-speed processor. Still you get three times more MIPS out of the same hall - good economy.

Of course, the modern way of doing it is not to build several processors, but to stuff multiple cores onto the same chip — i e multicore. Still, the driver is the same - you get more MIPS out of you data centre by lots of low-frequency cores that out of substantially fewer high-frequency cores.

This is not just theory. Several of the large web-sites of the world use new keys like MIPS/W or MIPS/m3 when benchmarking their data centres.

Actually, the same goes for laptops where heat must be limited to prevent user burns. There two slower cores on the same chip can give higher computational power and still radiate a lot less heat.

Better Capacity and Worse Response Time

Getting back to response time and capacity — how are these affected? Well, the data centre upgrade might consists of pulling out old single-core processors and replace them with quad-core processors with 20% lower frequency. Power consumption goes down, making it possible to stuff more processor into the same space.

Looking at the data centre from the perspective of all clients we serve, or all transactions we process — we get a dramatic increase in capacity. However, from the perspective of a single request or transaction we experience a slower processor — so response time or latency will get worse.

Where processors and data centres earlier where sports cars getting faster and faster, they now rather resembles school busses that do not run nearly as fast but can carry a lot of people at the same time.

Of course this will affect the non-functional attributes of our systems running on these data centres. Unfortunately, the effect is not that straight-forward, but I am pretty sure we are slowly running into trouble and need to do something about it.

Yours

Dan

ps This is one of the reasons it is important to keep apart response time and capacity, instead of stuffing them both under "performance".

Two Types of Performance

2010-08-27T09:00:00.041+02:00

Dear Junior

In architecture one of the most important tasks is to keep an eye on the non-functional (or quality) attributes of the system. Often this importance is enhanced by stakeholders holding up some non-functional requirement (NFR) saying "the system must really fulfill this". Unfortunately, these NFRs are often quite sloppily formulated and a key example is "performance". I have stopped counted the times I have heard "we must have high performance".

I think a baseline requirement for any NFR is that it should be specific. There should be no doubt about what quality we talk about. In my experience "performance" can mean at least two very different things. Instead of accepting requirements on performance I rather try to reformulate the NFR to use some other wording instead. I have found that the "performance" asked for often can be split into two different qualities: latency and throughput.

Latency or Response Time

With latency or response time I mean the time it takes for some job to pass through the system. A really simple case is the loading of a web page, where the job is to take a request and deliver a response. So we can get out our stop-watch and measure the time it takes from the moment we click "search" until the result-page shows up. This latency is probably in the range of 100 ms - 10 s. Of course, this response-time is crucial to keep the user happy.

But latency can also be an important property even without human interaction. In the context of cron-started batch jobs it might be the time from the input-file is read until the processing has committed to the database. The latency for this processing might have to be short enough so the result does not miss next batch downstream. E g it might be crucial that the salary-calculation is finished before the payment-batch is sent to the bank on salary day.

In a less batch-oriented scenario the system might process data asynchronously, pulling it from one queue, processing it, and pushing it onto another queue. Then the latency will be the time it takes from data being pulled in until the corresponding data is pushed out at the other end.

All in all, the latency is seen from the perspective of one single request or transaction. Latency talks about how fast the system is from one traveller's point of view. Latency is about "fast" in the same way as an F1-car is fast, but will not carry a lot of load.

Throughput or Capacity

On the other hand, throughput or capacity talks about how much work the system can process. For example, a news information portal might have to handle a thousand simultaneous requests — because at nine o'clock coffee break a few thousand people might simultaneous surf to that site to check out the news.

Throughput is also important in the non-interactive scenario. Each salary-calculation might only take a few seconds, but how many will the system be able to process during those 10000 s between midnight (when it starts) and 02:45 when the bank-batch leaves. If we cannot process all 50 000 employees, some will complain. To meet the goal we need a throughput of five transactions per second.

In other words, where latency was the performance from the perspective of one client or transaction, then throughput is the performance seen from the perspective of the collective of all clients or transactions, how much load the system can carry. Here "load" in the same way as a bus will take a lot of load transporting lots of people at once, even if it is not fast.

Fastness vs Load Capacity

Both F1 cars and heavy-duty trucks are no doubt "high-performing" cars. But they are so in completely different ways. To have a F1 car showing up at the coal mine would be a misunderstanding that only could be matched by the truck at the race track.

So, I avoid talking about "performance" and risk misunderstanding. Instead I try to use "latency" and "response time" to talk about how fast things happen — while thinking about an F1 car. And I use "throughput" and "capacity" to talk how much load the system can handle — while thinking about a bus full of people.

What is the latency for transporting yourself between Stockholm and Gothenburg using a F1 car or public-transport bus? What is the throughput of transporting a few thousand people from Stockholm to Gothenburg using an F1 car or public-transport bus?

Yours

Dan

P s Now when we are moving to multicore, we will see increasing capacity but latency leveling out or getting worse. This in itself will be a reason to move to non-traditional architectures, where I think the event driven architectures (EDA) is a good candidate for saving the day. My presentation at the upcoming JavaZone will mainly revolve around this issue.

Agile is Different

2010-08-23T11:30:00.001+02:00

Dear Junior

From time to time I hear, see, or read how Agile is explained as "it is no news" or "it is the same old good project management, with some twists". My impression is that this is done as an attempt to make Agile less scary to make it easier to "sell" to management. I think this is a serious mistake. I think that in subtle but fundamental and important ways - at its heart and roots - Agile is different.

The difference is subtle because you cannot observe it directly. Any and all of the things you can see in an Agile-honouring organisation can easily be copied. There are software delivered in short and regular intervals, there are team retrospectives, and daily team-meeting. And all of these practises can be used in a traditionally managed organisation as well, but it does not make that organisation Agile.

To me the fundamental difference is in how you look at humans.

Traditional management use humans as building parts to build a software-producing machine, or a factory. In this machine people are the moving parts and they are strung together by a processes that dictated their interactions. At the end of the process, software emerges. It is very mechanical. For example, in this world it would be very strange if people would arbitrarily start changing the process - then the designed process might just break and who knows what would happen. So that cannot be allowed.

I might be guilty of over-exaggerating, but the practice of constantly referring to people as "resources" to me unveils an outlook on people that I find scary.

The view of humans in Agile is different. In Agile we acknowledge that it is the engagement and skills of people that make things happen. We make it a first-order concern that people should feel motivated and proud. And instead of a mechanical world view, we rely on a more organic view of organisations. If people want to change the process they are not only allowed, they are encouraged to do that — even if we do not know the precise effect it will have on the overall system.

To explain this, I think it is easiest to look at the Agile Manifesto. The first of its values is:

Individuals and interactions over processes and tools

This is not a small thing. Here lays a fundamental difference in how we look at people and organisations. A traditional process is defined by a single person at a single point of time. However, the wisdom and insight of that person at that time is nothing - absolutely nothing - compared to what can be achieved by having each involved person thinking and discussing with their peers - and doing so continuously. And if given the choice between an ever-so-well defined process on one side and trusting the wisdom of the crowd on the other hand, we chose the crowd any day of the week.

It is a little bit like democracy. We could trust a wise and benign emperor. However, we think we get a better result if all citizens engage in an open discussion. We create and change our laws according to that discussion - even if we do not know the result in advance.

In this perspective Agile is a celebration of the wonderful and mysterious system that emerge from initiatives that rise out of interaction between people that care.

This is also what can be seen in the fifth principle of the Agile Manifesto:

Build projects around motivated individuals.

Give them the environment and support they need,

and trust them to get the job done.

We actually trust that people want to work. We acknowledge that we need to give them the proper environment and tools, but that will be enough. There is no need to command and control. Things will just happen. It is a leap-of-faith to let control go. Organisations with traditional management dare not take that leap. Agile does.

So, when looking at a traditional organisation or project and comparing that one with one honouring Agile, you might not see much of a difference. But if you are inside of it, you feel the difference. It is there - at the heart. You feel trusted and empowered. And if you look closely you might see it as well - the small smile on peoples faces.

The difference is subtle. But it is fundamental. And it is important.

I am convinced that at its roots and heart - Agile is different.

Yours

Dan

Two Observations on Overtime in Traditional Projects

2010-08-20T08:15:00.006+02:00

Dear Junior
I have seen quite a few classically managed projects, and through friends and collegues I have been in contact with even more of them. I would just like to make two observation on the practice of working
overtime in different phases of such projects.

* It is very uncommon, actually I have neither experienced nor heard of, that project management orders overtime for requirements analysists so that they will deliver the full and finished requirement documentation on a specified date.

* It is very common, actually I have experienced it several times myself and heard of it numerous times, that project management orders overtime for programmers so that they will deliver the full and finished implementation on a specified date.

From these two observations it is probably possible to deduct lots of interesting conclusions. I will refrain from doing so here and leave it up to you as an interesting mind-game.

Yours

Dan

DDS Value Object Presentation Video

2010-08-18T08:01:00.002+02:00

Dear Junior

At the OWASP AppSec conference I had a presentation on value objects and Domain Driven Security. It turned out into a 35 minute code kata with refactorings, where I used value objects "DDD style" in two ways.

Firstly, I used them to create a design that makes indata validation come natural in the context of Injection Flaws. I used the classical SQL Injection login-attack as an example and applied "hard type" value objects in the same way as we have discussed before.

Secondly, I addressed a Cross-Site Scripting (XSS) scenario. Here I noted that even if XSS is often perceived as a problem with "bad indata", there are cases when those data are perfectly valid. So, instead I choose to look at it from an output-encoding perspective - not "bad indata validation", but "bad outdata encoding".

To be able to do something about the outdata encoding, you can think about the client side browser as a subsystem of your system. Seen that way, the presentation tier plays the role of the API to that subsystem. Then it becomes obvious that we should enforce proper encoding at the border of that subsystem - i e in the API we use for calling it.

Modifying the presentation tier API from "soft type" strings to "hard type" value objects made it obvious where proper encoding should go - thus solving the XSS problem.

The full video coverage from the conference has been released, and my presentation can be found at the OWASP website.

Yours

Dan

Blitz Retrospective

2010-08-12T09:02:00.001+02:00

Dear Junior

One of my favourite method tools is a retrospective format I call the "Blitz Retrospective". I have used it in a lot of situation where it has been impossible to do "large" retrospectives. For example, in some situations, the teams have been very "retrospective sceptic". In other, there has been a formally appointed "scrum master" who was not very open to suggestions, so suggesting changes to his retrospectives would not achieve much. In yet other, the sprints have been so long I have not had the patience to wait a few weeks. In all these situations it has been possible to suggest, and do, a "really quick retrospective".

The "Blitz Retrospective" only takes 25 minutes, and its focus is to find *some* change the team think would improve things. The format consists of three parts: the startup, collection of ideas, and vote.

Startup

If it is the first time, the startup is spent explaining the format so everybody feels comfortable with what will happen. I clarify that the purpose is to find one single idea of improvement that we will try out next week.

A question that always come up on purpose is why we are limited to one idea, and what happens with the rest. Well, if the timespan is only a week, it would be foolish to focus on more that one thing - it would just result in none of them being changed.

As for the rest of the ideas that come up - they will be there in the back of peoples minds and might change things implicitly - but they will not be actively in focus. If it is not the first time, I usually quickly rehearse the procedure, but I spend the time on evaluating the last retrospective's "winning idea" - more on that later.

The startup in full should not take more than five minutes.

Collection of Ideas

Second section is the collection of ideas. To help the team members I split a whiteboard in three sections labeled "Continue/Increase", "Start/Try", and "Quit".

First section "Continue/Increase" is for things we already do and that support our work. Second section "Start/Try" is for ideas that we think we should benefit from, but we do not do it already, at least not in any significant amount. Last section "Quit" is for things team members think we do, but should stop doing as it does not serve us well, or even harm us.

Then team members are free to write down suggestions in any of these sections. Usually, I let them write on stickies, so that they can write a short statement on the sticky and explain it briefly when they post it on the board. If a short discussion emerges, then fine. However, if it seems to turn into a debate, or risk running long, I cut it short pointing to the purpose (find one idea) and the procedure (there will be a vote later).

I also try to convince the team to keep the suggestions very concrete and limit them to thinks they directly can effect. For example, replacing the ventilation system of the building might really improve, but selecting such improvements is just asking for failure. In other retrospective formats such ideas are really valuable, but the purpose of Blitz Retrospective is basically to get acceptance for retrospectives - and then they must make a difference. We want things we actually can do, and that we can do within a week.

The collection of ideas could take ten to fifteen minutes. On occasions when time has not been an issue, I have let it run on longer - but usually most ideas are on the board after ten minutes.

Grooming the List of Suggestions

When getting closer to the end of collection of ideas, I take a more active role, starting talking about the stickies in the "Quit" section.

It is extremely valuable to get the "Quit" feedback, and I really encourage people to post such suggestions. However, I am a firm believer of that you should focus on "telling the good stories", because the stories you tell (and repeat) will become part of the "team lore" and shape the atmosphere of the team work. This is the basic idea behind the organisational philosophy "Appreciative Inquiry".

So, telling "bad stories" will basically make people feel bad - but not do much good in the long run. Telling "good stories" will culture a nice team atmosphere as well as enforcing the good habits.

Therefore I take on the role as the "positivist fanatic" and try to rephrase each sticky under "Quit" into "positive" ones. For example, if a quit-note says "stop coming late to meetings", I might suggest a start-note "meetings begin exactly on time - even if people are missing".

Before throwing away the "Quit" not I ask the original poster whether the new notes have captured the original purpose. Surprisingly often there is an additional aspect I had not understood. In the "begin on time" example, the poster might say: "It is not only about starting on time, it is also that every time someone arrive (late) there is a start of chit-chat and small talk that disrupts the meeting". To capture this there will be a second note start-note "keep meeting focus when people arrive mid-meeting".

I also ask if there are some suggestion that the poster think is a duplicate of some other - if so they can have it removed. Merging two similar suggestions to one of course gives them a better chance to "win".

Still when "grooming", the board is still open for new ideas, it is just for the team members to step up, post a sticky, and present the idea.

At the end of collection of ideas there often are a load of ideas and we only have a few minutes until we shall leave the room with one selected idea: time to vote.

Voting

For the voting, I simply rearrange the stickies and let each team member give three suggestions one vote each. The sticky with the most votes is the winner and is what the team should try for improvement the coming week.

As for the rest - things might improve just by having vented them, but they are not in active focus.

Evaluation

Next week I use the startup section for a quick evaluation. Here I want to separate two parts.

One question is whether we did as we intended at all. Not trying isn't necessarily failure. Things change quickly from time to time and we might have had valid reasons not to do it. Even "did not have time" is a valid reason - and suggests that we should put aside time.

If the team did try the suggestion, the second question is whether it helped us or not. If it was helpful, we try keep doing it. If it was not helpful, then it is important to keep in mind that it was an experiment - and such should have positive and negative outcomes from time to time.

If the suggestion was tried and found helpful, we should find a way to formalize it. Technical stuff might go into a new check on the build server, working habits might go into the team's "Team Rules" or "Working Agreements" - whatever the name. The important part is that we do something that makes it plausible we will continue doing this good thing we have just found.

Getting Retrospectives Going, at All

I have found that running a few Blitz Retrospectives often results in an acceptance of having retrospectives at all. In combination with the not-very-scary format "ok, we can spare 25 min after Friday lunch", it is a good way to get retrospective started at all.

Of course, such a brief format misses many points - things a longer and deeper retrospective will catch. But the purpose of Blitz Retrospective is not to catch those - it is to win acceptance for having retrospectives at all an pave way for deeper retrospectives down the road.

I have found that the format works well for that purpose.

Yours

Dan

ps Ester Derby and Diana Larsen have written a great book named Agile Retrospectives that is really helpful once you have gotten past that initial resistance, have established retrospectives, and want to elaborate them, specialise them, or just improve them in general.

pps Tobias Fors has made the point (blog post in Swedish) that feeling safe and secure is fundamental to engaging in a retrospective. He suggests the fundamental rule being "Everyone did their best given the conditions", and I have seen it helpful to start each retrospective with writing some similar statement on the whiteboard.

Typewriting is not Storytelling

2010-08-04T12:45:00.008+02:00

Dear Junior

If you would observe a famous author during a workday to get insights in how such people work, you might come out with a report along these lines.

"After breakfast, Famous Author sits down at the typewriter. She then punches the keys, using the tip of her fingers, repeatedly. From time to time she picks a new blank page and roll it into the typewriter. She continues until early afternoon, except for a lunch break, whereafter she walks around in town taking pictures of people."

Even the Famous Author herself might describe her workday in a similar way ("do always write at least ten pages a day" or "only write when inspired"), describing the structure of the work, or the ceremonies surrounding it.

Correct as these descriptions might be, they totally miss the point. They tell you nothing about weaving a plot, about evolution of characters, about where to start the story, about how to finish it, or other things that makes the work worth reading.

Unfortunately, I have the feeling that many descriptions of agile practices make the same mistake.

Yours

Dan

In Agile, the Practitioners Own the Process

2010-07-23T11:43:00.018+02:00

Dear Junior

The term "process management" might be one of the driest around in system development, but in the shift to agile, the term hides a secret both surprising and delightful.

Demystifying "process", is nothing but our daily habits, what we do when creating systems. Examples of process include how we decide when we commit stuff, how we manage our requirements, whether we do pair programming, when to do code reviews, how we decide something is finished , etc. It is really about our every-day habits of life when working.

Put that way "process management" becomes how we decide what habits we should have, or how we change them, like if we should introduce code review, or take them away, if we should be more rigorous on "no failing tests" when committing, etc.

Put that way "process management" is simply how we get better at development over time.

Process Management Traditional Style

In traditional process management each organisation has some kind of authority that decides what process we should follow. It might come in the form of a "Chief Process Officer" or as a "Project Office", but the point is that somewhere we decide the process. The groups that do the daily work of creating and maintaining systems are then to follow this process. These groups can be projects, maintenance groups or have some other organisational structure. For simplicity let us call them "practitioners" for short. So, project office decide the process, practitioners follow the process.

In many cases "project office decide, practitioners follow" is an illusion. There actually exist (take my word for it) project offices that write a process behind locked doors and then hand out a binder to the rest of the org, while believing the process will be followed. Truth is that those documents will never be read outside the project office, and will obviously not be followed.

As an example, the project office might have decided that all code should be reviewed, and then fully believe that so is done — because it is in the process. The truth might be that no code is ever reviewed at all and the practitioners might not even know code review is part of the formal process. This "alienated project office" is of course a process management anti-pattern.

A good process office traditional style will of course behave differently. It will stay in touch with the practitioners and check whether the process is followed. If it is not, they will question why it is not followed, and explain the rationale of why they process look the way it does. It will also gather feedback and change the process to better fit changed of discovered needs.

However, even if the project offices adapt the process upon request, it is still in charge. If a practitioner what to change the process (e g "continuous, automated deployment to test environment"), she will suggest that change to the project office that will evaluate it. If the project office thinks it is a good idea, they will change the process, and the practitioner can start following the improved process. However, the practitioner cannot single-handed change the process just because she thought it was a "nice idea, worthy to give a try".

Process Management Agile Style

Agile development on the other hand relies heavily on the idea of continuous improvement. The basic idea is that that the process you have today is just the best you have come up with this far. Further more, the challenge of today is to find out what you should do different tomorrow. Agile development also honours doing experiments — if you get an interesting idea, then try it out for some time. At next retrospective, you evaluate the change and keep the it if it proved beneficial — or throw it away and try something else.

Experimentation is not only a right in agile, it can also be claimed to be a responsibility. Taking it to an extreme, it can even be claimed that if you have not changed your process in some significant way the last year, you are probably no longer agile. It is probably a sign you no longer do experimentation coming up with no ideas to try. That you have found "the perfect process" is just to improbable to be likely.

Here comes the mind-shift: for the practitioners to do this experimentation, they should have a mandate to do these continuous changes to the process. In other words, they should be empowered to change the process (ever so slightly) at their own will.They should not need to ask project office for permission. We want to keep the threshold for new ideas as low as possible.

Rephrased, the practitioners are actually unilaterally allowed to change the process. They now own the process they follow themselves, it is no longer owned by the project office.

Project Office Still Valuable

This does not mean that the project office becomes obsolete. Quite the opposite, it might even become more important. When all practitioners can form their process, it becomes even more interesting to have some group that pays attention to all these changes, gather the changes that where successful, and can promote cross learning between practitioners of different groups.

A team might for example struggles with that some pieces of the code is only understood by one single developer. Independently another team has solved a similar situation by using pair programming whenever such code is touched. The project office might then recognise the situation and suggest that the struggling team try that method to see if it will help them.

At this point of time the Project Office might change its name to enforce their new role. Some have re-branded themselves as the Agile Office, which I think is nice turn. Mike Cohn has reported that at one of his clients they called this group the "Agile Enablement Team" to emphasise that their goal is to enable other to become more agile.

What about Standard Process

In agile process management you can still have a standard process. The difference is that the standard process will not be pre-defined, but rather emerge. The project office will gather ideas that have worked at some team, e g "pair program on baldy understood code", and suggested it to some other team. If it worked at the second team as well it will promote the idea a little more active, and if it fits the organisation as a whole, it will soon enough be a standard among most or all teams. At this point "pair program on badly understood code" has become part of a de facto standard.

Over time, more a more practices will emerge and spread to all teams and a standard process will be formed. But, it will only do so if those practices actually work well throughout the organisation. And, it will not be cut in stone, but open to be challenged and evolved in the same dynamical manner.

So, the standard process will not become a standard process because someone say so, instead it will become the de facto standard process because a majority of the teams have tried and found the same set of ideas valuable and fit within the organisation. Here the project office plays a key role in facilitating and guiding this evolution so that the good ideas are spread and that the practises have a chance to converge instead of diverging.

The Shift

In traditional process management, the process is owned by the process office and followed by the practitioners.

In agile process management, the process is owned by the practitioners and facilitated by the process office.

Small mind shift in theory, huge difference in practice.

Yours

Dan

P.s. The situation of the Project Office in an agile environment resembles very much the situation of the architect who is unable to enforce architecture, but instead must work with the practitioners to explain and evolve it.

Refactoring as Time Machine

2010-07-15T09:58:00.003+02:00

Dear Junior

If I had a time machine, I would not need to worry very much on how to invest my retirement pension plan. I could just pick any random fund, and if that one turns bad, I could just travel back in time and pick one that will succeed instead.

Now, who said that time travel should be free? If nothing else, the power to run it should send your electricity bill to unusual heights. So each time travel will certainly come with a fee. However, if it enables me to "have made" a better choice of pension plan thirty years ago, it would probably be worth it.

The Existential Angst of Making Decisions

How wonderful would that be, to not have to spend effort and energy on finding what hopefully is the best, and then have to worry about having made the wrong decision.

Unfortunately, I have no time machine. And chances are I will not have one either when time comes for me to retire, about thirty years from now.

So, boring and uninteresting it might be, I have to spend time and energy on investigating different choices to make a good one. And live with the angst.

In System Development We also make Decisions

When developing software, I constantly have to make a lot of decisions, small or big. It might be just what to name a method, or it might be how to split the system in tiers; still decisions, decisions, decisions. And, all those choices affect how easy the system will be to work with in the future - picking the wrong choice, and I will have a hard time.

Sometimes I find that I have earlier made a design- or architecture decision that "seemed like a good idea at the moment", but now feels like crap. Perhaps I should have spent a little more time and effort thinking over all possible futures?

In System Development We Have a Time Machine

To my delight, I have found that in system development I have the time machine I lack in financial investments. Often some not-too-complicated refactorings can "take me back in time" and change that template method solution (that does not suite me well any longer) to a shiny state and strategy pattern (that does suit my current needs).

Of course, the refactoring time travel is not for free - there is a fee to be paid in the work I have to do and the testing to ensure it went well. However, that fee is much lower than paying the price of continue working with a code-base that is not well aligned with my best-idea-at-the-moment.

Obviously, I still have to spend enough time and energy to find a decent solution when making a choice. The difference is that I do not have to spend the enormous effort to cover all options. The possibility of making future refactorings simply shrews the economics of making a decision. Refactorings make it possible to settle for a good solution now, keep the speed up, and pay the small fee for changing your mind in the future. And I can skip the "angst" part.

In that way refactoring is a wonderful little time machine.

Yours

Dan

Ps The ability to do refactorings in the future makes it possible to keep options open longer, thus deferring decision-commitment. This is what I see as the central message of the Real Options-movement.