Vladimir Galore - Lots of Threaded Scala Actors Waiting for Godot

Dear Junior

When we discussed the model for programming actors in Scala we saw that the thread-based model could be compared to the very actively waiting character Vladimir of Waiting for Godot. I also mentioned that the more drowsy, laid-back Estragon of the same play could be a good picture of the other model, the event-based.

However, before proceeding to the event-based actors, I think it enlightening to dive a bit more into the thread-based and see what the problem is.

The point of actor-based programming is that you want to mimic a "society of collaboration" where each actor performs a focused task. And you do not just want one actor per function (or type of task), you actually want one actor per task. So you want a lot of them.

This is similar to object orientation - you do not want one object per class (e g representing phone numbers), you want one object per instance of the class (one for each phone number). Actually, it can be claimed that the restricted actor-message model is closer to the original ideas of object orientation than the object-method model we have become accustomed to.

So we really want loads of actors. Let us see how many Vladimirs we can create before my poor laptop cringes.

Let us revise the code for Vladimir in his waiting for Godot. As we will create a lot of Vladimirs I have given each an id.

class Vladimir(id : Int) extends Actor {
  def threadid = {
    Thread.currentThread.getId
  }

  def name: String = {
    "Vladimir" + id
  }

  def act() = {
    println(name + " is waiting " + threadid)
    receive {
      case Godot =>
        println(name + " saw Godot arrive! " + threadid)
    }
    println(name + "'s wait is over " + threadid)
  }
}

case class Godot

Now we also need to create loads of them. Let us make a list of integers and turn each of them into an instance of Vladimir.

object vladimirgalore extends Application {
  override def main(args: Array[String]) {
    val actors = Integer.parseInt(args(0));
    val ids = 0 until actors // [0,1,2 ...]

// turn each int to an actor using the int as id
    val vladimirs = ids map (id => new Vladimir(id))
    println(actors + " actors on stage")
    for(vlad <- vladimirs) { vlad.start }
  }
}

scala godot.vladimirgalore 6
6 actors on stage
Vladimir2 is waiting 12
Vladimir0 is waiting 10
Vladimir1 is waiting 11
Vladimir3 is waiting 13
Vladimir4 is waiting 17
Vladimir5 is waiting 18
^C

Never mind the order of the output - actors are threads and are thus entitled to run scheduled in any order. What we see is six actors, each an instance of Vladimir, and each given a thread of its own. And every one of them are in a wait-state waiting for the message "Godot". Let us relieve one of them from its wait, just to see one completion. Let us send Vladimir4 the happy message of Godot's arrival.

object vladimirgalore extends Application {
  override def main(args: Array[String]) {
    val actors = Integer.parseInt(args(0));
    val ids = 0 until (actors)
    val vladimirs = ids map (id => new Vladimir(id))
    println(actors + " actors on stage")
    for(vlad <- vladimirs) { vlad.start }
    vladimirs(4) ! Godot
  }
}

danbj$ scala godot.vladimirgalore 6
6 actors on stage
Vladimir1 is waiting 11
Vladimir2 is waiting 12
Vladimir3 is waiting 13
Vladimir0 is waiting 10
Vladimir4 is waiting 17
Vladimir4 saw Godot arrive! 17
Vladimir4's wait is over 17
Vladimir5 is waiting 17
^C

Ahaa. Vladimir 4 was started with thread 17 - which used "receive" to register the message handler (the code block containing the "case"). It was also thread 17 that later executed the message handler, doing the pattern matching of the case and running the corresponding code for "case Godot". Finally it was thread 17 that continued the code after the handler-block. Same thread all the way - that is why they are called thread-based. They do not only behave as if they where a thread, they are actually implemented using the same thread all the time.

Accidentially, thread 17 managed to complete the act-method of Vladimir4 so that specific thread could be reused for Vladimir5. However, in all other cases, a new fresh thread was required.

Now let us put loads of actors on stage. For clarity we remove the release of Vladimir 4 so that every actor will be waiting and all threads be locked up.

Let us see if we can put 1000 Vladimirs on stage and set them acting.

object vladimirgalore extends Application {
  override def main(args: Array[String]) {
    val actors = Integer.parseInt(args(0));
    val ids = 0 until (actors)
    val vladimirs = ids map (id => new Vladimir(id))
    println(actors + " actors on stage")
    for(vlad <- vladimirs) { vlad.start }
    // vladimirs(4) ! Godot
  }
}

danbj$ scala godot.vladimirgalore 1000
1000 actors on stage
Vladimir0 is waiting 10
Vladimir3 is waiting 13
Vladimir2 is waiting 12
Vladimir1 is waiting 11
Vladimir4 is waiting 16
Vladimir5 is waiting 17
…
Vladimir115 is waiting 128
Vladimir116 is waiting 129
Vladimir117 is waiting 130
Vladimir118 is waiting 131
Vladimir119 is waiting 132
…
Vladimir251 is waiting 264
Vladimir252 is waiting 265
Vladimir253 is waiting 266
Vladimir254 is waiting 267
Vladimir255 is waiting 268
^C

So, the system hangs on Vladimir255 even though it has not yet started all the 1000 actors I asked for. 

Hmm … "Vladimir0" to Vladimir255" - that is 256 actors that have been started before the system hangs. Such a number is hardly a coincidence … Here my colleague George Spalding came to the rescue by pointing out the relevant JVM properties, in this case "actors.maxPoolSize (default 256)". So, as I understand it Scala will not allow the JVM to allocate more than 256 threads for actor stuff. This means that when 256 Vladimirs had been started, then all 256 threads where sitting waiting for receiving Godot.

receive {
      case Godot =>
        println(name + " saw Godot arrive! " + threadid)
    }

And if the runtime-system refuse to allocate more threads, then no more actors will be started.

Let us run it again, with modified properties, increasing the maximum number of actor threads.

scala -Dactors.maxPoolSize=10000 godot.vladimirgalore 1000
1000 actors on stage
Vladimir0 is waiting 10
Vladimir3 is waiting 13
Vladimir2 is waiting 12
Vladimir1 is waiting 11
Vladimir4 is waiting 16
Vladimir5 is waiting 18
…
Vladimir997 is waiting 1010
Vladimir998 is waiting 1011
Vladimir999 is waiting 1012
^C

Ok now it worked… what about 2500?

danbj$ scala -Dactors.maxPoolSize=10000 godot.vladimirgalore 2500
2500 actors on stage
Vladimir0 is waiting 10
Vladimir3 is waiting 13
…
Vladimir2498 is waiting 2511
Vladimir2499 is waiting 2512
^C

Seems to work … and 5000?

danbj$ scala -Dactors.maxPoolSize=10000 godot.vladimirgalor 5000
...
Vladimir2538 is waiting 2552
Vladimir2539 is waiting 2553
godot.Vladimir@35a631cc: caught java.lang.OutOfMemoryError: unable to create new native thread
java.lang.OutOfMemoryError: unable to create new native thread
 at java.lang.Thread.start0(Native Method)
 at java.lang.Thread.start(Thread.java:658)
 at scala.concurrent.forkjoin.ForkJoinPool.createAndStartSpare(ForkJoinPool.java:1609)
 at ...
 at scala.actors.Scheduler$.managedBlock(Scheduler.scala:21)
 at scala.actors.Actor$class.receive(Actor.scala:512)
 at godot.Vladimir.receive(waitingforgodot.scala:38)
 at godot.Vladimir.act(waitingforgodot.scala:49)
 at ...
 at scala.actors.ReactorTask.run(ReactorTask.scala:36)
 at ...
 at scala.concurrent.forkjoin.ForkJoinWorkerThread.mainLoop(ForkJoinWorkerThread.java:340)
 at ...
Vladimir2540 is waiting 2553
^C

Nope it crashed.

But … systems always reveals interesting information when breaking down.

In this case it was the scala.actors.Scheduler that tried to start a new Thread in order to serve the Vladimir "receive" inside its act-method. And creating this "new native thread" was too much load for the JVM that crashed with OutOfMemoryError.

Let us run this once again, just below the limit where it crashes.

danbj$ scala -Dactors.maxPoolSize=10000 godot.vladimirgalor 2539

and have a look at process status

danbj$ ps -m -O rss
  PID    RSS   TT  STAT      TIME COMMAND
 8164 264280 s001  R+     0:06.80 /usr/bin/java -Xmx256M …

OK, "RSS" stands for "resident set state" and is basically "real memory". So, memory use is 264280 kB. Also note that the scala runtime has set the JVM "maximum heap size" (Xmx) to 256M. These two numbers are strikingly close to each other. My conclusion is that it seems to be the allocated heap that has filled up.

This does makes sense, a few thousand threads will need one allocated stack each, and that will eat the available space pretty quickly. Trying out with different number of actors give us some data on how much stack is needed per actor.

actors    RSS (in kB)
    1    62752
  500    95676
 1000   134132
 2500   239248

So, apart from that the system needs 62M just to start, every Vladimir (thread-based actor) seems to need just below 70k each. That is not very slim, not if we want to create truckloads of actors.

Obviously, the thread based actors have their advantage in being pretty easy to understand. But they definitely have their drawback in eating system resources by asking for one thread and a stack allocation each.

But, hey … in our benchmark all the threads where in wait-state. Could they not have been pooled in some way? We could have gotten away with just a few threads, and we would have been able to create many more actors!

Sure we could, and that is exactly what event-based actors do. Instead of being on their feet like Vladimir, we want them to drowse off and take a nap like Estragon. And in the mean-while the thread could be used by some other actor that has something in its inbox it wants to process. Unfortunately, they will also loose a little bit of their sense of time like him. But that will have to wait to another letter.

Yours

Dan

ps Check out what it looks like using the event-based execution model - with a very similar programming model.

Multicore, Power, and Performance

Dear Junior

Next time your data centre upgrade, you will of course get better performance out of your deployed systems. However, as there are several kinds of performance it can be interesting to think about what will happen to response time and capacity. Chances are capacity will improve, but response time might decline. The reason is spelled multicore.


In ancient days of yore, long gone, processors improved by running on higher clock frequency. Higher frequency made them able to process more stuff per unit of time. New processor meant higher frequency and thus better response time. If high capacity was needed, it was achieved by the processor thread-switching between several requests serving them all within reasonable response time.


Those days are gone. Nowadays high-speed processors are no longer in demand. The reason for this is power consumption. The power-consumption of a processor is roughly proportional to the frequency, squared. So for thrice the speed, you pay nine times the power. 


Slower Saves Power


There are of course several problems with high power consumption such as electricity bills and environmental issues. However, the driving force is cooling. Every single Joule that is fed into a processor in a data hall is transformed to heat. That heat needs to be transported out of the hall by the cooling system - and that is the limiting factor.


In other words — if you populate your data centre with high-speed processors you will hit cooling capacity at one point. However, if you use low-speed processors (frequency third of the high-speed), you can stuff in nine times more in the same data centre. So, nine times more, each giving a third of the MIPS compared to high-speed processor. Still you get three times more MIPS out of the same hall - good economy. 


Of course, the modern way of doing it is not to build several processors, but to stuff multiple cores onto the same chip — i e multicore. Still, the driver is the same - you get more MIPS out of you data centre by lots of low-frequency cores that out of substantially fewer high-frequency cores. 


This is not just theory. Several of the large web-sites of the world use new keys like MIPS/W or MIPS/m3 when benchmarking their data centres.


Actually, the same goes for laptops where heat must be limited to prevent user burns. There two slower cores on the same chip can give higher computational power and still radiate a lot less heat.


Better Capacity and Worse Response Time


Getting back to response time and capacity — how are these affected? Well, the data centre upgrade might consists of pulling out old single-core processors and replace them with quad-core processors with 20% lower frequency. Power consumption goes down, making it possible to stuff more processor into the same space. 


Looking at the data centre from the perspective of all clients we serve, or all transactions we process — we get a dramatic increase in capacity. However, from the perspective of a single request or transaction we experience a slower processor — so response time or latency will get worse.


Where processors and data centres earlier where sports cars getting faster and faster, they now rather resembles school busses that do not run nearly as fast but can carry a lot of people at the same time.


Of course this will affect the non-functional attributes of our systems running on these data centres. Unfortunately, the effect is not that straight-forward, but I am pretty sure we are slowly running into trouble and need to do something about it.


Yours


   Dan


ps This is one of the reasons it is important to keep apart response time and capacity, instead of stuffing them both under "performance".

Two Types of Performance

Dear Junior

In architecture one of the most important tasks is to keep an eye on the non-functional (or quality) attributes of the system. Often this importance is enhanced by stakeholders holding up some non-functional requirement (NFR) saying "the system must really fulfill this". Unfortunately, these NFRs are often quite sloppily formulated and a key example is "performance". I have stopped counted the times I have heard "we must have high performance".

I think a baseline requirement for any NFR is that it should be specific. There should be no doubt about what quality we talk about. In my experience "performance" can mean at least two very different things. Instead of accepting requirements on performance I rather try to reformulate the NFR to use some other wording instead. I have found that the "performance" asked for often can be split into two different qualities: latency and throughput.

Latency or Response Time


With latency or response time I mean the time it takes for some job to pass through the system. A really simple case is the loading of a web page, where the job is to take a request and deliver a response. So we can get out our stop-watch and measure the time it takes from the moment we click "search" until the result-page shows up. This latency is probably in the range of 100 ms - 10 s. Of course, this response-time is crucial to keep the user happy.

But latency can also be an important property even without human interaction. In the context of cron-started batch jobs it might be the time from the input-file is read until the processing has committed to the database. The latency for this processing might have to be short enough so the result does not miss next batch downstream. E g it might be crucial that the salary-calculation is finished before the payment-batch is sent to the bank on salary day. 

In a less batch-oriented scenario the system might process data asynchronously, pulling it from one queue, processing it, and pushing it onto another queue. Then the latency will be the time it takes from data being pulled in until the corresponding data is pushed out at the other end.

All in all, the latency is seen from the perspective of one single request or transaction. Latency talks about how fast the system is from one traveller's point of view. Latency is about "fast" in the same way as an F1-car is fast, but will not carry a lot of load.

Throughput or Capacity


On the other hand, throughput or capacity talks about how much work the system can process. For example, a news information portal might have to handle a thousand simultaneous requests — because at nine o'clock coffee break a few thousand people might simultaneous surf to that site to check out the news.

Throughput is also important in the non-interactive scenario. Each salary-calculation might only take a few seconds, but how many will the system be able to process during those 10000 s between midnight (when it starts) and 02:45 when the bank-batch leaves. If we cannot process all 50 000 employees, some will complain. To meet the goal we need a throughput of five transactions per second.

In other words, where latency was the performance from the perspective of one client or transaction, then throughput is the performance seen from the perspective of the collective of all clients or transactions, how much load the system can carry. Here "load" in the same way as a bus will take a lot of load transporting lots of people at once, even if it is not fast.

Fastness vs Load Capacity

Both F1 cars and heavy-duty trucks are no doubt "high-performing" cars. But they are so in completely different ways. To have a F1 car showing up at the coal mine would be a misunderstanding that only could be matched by the truck at the race track.


So, I avoid talking about "performance" and risk misunderstanding. Instead I try to use "latency" and "response time" to talk about how fast things happen — while thinking about an F1 car.  And I use "throughput" and "capacity" to talk how much load the system can handle — while thinking about a bus full of people.

What is the latency for transporting yourself between Stockholm and Gothenburg using a F1 car or public-transport bus? What is the throughput of transporting a few thousand people from Stockholm to Gothenburg using an F1 car or public-transport bus?

Yours

    Dan


P s Now when we are moving to multicore, we will see increasing capacity but latency leveling out or getting worse. This in itself will be a reason to move to non-traditional architectures, where I think the event driven architectures (EDA) is a good candidate for saving the day. My presentation at the upcoming JavaZone will mainly revolve around this issue.

'As a' as stakeholder, not actor

Dear Junior

The story format ‘As a <> I want <> so that <>’ is quite well established within the agile sphere by now, and almost the standard format in Scrum. In a way it is a pity because even though I like the format, and use it myself extensively, almost exclusively, there is a risk that we are setting in the mould. I have even heard equivalents of “if you are not doing asaiwantsothat, you are not agile”. I think we should be doing more experimentation, we should be avant-garde, not dogma. Nevertheless, well established as it is, I think it is misunderstood.

Old habits die slowly, and so old mental models. One old mental model is the one of the “use-case”, where some actor performs some action, and the system reacts in some described way. The format ‘As a <> I want <> so that <>’ can be used to describe a use case by inserting the actor in the as-a-clause “As a(n) <actor> I want …”, where the I-want-clause often takes the form of “be able to <GUI-enabled feature>”. My point is that even if the format can be used in this way, it is not limited to that usage.

If we use the format to express enterprise value, we can express things like ‘As a head of accounting department, I want nightly synchronizations and comparisons with our bank accounts, so that we early can catch failed sales”. Barring that we know little about ‘synchronizations with bank’ or ‘failed sales’, we can see that this is not a use-case: the as-a is not the actor, it is the person feeling the pain of this functionality lacking, and based on this way of expressing the story, it will be much easier to get the Head of Accounting to sponsor and promote the story.

Opening up this door we see that there are multiple ways to address other difficult stories. Expressing quality characteristics (aka non-functional requirements) can be done in the same was “As a marketing manager, I want the system to be able to handle 100 000 new users registering the same evening, so that it does not break down when we run our successful advertising campaigns”. You can also express ‘tech stories’ “As a QA manager, I want the system to be put under continuous build with the tests run automatically, so that we early know if a quality problem arises”. You can even express ‘soft issues’ like training “As a Development Manager, I want all team members to get basic training on version control so that we get rid of all those merge-mistakes that we suffer “. In all these examples, we have a stakeholder expressing something that would give enterprise value to their aspect of the development effort, thus something a team could estimate in effort and a product owner could prioritize.

Of course, we can now notice that the ‘actor’ interpretation is just a special case where the stakeholder happens to be an end-user of the system.

When working with this format I have also found it helpful to drop the ‘a’ in ‘As a’. Phrasing it “As Head of …” makes it a little more succinct, and seems to give the stakeholder him/her-self a more direct bond when reading it. Phrasing it “As a Head of …” seems to be a little bit more ‘distant’ or ‘observing’ way, and does not (in my humble experience) to catch on in the same way.

So, obviously we are nowhere near the perfect format for “requirements”, we should not restrict our formats unnecessarily, we should look for new way of using those we have, and we should definitely do more experimentation.

Yours

Dan

ps Mike Cohn has written some related blogs and was kind enough to point me too them. Check out Non-functional Requirements as User Stories, Advantages of the “As a user, I want” user story template, and Writing User Stories for Back-end Systems that muse on adjacent themes.