Multicore, Power, and Performance

Dear Junior

Next time your data centre upgrade, you will of course get better performance out of your deployed systems. However, as there are several kinds of performance it can be interesting to think about what will happen to response time and capacity. Chances are capacity will improve, but response time might decline. The reason is spelled multicore.


In ancient days of yore, long gone, processors improved by running on higher clock frequency. Higher frequency made them able to process more stuff per unit of time. New processor meant higher frequency and thus better response time. If high capacity was needed, it was achieved by the processor thread-switching between several requests serving them all within reasonable response time.


Those days are gone. Nowadays high-speed processors are no longer in demand. The reason for this is power consumption. The power-consumption of a processor is roughly proportional to the frequency, squared. So for thrice the speed, you pay nine times the power. 


Slower Saves Power


There are of course several problems with high power consumption such as electricity bills and environmental issues. However, the driving force is cooling. Every single Joule that is fed into a processor in a data hall is transformed to heat. That heat needs to be transported out of the hall by the cooling system - and that is the limiting factor.


In other words — if you populate your data centre with high-speed processors you will hit cooling capacity at one point. However, if you use low-speed processors (frequency third of the high-speed), you can stuff in nine times more in the same data centre. So, nine times more, each giving a third of the MIPS compared to high-speed processor. Still you get three times more MIPS out of the same hall - good economy. 


Of course, the modern way of doing it is not to build several processors, but to stuff multiple cores onto the same chip — i e multicore. Still, the driver is the same - you get more MIPS out of you data centre by lots of low-frequency cores that out of substantially fewer high-frequency cores. 


This is not just theory. Several of the large web-sites of the world use new keys like MIPS/W or MIPS/m3 when benchmarking their data centres.


Actually, the same goes for laptops where heat must be limited to prevent user burns. There two slower cores on the same chip can give higher computational power and still radiate a lot less heat.


Better Capacity and Worse Response Time


Getting back to response time and capacity — how are these affected? Well, the data centre upgrade might consists of pulling out old single-core processors and replace them with quad-core processors with 20% lower frequency. Power consumption goes down, making it possible to stuff more processor into the same space. 


Looking at the data centre from the perspective of all clients we serve, or all transactions we process — we get a dramatic increase in capacity. However, from the perspective of a single request or transaction we experience a slower processor — so response time or latency will get worse.


Where processors and data centres earlier where sports cars getting faster and faster, they now rather resembles school busses that do not run nearly as fast but can carry a lot of people at the same time.


Of course this will affect the non-functional attributes of our systems running on these data centres. Unfortunately, the effect is not that straight-forward, but I am pretty sure we are slowly running into trouble and need to do something about it.


Yours


   Dan


ps This is one of the reasons it is important to keep apart response time and capacity, instead of stuffing them both under "performance".

Two Types of Performance

Dear Junior

In architecture one of the most important tasks is to keep an eye on the non-functional (or quality) attributes of the system. Often this importance is enhanced by stakeholders holding up some non-functional requirement (NFR) saying "the system must really fulfill this". Unfortunately, these NFRs are often quite sloppily formulated and a key example is "performance". I have stopped counted the times I have heard "we must have high performance".

I think a baseline requirement for any NFR is that it should be specific. There should be no doubt about what quality we talk about. In my experience "performance" can mean at least two very different things. Instead of accepting requirements on performance I rather try to reformulate the NFR to use some other wording instead. I have found that the "performance" asked for often can be split into two different qualities: latency and throughput.

Latency or Response Time


With latency or response time I mean the time it takes for some job to pass through the system. A really simple case is the loading of a web page, where the job is to take a request and deliver a response. So we can get out our stop-watch and measure the time it takes from the moment we click "search" until the result-page shows up. This latency is probably in the range of 100 ms - 10 s. Of course, this response-time is crucial to keep the user happy.

But latency can also be an important property even without human interaction. In the context of cron-started batch jobs it might be the time from the input-file is read until the processing has committed to the database. The latency for this processing might have to be short enough so the result does not miss next batch downstream. E g it might be crucial that the salary-calculation is finished before the payment-batch is sent to the bank on salary day. 

In a less batch-oriented scenario the system might process data asynchronously, pulling it from one queue, processing it, and pushing it onto another queue. Then the latency will be the time it takes from data being pulled in until the corresponding data is pushed out at the other end.

All in all, the latency is seen from the perspective of one single request or transaction. Latency talks about how fast the system is from one traveller's point of view. Latency is about "fast" in the same way as an F1-car is fast, but will not carry a lot of load.

Throughput or Capacity


On the other hand, throughput or capacity talks about how much work the system can process. For example, a news information portal might have to handle a thousand simultaneous requests — because at nine o'clock coffee break a few thousand people might simultaneous surf to that site to check out the news.

Throughput is also important in the non-interactive scenario. Each salary-calculation might only take a few seconds, but how many will the system be able to process during those 10000 s between midnight (when it starts) and 02:45 when the bank-batch leaves. If we cannot process all 50 000 employees, some will complain. To meet the goal we need a throughput of five transactions per second.

In other words, where latency was the performance from the perspective of one client or transaction, then throughput is the performance seen from the perspective of the collective of all clients or transactions, how much load the system can carry. Here "load" in the same way as a bus will take a lot of load transporting lots of people at once, even if it is not fast.

Fastness vs Load Capacity

Both F1 cars and heavy-duty trucks are no doubt "high-performing" cars. But they are so in completely different ways. To have a F1 car showing up at the coal mine would be a misunderstanding that only could be matched by the truck at the race track.


So, I avoid talking about "performance" and risk misunderstanding. Instead I try to use "latency" and "response time" to talk about how fast things happen — while thinking about an F1 car.  And I use "throughput" and "capacity" to talk how much load the system can handle — while thinking about a bus full of people.

What is the latency for transporting yourself between Stockholm and Gothenburg using a F1 car or public-transport bus? What is the throughput of transporting a few thousand people from Stockholm to Gothenburg using an F1 car or public-transport bus?

Yours

    Dan


P s Now when we are moving to multicore, we will see increasing capacity but latency leveling out or getting worse. This in itself will be a reason to move to non-traditional architectures, where I think the event driven architectures (EDA) is a good candidate for saving the day. My presentation at the upcoming JavaZone will mainly revolve around this issue.

Not Enforcing Architecture

Dear Junior

As I often enough take on the role of software architect (whatever that is), I am from time to time asked by other architects how I do to enforce architecture, or certain patterns, or coding conversions. They seem to see a problem that they spend a lot of energy and effort in defining these rules or guidelines, and developers in general seem not to follow the rules, so how to enforce them?

The question always takes me aback. Besides really not recognizing the scenario, I really have some trouble with the question itself, especially the usage of the word "enforce".

To start with, if you believe in that there is something like "free will", you cannot enforce anybody to do anything. From a philosophical perspective anyone is always in a position to say "no" - of course taking the consequences, but still refusing to be forced. The only way to really force someone is to ensure that the consequences are catastrophic, so that refusing is not an option. Total annihilation of mankind would be such a scenario. Or, from a personal perspective, pointing a gun to someone's head would suffice in depriving them their free will, thus enforcing some behaviour.

Well, pointing a gun to the developer's head will probably not be a feasible way to enforce some coding standard. In particular it has the drawback that you can only enforce it at one programmer at a time, or two if they are seated within arm length distance and you have two guns. Apart from that, I think it would be illegal in most countries I have worked.
So, the programmer must follow your architecture out of free will taking into consideration the consequences of not adhering. Let us get back to this later.

Secondly, the word "enforced" is normally most used in the context of "law enforcement" or "enforcing legislation" referring to the power of police, courts, and the rest of the system of justice to uphold the rules of society - which is usually considered to be a good thing. Let us pause for a moment and consider why this is considered good, even though it is a restriction of our freedom of choosing our actions. The legitimacy of the legislation in our democratic societies derives from the fact that it has been decided in an open process after careful discussion and debate between delegates elected to represent the people being governed by the laws.

So, how often does this structure apply to the relation between the architect, the architecture, and the developers? Is the architect elected by the developers in free elections, and the architecture debated and voted upon? In most of the cases when the architect asks for advice on enforcement, the answer to these questions is always "no" in my experience.

If we are to find a government system that resembles the structure of most organisations with architect and developers, it is despotism that is closest. The despot makes decisions he (most of the cases) consider wise, the subjects have no say in the question and must obey. Of the different kinds of despotism, the situation is actually best matched by a kingdom where power of the king stem from a higher power (i e a God in historic societies, and Upper Management playing that role in development organisations).

I am sorry, but if someone asks me for advice on how to enforce legislation in a system of one-ruler dictatorship - I will simply discretely not cooperate.

Back to the free will and making people follow rules. If you do not point a gun to their head, the reason for developers to follow "your" architecture is the consequences of not doing so. The architect might get angry at them, they might have an unpleasant discussion with the boss, or they might eventually be fired. Basically, they will follow the architecture out of fear. And, having seen quite a few teams in various situations I can tell, fear is not the best motivator for producing high quality software.

So, at the bottom line: I will not enforce architecture. I will not help enforcing architecture. I think it is a bad idea to enforce architecture. I think that if you want to enforce architecture you are working in the wrong direction.

Do not enforce architecture. Do not even consider doing it. Find a better way instead.

Yours

Dan