It is not just an Integer, it is a Domain Primitive

Dear Junior

Right now I am sitting on train number 928, travelling from home on my way to Arlanda airport and onwards to Explore DDD Conference to present on Domain Primitives together with my friend and colleague Daniel Deogun. Train numer 928 … if I would write a program handling trains I would probably come up with a class named TrainNumber somewhere sooner or later.

Some people might object ”come on Dan, that’s overdoing it, 928 is just an integer” and suggest representing it as a simple int. Me forming a separate class - am I doing it more complex that it is?

For a moment, let us dive into ”it is just an integer”. What does it mean to be ”just an integer” anyway?

Obviously integers are not just a bunch of numbers in a pile. One of the first intellectual things we teach our kids is to count stuff ”one, two, three, four …” pedagogically moving our adult index finger from one toy car to the next. Clearly the fact that ”three” comes immediately after ”four” is something very fundamental for us.

Does that hold for train numbers as well?

Another important property of integers are that we can add them to each other: 4+5=9. This is a really useful property that enable us to take some new-bought apples (five of them) and put them in the fruit bowl where there were some apples since before (four of them). We now know there are nine apples in the bowl. And it neither matters how many apples there were in the bowl, nor how many apples we put in, the result will always be a integer amount of apples.

Of course, integers can be negative. Negatives denote a shortcoming, a lack, or a debt. According to the bank, I own an impressing number of negative Swedish kronor related to my house loan. And when I add Swedish kronor to that negative pile, at time of mortgage, that negative number increase towards zero. Hopefully it will be zero some day.

Does that hold for train numbers as well?

We can also multiply integers: collecting four piles of plates, each six plates high, yields a pile of twenty-four plates. 

Does that hold for train numbers as well?

Zooming out, using mathematical notation: the integers form ”an abelian group under addition”. This means for example that there exists an ”identity element” (denoted 0) such that it doesn’t matter if you add it to another integer: 0+a = a. It also means that for every other element (not 0) there exists an ”inverse” which is the ”opposite”; so as 4 is an integer which is not not 0, there must exist an integer (which we name -4) such that 4 + (-4) = 0.

Does that hold for train numbers as well?

Being ”just an integer” does not seem so simple any longer, doesn’t it? It turns out that integers is a very rich type with a lot of properties (and we have not even started talking about prime numbers yet).

Does really train numbers have all these complex properties?

Of course not: There isn’t necessarily any specific meaning in that train number 928 is ”adjacent to” train number 929. Adding train number 928 and 356 does not necessarily yield a train number. Train numbers are very seldom negativ. Multiplying two train numbers doesn’t  make sense at all (a train-square?). Train numbers does certainly not form an abelian group.

Train numbers are much simper than integers. It’s basically just a set of distinct elements, without any operations or relations apart from comparing two of them.

Train numbers are simply something that is a primitive in our domain. It’s one of the fundamental concepts that the train domain stand on, its parts carry no meaning. In that regard it fulfils the same kind of role that ”language primitives” fulfil in programming languages, but in the train domain. It is a ”train primitive”.

Turing our heads toward programming, think about all the things you can do with an int - just have a look in the Math-class for inspiration. This means that whenever someone think ”train numbers are just integers” they have to take care not to do any of these things that can be done. The TrainNumber class on the other hand can only be used for what it is intended for, nothing else.

Saying that train numbers is a domain primitive, and giving it a class TrainNumber, is not to make things complex. It’s to make things simple.

Yours

   Dan

Impressions from Jfokus 2017: Internet of Things (IoT)

Dear Junior

The annual developer conference Jfokus is always an almost overwhelming source of impressions. There is always lots of interesting presentations, and today was no exception. It's impossible to cover all of them, but today I attended three presentations that together formed a nice whole as they covered different aspects of IoT - Internet of Things.

During the IoT keynote Kevin Hoyt went very down to earth. He covered the state of IoT devices today using different means of connectivity as his starting point. My main take-away was his very thorough coverage of each "typical" device for Bluetooth-, WiFi-, and GSM-connectivity. For each type he demoed the device, showed the development environment, and discussed programming concerns: for example the importance of entering deep sleep when working on a device using Bluetooth and powered by battery. Specifically I took away how the libraries have evolved to make connectivity and features like deep sleep and wake-up-on-connect.

The other presentation that fitted nicely into the topic was the presentation Shahid Raza of RI.SE (Research Institute of Sweden), formerly SICS (Swedish Institute of Computer Science) held under the title "Is IoT Security a Nightmare?". I simply assumed that the question was purely rhetorical, and had expected a "yes" with some elaborations on the subtler points. Boy, was I mistaken. The overarching message of the presentation was a resounding "no". In fact the current protocol stack for IoT has evolved a lot and now even support end-to-end security of communication, even when the data package has to make multi-hops from one device to another before ending up at its final destination. The protocols for doing this are essentially structured in the same way as we do with certificates and HTTPS over TLS. The largest surprise for me was that the computation power needed for cryptographic computation is not a problem. I was under the impression that the power needed for those computations would take too much of a toll from the batteries of battery-powered devices, but apparently not. The largest toll is apparently the radio-transmission of data packages. And, the protocol folks have worked hard to compress the protocol stack so that the overhead-parts of each package (delivery info and security) is compressed so that there is more room for the payload. In that way, fewer packages are needed for the same amount of payload, and that is where you conserve a lot of battery power.

The third presentation was on the just-released Bluetooth 5 standard and was held by Ioannis Glaropoulos. He presented how the new standard has increased throughput of Bluetooth to 2 Mbps, making it feasible for e g streaming audio, which makes the protocol useable for more use-cases. Counter to intuition this might actually decrease battery consumption because the same amount of data can be sent in a shorter amount of time, diminishing the risk of interferences and resending the data. So the amount of energy needed per byte sent might be lower. Bluetooth 5 also features a longer range. For indoor use the main advantage is simpler topologies: many devices can connect directly to a central node or gateway, instead of having to form a mesh where some devices need to route-forward messages from more distant devices. This also conserves energy for the intermediary devices that doesn't need to wake up to forward messages, but can stay in deep sleep until they have something they want to transmit.

Put together these three presentations definitely broadened and deepened my understanding of where Internet of Things stand today. Something that is a worth outcome of a day at a conference.

Yours

   Dan

Writing a book - Secure by Design

Dear Junior

I'm sorry for not writing to you for quite some time. But, you see, I have been working on a book together with my two colleagues and friends Daniel Deogun and Daniel Sawano. Late this summer we decided to try to write down our collective experiences and insights in design and security as a book. We are far from finished, but by now the first few chapters are released from the publisher Manning as part of their "Manning Early Access Program" (MEAP).

The book is named Secure by Design and our main message is how good design and good code can help developers avoid security problems. We have found that lots of security vulnerabilities are due to bugs that could have been avoided if the design of the code had been different. 

You as well as I know that it is a lot easier to think about "good design" when developing features, that it is to think "security" at the same time.

Of course not all good design lead to avoiding security vulnerabilities. So we have gathered our experience on what design-tricks that are most effective in giving security as a side-effect. It's probably no big surprise to you that Domain-Driven Security is an important part of this, but there are many other parts as well.

We still have a lot of material to cover. But, what we have started with things that are close to the code, such as building Domain Primitives, how to structure validation, immutability etc. I guess you will find several of these ideas familiar, as we have discussed them earlier - but there are also some new insights I have not yet had the time to write to you about. Well, not to mention all the ideas from Daniel and Daniel which you and I haven't had possibility to reflect on earlier.

In the later parts there will be more material on integration, security benefits from cloud thinking. We also want to share our ideas around architecture such as security concerns for microservice architecture or how to handle legacy codebases.

We will continue to write during the spring and early summer. Hopefully we will see the book in print sometime during the fall this year.

Yours

   Dan

PS The early-access was released last night; you can check the book at https://www.manning.com/books/secure-by-design

Idea Mingle to Bootstrap Open Space

Dear Junior

Some of the things I do on a regular basis is to facilitate workshops and events of different kinds - open space being one of them. I'd like to share with you a small trick-of-the-trade I have evolved while facilitating those: the Idea Mingle.

I really like the energy the open space format brings to the discussions. People are familiar with the format of participating in a discussion, and also shy or introvert people are given room to participate in a comfortable way.

However I have sometimes found that the initial part can be a little bit slow, the part where you pitch topics for discussion. This is seldom a problem with groups that are used to open space. Everybody involved understands what "level of ambition" is needed for posting a topic - acknowledging the fact that the really interesting part will be the discussion that unfolds around the topic. However, with groups unused to open space I have found that people can get shy. Posting a subject might seem pretentious: "what have I that everybody else should find interesting". So, when asked at point blanc "what do you think should be discussed", their mind goes blank.

To give the "pitching phase" of open space a soft start I do a "Idea Mingle" that gives people a chance to evolve their interests into discussion topics before it is time to post them. It is a "mingle party" with the purpose of distilling ideas for discussion, thus "Idea Mingle".

The instructions I give are simple: 

• Look around the room and lock eye contact with someone you have not yet spoken to today. 
• On my call, pair up
• Introduce yourself by saying "Hi, my name is …" to each other
• Continue by saying "I think it would be interesting to discuss something around …" and fill in a field. It does not need to be specific, it can be broad or vague, e g "something about testing".
• Have a short discussion digging a little bit into each others interests
• After two minutes I will break the discussion

The basic idea is that the one-on-one-format is a more comfortable format for talking about an idea than to do it in public. Speaking to a stranger is still an uncomfortable situation for a lot of people, me included. However, the short format and the very focused scope makes it managable - as opposed to the usual mingle setting "find a stranger, talk to him/her, and be nice and interesting for an undefined period of time". Just the thought gives me creeps.

Now, after the first one-on-one, all participants have bounced one of their interests on another participant. Almost certainly they have received acknowledgment that "it is an interesting field" and most have probably got feedback of the form "testing is interesting; I myself is interesting in how to use automation to make it easier".

OK, so people have now got a slightly refined idea about an interesting discussion. But, for most of them it is not yet ready to publish as a open-space-discussion-subject.

Allow me a slight detour. 

At a workshop class with Lyssa Adkins and Leslie Stein there was a small knowledge-sharing exercise. During that exercise I was given the task to explain "sprint retrospective" to a few of the other participants. I was given literally no time to prepare, and my presentation time was limited to five minutes (if I remember correctly). It was really awkward.

Immediately after the first group of listeners, I was sent a second group of a few class-mates, and had to explain to them. This time it was less awkward. A third group was sent to me, and this time I basically knew what I should say or not. The fourth time, the explanation went smooth. So, in four iteration a very awkward rambling refined into a pretty crisp micro-presentation.

Back to open space and Idea Mingle.

The next instruction for Idea Mingle is: Now, lock eye-contact with someone else. And, we do the same thing once again.

At the end of this second one-on-one the idea might have evolved to "automation in testing, and the trouble with databases".

And then we repeat the one-on-one a total of four times.

At this point of time, many of the participants have received acknowledgement that they are interested in a field others also are interested in. And, a lot of them have refined an initial rough idea into something that is a more specific topic, e g "How to involve DBAs in test automation" - a topic ready for discussion.

And we are ready to form a queue for pitching topics to discuss.

Also, everyone is a little bit frustrated over that each one-on-one was interrupted just when they were getting started. So, all participants are eager to start the open space discussions.

Yours

   Dan  

The Way Agile is Fast - Fast as in Orienteering

Dear Junior

It is often claimed that agile development is faster than the alternative approaches. However, "faster" used this way might be misunderstood. For example, we all know that running can be tiring, so running faster can be exhausting. 

Does agile development mean that we will exhaust the poor developers? And do so sprint after sprint? Does it mean we induce stress ulcer? Or will the developers have to cut down on testing, design, and quality to meet this faster pace of deliver?

Quite the opposite. Agile is faster in another way. It is not faster as in a long-distance runner being faster, rather in the way an orienteering runner might be faster - by making smart choices of route.

In agile we realise that developing systems is not running down a pre-routed track. It is more alike to make our way through wild terrain. When doing this it is crucial to early see if we are moving down a dead-end route, and then stop ourselves. To run down a dead-end route without getting anywhere is the most wasteful exhausting thing an orienteering runner can do.

In agile we try to adress this at as many levels as possible. To prevent the individual developer go down a dead-end without realising it, we have pair-programming or daily stand-ups as reality checks. At a team-level we have short sprints, or watch the kanban cycle-time, to prevent us from working on stuff in a bad direction for too long. At project- or initiative level we do impacts map and similar to ensure we do stories that matter. At the level of finance/funding and portfolio management we track effects to stop us from continue spending huge effort on stuff that give no or low effect. 

So, agile does not make us faster by making us run at a more exhausting pace. Agile makes us faster by giving us tools to take smart routes, preventing us from detours.

Yours

   Dan

ps This is of course related to how Domain Driven Design makes us faster, even in the short run. Using the orienteering metaphor, what you to is to slowly jog for some a short period of time while exploring the map for different routes.

Coding Kids, it is about Democracy

Dear Junior

These days it seems like a lot of people think it is important to teach kids to code. I would like to chip in my idea about why this is important. To me, it boils down to a question about democracy, but let us not start there.

Code as a Profession

The first argument is about job market. There is no doubt that today we are already short on good system developers, and all predictions and forecasts say that this is going to get worse. So, going for a career in system development seems like a good hedge. This is the argument seen from the perspective of the individual kid. On a side note, I'd like to add that I personally think that system development is an interesting mix of creativity, intellectual stringency and cooperation; so apart from being a job that will probably be in demand it is also a job that is quite fun, when done right.

This argument also have the macro-economic side: companies need system developers to be able to continue their development. With a lack of skilled developers, the companies innovation pace might come to a halt, which would harm society at scale.

So, learning to code is obviously a good way to take the first steps towards a job in systems development.

A historical parallell is that there was a time when reading and writing was only relevant for a small 

were jobs where reading and writing were essential skills: handling written text where in those days mainly done by priests and medieval administrators where in those days that where handling written text. Written text was irrelevant to the rest of society.

This as definitely been true for code. Code was essential for the coding professions - a limited number of engineers and researchers - but not outside those circles. Code was irrelevant to the rest of society. 

However, I think that has changed. Code is no longer important only for those that work as programmers. Today the ability to code to some degree is fruitful to other professions.

Code in more Professions

Thus, second argument is that more jobs will involve code. If we look outside the profession of system developers, there are already places where coding helps people do their job better. Imagine Susan in HR, who is working on gender equality issues and is curious about how the salary level compares across different department, different ages with respect to gender. 

Susan has all the data at hand, perhaps in a few spread-sheets, perhaps in some simple database. Now, she could ask the analysis department to do some statistics which she could analyse to see if there is something to her hypothesis. That would probably take a day or two before they come back. But, if Susan herself can put together some SQL-queries or some scripts, she can get the result the same afternoon. The difference in how well she can do her job - analyse if there are salary differences that should not be there - is significant.

Knowledge in coding is not just a profession in and of itself. It is also already a helpful tool in other professions, and I think it will soon be essential.

A historical parallell could be at the beginning of the industrial revolution. At this point written text was not something that was relevant for the text-professions priests and adminstrators. Suddenly the factory workers where supposed to be able read, reading written instructions or work orders. To start with only the foremen needed this, but soon enough literacy was expected from all workers.

I think this is where code and society is today.

Code in Society

Looking forward, I think it will not stop here. I think understanding of code will become even more important.

Let me start with the historical parallell. After industrial revolution society evolved together with the usage of text. Text weaved into the fabric of society. People who could read were able to follow the news in the spreading news-papers. People who could write were able to participate, write letters to the editors. In parallell civil associations arouse such as workers unions or the anti-drinking movements. These were all possible because their member were able to read and write, to document their meetings in protocols, to create pampflettes to spread their ideas etc. The raise of modern democracy was fuelled by people knowing text.

People who could not read or write were soon standing at the side watching society evolve, more or less unable to participate.

I personally had a revolving experience. An a journey in China in the mid-90s I found myself standing at a cross road in the middle of Beijing. Everywhere I looked I could see text, massive amounts of text. Every shop I could see had signs promoting their wares, there were banners and posters, there were road signs, there were advertisements for papers. Texts everywhere, massive amount of information, a world that everyone around me navigated trough without effort. And I did not understand a single thing. 

Here and there I could see a sign I could recognise, the sign for "north", the sign for "water" etc. But I was feeling totally alienated. Watching people around me naturally navigating on this ocean of information, it felt like being surrounded by magic.

It was not hard to relate to what it must have felt like being an illiterate while society quickly evolved, suddenly just assuming that people in general would understand text and could participate.

I think this is where code is going. 

A World of Automated Processes

Already today we are surrounded by automated processes - powered by code. Most of them are pretty trivial, for example paying a parking using your credit card by checking in and checking out. Some of them are a little bit more complicated, e g when you order goods online and you can track the goods until it arrives at your post office, coupled with status-notification to your mail or phone. Some are pretty complicated. In Sweden the taxation rules for partnership incorporated companies are somewhat complicated - there are several set of rules to chose from and they give different results. However, when declaring taxes on-line on the tax authority web site - then the system helps you chose the set of rules that are most beneficial to you. Even though I know it is just code, it feels a little bit like magic.

We are probably only in the beginning of this evolution. Within a few decades, automated processes will be in much more places, they will be much more complex, and they will probably be interlinked. They will be part of the fabric that constitute society.

Those people that know code, they will be able to understand, to participate, to protest, to support. The people who do not understand code will be standing at the side, watching society evolve, more or less unable to participate.

I realise this might seem like jumping to conclusions. But think back. When industrialisation was new and reading was mostly used by foremen to read work orders and instructions - did it seem probable that reading and writing would transform society, making those skills essential to participate in and effect society at large? 

I think that understanding code will be the literacy of tomorrow.

Democracy

Teaching kids to code is not just about opening the door to the programming professions; it is not just about giving them a better chance to do their non-programming jobs well; at stake are giving them the foundations they need to participate in society; it is about democracy.

That seems pretty important to me.

Yours

  Dan

CSR, Knowledge, and a Student Conference

Dear Junior

At Omegapoint we have for a long time struggled with the question: How can we contribute towards society? Or, put otherwise, how can we pay back to the society to which we owe so much? To use the business lingo phrase: How do we exercise our Corporate Social Responsibility (CSR)?

There is the obvious non-answer: We already contribute to society though our business. We pay taxes without fuzzing or hiding. What we do for our customers must be good in a broader sense, otherwise they would not pay for it. So, obviously we contribute.

But our moral standard is somewhat higher than that kind of waterline mark.

Of course there is always the easy way out: Donate money to some beneficiary organisation. And of course we have done so. But it does not feel completely satisfactory. Anyone can donate spare money: should we not pay those money as salaries and let our employees donate at their own discretion? What could we as a company contribute with?

Also we have done pro bono work: helping NGOs with their web-sites and support systems. Also we have given pro bono aid and coaching to local startups. Of course this is better, but we were still not completely satisfied. Lots of other companies could do this, it did not feel unique to us.

Slowly has the realisation dawned on us. We define ourselves as a knowledge company with an attitude of openness. The epitome of this is our semi-annual two-days competence conference, affectionately named OPKoKo where we run presentations, workshops, and discussion - all powered by stunning colleagues and some occasional guest. This is the soul of Omegapoint, this is what we should share.

Sharing with whom became pretty obvious. At Omegapoint almost everyone has studied at a technical university, apart from some remarkable autodidacts. In Sweden university education has no tuition fee at any of the universities - it is funded through taxes instead. This means that even the best universities have a steady flow of talented young people from all corners society. Omegapoint as a company is build on this foundation. We should pay back to Swedish Academia, and its education of students in particular.

Once question phrased, answer was simple. This spring we will run our first student conference: Studentkonferens 2015 by Omegapoint Academy.

We have put together a one-day conference with three tracks: programming, security, and testing. We will bring our experts from the field to spend a day presenting, discussing and socialising with the students. 

We have tried to pick subjects that are as relevant to students as possible - not far-away is about "strategic architectural enterprise business process modelling", but rather about stuff they can relate to and hopefully use immediately. We ended up choosing "Craftsmanship" as the theme.

Now, "Craftsmanship" is a term that has been thrown around our community left and right, and I am definitely sceptical to some uses of the word. Still, I think the original message of craftsmanship holds: love the craft, pay attention to what matters, be pragmatic.

Yesterday we announced the conference to the public during the Datatjej2015 conference run by female-CS-student association Datatjej. This is not a coincidence. 

At Omegapoint we always hire on merit. The idea to select from non-relevant attributes as gender or ethnicity is completely foreign to us. However, we also do know that women have had (and still have) a harder time in our industry than men. Thus, we want to encourage initiatives that encourage young women to make a move in our industry. Hence, it makes sense to draw some attention to Datatjej2015 by selecting that event for announcing our own initiative.

Now, this is going to be fun. Also, I have been given the honour to present the opening keynote, presenting my view of Craftsmanship in software development. I am really looking forward to all of it.

Yours

  Dan

PS To pay credit where credit is due, I really must praise my colleagues Lotta Hammarström and Daniel Deogun. They where both part of the original idea, and have really worked hard to it come alive.

Make Concepts Explicit when Fixing Bugs

Dear Junior

To find implicit concepts and make them explicit is a very powerful way to improve code. But, sometimes it is hard to judge beforehand which concept is important enough to make explicit. We try to make good judgements, but sometimes we miss.

But, no reason to despair, we can take Time as our helpful aid. So, instead of everything perfect from start, we continuously perfect it whenever we find a reason. Two of the topmost reasons are either a bug appearing or further development of the code.

Let me for now fokus on the case of when a bug appears. If the code does not behave the way we intended it to do, then there is most often a place in the code that was to convoluted. In that convoluted state, subtle misunderstandings and simple mistakes could hide. The solution is to clarify the code until it is obvious whereof the mistake consists.

Things get clearer by examples, so let me clarify what I mean with a very minimal example, the story about a bug and its fix.

In real life, more or less a decade ago, I worked on a project where we did a batched import of records coming from an external source. The code looked roughly like this following in the class ImportCron.

public class ImportCron {

    private RecordImporter importer;

    private Logger logger;

    void runImport()  {

        List records = importer.listWaitingRecords();

        logger.log(Level.INFO, "Importing records: " + records.size());

        for ( Object rec : records) // imports each of the records in turn

   …

    }

Now, the import was run once a minute, but most of the time there where non records waiting.  Problem was the import log became filled with rows saying "Importing records: 0" so we had to grep out the non-zero lines each time we wanted to look at what was happening.

One of those days I got tired of this, fired up the editor, and wrapped the logging in an if statement to filter out all those zero-import writes.

    void runImport()  {

        List records = importer.listWaitingRecords();

        if (records.size() == 0)

            logger.log(Level.INFO, "Importing records: " + records.size());

        for ( Object rec : records) // imports each of the records in turn

   …

    }

A few days later we released during lunch, which was the best time for our users. The deploy went smooth, apps started up as expected, and soon it was time for the first import "tick".

Within a minute, import was run the first time, and the log line read:

<timestamp> Importing records: 0

While standing confused, a minute went by, and the log line read:

<timestamp> Importing records: 0

Another minute went by … and no log line.

While firing up my editor to check the code, another minute went by and a third log line appeared:

<timestamp> Importing records: 0

A colleague of mine looking in the database calmly reported "We just imported three records".

It only took me a split of a second to realise my mistake once I had the code in front of me.

    void runImport()  {

        List records = importer.listWaitingRecords();

        if (records.size() == 0)

            logger.log(Level.INFO, "Importing records: " + records.size());

        for ( Object rec : records) // imports each of the records in turn

    }

Obviously my spine reflex for coding "check for zero" made me write the boolean expression the wrong way around. By the way, did you capture that mistake at first glance in the code on my first description? Confirmation bias is a nasty thing.

Now, I must say this kind of bugs are pretty uncommon, bugs that are typos or simply mispunching the keys. Most bugs in my opinion are rather that pieces of code subtly misunderstand each other.

Well there are at least two ways of fixing this code. The obvious, and fastest would be to quickly change "==" to "!=". However, humbled by the mistake I had done in the first place, I realised that that kind of hasty coding was what got me in the trouble in the first place.

One of my coding mantras since long have been "Code should mean something, not just do something". So, a better way out would be to make the code more meaning-ful. From Eric Evans I learned the phrase "Make implicit concepts explicit", which says the same thing in this context, but gives a better guiding direction forward.

I took to the challenge of fixing the bug by finding what implicit concepts had been missed, and making them explicit until it was blatantly obvious that the code was wrong. An added benefit would be to be able to make a test proving the code was wrong.

Extracting the boolean condition to a method of its own would force me to spell out the meaning of that piece of code.

    void runImport()  {

        List records = importer.listWaitingRecords();

        if (containsRecords(records))

            logger.log(Level.INFO, "Importing records: " + records.size());

        for ( Object rec : records) // imports each of the records in turn

    }

    static boolean containsRecords(List records) {

        return records.size() == 0;

    }

Granted, this is more code than I started with - but I think the code is more "to the point" (phrased inspired by Rickard Öberg, another of the great programmers).

At least this made me able to write a test

    @Test

    public void shouldConsiderEmptyListNotContainingRecords() {

       Assert.assertFalse(ImportCron.containsRecords(Collections.EMPTY_LIST));

    }

Well, at least this is what the test would look today - at the time JUnit looked slightly different.

Anyway, now I have a failing test. Claiming "the empty list does not contain records" is simply false - as proven by the test. We safely update the code to fix the bug.

    static boolean containsRecords(List records) {

        return records.size() != 0;

    }

Upon which the test switched to green as expected.

During this "refactor -> put under test -> fix" something interesting has happened. The concept "import list contains records" that hitherto had been implicitly represented by the technical "records.size() != 0" has now been made explicit and given a name "containsRecords".

It might be claimed that the main benefit in this example was derived from the drive to put code under test before making a change. Undeniably, that is a point, but I think it only tells half the tale. 

Simply putting things under test can be done in a myriad of ways, and I have seen several very technology-based efforts. I do not think those efforts have paid off handsomely. Mostly the code get cut along technical lines to push in tests in the gaps. But, the code does not get more comprehensible over time.

Focusing on "make implicit concepts explicit" have been a more productive course for me when working with code.

Yours

   Dan

ps My team-mates did not force me to skip lunch to fix the bug. We went out together, and when we came back satisfied and rested, I sat down and fixed the code. We released the fix during lunch-time the day thereafter.

Make Implicit Concepts Explicit in Code

Dear Junior

In our letters on software development we have touched upon the idea several times, but I think it might be worth spelling it out - the idea to make implicit concepts explicit; something I see as one central message of Domain Driven Design applied to coding.

Let me pick that phrase apart for a moment. When coding we have a lot of concepts in mind, e g when writing code that handles people I might have a class namned Person. In this case the concept of person has an explicit representation in the code - i e an explicit concept.

A person might have a birth date, represented in code as a data-field Date dayofbirth in the Person class; another example of an explicit concept.

We might have business logic restrictions that are based on the age of the person, e g you have to be at least fifteen years to access some content. So there will be code calculating this.

    void contentRequest() {
        ... 
        boolean access = ((new Date().getTime() - customer.dayofbirth.getTime()) / msPerYear) >= 15;

Again we have a concept represented in code, this time age. However, this time the representation is not explicit - there is nothing in the code saying "age". Age is an implicit concept. 

There is nothing strange with having implicit concepts. In the short code snippet we have several implicit concepts: current point of time, point of time of customer's birth, and age-limit for content are just three obvious. This is not a problem. The programmer reading the code will intuitively re-construct the relevant concepts.

However, the design guideline from Domain Driven Design advice us to keep an eye open for important concepts - and if we find the represented implicitly, then make that implicit concept explicit. 

Let us look at the code snippet again.

        boolean access = ((new Date().getTime() - customer.dayofbirth.getTime()) / msPerYear) >= 15;

Of the implicit concepts dwelling in this like, which of them seem important enough to make explicit? 

The concept of "current point of time"? Probably not.

The concept of "milliseconds since birth"? Nah, not that either.

The concept of "milliseconds since birth expressed as years, rounded downwards"? Hey! I know this! We already have a word for it - we call it "age"! And we talk about it all the time!

Well, that sounds important enough. Let us start with giving that value a name. 

    void contentRequest() {
        ... 

        long age = (new Date().getTime() - customer.dayofbirth.getTime()) / msPerYear;

        boolean access = age >= 15;

By the way: this is where I really like the IntelliJ shortcuts "cmd-w" to widen the selection until the expression is selected, then "cmd-alt-v" for the refactoring to extract the selection as a variable - naming it "age" in the pop-up. It literally takes less than 10 seconds.

Now, we actually have made the implicit concept age into an explicit concept. Mission completed.

Well, not completely. We need to find the concept a good home where it can lead a good life. Right now it is stranded in the middle of some access computation. At least we can give it a method of its own.

Applying another of my favourite refactoring "Replace Temp with Query" yields this code.

    void contentRequest() {
        ... 
        boolean access = customerAge(customer) >= 15;

    private long customerAge(Person customer) {

        return (new Date().getTime() - customer.dayofbirth.getTime()) / msPerYear;

    }

Better, but can be improved to make the concept clearer. If we talk about the "age of the customer" it will vary from time to time, and it might cause confusion: the age when the customer requested access to the content, the age when content was first accessed, when it was last accessed, the age now? Better to clarify.

Of course we could rename the method to "customerAgeNow". However, I do not feel comfortable having my concepts depend implicitly on external things like the system clock. I prefer to make those dependencies explicit. So we make the time-in-question a parameter. 

    void contentRequest() {

        ... 

        boolean access = customerAgeAt(customer, new Date()) >= 15;

    private long customerAgeAt(Person customer, Date timepoint) {

        return (timepoint.getTime() - customer.dayofbirth.getTime()) / msPerYear;

    }

 

This design also improves testability drastically. The tests can now use explicit test-data, and need not rely on checking, or changing, the system clock.

Kudos once again to the lovely refactoring support of modern IDEs. Another less-than-10-seconds-refactoring: two cmd-W to select "new Date()", cmd-alt-P to extract as parameter, change name to "timepoint" in pop-up. Enter. Done.

By now it is pretty obvious that the method "customerAgeAt" suffers from feature envy. The most relevant data it operates upon is the Customer, but it resides somewhere else. We should consider moving it.

Should we move it, the Customer class would get a method "ageAt", taking the liberty to rename it slightly. That method certainly makes sense in this context, but does it so in other contexts? As we do  not have the rest of the codebase at hand, we will just have to pretend that "ageAt" makes sense in other context - and might actually be useful there as well.

This is really one of my favourite refactoring: move method. Again extremely smooth thanks to modern IDEs. Literally two keystrokes (F6, Enter - in IntelliJ) for the move, and a few for renaming. 

    void contentRequest() {

        // ...

        boolean access = customer.ageAt(new Date()) >= 15;

class Person {
    Date dayofbirth;
    // ...
    long ageAt(Date timepoint) {
        return (timepoint.getTime() - dayofbirth.getTime()) / TimeUtil.msPerYear;
    }
}

You might not believe me, but I remember the days of yore when you actually had to do all the involved steps manually; cut-n-paste the method body and header, change the list of parameters, update the code to use the fields of "this" instead of the argument, change all client calls (which might be several).

Now, we have finally ended up with a version I feel comfortable with. The concept "person" has now an conceptual attribute "age at" which has an explicit representation in code. In code we have "enhanced" our language of what we can talk about directly. So, when we discuss the age of a customer with business people, it is likely that we consistently mean the same thing - even if we for sure still can misunderstand each other.

As a side effect the code checking the access has become a lot clearer.

        boolean access = customer.ageAt(new Date()) >= 15;

This is a line of code that can be shown some person from the business side and explained by reading it out "access is granted if the customer's age at 'now' is at least fifteen years". Given that support, they can see by themselves - something that really builds trust. 

The time invested was not huge. Coding-wise all refactorings took less then a minute together. The time to slowly realise that "age" is an important concept in this particular domain is not counted. However, that insight will have to dawn on the programmer sooner or later, and when that happens, that extra minute is well invested.

To make concepts explicit does not only apply to code, it can be fruitfully applied to other areas as well, e g capturing and writing requirements/specifications. 

As we all stand on shoulders of gigants, I also must give credit where credit is due. The phrase "Make Implicit Concepts Explicit" I have picked up from Eric Evans, the thought leader of Domain Driven Design. 

In conclusion: to make implicit concepts explicit help us to over time close the gap between the code and the understanding of the business domain. It is not uncommon to in the process find bugs or crucial misunderstandings, that then can be adressed in a proactive manner instead of popping up as nasty surprises later on. 

Yours 

   Dan

Agile Projects should have Effect Targets

Dear Junior

Measuring projects and setting targets for them is a tricky business. And, it does not get easier when agile projects enter the scene. This is not really because agile projects are strange per se, but because they are different from non-agile projects.

Setting targets and measuring projects is also an important business. I have seen several agile initiatives fail. Most of them have failed because they did not succeed in setting goals for projects, and monitor their progress. And if you cannot do that, you quickly lose the confidence and support from upper management. Agile initiative terminated, or left to dwindle away. End of story.

However, it need not to be so. Agile projects can have measurable targets, and they can be monitored - you just have to do it right.

There are some bad news and some good news.

The Well-Established Chaos Rod for Measuring Success

The bad news is that agile projects cannot be measured using the de-facto established standard rod for project success.

The standard rod for measuring projects is "on-time and on-budget, with all features and functions as initially specified", as used by e g Standish Group in their much-to-cited Chaos Reports. Let us call this the "Chaos rod". Most organisations use some version of the Chaos rod for measuring their projects.

As we have discussed earlier it is pretty obvious that "all features implemented" is not a good way to measure "project success" - not for any project. Nevertheless, this is the standard rod.

Damned if you do, damned if you don't

Of course agile projects will fail if measured using the Chaos rod. The reason is simple - agile projects does not manage to keep their hands away from fiddling with the original specification. Agile projects remove stuff, change stuff, add stuff - agile projects are in a constant state of scope-creep.

This is no coincidence - it is how agile projects are designed.

Think of it this way: If we learn something during the run of the project - shall we let that insight effect the plan of what we intended to do? Or shall we ignore that insight, sticking to the original plan even though inferior? Of course we will adapt! Anything else would be ridiculous.

An agile project anticipates that such insights will emerge, so its processes are designed to harness and leverage upon those insights: demos, retrospectives, re-prioritisation of product backlog etc. These practises are all aimed at constantly refine and redefine the scope. But then, we have derived from the original specification "all features and functions as initially specified".

Thus, any agile project longer than a sprint will fail - by definition. That is, if you use the Chaos rod for measuring success.

To put it bluntly, if you measure an agile project using the Chaos rod, it will fail. Either the agile process will fail, or the project as defined will fail.

Either, the project adapts to its measuring rod and blindly follows the "functions as specified", throwing whatever insight gained aside. Then, "project" will succeed, but "agile" will fail.

Or,  the project will work according to its agile-minded processes, and then certainly the result of the project will not be "all features and functions as initially specified". Then, "agile" will succeed, but "project" will fail. I e project will fail as measured by Chaos rod.

Damned if you do, damned if you don't.

A New (or Old) Hope

The good news is that the Chaos rod is not the only way to measure projects. As we have discussed earlier, there has always been two ways to measure projects: feature list (Chaos rod) or measuring business effect, what we can call the Effect rod.

Let us take our earlier example with an on-line book store. They had an idea that recommendations of the type others-have-bought would increase their sales, so they set of some money for that projects.

Using the Chaos rod they would have created a feature list around others-have-bought recommendations. The project would later be evaluated on how well it implemented the list. But let us look at a better idea.

Using the Effect rod they might set a target that customers will by 0.35 more books per checkout on average. The Effect rod is used to state the value of a project. These 0.35 books can probably be converted to money, that makes the project worth the effort.

The project might start out with an idea that others-have-bought recommendations would cut the cake. After the first small stories, deployed to production and put into hands of customers, the team learns that some kind of category would be helpful. To facilitate this they implement a simplified "search similar" as one of their features.

Suddenly the number of books in the checkout carts raise to a level 0.4 above the pre-project baseline. And the level sustains, it was not just random noise - the change is statistically significant.

The project has fulfilled its target according to the Effect rod, and is declared a success.

Measuring using the Chaos rod "on time, budget, and specification" - the project would have been deemed a failure, because it did not deliver the initially specified others-have-bought.

So, agile projects can be measured. You just have to use a measurement that is well suited. And, to be honest - which is better anyway.

A sad side-note is that I know of several projects which have worked in an agile manner, and delivered enormous business benefit - but in the project report the project lead has had to apologise repeatedly for not meeting the project target as defined in the project specification - a feature list.

Granted, to measure projects on business effect is not a new idea in any way at all. The idea was certainly there before the Agile Manifesto was written around the turn-of-millennia. What is new is that this way of measuring is essential to provide rigour to agile projects.

For Agile to succeed at larger scale, we certainly need lots practises around agile-minded processes. Measuring agile projects on effects is certainly one of them.

The way to measure agile projects is to set targets for business effect.

Yours

   Dan

PS Interesting enough, there is a sub-category "agile projects" in the later Chaos Reports, but the rate of success is not 0%, it is around 40%. I wonder what is going on here.

PPS Within the agile community, the practice of projects is debated - at least in the sense of the common description "temporary organisation with limited time, resources, and ambition". So, it would probably be better to use some other term, e g talking about "initiatives". However, for convenience, I have kept the often-used and familiar term "project". Check out the twitter hashtag #NoProjects.