Deep Learning as a Strategy for Information Age

Dear Junior

Last friday we had the pleasure of having CS students Sofie Lindblom and Anton Arbring as guests visiting the monthly competence day at Omegapoint. 

After this visit, Sofie have done me the honour of musing on the theme of our letters by writing an open letter "Dear Senior, Letter to a Senior Programmer". 

That post is so full with interesting topics it would take a day just to briefly discuss them. But those topics are also way too important to leave uncommented. So, to do something, let us pick one important thing and discuss it. I pick the topic about "what to learn".

Too much information out there - as always have been

Sofie writes:

But there is too much information out there to know where to start. I am not stupid, I did very well in all programming courses and is a fast learner. But I feel exhausted by the amount of information available.

To start somewhere, let us start with the vast amount of information, technology, frameworks, etc that are out there. Obviously there is no way to take in all of that. If we want to use metaphors, it does not suffice saying "drinking from the fire-hose", it is rather to try to gulp the Nile. 

Good part is that the situation is not new. Of course there are more information out there now compared to 15 years ago when I left university. But even then the amount of information available was too much for any individual to comprehend. And the situation is even older. The proverb "so many books, so little time" is not fresh-out-of-the-presses. 

For those leaving university today, there will be truckloads of technology you will be using at work that you did not learn in class. But that situation is not new either. When I left university, I had not used a relational database in any single class or lab. Still, most systems (not all), I have worked with professionally have included SQL databases of some sort. Actually, one of my first jobs was to teach a class on the Java database API JDBC. How did I manage?

The obvious solution is to replace "know everything" with "able to comprehend". We cannot know everything beforehand, but we need to be able to understand any technology we come across with just spending a reasonable amount of work.

Killing a meme

There is a meme around in this information age that basically goes "you do not need to know, you need to be able to find information". I want to kill that meme in the context of system development.

The meme might very well be true, with Wikipedia and the rest of the web at our fingertips we will be able to find data like "first historically recorded solar eclipse" (5th of May 1375 BC in Ugarit). Nevertheless true, it is worthless to us as software professionals. Because what we need is not data or information, but understanding.

Deep knowledge feed deep knowledge

Now, this is only my own meandering experience, but I have found it invaluable to know a few things really well. Deep knowledge has interesting side effects. Suddenly you see some pattern apply to a new domain. It seems like no domain of knowledge is an island. Even if facts do not carry across boarders, some structures of thinking and reasoning actually apply.

This is really vague, so let me throw some examples to clarify. When studying law I suddenly found that my studies of formal logic really helped me. I studied negotiation theory and found how it applies to finding a good architecture for a software system. I studied compiler technology and found it helpful when studying linguistics. Through my lifelong studies of math, I see wonderful aspects of beauty in the world every day. (OK, the last is a little bit off topic - but it makes my life richer, and that is worth something)

The strategy I try to apply myself is to study subjects in depth, to the level when I have to think hard about them. The specific knowledge might not be immediately applicable - I will probably not have any specific use of knowing e g how to count the number of ways to paint a cube using several colours. However, thinking hard has probably etched new traces in my brain - and those ways of thinking will probably pop up applicable in a new domain.

To fall back on metaphors again. As software developers we need to dig deep to understand a new technology. To get down to depth we are not very helped by having dug a meter deep over a large area. But if we have dug a few 20 m deep holes in other places, there is a good chance that we can dig a short tunnel at the 20 m level from the bottom of some other hole. 

How did I survive that first job-gig teaching an API that I had myself never used before? Well, having studied functional programming in depth (using e g ML) had made me comfortable with the ideas of abstract datatypes. So the idea of an API was not unfamiliar. Having studied linguistics I was very familiar with formal grammars of languages so SQL syntax was not strange. Having studied compiler technology I could understand the semantics of SQL. Having studied algebra and set theory I could easily pick up how SELECT and JOIN worked.

It took me a few days to read the JDBC API and specification combined with some small hacks to validate that I had got it right. And after those few days I did not only knew about JDBC, I actually understood it well enough to be able to teach it in a reasonable way. Not being an expert, but reasonably competent.

Without the deep knowledge in some very obscure subjects (linguistics, set theory, compiler technology etc) I would have been utterly lost. No skill in "searching information on the web" would have helped me the least.

Sofie writes

The more I learn, the more I realize how little I know. It creates contradictory feelings towards the field I love. To twist it even further the part I love the most is that you can never be fully learned and that there is never a ”right” answer.

I understand the frustration. But I am not sure I would like to have a field where there was a right answer, a proven best practice - many in our field dream of such. 

However, to me a large portion of the beauty of the field is that we are constantly pacing unchartered terrain. The challenge is to constantly search your tool-box for something that seems applicable, to adapt, to improvise, to search, to try, to fail, to back up, to learn, to grow, to try again, to discuss, to exchange ideas, to finally nail it.

This is nothing but my own personal experience, but if I should offer an advice to handle the world of information we have around us it would be the following:

Find things to learn that you find interesting and that challenge your intellect. Take the time, pain, and pleasure to learn a few of those things to depth. The deep thinking will etch your brain in ways that will help you enormously whenever you approach a new field. And enjoy the pleasure of deep understanding when it dawns on you.

Yours

  Dan

PS Should you come across Sofie and Anton, take the time to have a discussion with them. And do not stop at a chat about everyday things - they have really interesting ideas to delve into.

Psychology of Tests, Testing and TDD

Dear Junior

There have been lots of talk about the benefits of test driven development, but there is one element that I think is interesting that I not have seen a reference to - the psychological process.

Pretend that we dive into the head of a programmer (e g me) and see what happens during testing. Let us firest have a look at "classical" post-fact tests and then at the TDD way of writing the test before the code.

If I write some functional code and consider myself finished I will without doubt feel immensely proud of myself - in other words my ego is on a high note. Then I sit down to test it, and let's see what can happen: either I find bugs and flaws, or I don't. If I do not find any bugs, I do not get any extra credits in my own books - I have been exactly as good as I thought I was (but I might feel that I have wasted some time). If I do find bugs I get a slap in the face - I have not been as good as I though I was (and have committed hubris, which is a severe sin on the list of things considered as sins in Sweden). So, even if testing will improve the general level of the result, the activity of testing (per se) will never give an upside, can at best have a zero outcome, and at worse a big minus. Depressing.

So, how do I treat activities that give me no upside, but possible downsides? Naturally, I tend to avoid them, and over time minimize them.

So, I tend to shy away from testing and writing tests; even though I am firmly convinced of their necessity. Depressing.

Driving the development through tests breaks that negative feedback loop. If I write the test before the code, naturally it will fail: I have no other expectation, and the red bar is not a failure. Later on, I write some code, which make the test go green, and I have actually finished on a high note. Great!

Well, actually I have not finished as any TDD-savvy would quickly point out. I have not refactored away duplication and other code smells yet. But doing so will satisfy another part of my ego, the code aesthetics lover that resides inside me. So, satisfying that part of my ego will end the job on a yet higher not.

OK, to be brutally honest: are there no part of TDDs "red-green-refactor" that have the same "can only go wrong" drawback as traditional test-last? Well, there actually are: the refactoring. At best it says: you did not mess up, at worse: you think you where so smart but you failed. So, how come I do not tend to neglect refactoring?

Well, I have already given the answer: I love refactoring, if nothing else for the pure love of nicely crafted code. But also, the refactorings are not only the last cleanup of one part of the job. They are most often also the stepping-stone from which I launch into the next part of functionality to write - so I get immediate feedback from the refactoring as well.

I think writing tests in general, and unit tests in particular are really important. Considering how many that agree, and how seldom it is practiced, we must look for explanations: ignorance, arrogance or malice do not suffice. To see how we experience "writing tests" has helped me to shed some light over the mystery.

Whatever, I think one of the drives for me to do TDD is that is make all parts of the development cycle to be fun.

Yours

Dan