Hamann Distributed

Making mistakes at scale so you don't have to.

Dear Lucas

Yesterday, I found an E-Mail in my Inbox.

Very impressive profile, Dominik! Any advice for a relatively young programmer who would like to be someone with similar skills in the next 10 years?

Lucas W.

Now, I admit I was a little flattered. I really don’t think I’m that good, but looking back at the last six years and realizing again how I went from a broke, bipolar and triple failed university student to Lead Architect and eventually CTO and Co-Founder within this short timeframe, I can’t help to think that there must be something I did right and I could share with others. My impostor syndrome would beg to differ, but in fact there wasn’t so much magic involved – but lots of hard work, meeting the right people (among many more wrong ones) and taking opportunities that popped up along the way, even if they had some risk involved. I could totally do it and so can you.

My first intention was answering him personally back on Linked.in – but from my experience here’s one tip for starters already: If someone asks you a question that many more people might have, do yourself a favor and write it down for posterity. Put the answer into your code, wiki or blog for everyone to read for reference and you might make more than just one person’s day.

So here’s my advice to you, Lucas. I’m sorry, there is no TL;DR. There are no shortcuts. This is the real deal.

Ambition and Culture

Lucas, you asked me for a 10 year career path. I think you should be bold and shoot for what you can achieve in one year instead. Before I go into any of the numerous technical aspects of your question, I think there is one important lesson for you to take so far already – which is: Don’t stand still. The fastest way to learn ANYTHING is to learn it from people who already know what you want to know – and are great at both teaching and learning. You need to surround yourself with those people if you really mean it. These are always people who are “better than you”, and you should try to get them to mentor you.

How to do it? Simple. Approach them even if you’re not working directly with them. Be nice, interested and try to solve some of their problems on your own. Ask them good questions and come back when you’ve done your homework to ask some more. Don’t ever ask them stupid questions, which you could answer yourself by Googling. Don’t ever ask a question twice. You know you’ve found a good mentor when you get continuously harder tasks and candid feedback from them. Within your company, try to get into the fastest and most respected team. Whenever you think your learning curve has peaked and is on its way down, look around for new opportunities.

In some great companies, it’s possible to evolve into ever more challenging jobs with increasing responsibility, but unfortunately often it’s not. When you feel like you’re hitting the glass ceiling, move on and look for something else to continuously broaden your horizon and stretch outside your comfort zone.


High degrees and certificates will only take you so far in programming and often aren’t worth the hassle. I once did the Zend Certified PHP Engineer to prove I wasn’t a total failure in programming, but I skipped the expensive course and just sat down with a book and some test questions to prepare. More importantly, after years on the other side of the hiring table I’ve learned that people with the most certificates and the highest degrees are more often than not the least productive ones, because somehow, a lot of them seem to rest on their past successes and stop learning.

Remember, you’ll be hired for your papers, but you’ll be promoted for your output – so don’t spend too much time on the former.

Read Hacker News

Why Hacker News? Quite simply, it’s where some of the best people meet and share their knowledge about the whole technology ecosystem. There are other aggregators, maybe even in your native language. However, you should really start embracing English as your work language if you didn’t already and the best way to become proficient is reading and writing a lot.

Some people say Hacker News has become boring and repeating. These are usually the people who have become experts themselves by reading there for years – I still find lots of new, interesting and important information about technology and startups there.

Read about half an hour every day to stay on touch and learn something new, then get on to work on other things. Reading more won’t make you better after a certain point, only practice will.

How to learn

I’ll give you a truckload of things to learn below, and learning is the most important thing you will ever do as a developer. But remember that learning works best (some say: only) as a spiral. You’ll learn something, not quite understanding everything and move on to the next topic. Maybe you’re a little frustrated. This goes on and on until you come back to the first topic. You’ll look at the topic again and see that a lot of things explain themselves through the new connections, abstractions and transfers you made on your detours. This spiral will continue forever. If you’re not embarrassed by work you did years ago, you probably haven’t evolved or developed much.

Don’t try to learn any of the topics below “fully” before moving on. It’s simply not possible. Try to get one focus area every day and rotate.

Get used to the feeling of not understanding certain things. It’s normal. You will eventually.

Get used to the feeling of not knowing the exact paths to the goal you’re after. It’s normal. You’ll find out on the way.

The Mix

Then there’s “the learning mix”. It’s the optimium mix between reading, working, toying and communication.

Try to get the balance right, because combining all four belong to becoming a great developer.

You need to read, because there is a lot of information that you need in order to make good choices in every possible field.

You need to work because only applying your knowledge to a concrete codebase will complete the learning cycle and give you actual experience.

You need to toy and play. No, I’m not kidding. This is also work but it’s disguised in a much more undirected format, often with no clear goal at all. This is all about looking behind the curtains, trying unusual solutions, long shots which most probably wouldn’t ever work (and finding out why). Finding shortcuts or doing things with frameworks, languages or technologies that are not used at (or not even recommended for) “daily” work. Lots of people – especially women – are conditioned quite early by society not to needlessly toy around but to be “serious” – yet toying around is essential to becoming a great developer.

It’s really about expanding your horizon and it’s even appropriate to do some of these things at work: Small tools and one-off tasks are great use cases for more experimental technology. Some of the more time consuming things you should do better do in your free time.

Last but not least, you need to communicate. A lot. You need to ask, tell, understand and be understood. Also, get the mix right. You should hang out about one third within your core technology group (other devs, mentors, bosses), one third within extended technology (other teams, admins, BI guys, meetups) and one third outside of technology to understand other business functions and their needs better.

Lunch is usually your friend.

Learn 10 Programming languages over the years

Why ten? Well, I just pulled a number out of my hat that feels about just right to start to get an intuitive grasp on code. You’ll learn that all programming languages share some features that are common but all have distinctive advantages and disadvantages in different situations. It also helps you get over stupid language wars and see the other sides of the table. It’s really about developing a technology “feeling” that will help you make good decisions later. Make sure you get some diversity again.

Choose something low level like C or assembler, something high level with a framework (Rails, Django), something object-oriented (Java, Objective-C), a functional language (Haskell, Clojure), something mixed-paradigm (Go, Scala), something arcane (COBOL, BASIC) and something brand-new or experimental (Rust, Nimrod). Choose a language targeted towards certain niches (Julia, Ada, Autohotkey <– don’t laugh, it was my venture into programming), and maybe something exotic like Brainfuck or LOLCODE. Be sure to learn JavaScript and why it’s both ingenious and awful at the same time. Learn at least the basics of SQL, HTML and CSS – I don’t count them as real languages, but you’ll come across them pretty much everywhere, even if you don’t see yourself as a “web developer”.

You don’t have to learn them all to full proficiency, but you should try to get something specific done in every one of them and move along. Stay around some more time with languages you like, learning them at a deep level – and leave others behind, mentally noting why you didn’t like them. This is really subjective. I for example can’t stand Lisp dialects because I find the parentheses ugly and the syntax unreadable. You will find other programmers totally excited about its expressiveness.

I won’t tell you PHP is a bad choice, lots of others will happily do that for me. Facebook, Wikipedia and Wordpress are examples showing you that the language works. Yet not everything which is widespread is great as well. Don’t believe others, find out yourself. PHP used to make me happy, fast and productive in my early days, but it doesn’t anymore.

Your preferences will change as you get more experienced. Often, beginners start with dynamic languages and hate types in the beginning (mostly because you don’t fully understand them and they’re a hassle to work around), yet start to appreciate them when you get more experienced and work in larger teams. YMMV.

Learn about data structures and databases

Data is everything you will ever create and move around in IT. If you are a self-made programmer like me, you will have most headache but also lots of leverage with understanding data structures. This is the boring stuff of every CS class which you cram into your head and then forget – but it’s only really interesting if you come across concrete problems where you’re running into the limits of your data structures. You should definitely know what sets, maps, arrays, trees, tries and lists are, how they look in computer memory and what their complexity cost in Big-O-Notation is. I found it much easier though to learn them on the job when I actually used them.

Databases are important as well. If you don’t know anything about databases yet, don’t be a hipster and just use Postgres. You’ll be sure to learn something solid, performant and generic which will stick around. But if you’re already planning beyond Gigabytes of data, hundreds of concurrent connections, below a hundred milliseconds of latency or all of the above, you need to understand more scalable database technologies and their limitations. Read about MapReduce, DynamoDB, BigTable, sharding, elasticity, indexes, immutability, ACID and BASE, and these are only the very beginnings.

To really make good technology decisions, you need to talk to your business about the eventual requirements of the system, because you WILL have to make some tradeoffs eventually and you should be aware of them. Think of good questions before you try to find answers, i.e. “How often will data get updated?”, “How much downtime is acceptable in a day / year / minute?”, “How fresh does the data need to be?”. I can give you 20 of these questions drilling down into your real needs right now, and all of this will reflect in the architectural choice I will make.

I was once asked by a CEO why we couldn’t just “put everything into a really fast database and then get results from arbitrary queries in realtime” (note that he was talking about 500 GB a day). I told him that he could sell our company for several billions if we had that technology.

Learn about your craft

In the middle of this page, you will find a mind map about everything you should master as a Rails developer. Forget the “Rails” part for now, 80% of this map describes the neccessary skillset for every language and framework you can think of. You should spend at least a few hours for every single leaf of this tree to understand the concepts behind them. A developer who can program perfectly but can’t use VCS, Continuous Integration and Testing in an Agile Environment is pretty much useless in 2014.

Really, good craftsmanship cannot be underestimated. You should choose to (deeply) learn a powerful editor like Vim/Emacs/Sublime/IDEA and learn 10 finger typing. I bought this keyboard and learned it the hard way – within 2 weeks and at the venerable age of 26 years. Both measures roughly tripled my coding & writing performance and I didn’t ever look back.

Learn about people

Dealing with people is the single most undervalued skill in IT with the most leverage. This is both the shortest and the longest chapter of this text, because everything that needs to be said about this topic has been written 78 years ago already – and the only thing you should really do is get a copy and read it. Don’t be put off by the title, remember this book is as old as your grandparents, yet people didn’t change a single bit…

Learn about your business

Someone who can take any spec and write it down into working, readable and performant code is a great programmer.

Someone who can take any business problem and write it down into a well-modelled spec before coding it down is a great engineer.

You’re free to stop at being a good programmer, but good engineers are paid double the price. Really, the top three skills you can learn if you want to progress your career are

  • Taking a business problem and turn it into a working software model

There are lots of good ideas out there, yet execution of all of them depend on your technical transfer skills. Nothing is quite as important as learning to read the language of business people, form them into a domain model and explain and reevaluate the model together with the business people before even writing the first line of code. A great intro to this topic are the first chapters of Eric Evans’ “Domain Driven Design”. This also means that you should get familiar with the business itself that you’re currently in. Learning which KPIs are important and have the most impact to your business will help you find much better technical solutions.

  • Explaining technical dependencies and challenges to non-technical people

It’s easy to underestimate how hard this really is. Yet, the ability to explain everything to a 5-year-old and negotiating features in order to keep down complexity are incredibly important for you to learn. A great example I had: Product Managment specified that they wanted a user that had seen a banner not to see it more than five times within the following 24 hours. I asked if it would be okay to only show the banner to them five times per 24-hour-interval. They said “but that’s almost the same!”.

Yes. From a product side this was almost equal, but of course technically, the latter was much less complex, saving lots of programming hours and computing complexity by simply rethinking the specs and providing an alternative. Learn how to teach the concepts of technical debt, technical investment and software rot and incorporate them into your estimates. Speaking of which…

  • Making good estimations

Estimating is usually one of the things even great programmers learn last. Yet it is crucially important to know for the business how long things will take. Learn to express yourself in terms of complexity and uncertainty. It’s helpful to continuously practice the estimate-observe loop with your own tasks and also learn about how large and complex projects are fundamentally different to a one-man-show. A great rule of thumb for larger projects from “The mythical man-month” which I often use is: 1/3 of the time is spent on design, ¼ on early testing and debugging, ¼ on late integration and acceptance testing and only 1/6 (!) on actual coding.

Take this into account and multiply the resulting time by 1.2 for every team member before dividing through the team to account for communication losses. If you have time, read the whole book for some great insights on planning software projects. Same thing here: You won’t believe how accurate most parts of a 40 year old book on software development still are today…

Learn to glue

30 years ago, if you wanted to write an app, you wrote everything yourself, from the device driver routines to the graphics buffer. Fortunately, thanks to the Github revolution, nowadays you can find an open source package for just about anything. And systems are so much more complex now that you’d be stupid not to. Not just that – most of the times, these libraries are written by people smarter or more experienced than you are.

On the other hand, with great power comes great responsibility. These packages often will not exactly do what you want or what you need and you’ll have to make important decisions. Do you commit back to the project and hope for a pull request? Do you integrate parts of it as an API? Sometimes it’s a good choice to go for a full stack framework. Other times it’s better to hand-pick and compose some smaller packages to achieve success.

On top of that, you will have to manage the dependencies as well. They will get out of date, glitches and security bugs will pop up and often they might have changed their interface inbetween so you have to change all your code in order to upgrade. Some code will rot and get abandoned. If it’s a small library, that might be okay, but if it’s your CMS or the fancy bleeding-edge framework you built your website on back then, you might be screwed.

If on the other hand you choose to build everything yourself and end up with a huge stack that’s reinventing the wheel instead of using standard components, it’s getting continuously harder to onboard people, up to the point where I’ve seen this literally killing companies.

All these problems require a completely new skill, which is searching, composing and glueing software components to your own stack the right way. There’s even a conference solely dedicated to this topic, that’s how hard this is in itself. If you have several repositories doing what you want, you need to be able to estimate their traction by evaluating the number of stars and forks, frequency of pull requests, number of open issues, commit frequency and date of last commit and if it is maintained by a company (yet not too much to cripple it), just to mention a few.

Get good at this stuff and if you really decide that you write a component yourself, you should already know the three best repositories closest to solving your problem and how they are not cutting it quite yet for your use case.

Learn DevOps

In the days of Infrastructure as a Service, it’s become so easy to start up your own server / cluster / database, that there is simply no excuse not to try it yourself. The entry barrier is so low through services like AWS or Heroku, that you need to be proficient in provisioning a prototyping environment yourself. Some people “admire” my Hadoop skills and complain they never “had the opportunity to work with it” before. I call BS. For just a few dollars, you can run your own map-reduce jobs on Amazon EMR and experiment with huge freely available open-data test sets if you were really interested in it. Get an account NOW and understand which problems the dozens of AWS services solve before you try to roll your own.

Learning how to properly operate, monitor, deploy, secure and maintenance real server systems – even superficially – will get you a much deeper understanding on how your code will actually run and perform when in the wild.

Also, you’ll get some humility for Admins who do this stuff full time.

Learn about technologies like Vagrant, Docker and Fig and their underlying concepts of immutable infrastructure to get into the modern mindset of 12factor apps.

Yes, all of this didn’t even exist in 2011.

Yes, it’s hard to continuously follow these important trends and keep up with the pace. You signed up for the ride.

Go on from here

Lots of what I wanted to say as well has been already said – for some more good stuff that I didn’t cover, go over here:


I couldn’t have said these things any better myself, thanks Peter.

Thanks for reading and I’m looking forward to see what your career has to offer.