How to Deal with Unexpected Problems

We live in a cruel world, where a lot of unexpected things could happen. So people that build reliable software tend to be really pessimistic about it and plan for the worst. That costs a lot (think of all the atomic bunkers built during the cold war).

However, real life is much worse than any plans. It always proves our assumptions to be wrong. Expected bad things rarely happen, but unexpected weird things do happen. Here are some samples from software world:

database connections timing out;
Virtual Machines going corrupt;
customer bringing in 100x of load that you planned capacities for;
3rd party services replying at the speed of 1 byte per second;
somebody pressing the wrong button.

SLAs, probabilities and estimates always lie. They are nothing more than expectations based on the assumption, that worst case scenario would be as bad as the one you've seen earlier (or just little bit worse). It's impossible to plan in advance. Get used to it.

However, here's the approach that you can bet on (in fact it has been proven to be successful by thousands of years of evolution that brought fourth Homo Sapiens specie).

Stop spending time trying to prepare for everything in advance. This will be waste of time and resources (think of all the dinosaurs that didn't survive Ice Age, although some were quite packed with muscles and claws).

Instead, learn to evolve and adapt fast: reduce development and delivery friction and ensure that it is easy and cheap to make small changes with confidence. This will give you ability to react fast when life throws something unexpected at you.

Make sure that you know your world well - deploy early and have means to keep a steady eye on your deployments, to know what is happening inside.

Evolution is about survival of the fittest. So ditch ivory-tower dreams of being prepared for everything and just stay real and fit.

Who knows, you might even buy yourself some time to enjoy the sight of competition dying out due to being unprepared for the unexpected.

Published: February 21, 2012.

🤗 Check out my newsletter! It is about building products with ChatGPT and LLMs: latest news, technical insights and my journey. Check out it out