The Long Line Behind Every Idea
Why every system, from philosophy to software, inherits its shape from what came before - existentialism, Tailwind, Gall's Law, and Canada's failed payroll software.
I first got familiar with existentialism as a philosophical school during the pandemic.
Long walks were my only source of entertainment, and after I caught up with all my technical podcasts and Hardcore History episodes, I turned to topics that had long been in the periphery of my interests.
I clicked on a random philosophy podcast episode about existentialism and spent the next three hours learning about Sartre, de Beauvoir, Camus, and the meaning of life. But throughout the whole time, I had one question on my mind.
How did they come up with this?
Do you wake up one day and start thinking about the meaning of life out of the blue? How do you get such detailed thoughts on such a niche topic?
After that podcast episode, I went down the existentialism rabbit hole and noticed that philosophy books rarely dive into the main topic straight away. They start way back with the philosophy schools before what the book is actually about, and build up to the idea they want to present.
Sartre is responding to Heidegger, who is responding to Husserl, who is responding to Kant.
When you reach the part about existentialism, you have all the historical context about why these people paid so much attention to the meaning of life.
I’ve had many of the same questions about software development throughout the years.
How did they come up with this obscure library?
Why did they create this niche tool?
How did they end up with this architecture?
The Evolutionary Path of Technology
A few years ago, I came across Adam Wathan’s article about utility CSS classes as I was ready to go to sleep. Weird bedtime read, I know.
Had someone sent me a link to Tailwind, I would’ve just closed the tab and left the message on “seen”. But Adam describes his idea the same way philosophy books ease you into existentialism.
He builds up the context about semantic classes, the problems with them, and when you read about his alternative, you want to jump out of bed and go write utility classes. I certainly did.
He doesn’t just explain what he’s building, he’s explaining why he’s building it
By the end of that article, everything had clicked. While I was still skeptical of the paradigm, I gave it a try, and to my surprise, it worked! I became a huge Tailwind advocate, and I wholeheartedly believe that it makes for the most maintainable styles.
The more I read about art and science, the more I see how everything we come up with is a response to the trends and movements before it.
Ideas don’t come out of thin air.
They don’t spring out of a vacuum. Maybe Nuton is the only exception, but even he had the aid of an apple.
Gall’s Law and the Myth of the Rewrite
If I can make a slight logical leap here, I think skipping this evolutionary technological path is the underlying reason that makes Gall’s Law relevant in software engineering.
“A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.”
— John Gall
John Gall was a pediatrician with a front-row seat to the bureaucracy in the medical field, which showed him how systems fail and succeed in real-life conditions.
He noticed that complex systems have too many interconnected points that designers can’t anticipate, and this naturally leads to failures. This law has become almost gospel in software engineering, and you will hear it cited as an argument against big-bang rewrites or overly ambitious architectures.
I find it to be true.
You can only have Tailwind’s utility classes after you’ve struggled to maintain semantic class hierarchies for years. No one wakes up one morning thinking:
“You know what would make CSS maintainable? Having 20 shorthand classes on every HTML tag that require their own build pipeline and tooling. That’s what.”
The Phoenix Project Disaster
Building a complex software system from scratch is akin to coming up with a fundamental theory without as much as an apple to hit you on the head.
Canada attempted to do that with its Phoenix project - an ambitious government-wide payroll system that became a total disaster.
Can you imagine writing software that bad that people protest against it?
What was an attempt to save taxpayer money turned into a black hole for funds that has cost more than $5.1 billion so far, and still has a backlog of 400,000 unresolved issues.
That’s not just bugs or problems in the UI, too.
These are underpayments, overpayments, and, in some cases, no payments at all for public servants. The government has already awarded a contract to another company to replace the Phoenix system without even being able to integrate it properly.
It’s a modern Tower of Babel story that needs to be studied in every software engineering class.
Small-Scale Catastrophies
A company I worked at won a bid for an ERP system, and even though I was working on smaller peripheral projects, I saw the development of that product devolve into chaos in real time.
The engineers made a grand design for the system up front, but I could see their initial enthusiasm evaporate with each month. They constantly had to refactor and rework core modules because they found use cases they hadn’t thought of before.
“In this one office we do the process differently. Ah, yes, this supplier that requires all their payments up front, sometimes we do that.”
They patched up edge cases by the dozen.
The new system became slower and slower, and engineers quit taking that hard-fought domain knowledge with them.
What the company planned to ship in a year took five years, churned some great developers, and its initial budget looked like a down payment.
Scaling Understanding Before Scaling Systems
If we go back to Canada’s Phoenix payroll system, a software development approach that would satisfy Gall’s Law would’ve been to handle the payments for a single city or a small town first.
They could get a foot in the door and understand the government payment system on a smaller scale without all its edge cases on a nationwide scale. They wouldn’t have to tackle the complex domain modeling, scalability, storage, and infrastructure all at the same time.
It’s easy to design a system on paper only for teams to start arguing about ownership and domain boundaries. It’s easy to scale a system on theory only for problems to arise where you didn’t anticipate them.
“But I’m a great engineer and I can foresee these problems!”
— someone who can’t foresee them
That’s what everyone thinks. Everyone who has failed to design a complex system from scratch has said these words. A part of me certainly thinks the same way, too.
But the data says different.
I’m sure that the engineers working on Phoenix were competent. But creating a project of this magnitude, a system that spans an entire government, that’s an uphill battle even for the strongest engineering teams.
Magic Arena and the Power of Incrementalism
A good example of how this is done is Wizards of the Coast’s Magic Arena, a virtual version of the popular Magic: the Gathering card game they created as a successor to their older Magic Online platform.
Magic is a game with a 30-year history, rules too complex for AI to play, and more interactions and edge cases than you can think of.
So Wizards (intentionally or not) used Gall’s Law to their advantage.
When they decided to modernize their digital experience, they didn’t pull the trigger on a big bang rewrite that needed to achieve feature parity with their older platform.
They started Magic Arena with a single game format that only had the most recent cards and left the older ones in MTGO.
That meant less complexity to handle, fewer edge cases, but still a working product.
Eight years later, they still don’t have all their formats in the new platform, and it will probably take them eight more years until they do. But they have a profitable game that’s making their playerbase happy.
I Can’t Say “Context” Enough
Companies that adopt microservices from day one think they’re also adopting the hard learned lessons from the past, but by skipping a service-based architecture first, they don’t take into account the evolution of the company itself.
These architectures are never applied “by the book”.
Every organization ends up developing its own flavor based on its history, its constraints, and the scars it has collected along the way.
I saw a great example of this during my time in the media industry.
Software teams in media rarely use single-page applications for their user-facing products. They still used React as a templating engine and a way to add interactivity to a page, but they didn’t take advantage of client-side routing.
“Surely, these people don’t know what they’re doing. I need to show them the way.”
I struggled working with that architecture because everything in me told me it was wrong.
It took me a while to figure out that:
Users browse websites atomically
Caching and scaling benefits far outweigh those of client-side routing
Pages have limited interactivity
People knew what they were doing. Their ideas didn’t come out of thin air. I just didn’t have the context to understand them. I had a knee-jerk reaction, I didn’t like the architecture, I questioned it, I proposed alternatives, and put the occasional rewrite that I could do in a month on the table.
But after some time in the domain, things clicked.
I understood the business, the constraints, the history, and the “why” behind the “how”. This project worked because its architecture didn’t appear in the tech lead’s dream one night - it was based on its simpler predecessors.
MVPs Are Misunderstood
You have to pick a starting point, though, or you’d have to build yourself a computer first in order not to miss an evolutionary step.
This is where the concept of the MVP comes in, a largely misunderstood idea in large companies.
The “minimum viable product” is a startup’s way to validate whether an idea has legs before they go all out to build it. Many corporations adopted the name, but only went as far as using it for the first version of their product without applying the philosophy behind it.
The MVP can be used as a starting point for this evolutionary process. What’s the smallest scope that allows you to start the learning process?
For Wizards of the Coast, this meant the smallest game format. For Canada’s payroll system, it could’ve been a single city. For your company, it’s something entirely different.
Breadcrumbs for Future Engineers
Documentation goes a long way to help people understand this evolutionary context and not reinvent the wheel, but when people hear that, the Confluence logo pops up in their mind, and they get an allergic reaction.
Documentation doesn’t mean long, boring writing that people need to summarize with an LLM.
All you need is a Markdown file in a repository.
You can write it in the README, that’s good too. Just find a place where you can leave breadcrumbs that people can follow. Don’t enforce a format, don’t create a dogma around it - write things down in an obvious place.
Each doc should answer three questions. Who? When? Why?
That’s all you need.
Imagine that you’re providing the backstory so that when a future colleague of yours stumbles on your work many years later, they can understand what you were on about. Otherwise, they’ll have many of the questions I did when I first started reading about philosophy.