It's Not Technical Debt, It's You

Nothing states this more clearly than resumes developers write. Take a quick look at the resumes you have recently received, or even your own resume, what do you see listed mostly? A list of programming languages, frameworks, and server technologies for the most part, some previous employers then a couple of words about projects you worked on.

It’s as if the tools we use define us as professionals and define the products we build with them. When you see the title An amazing [PRODUCT NAME HERE] written in Rust, you think of a product that is fast, reliable and probably more secure. But the language used in and of itself does not guarantee that, and it also does not preclude other languages from producing products that are also fast, reliable and secure.

This tool-focused thinking doesn’t sit right with me. And to use a naive analogy, a carpenter’s resume would not list brands of power tools, would it? It probably would have a nice catalog of showcased work and testimonials from satisfied customers.

Unfortunately, this fascination with tools is not restricted to developers themselves; CTOs, CIOs, Sales, and customers share the same interest. Due to the promise associated with technology names.

Don’t get me wrong, everyone should use the right tool for the job, dedicating time and effort to master it, and customers should be interested in the details of the technology that drives their businesses. However, we need to realize that this over-interest is probably one of the major sources of inefficiencies in our software teams. And the fact that this over-interest is shared by so many departments in companies is why pointless work—work that has no place being allocated time for—is sometimes given top priority in planning sessions.

How often have you seen a perfectly working solution being refactored because it was legacy, and that technical debt needs to be repaid? How many perfectly working HTTP APIs got replaced with a gRPC API because the application needs to scale? I can go into countless examples I’ve seen –I kid you not- of rewriting the whole codebase to make it “more modern”, to changing whole backend DBs from traditional SQL to NoSQL. I’ve seen message-based systems replacing perfectly reasonable sync HTTP. All of these had reasonably-sounding technical rationale behind them, revolving around repaying the Mighty Technical Debt.

The problem was that not a single one was really necessary. And the cost of rewriting, retraining, stabilizing, overcoming unforeseen new bugs and pitfalls was definitely not worth it, I’ve witnessed a company of 300 employees collapse under the cost of rewriting its core product to make it more “modern”. And in all of them the technical goal could’ve been achieved by simple optimization techniques or minor to moderate refactoring. There are of course legitimate cases for rewrites, and in a few cases, I’ve recommended them, but those were 2-3 cases during a career of 20 years.

In this article I will not discuss the definition of Legacy or Technical Debt, I will not try to find a reason for our fascination with frameworks and languages, or engineers’ overwhelming desire to destroy and build anew. Those discussions require their own space that is not here.

I will however try to list and discuss the reasons why software teams do convert their fascination into (un)practical software changes. And I have identified a few general reasons. Some might lump them into less, and others might want to split them into more, but I'm satisfied with my list, so I’ll go with it.

---

1. Resume/Promotion Driven Development

All professionals want to beef up their resumes, and developers are no exception. Perfectly normal. The issue arises when developers start making decisions based on what looks better on a Resume or a Promotion Document rather than what delivers best value to customers or a company.

So, you’re building a new API for an internal component, the whole company uses HTTP APIs, they work perfectly well, developers understand them, everyone can debug them, tracing tools are fully compatible with them. But you really want to put gRPC on your resume, or you really want that promotion, and a big chunky paragraph on your promotion document on how throughput got increased with gRPC does sound tempting, so you convince your teammates and go for it. Now, whether gRPC delivers on its promise, or if the overall system got a benefit of the gRPC API for one of its components, well, that’s secondary.

And of course, the extra bugs, the extra work needed to adapt observability tools, the extra time needed to retrain developers on the new technology, is often invisible to the organization. Unless it has the maturity of process to see those usually difficult to spot inefficiencies.

This motivation could also manifest in misplacement of effort and resources on a broader scale.

In a previous company there was a quite an unstable component, it failed spectacularly every couple of weeks, and on every failure a bunch of developers needed to work until 2:00 AM to get it fixed. In addition, this failure would frequently cause a delay of feature deployment to users.

I was an Engineering Manager during that time and I was talking to my manager and arguing to allocate some time to get this component fixed once and for all, it would take us a few weeks from 1-2 engineers, and we could get it fixed. This would boost team morale, and deliver value to our customers by giving us more time to focus on delivering value rather than propping up a collapsing component.

The response I received was Yes, but such work will not help the developers get promoted, working on [this new technology] will. Let’s allocate the engineers to work on the new technology. It was quite blunt to be honest and made it clear how misplaced –IMHO- the priorities in that company are.

Later I realized that this manager was looking for a promotion himself and he knew that overseeing a switch to that trendy tech is going to help him get it.

That made me realize that Promotion Driven Development is quite a powerful force as it has a bubbling-up effect. The work done on trendy tech also bubbles up to the manager’s own promotion document and creates an incentive for the wider team to misplace their priorities.

---

2. Sales Driven Development

In an attempt to boost sales or undercut competition, CTOs and Sales reps push for trendy tech to be part of the stack. If customers are asking How will your platform handle our huge traffic? or How will you handle our geographically dispersed customer base, they better have the word Cloud in their answer, preferably with a big provider name attached. They might not be able to close the deal otherwise, especially if their competitors have it all over their website and marketing decks.

This demand could also be generated by investors, when talking to the CEO or CTO, they might explicitly demand -AI for example- be part of the tech stack otherwise their investment might be in jeopardy.

So, in the next company all-hands you’ll see a few slides on AI (Cloud is old news anyway) and the need to incorporate it into the offerings. And that’s OK if there is a natural place for it, if it solves an itch for customers. If it’s just for sales, then I would expect that We bring 20 years of proven industry experience, a fast, reliable and user-friendly platform that our customers love, and integration with over 1,000 partners—making us the top choice in the market, if not the Only one. is a better sales pitch than AI, AI, AI! but that’s just me.

I’m not judging salespeople, nor am I diminishing the real pressure of market hype. Just advocating for these pressures to be labeled correctly and used correctly when discussed with product and tech teams. For example, a feature requested by Sales could be labelled by them as Hype Response, or Deal Breaker, or Sales Booster. Denoting its value in the sales cycle.

Proper, open, and honest communication is essential in companies and labeling everything correctly can drive a meaningful discussion where solutions could be made that serve all interested parties without sacrificing company efficiency and technical soundness of its products.

---

3. Curiosity

We developers like to tinker, it's part of what defines us as developers, we are also a part of a wider community of constant innovation and sharing. We are bombarded daily with news about cool tech, or new patterns, or a fascinating Cloud Offering from AWS, Azure or Google. And we just want to try them. See if we can solve the problems we’re facing at work with them. Our batch processor is quite slow, maybe if we rewrite it in Rust it’ll get faster. Our CD pipeline is quite clunky, maybe we should move to Spacelift, see how it improves things.

Of course, rewriting components and migrating pipelines require more than just curiosity from a developer, it's a lengthy process that involves many parties, but curiosity could be the trigger and initial motivator.

The scope of change could also be smaller, a developer could silently try to use a new JS framework for her independent task, and while she’s at it she could make the backend API a GraphQL API, because why not!?

As we’re making such decisions, we should be aware of our motivations, developers impulses need to be respected and even encouraged, this is how you build motivation and innovation in teams. However, as developers we should always be self-aware and reflect on our motivations, ask ourselves if this change that we’re introducing is going to be a sound investment, bringing more value than it will cost. Or am I doing it because it's fun? It’s OK if it was purely for fun, we just need to be conscious of that fact.

---

4. Conviction

Some of us are simply opinionated, and some of us are dogmatic. Whether it's the unshakable belief in Microservices or Monoliths, Relational DBs or NoSQL, Python or Java. For some of us the problem is always obvious, and the solution is even more so, “Microservices are bloatware, let's lump everything into a monolith”, or vice versa.

No matter the problem, we could coerce a solution that is compatible with our convictions. In my opinion this is probably one of the most dangerous drivers for change. As it’s almost impossible to reason with, especially if it's from an authority figure.

Can you argue with your CEO or CTO or the Chief Architect if they force a certain conviction on you? Even if you muster the courage to openly disagree, dogmatic opinions are not based on scientific evidence and therefore cannot be influenced by logical reasoning.

---

5. Financial Insecurity

This is a big one, and I must admit that I’ve felt it myself at multiple moments in my career. You spend a few years at a "boring” company that focuses on certain robust but “old fashioned” tools to build its products, that reflects on you and your skill set, while the world is busy with NoSql, you are busy with examining SQL execution plans to optimize a slow query, while the world is migrating to the cloud and hyper-scaling, you are modifying Windows scripts to enable Reverse Proxy on IIS.

You feel boring, outdated, a Dinosaur! And anxiety starts creeping in. “Am I hire-able anymore? Can I get a job if my company lays off some people? They mentioned their financials are not as they hoped at the beginning of the year. I should do some learning and side projects to get my skills up to date. But I barely have time to breathe these days”.

So, the obvious solution is to push for technical change within the company itself. The next project is going to be with Kubernetes, or some fancy AI component.

---

6. New Hire/Promotion Anxiety

Imagine that you get hired as an Architect, or a Staff Engineer or maybe you got promoted into a similar role in your company.

The hiring manager explained to you that their software has been struggling with the current load of customers and their plan was to quadruple the number of customers by the end of next year.

Their code/architecture/infra needs to be redesigned to handle such growth plans and they already have budgeted a great deal for Cloud costs and already hired a bunch of cloud DevOps/DevSecOps engineers and even a FinOps guy to help you with your revamp, there is also a draft Architecture prepared by the previous architect and overseen by the CTO, Director of Engineering and a select committee of the company’s top talent. It's all ready for you to review and make amendments where you see necessary.

You, being smart and experienced, would take some time to study the current software and analyze its bottlenecks and what needs to be fixed. You quickly realize that the throughput can be increased by 10x, mean response time can be cut in half and loads on backend servers can be reduced by 66% if the following was done.

1. Add 3 indexes to key database tables.
2. Reduce web traffic by using caching and compression on specific APIs.
3. Refactor some code that tries to be too smart with threads.
4. Lump 2 tightly coupled microservices into 1 and reduce messaging overhead in synchronous integration scenarios.

The draft architecture on the other hand, is:

1. Totally unnecessary.
2. Overcomplicated, hard to understand and implement correctly by developers.
3. Will introduce tens of fault lines where things could go wrong.
4. Will require migration costs.
5. Will increase cloud bill for its extensive use of vendor specific messaging and object storage.
6. Will unlikely result in better performance, in fact, it has a good chance of slowing things down.

IMHO, 9 out of 10 architects will opt for the big plan proposed by their colleagues. Because of the known phenomenon - that I’ve just made up - New Hire Anxiety Syndrome, the person under NHAS has the following pressures:

1. I’m new here, maybe they know something I don’t know, maybe someone tried my approach in the past, and it failed. Am I really that confident about my findings?
2. Clearly these guys want a fancy architecture they read about it in books and Thoughtworks blogs, if I don’t produce something they expect I might be considered unqualified for the job. Maybe they’ll think I don’t know what a Service Broker is.
3. It's a tough market and I need to pass my probation.
4. The new architecture “looks” cool, it would be cool if I can oversee implementing it.
5. My proper solution would make me look like a dinosaur to those young engineers. They’ll probably start making COBOL jokes about me soon.
6. The CTO, and all senior staff, really like this new architecture. Do I want to start having conflicts on my first day of the job.

NHAS is a powerful force, and it could do great damage if not alleviated. And in the absence of proper communication, and invitation to open discussion where ideas and opinions are freely shared and accepted, it won’t be alleviated.

---

7. Peer Pressure and Bragging Rights

It's not only for school kids, peer pressure is a real thing even for work professionals. We attend seminars, conferences, post on social media, we write blogs, if most of your network are dabbling with Kubernetes, you will want to dabble too. If they wrote their own game engine, you’ll feel the need to write one too.

When 3 ex-colleagues meet after a few years, and talk about what they’ve been doing lately, it’ll probably go like this.

Alice (in a bragging tone): We’re recently revamped our architecture and business application to make them follow Flow Architecture. It was so challenging, but we did it.

Bob (in a more pompous and bragging tone): I worked on a project to optimize our mobile app to run smoothly in rural areas of impoverished countries. The challenges with slow intermittent connectivity bundled with slow low-end devices used by people there were so enormous, but it was a very cool challenge.

You: We’ve... umm... Upgraded our ... umm … backend to the latest Java LTS version.

Most developers don’t want to be that person. And the temptation is too great to introduce an extreme optimization plan for Q1 next year for their apps, even though they’re targeted mostly to accountants with modern laptops connected to enterprise grade Wi-Fi.

---

8. Proving it to Yourself

In addition to curiosity, a strong internal motivator is the Proving it to Yourself factor. Popular tech usually comes with its own challenges, a steep learning curve maybe, or maybe complicated infra setup, or some post setup monitoring to make sure it's working as expected. These challenges are a big motivator to smart individuals. And the constant pressure of can I do it? might become too much to bear.

There is something highly rewarding when a big effort is put into reading, trying, making mistakes, fixing them, then getting something into production. The rollercoaster ride of self-doubt moments when starting something new and complex, the minor highs you get while working and fixing small hiccups on your way to the euphoria of completion, is a powerful temptation. Usually in the absence of meaningful work in the company that provides similar thrill, developers will start to seek those rides on their own.

---

Conclusion

So, having listed those issues, what now?

Software development is predominantly a human activity. Each one of us is susceptible to a myriad of temptations, pressures, and anxieties. And this is what makes our field a very creative one, closer to art than science most of the time.

However, it's not purely art, and there are cold facts to face when it comes to markets, financials, legal commitments, etc.

Any healthy organization should have open and direct discussions when choosing what to focus on for the coming period. Coming into those discussions in a state of full awareness of one's own motivations for proposed projects is invaluable to making sound strategic decisions. Committing valuable resources to the wrong location could take the company under, and committing to the right projects at the right time could bring unimaginable success.

It’s also important for everyone to understand where the other person comes from. If your tech team feels restricted in an outdated tech stack, maybe it will be beneficial to give them the space to modernize and restructure. Label that effort under “Retention investment” if nothing else.

If Sales are under pressure to close deals, maybe tech teams should sprinkle some AI pixie dust on the product. A compromise between a flat NO and reinventing stable features to boost the sales pitch.

If your developers feel the need to stuff their work with flashy technology titles to get some form of recognition, you probably should reevaluate your evaluation methods so meaningful work can be made visible and recognized, even if it sounds boring.

Maybe you should also reduce pressure on new hires, give them the time to formulate their own opinions and share them openly with their colleagues.

Software development is like Jazz music, it's mostly played by ear and full of improvisation. I can’t, even if I really wanted to, give a concise list of solutions to everything that influences a developer’s drive. All I can do is highlight my observations, give them titles and allow everyone else to build and improvise on top of them.

This article is about the amazing humans who build software, about us understanding others, and, hopefully, our own self better.