The No. 1 Reason for Developer Burnout — Too Much Freedom
Freedom in software companies is a double-edged sword
Background
I studied college with a major in Biotechnology. In the third year, we had to do a project on which we’ll be graded. Our team, upon reading a dozen papers and consulting with the professors, decided to build an electrochemical cell that uses corn cob to generate electricity. We’ve decided to cultivate a specific family of bacteria that metabolizes glucose present in the corn cob releasing electrons in the process to produce electricity. Our professor who was our guide approved the idea.
The biotechnology lab was a recent installation in our college. Thousands of dollars were invested in buying chemicals and setting up the laboratory. The chemicals were expensive; some enzymes cost as much as $300 for a single 25ml bottle. They were too expensive to be trusted in the hands of young undergrads. Naturally, access to the lab was limited to the PhDs, and students can only work in the lab under their supervision.
The supervisors would lock the lab when they were out for lunch or any other work. So whenever we’d want to work in the lab after our classes, we had to wait for them to come back.
To grow bacteria, we had to first sterilise the petri dish and prepare a broth with the right mix of nutrients. Then we would add a tiny speckle of already-grown bacteria and they would start multiplying exponentially and fill the petri dish within a few days. However, a single mistake like not wearing gloves while working would lead to contamination and the bacteria would die. So we had to be extra careful.
All in all, we were successfully able to grow bacteria and generate electricity for the project. But the whole process was slow, restrictive, and inefficient.
Why I chose software engineering
For next year’s project, I chose to work on genomic data analysis. Essentially, I had to pull data from open-source databases and do some basic statistical analysis.
I could work anytime from anywhere. All I need was my laptop and a stable internet connection. Moreover, I could solve the problem however I wanted to as long as it leads to the end goal. I could choose to use any framework and nobody could care less. I enjoyed the freedom. This had a stark contrast to the wet-lab work I was doing previously.
Happy with the work, I decided to pursue the field further and landed a job as a machine learning engineer right after graduating.
Little did I know this newfound freedom would eventually lead to burnout.
The cost of freedom
One of my first projects in the company was to build a face-mask detection system. Covid was starting to rescind and people were going back to the outside world. The client wanted to monitor if everyone at his business was wearing the face mask properly which was mandatory.
It was a relatively easy problem and we chose to go with an object detection model. The client had given us total freedom. We were free to decide on the model architecture and the implementation framework.
We went ahead with the RetinaNet model and achieved an accuracy of 95%. The client was happy with the number. However, upon further inspection of a few images, the client told us while the model was good, it was failing at certain edge cases where people were wearing the mask but weren’t covering their noses.
It was a hard edge case to solve. We were happy to try out new models and algorithms, but we had a tight deadline as the client had to present the work to the board.
So we were working 13-hour days on average, and sometimes even during the weekends. We explored the latest research papers and implemented them. Sometimes they would work, but more often we wouldn’t see any improvement. Finally, we delivered the project with a 3% improvement and closed it.
This made me realise the cost of freedom. While we were free to do whatever we wanted, we had to deliver a good solution within the tight deadline — and this is the number one reason why developers burn out.
Knowledge workers aren’t black boxes
Peter Drucker was one of the most influential thinkers of the 20th century who coined the term “Knowledge Work”. He argued that knowledge workers need to be given the necessary autonomy to manage themselves. They’re often more knowledgeable than their supervisors in their specialised areas. The best way to utilise them is to give them clear objectives and leave them alone to accomplish the work however they saw fit.
This might have been the best approach in his time, but today life is much faster. To build great products rapidly in a competing market, it’s a bad idea to let developers wander and figure out the path themselves. The worst thing you can do is give them an illusion of freedom while imposing tight deadlines. Without structure and direction, the path will only lead to burnout.
Let’s talk about some possible solutions.
The solution — Iterative Development
The first step is to stop treating developers like black boxes. The product manager should at least try to understand what the developer needs to do and come up with a realistic, definitive plan for building the solution.
Moreover, software development is an iterative process. Instead of spending hours discussing the right framework and architecture, the team should start by building an MVP. Once it’s deployed in production, it should be monitored and analysed. The insights drawn from the analysis should inform the next cycle of development. This is an infinite feedback loop as technology keeps on improving.
François Chollet, the creator of Keras, calls this the loop of progress.
The best ideas rarely spring from a sudden “Eureka” moment or from an ambitious brainstorming session. They’re born from years of iterative refinement. — François Chollet
While this process sounds simple, it’s rarely followed. Many teams want to quickly get the project done so they can move on to build the next big thing. This is a natural human tendency that is hard to combat. However, this leads to technical debt which may result in stagnation.
Conclusion
Software development isn’t special. It should be treated just like any other job. Instead of leaving developers to their own devices, a definite plan with structure and direction should be laid out and followed to avoid burnout.
If you think about it, software engineering is a relatively new field as compared to, say, construction of buildings which is being done for millenniums. So it’s natural we don’t have strong definitive rules. However, with the rapid pace of development, It’s only a matter of a few decades since we get there.
I hope you’ve enjoyed the article. Let’s connect.
