How Culture Impacts Technology Choice: A Review of Netflix’s Use of Microservices

[Originally published in TheNewStack on May 25, 2021]

I recently had the opportunity to read the book “No Rules Rules: Netflix and the Culture of Reinvention” by Reed Hastings and Erin Meyer of Netflix, and it dawned on me that while this book wasn’t at all focused on Netflix’s technology, the global company-wide culture had a significant impact on its technology choices. The book focuses on the many times Netflix had to reinvent itself and transform its business in order to revolutionize the entertainment industry. This revolution was made possible by an equally radical and formerly unheard of approach to a company culture that not only influenced how we build technology today but also thinks about company culture on a global scale.

Needless to say, Netflix’s technology choices went on to influence the way the entire industry thinks about architecture built for a massive scale. In this post, I’d like to take a look at how Netflix’s culture was the backbone that enabled its bleeding edge technology decisions — and specifically microservices. Some of what I write here is actually the direct influence of my own personal love-hate relationship with microservices. This book afforded me a newfound perspective on how to succeed at building challenging technology architectures to enable the business, by ensuring the right cultural mechanisms are in place first.  I’ll posit that if not for Netflix’s unique approach to company culture, their technology choices would not have been possible either, and ultimately neither would their success.

My Backstory

So just to dive into my inspiration for this post a bit more, my love-hate relationship with microservices began when I was a software engineer at eBay trying to tame a crazy monster of a monolith. It was so complex and vast and no one really understood all of it, that I learned how truly difficult it is to break up a monolith that is the beating heart of a business.

Before kickstarting Komodor, I had a couple more opportunities to work on building microservices. The first was a gradual breakdown of a monolith, and the other was building a microservices architecture from the ground up. All of this experience provided me the opportunity to build the architecture that would drive our business from day one at my new venture, and I try to apply everything I have learned all the time.

And just so we’re on the same page, if you need a refresher on microservices, a great place to start would be this foundational piece by Martin Fowler of ThoughtWorks. Another good read would be this newer piece by Segment, which provides a much more practical perspective on migrating to microservices and the challenges involved.

Netflix’s 6 Principles of Building Culture AND Great Technology

So how does this all come together? In their book Reed Hastings and Erin Meyer outline the core cultural principles that made their success possible, I’d like to compare these side by side with the cultural practices that enable engineering organizations to successfully deploy and maintain microservices architecture — which wasn’t actually the premise behind the book, but it certainly got me thinking about the places that engineering and culture intersect.

1. Principle One: Build Up Talent Density

Netflix did not compromise on talent when it came to hiring. It hired the best possible people for each position, which was the foundation to their success. This is true by orders of magnitude when it comes to microservices architecture. Microservices require a great capacity for learning, as the people maintaining the architecture will continuously encounter new challenges and problems they’ve never confronted before. If we’ll be completely honest, it is just not possible for all developers and DevOps engineers to be the best in their field.

While some companies have some pretty great engineers, Netflix has world-class engineers. This made it possible for Netflix to overcome challenges that many other engineers likely wouldn’t have had the capacity to work through as pioneers of complex architecture. It had some of the best engineers in the world working on the newly discovered challenges microservices surfaced, they overcame these and later enabled the industry to do so as well by sharing from this experience.

2. Second Principle: Begin Removing Controls

Netflix’s radical approach to culture can be seen in the most basic of company guidelines from expenses through vacation approvals, employee trust was built-in to the culture as a foundational principle. That said, this was only made possible by having the right infrastructure in place to support this culture as well.  We’ll get into that later.

This cultural practice is one of the core factors that can influence the success of managing microservices architecture at companies that are not Netflix. I’ve found that even companies that have made the decision to go down the microservices route do so while old processes and mechanisms still exist, and this highly hinders the ability to properly take advantage of the migration. If companies still maintain rules like approvals for each new change or commit introduced into a system, or even reverting changes, they essentially have a distributed monolith culture.

So on the one hand, they have an architecture that can scale autonomously — but their people can’t. You need to trust your people to make the right decisions in order to extract the real benefits and velocity microservices make possible. Without this inherent trust and autonomy, you’ll essentially have the worst of both worlds. Netflix took this approach with all things, not just engineering, and it repaid them kindly in achieving success.

3. Third Principle: Maximize Candor Using Feedback Circles

Many times the decision to move to microservices is a top-down decision. What oftentimes happens is that the decision-makers don’t really have any concept of the complexity of the task at hand — breaking up their architecture into smaller and smaller pieces. On the flip side, those who need to execute on the top-down decision don’t have the agency to influence the process. The move to microservices often surfaces a lot of complications, where things that previously worked suddenly stop working. A critical piece in moving quickly and resolving these issues, is short feedback loops.

At Netflix, they were required to raise a flag and communicate problems when they surfaced, whereas other companies often do not enable a culture of such candor. Companies that don’t have an atmosphere that enables this kind of communication and candor will be set up for failure with complex architectures, where their teams will constantly be firefighting problems instead of architecting more long-term solutions built on collaborative thinking and communication as oftentimes the voices of the people in the trenches aren’t heard.

4. Fourth Principle: Build Supporting Infrastructure

To be successful, you need to know that as a first step you’ll be spending a lot of time in building supporting tools for your developers and DevOps teams to enable them to be productive in this transformation.

Like all “shift left” practices, if you only remember to do so after the fact, you will find it costing you 100x in time, resources, and even budget to build the supporting mechanisms to enable the teams to succeed. Succeeding with a loosely coupled architecture, that requires a lot of trust in your engineers’ decision-making, requires you to provide those at the bottom of the chain with more “powers.” To support their innovative approach to expense reporting and vacation requests for example, Netflix built the proper checks and balances into the system in advance, to ensure on the one hand that it couldn’t be abused, but that it could also scale with them as they grow and maintain this critical piece of their culture.

5. Fifth Principle: Celebrate Success and Sunshine Failures

I think the industry as a whole learned the importance of celebrating successes, and today nearly every organization that prides itself on its company culture has some way of demonstrating appreciation for good work. However, on the flip side, one thing that companies can still learn from Netflix is their approach to failures. At Netflix failures are not brushed under the rug, they are analyzed, to be able to learn from each experience in order for the organization to grow and evolve, and not make the same mistakes twice.

When you migrate to microservices you are going to have A LOT of failures at first, and if you don’t take the opportunity to learn from these, you will be destined to repeat the same mistakes. In the spirit of this pillar at Netflix, in a landscape of many partial successes and common failures — it is important to be self-aware and have the sensitivity to learn from these failures to figure out the root cause, and ultimately how to fix it, so it doesn’t repeat itself.

6. Sixth Principle: Leading with Context, not Control

With monolithic architecture, it is very easy to have control, as there is only one service and everyone for the most part understands what is happening in their domain and aligns themselves to the practices in place. With microservices engineering management simply cannot understand everything that is happening in the system as a whole, and each team needs to have a high degree of autonomy. That’s why transparency is extremely important coupled with providing a lot of context around what is happening, when it comes to being able to make decisions in this fast-moving engineering environment.

With microservices, it is impossible to micromanage and therefore you have to provide the general vision and direction and trust that your teams will align to these shared goals. Netflix understood how to do this really well, and this instilled a bottom-up joint purpose that fueled much of their success.

What We Have Learned

So basically to wrap it up, microservices has proven to be a very powerful architecture to power modern, complex business, but you need to remember that to achieve success with technical architecture, it is no less important to “engineer your culture” (yet another great Netflix mantra). You need to have a good understanding of the culture you need to have in place to support distributed, advanced microservices architecture and for everything to ultimately work as expected. Having the technology without the culture will just result in many unhappy and overworked engineers.

Great culture without modern engineering and architecture practices results in frustration in the inability to move at the desired velocity and deliver services as rapidly as engineers would like. Advanced architecture without the culture sets you up for failure, as you will not be able to capitalize on the benefits of microservices. So if you put in the effort to craft a truly great culture from the ground up, this will be foundational in enabling your people to thrive.

So eventually when we build great culture alongside an excellent product, this will drive those who are together with us in our purpose to get creative, and will repay us generously in both innovation and business.