#008 – Kubernetes for Humans Podcast with Alexandru Dejanu (Systematic)

Itiel Shwartz: Hello everyone, and thank you for coming back to the Kubernetes for Humans podcast.

Alexandru Dejanu: First and foremost, thank you for having me. It’s an honor. I’ve never been on a podcast before; usually, I’ve been to various conferences and meetups, giving presentations mainly about Kubernetes, platform engineering, and cloud-native technologies. So, it’s a nice opportunity, and it’s nice to meet you all.

Itiel Shwartz: Great to have you. It sounds like you’ve had a long journey in different, highly regulated industries like healthcare and finance. Before we dive into what you’re currently doing, can you share why you got into backend development, then moved into operations, and finally, your first encounter with Kubernetes?

Alexandru Dejanu: My entry point was through development, due to my studies. When I started, the development lifecycle was pretty straightforward. You had your application, which was packaged as a binary, and then you deployed it to a server. Deployment was simple—just copying the binary to the server. You had your regular components: source code, build process, and application servers. Operations teams typically handled the setup.

Slowly, I noticed a trend towards adopting various technologies to streamline this process. I remember working with Cloud Foundry at one point. A few years ago, if you looked at orchestrators or platforms, Cloud Foundry was one of them. I saw the power of built-in deployment and how it made deployment easier and faster. That experience led me to move towards operations, where we started building in-house tools for deployment and maintaining solutions in production environments. Then, Kubernetes came along and became an industry standard.

Itiel Shwartz: Cloud Foundry was a way to deploy applications before Kubernetes. Migration is always challenging—no one wakes up wanting to migrate their core infrastructure. Can you tell me about the moment your team decided to leave Cloud Foundry and start exploring Kubernetes? Why did you make that decision, and what were the trade-offs?

Alexandru Dejanu: That decision was closely tied to platform engineering. One of the pain points of that project was finding people with Cloud Foundry knowledge in the market. We saw similar challenges with other orchestrators like Apache Mesos, which led us to consider switching to something open-source that we could extend and customize for our needs. That was one of the biggest selling points for migrating to Kubernetes.

Itiel Shwartz: So, it was about finding people with the knowledge or the ability to maintain the system over time?

Alexandru Dejanu: Exactly. In a big organization, people come and go, but you have to maintain the platform and pipelines. If you can’t find the expertise in the market, you take the challenge and risk of training people, but it’s a significant investment in time. Over time, we started learning about Kubernetes, which was initially not for stateful applications—just vanilla Kubernetes for stateless stuff.

Itiel Shwartz: When was this, and which company was it?

Alexandru Dejanu: It was a telecommunications company, a pretty big one, around 2017 or 2018. The first major change we felt was the API-driven architecture of Kubernetes. It felt closer to development rather than traditional operations, which was refreshing.

Itiel Shwartz: I had a similar experience. I worked as a backend engineer at eBay and a small startup before discovering Kubernetes. It felt clean and easy to get started with, compared to tools like Chef, which were more cumbersome. Kubernetes made it easy to get a cluster up and running, though making it production-ready is a different story. I’m curious, do you think platform engineering can exist without Kubernetes?

Alexandru Dejanu: Initially, I thought yes, but reflecting on my experience, I’m not so sure. Kubernetes is a powerful vessel for building platforms. For example, in the ML Ops space, the platform might revolve around tools like Airflow, with Kubernetes integrated. But in large companies, not choosing Kubernetes seems strange. It standardizes so much and makes building that first layer of APIs much easier.

Itiel Shwartz: Agreed. I see it as challenging to build platforms without Kubernetes in a large company. Another trend I’ve noticed is the choice between building or buying a platform. Out of curiosity, I checked, and there are over 118 certified Kubernetes service providers. Kubernetes is often described as a platform for building platforms.

Alexandru Dejanu: That’s true. Platforms are very tailored to the business needs of an organization. For example, a telecom platform might look very different from a healthcare platform. In my experience, each company standardizes certain workflows, like uniform logging using tools like Fluent Bit, FluentD, and OpenSearch. But the platforms differ greatly from company to company, even if they’re built on Kubernetes.

Itiel Shwartz: That’s interesting. Could you share an example of a platform you were involved in building?

Alexandru Dejanu: Sure. In the financial industry, OpenShift is often the go-to platform because of its enterprise support and minimal customization. It’s a regulated environment, so the focus is on stability. But the experience differs even within the same industry. Kubernetes allows organizations to explore and extend the platform in unique ways.

Itiel Shwartz: If I were to join a company, I’d want to see the custom resource definitions (CRDs) they use. It says a lot about what they do. Can you share a specific project around Kubernetes or platform engineering that stands out?

Alexandru Dejanu: One of the biggest challenges was making the platform invisible to the customer—Kubernetes should be a seamless experience. Another challenge was cost control, especially when using managed services from a cloud provider. Education is key here, starting from the development side. You need to understand limits and requests for your workloads, and always run load tests for your applications. Workloads are dynamic, so it’s important not to lose sight of that.

Itiel Shwartz: Load testing is essential, but it’s often done once and forgotten, especially as the application evolves. How do you manage that?

Alexandru Dejanu: It should be part of your company culture and best practices. Ideally, you automate this process, but embedding it in your workflow is crucial. Application changes often, so you need to keep track of that through regular load testing, especially after big changes or before high-traffic events like Black Friday.

Itiel Shwartz: That’s a good point. The cloud-native landscape is so vast that it’s easy to lose focus when integrating new technologies. And as you said, it’s common to see workloads either under-provisioned or over-provisioned.

Alexandru Dejanu: Absolutely. And then there’s the debate about running without CPU limits. It’s a complex issue, and the human factor plays a significant role in these decisions.

Itiel Shwartz: It’s a trade-off, for sure. Let’s talk about the relationship between SREs, platform engineers, and developers. Who should own what, and how do these roles interact?

Alexandru Dejanu: I see SREs as an extension of developers and the project. Observability is key here, and it’s a hot topic. There are so many tools for monitoring and observability, and the challenge is dealing with tons of dashboards and tracing data. It all comes down to your monitoring strategy.

Itiel Shwartz: I agree. Building dashboards is an art. At my previous company, we had a complex Kibana dashboard that only a few people could understand. It had all the answers, but we had to create simpler dashboards for general use.

Alexandru Dejanu: Exactly. Creating effective dashboards requires a deep understanding of user experience. It’s a mix of knowledge and design, and it’s a challenge many companies face.

Itiel Shwartz: Do developers in your company create their own dashboards, or is that mostly done by SREs?

Alexandru Dejanu: It’s a mix. Developers create dashboards with application-specific metrics, while SREs focus on higher-level metrics like service mesh, latency, and cluster performance. There’s also the challenge of provisioning these dashboards and ensuring they’re under version control.

Itiel Shwartz: It’s a complex issue that sounds trivial until you have to manage it at scale. With the adoption of large language models, I’ve seen tools like New Relic working on machine learning for observability. What are your thoughts on that?

Alexandru Dejanu: I’m skeptical. Machine learning models might be good at spotting trends, but they lack the business logic and context that humans have. I haven’t seen anything yet that’s truly impressive in this space.

Itiel Shwartz: I agree. There are so many nuances that AI currently struggles with. While I believe the industry is moving in that direction, it’s still an open question whether these models can effectively replace human intuition and understanding in observability.

Alexandru Dejanu: Exactly. It’s a tough problem to solve, and I think we’re still a long way from a reliable solution.

Itiel Shwartz: Looking to the future, what do you think will be the biggest challenges in the next few years?

Alexandru Dejanu: One of the biggest challenges will be keeping up with the rapid changes in machine learning workloads and the Kubernetes scheduler. We’re seeing a lot of customization and extension of the Kubernetes API, and that’s leading to significant differences between clusters. It’s becoming harder to maintain a consistent experience across different Kubernetes environments.

Itiel Shwartz: That’s a great point. Kubernetes allows organizations to capture their unique business logic, but it also means that two clusters can look very different. It’s both a strength and a challenge.

Itiel Shwartz: Alex, are you coming to the upcoming KubeCon in Chicago?

Alexandru Dejanu: Unfortunately, I’ll stick to the European events, but I hope to attend KubeCon in the U.S. in the near future.

Itiel Shwartz: Are you aware of the Kubernetes Community Days happening in Romania next year?

Alexandru Dejanu: Yes, I know about it. It’s happening in May or April. It’s a joint effort here in Romania to create a Kubernetes community. I hope we can build something stable and construct a strong community around Kubernetes.

Itiel Shwartz: That’s great to hear. It was a pleasure having you on the podcast, Alex. Thank you very much.

Alexandru Dejanu: Likewise, thank you.

[Music]

Alexandru Dejanu is a jack of all trades master of None. Alexandru is a Senior Systems Engineer with 8 years of experience in various domains such as Automotive, Telecom, Banking, and Healthcare. Currently, he works as an SRE at Systematic, part of the Customer Operations team, helping both development and operation teams to have full visibility of the complete application lifecycle. Opinionated and tech agnostic, loves helping others and sharing knowledge be it on StackOverflow, HackerNoon, or Medium. 

Itiel Shwartz is CTO and co-founder of Komodor, a company building the next-gen Kubernetes management platform for Engineers.

Worked at eBay, Forter, and Rookout as the first developer.

Backend & Infra developer turned ‘DevOps’, an avid public speaker who loves talking about infrastructure, Kubernetes, Python observability, and the evolution of R&D culture.  He is also the host of the Kubernetes for Humans Podcast. 

Please note: This transcript was generated using automatic transcription software. While we strive for accuracy, there may be slight discrepancies between the text and the audio. For the most precise understanding, we recommend listening to the podcast episode