#018 – Kubernetes for Humans Podcast with Pavel Brodsky (Forter)

Itiel Shwartz: Hello, everyone, and welcome to another episode of Kubernetes for Humans podcast. Today with me on the show, we have Pav. Pav, can you please introduce yourself?

Pavel Brodsky: Yeah, hi, nice to be here with you. My name is Pav Brodsky. I’m the Group Manager of Developer Platform at Forter. We are responsible for providing a great developer experience for our developers and analysts.

Itiel Shwartz: Sounds great. For full disclosure, I used to work at Forter about eight years ago, before you joined, Pav, or just before you joined. Let’s start with a bit about your background—how did you start your career? What did you do? Let’s take it from there.

Pavel Brodsky: I started as a backend developer, working mostly in Python and some other languages as well. At some point, I became interested in how things work under the hood—started looking into Jenkins, Docker, and so on. When I joined Forter five years ago, the VP of R&D, Oren, and I thought it would be a good fit for me to join the infrastructure team. Back then, it was just a small team, not a group, just me and a couple of other guys. And that’s how it all started. I’ve been in infrastructure for five years now. We’ve transitioned from infrastructure to DevOps and now to platform engineering. But it’s always been about providing tools and platforms for developers.

Itiel Shwartz: That’s really cool. We have a lot of guests who come to platform or DevOps roles from various backgrounds—sometimes from legacy IT or other areas. But you come from a pure developer background, moving from Python into this role, which is a bit different. Can you share a bit more about how you made that transition?

Pavel Brodsky: I didn’t even think about it as being something other than a developer back in the day. When I joined, I think my official title was something like Senior Infrastructure Engineer, but I saw myself as a software engineer focused on a subset of tools we use for CI/CD and other functions. I didn’t see it as different from other backend work. Over time, I’ve learned to enjoy the other aspects of the work that are more operational. Today, most developers, whether backend or full-stack, do some Ops work—they need to run Docker locally, interact with CI systems, and so on. So, I see myself and my team as developers first. The fact that we’re lower on the stack is just an attribute of where we are, but the job is the same. Focusing on the platform is a way to have an immediate, wide impact across the engineering team, which I find gratifying. Plus, my customers are developers, who tend to be better customers than non-developers, so that’s just a bonus for me.

Itiel Shwartz: That’s cool. Let’s talk about that for a second because I think it’s interesting—you said your customers are developers. How do you see success? At the end of the day, if I’m a developer, I’m trying to write my feature, and my product manager is telling me to build something. It’s much more business-oriented. Everything needs to transition into dollars. For you, it’s more about developers. As we’ve just finished 2023, what is success for you when we start 2024? How do you reach those success criteria?

Pavel Brodsky: That’s something I think about a lot. As you said, everything needs to converge to dollars at some point. The two best ways for me to impact business metrics are cost and developer productivity. I can make systems more efficient, for example, by migrating EC2 workloads to Kubernetes. I can also impact developer productivity with metrics like Dora or Space metrics. If my lead time is lower, the developer’s time to market is shorter, generating more features, which in turn leads to more revenue for Forter. Similarly, deployment frequency and time to recovery from failures (MTTR) are also important. For a company like ours that aims to be public, having a strong posture that the market can appreciate is key. Another factor is churn—developers might leave unpleasant working conditions. For us, working conditions are about automation, modern UIs, and reducing manual tasks. If developers are happy, productive, and costs are decreasing, then I’m doing my job well. We have OKRs to measure that, so that’s how I’ll know if I’ve succeeded by the end of the year.

Itiel Shwartz: Dollars are easy to measure, but how do you measure the satisfaction of the developers? Do you have surveys or other methods? It’s a common issue many platform engineers face—how do I know if I’ve done my job correctly?

Pavel Brodsky: It’s a good question and something we’re thinking about a lot. We do have surveys, though we’re not 100% sure of their validity. We might evaluate something like DX, or getdx.com, which offers a service with market data to compare your results to other companies of similar size and maturity. We also conduct interviews and have a customer advisory board (CAB) with senior engineers from different teams and groups. They help us create a roadmap based on customer input, so we can say, “These were the ten most important things to you; we delivered eight of them.” We couple this with satisfaction surveys and objective metrics like Dora and cost to gauge our success.

Itiel Shwartz: That’s quite cool—I don’t think I’ve heard of a CAB for platform teams before. Makes total sense. Before we dive into Kubernetes, you mentioned that when you joined Forter, you joined as an individual contributor in a smaller company. Can you share a bit about Forter itself—what is Forter, what is the business model, and how has the company transitioned since you joined in terms of both infrastructure and engineering?

Pavel Brodsky: Sure. Forter is in the trust business. The most basic service we provide is transaction approval or decline based on what we perceive as fraudulent or not. For example, if you go on the Nike website and buy a pair of Jordans, the transaction goes through us, and we decide whether it looks legitimate or not. We do this at a very strict SLA. Since then, we’ve expanded into fields like payments optimization, 3DS support, and login/signups—not just transactions. If it’s a trust issue and we can determine if a person is legit or not, that’s our business. When I joined Forter, the engineering team was about 30 people strong; we’re now about 120. We probably would have grown more if not for the pandemic and market conditions. One of the most interesting shifts in engineering has been moving from a mindset where everyone did a bit of everything to a more structured approach. We used to have a strong core of experienced engineers who knew everything, but as we’ve grown, we’ve had to create tools and platforms that make it easier for new hires to be effective quickly. We’re moving towards industry-standard tooling, like replacing our DSL for cloud infrastructure with Terraform, and giving developers more autonomy while still maintaining compliance, security, and infrastructure standards.

Itiel Shwartz: Super interesting. Let’s talk a bit about Kubernetes. You guys weren’t using Kubernetes before, and now you’re in the middle of a migration. How were you introduced to Kubernetes? Why did you decide to migrate, and how is it going so far?

Pavel Brodsky: This is actually our second attempt at migrating to Kubernetes. The first one didn’t go well, and I even gave a talk on how to reverse a giant architectural decision. About four years ago, we needed to be able to run on multiple clouds, and we thought that if all our workloads ran on Kubernetes, it would make it easier to launch our platform on another cloud. For various reasons, the cloud initiative stalled, but so did our Kubernetes migration. We were naive back then, thinking we could migrate everything in half a year, which was foolish. We didn’t have the principles we have now, like ensuring our platform is composable, extensible, and self-served for developers. Back then, everything had to go through our infrastructure monorepo, which caused a lot of problems. We also tried to introduce Istio as a service mesh, which was a bad idea—it wasn’t supported in the way we needed, and the experience was terrible. We eventually rolled it all back.

Itiel Shwartz: So now you’re in the second iteration, which seems to be going much better. What’s different this time? Who is owning the migration, and what changes did you make from the first attempt?

Pavel Brodsky: This time, we have strong support from the top—the CTO and VP of Engineering are on board. They understand that Kubernetes is an established technology that’s worth investing in for cost, time, and productivity benefits. The developers care about how their applications are deployed, but not too much. They don’t want to know the details about node groups, policy engines, or how Helm is picked up by Argo CD—they just want to focus on business logic. So, we’re providing them with the simplest solution that gives them the capabilities they need without burdening them with unnecessary complexity. We’re also responsible for the migration, and once developers see the benefits, like faster deployments, they’re eager to migrate everything to Kubernetes. But initially, they’re skeptical, especially because of our previous failure. So, it’s on us to convince them that the system is production-ready. We’re treating this as a gradual process—we go team by team, schedule time with them, and ensure they fully migrate before moving on. We also plan to do workshops to teach people about Kubernetes, and once they get used to it, they love it. Then they become the evangelists, pushing for more migrations.

Itiel Shwartz: That makes sense. Good developers are usually skeptical of new technology. Kubernetes is complex because of its ecosystem—it’s not just about moving Docker containers; there’s networking, storage, credentials, and configuration to consider. It’s a more complex migration than it might seem. Finding early adopters within the organization who can become Kubernetes champions is crucial. Also, Istio is a great example of something that adds a lot of complexity—if you’re not a big company with a dedicated Istio team, it’s not plug-and-play. It requires a lot of trial, error, frustration, and expertise to make it work. Now, let’s talk about the second iteration. What did you do differently, what’s the status, and what can we learn from this?

Pavel Brodsky: This time, we’re sticking to some core principles. We want developers to understand what’s happening using Google and open-source documentation. Our setup is a pretty simple vanilla EKS cluster with built-in integration to our observability stack. Developers automatically get logs in Loki, metrics, and more, all working out of the box. The out-of-the-box part is crucial because it’s easier to integrate things in Kubernetes than in our previous VM setup. For deployment, we evaluated different tools but went with Argo CD. We’re big fans of Argo, so we’re also using Argo Workflows for jobs and Argo Rollouts for progressive delivery. So far, it’s working well for us, but it’s still early days. Kubernetes has strengths, like out-of-the-box capabilities, that our previous solution didn’t have. 

Itiel Shwartz: That’s cool. You’re avoiding creating another layer of abstraction, which allows developers to be more self-sufficient without needing you in the middle.

Pavel Brodsky: Exactly. And it decreases the onboarding time for new developers significantly. We’re using common tools for deployment and other tasks, like Argo, which makes it easier for developers to get started. In the past, we had really strict time limits and unrealistic expectations. Now, we’re more mature in how we approach milestones and timelines. We also focus on making sure everything we add is composable and extensible, with reasonable escape hatches so developers aren’t locked into our pipeline. We want to be enablers, not blockers. Now, developers are coming to us saying they want to launch things on Forter, and we’re able to support them more easily.

Itiel Shwartz: That makes sense. And just to clarify for listeners—installing something like Flink via Helm might be easy, but maintaining it is another story. Helm can give a false sense of simplicity because it’s easy to install, but long-term maintenance is challenging. So, back to the migration—how did you start the second phase? What was the first application migrated, and what’s your strategy?

Pavel Brodsky: We started with ourselves, migrating our own services first because we’re a platform team, and our use cases were simple. Then we found a design partner with high needs and a specific use case that benefited from Kubernetes, particularly in terms of fast and efficient auto-scaling. They were motivated enough to go through the journey with us, and now they’re happy running all their systems on Kubernetes. They’ve become our biggest evangelists. So, find someone who needs Kubernetes as much as you need them to be motivated through the process.

Itiel Shwartz: That’s cool. We’re almost out of time, but I’ll say Forter is one of Komodor’s customers, and we’re trying to help with the migration. Pav, what’s in store for the future of Kubernetes at Forter and in general?

Pavel Brodsky: Last year, outside of the platform group, we had one big customer. This year, our goal is to move from early adopters to more widespread adoption—maybe 30% of workloads at Forter will move to Kubernetes. In 2025, we hope to leave only the more difficult use cases behind. Stateful apps are still a challenge, but we’re looking into running ElasticSearch or Aerospike on Kubernetes. Adoption is our number one priority right now. We’re also excited about Komodor’s new data offering, which can help us show our customers how the system is doing and maintain trust in the solution. We’re adding more capabilities like progressive delivery and improving our Helm offering to make the migration as smooth as possible.

Itiel Shwartz: Helm is great, but maintaining it at scale can be challenging. Any last thoughts or recommendations?

Pavel Brodsky: I’ll send you links to a couple of lectures, and I have a blog that’s going to turn into a book soon, so stay tuned for that. For those making these migrations, be kind to your customers. They’re coming in knowing nothing and might be scared. It’s your job as platform or DevOps engineers to make them feel comfortable with this change. It’s a big shift, but they will love it in the end. Just work slowly and methodically, earn their trust, and it’ll be worth it.

Itiel Shwartz: Great words to wrap up on. Thanks a lot, Pav—this was a super interesting episode. It’s the first of three we’re doing on migration. I had a lot of fun. Thanks a lot!

Pavel Brodsky: My pleasure.

[Music]

Pavel Brodsky has been a Backend Engineer for years before switching to a DevOps role at Forter — the leading trust as a service unicorn startup. Three years ago, he transitioned into an Engineering Manager role in a team responsible for Forter’s CI/CD pipelines and internal developer platform. Since becoming an EM, he has been focused on maintaining happy and effective teams, and he is passionate about developer experience. 

Itiel Shwartz is CTO and co-founder of Komodor, a company building the next-gen Kubernetes management platform for Engineers.

Worked at eBay, Forter, and Rookout as the first developer.

Backend & Infra developer turned ‘DevOps’, an avid public speaker who loves talking about infrastructure, Kubernetes, Python observability, and the evolution of R&D culture.  He is also the host of the Kubernetes for Humans Podcast. 

Please note: This transcript was generated using automatic transcription software. While we strive for accuracy, there may be slight discrepancies between the text and the audio. For the most precise understanding, we recommend listening to the podcast episode