#041 – Virtualizing Kubernetes with Lukas Gentele (Loft Labs)

Itiel Shwartz: Hello everyone and welcome to another episodes of Kubernetes for Humans podcast. I’m here today at KubeCon London and I have Lukas with me. Lukas, happy to meet.

Lukas Gentele: Yeah, awesome to be on the live version of the podcast.

Itiel Shwartz: We try we try our best. So Lukas, can you introduce yourself, your company? And yeah, what do you like about Kubernetes?

Lukas Gentele: Yeah, I’m uh Lukas, CEO and co-founder at Loft Labs. We are the makers of vCluster. We got the open-source project started in 2021. launched another open-source project about two years ago called DevPod and uh yeah, we launched another product uh yesterday called vNode. the reason why you know we have these different products in different areas of Kubernetes is we’re trying to really virtualize the entire cloud native stack. We started with the core piece which is Kubernetes. Yeah. You asked what I like about Kubernetes. A lot of things. I think Kubernetes is an amazing tech. I wish it would have more multi-tenancy built in because I think it is one of the biggest obstacles in Kubernetes adoption and order to act to get the actual potential of Kubernetes you know like Kubernetes wasn’t designed to create you know 3,000 uh little shared clusters. No and a lot of people have to do that today. Yeah. Yet this is the reality right. Exactly. And we want to help them get to what Kubernetes should, you know, is designed to be, right? When you think of the origins like Borg and stuff like that, right? Borg is not like 5,000 Borg instances. It’s like it’s one big distributed system, right? And that’s when you get all these benefits out of Kubernetes. So we help people spin up larger clusters, run virtual clusters on top of it, and have secure multi-tenancy at that Kubernetes layer. And since yesterday with the vNode announcement, we also allow you now to virtualize nodes and have security at the node level to separate things by vCluster.

Itiel Shwartz: So you know like I know vClusters which is like a super like famous project. maybe share a bit more about yourself like how did you get into I know tech in general and more stackifically why or like the origin of the VC clusters like why should someone want a virtual cluster when you can have a real cluster right yeah how do you wake up in the morning and say we’re going to virtualize Kubernetes right yeah it’s not like the first thing that comes to mind right exactly and it doesn’t sound easy it’s not like it’s yeah I’m going to do it like in a day or whatever

Lukas Gentele: yeah yeah it was a journey I think uh you know we We actually ended up building vCluster as pod of a pivot. So we started with uh a PaaS offering. We launched an open-source project in I think 2018 that was called DevSpace CNCF sandbox project today. And you know we built this product called DevSpace Cloud which is a PaaS right? It’s kind of like a Heroku but for Kubernetes. Okay. We had a free tier lots of free tier users right? I don’t think we hit the uh you know product-market fit uh ever. uh there were a lot of I think you know I would say a lot of free users love the offering right for the hobby projects but not a lot of commercial uh traction and you know so we had to really you know when you’re running a PaaS you’re effectively doing hosting for people in the end right yeah you’re becoming your own like mini cloud provider whatever exactly yeah so we had to run kubernetes very efficiently because as I said we didn’t have a lot of commercial traction but we had a lot of free user traction right so we had to constantly think about you know like giving everybody a separate EKS cluster just wasn’t you know viable right so we had to see okay can we hand out namespaces how do we secure namespaces then we had a lot of folks that wanted to do things across namespaces right uh and then we’re like okay how do we open things up so you can communicate across namespaces but only within your tenant right and then you know we were running these large kubernetes clusters on a multi-tenant way and it was a huge pain it was a huge pain because vCluster didn’t exist Right? So we were thinking okay how would we actually solve this problem at its core right for ourselves but then we also saw this opportunity right because obviously the past you know wasn’t commercially viable so we were wondering if we have this problem for our past maybe others have this problem as well. how does you know big Fortune 500 company with their 30,000 developers right like split things up internally and turns out they’re alone they just create 3,000 EKS clusters right so you know that’s essentially why uh you know we saw a commercial opportunity that we thought could actually be bigger than the past that we’re working on and I think looking back today that was the right move yeah

Itiel Shwartz: I think I think it was the right move and and he also already impacted the industry as a whole I think with this virtualization. So you are taking what you built in house you’re like creating v clusters share with me maybe a bit about the adoption like who is using it why should they use it and why do you have two new project like why should I use that

Lukas Gentele: yeah yeah yeah I mean adoption is always tricky with open-source project you put it out there so many people are using it you know just none of them are paying you though right like that’s the tricky thing yeah it’s true it’s a it’s a rough transition So obviously the ones that are paying customer you know like we’re very familiar with those we’re we’re in 7500 companies um you know there’s tons of you know smaller midsize companies using us commercially right the open-source is a lot more vague and bigger though in terms of the community but we have about you know 4,000 people in our Slack community for example right I think we have like 10k GitHub stars something like that right um that’s a lot of traction in the vCluster open-source project and you know What what is always the most fascinating thing for me is when people hit the ground running, they become successful with vosture and they’ve never talked to us, right? Just like last week, we had somebody contribute, you know, a couple lines of code and they opened an issue to explain why they made that change and just casually they threw out there, we would really love to add this feature because we have a thousand virtual clusters running and this would really help and you know the username was like potato or tomato or something like you know like no profile picture nothing and I was like they run a thousand virtual clusters who is this right like which which company is this right? yeah, turns out it’s a really large insurance company like you know I was able to you know I Googled for a while found out who the person has reached out and said like thanks for your contribution by the way what are you doing with a thousand virtual clusters I’m super curious right but but it’s amazing to see like people that’s you know

Itiel Shwartz: I feel like you have to have some people have like a very light open-source right it’s not necessarily allowing you to do a lot of things right um and you know the opposite is you have too much open-source and you don’t get enough commercial. It’s like a fine balance, right? You’re always like fighting yourself in a way, right? Like finding the right balance. So it’s very important.

Lukas Gentele: So we’re very transparent with our, you know, community that we are a commercial company. We need to make payroll, right? And you know, everybody who’s a lot of our customers are building software as well. They want to get paid, right? And a lot of their offerings are actually, you know, proprietary software, right? Most of the world, right? It’s not like open-source whatever exactly. So it helps being transparent with people and say okay we have a lot of innovation in the open-source we’re giving it to you for free no strings attached you run with it but here’s the line to this is the commercial features this is where right

Itiel Shwartz: where is the line for you guys

Lukas Gentele: so the line for us uh really starts when you have a lot of virtual clusters apparently you can even spin up a thousand without big cluster but typically you know once you get to like you know 100 plus virtual clusters when you get to virtual cluster in production that needs uh to be very very you know hardened resilient right it’s those kinds of features that that are for us in the uh commercial offering the commercial offering has uh two pieces right it’s like a if you will it’s like a fork of the open-source with additional features in the vCluster to make every voster more scalable faster more resilient right but then we also have what we call the vCluster platform and that really helps you manage your fleet of virtual clusters fleet management for clusters Exactly. Yeah, that’s the kinds of things like you want SSO and audit logging for all your VM clusters, you hook them up to the platform, no problem.

Itiel Shwartz: That sounds really cool. So, so share with us, you had a huge announcement yesterday, right? Yeah. What was it? Why do I need something else if I already have vClusters? Yeah.

Lukas Gentele: What we announced is vNode. so we’re going a level deeper uh on the stack now, right? vCluster is control plane isolation. is really great if you want to give people a separate cluster without giving them a separate cluster, right? They can do CRDs, they can do multiple namespaces, but they’re still locked into a single namespace in the host cluster, right? and all of these higher level uh objects in Kubernetes like you know, CRDs, deployments, stateful sets, right? None of that actually has to do with spinning up a container, right?

Itiel Shwartz: I agree. So, what we have in the underlying, you know, traditional cluster is just the pods of that virtual cluster, right? and they get launched by the underlying cluster.

Lukas Gentele: So now you have the situation where you know two vClusters running in two different namespaces use the same underlying node in the in the traditional cluster right and yeah the question is how do you keep things separate right so you can rest pod you know uh take a lot of privileges away from them make it really harden the way you run images right etc but you’re it’s never going to be perfect in terms of keeping things isolate and you So you’re going to restrict users very heavily. The other thing you can do is you can say okay I’m going to configure my vCluster to run in the same cluster but I’m doing separate node build but that is a lot of overhead a lot of cost right it’s maintenance effort right it’s you got to configure that and it it makes it much more static so the ideal vision would be you can have the same node right you can have shared nodes but in a secure way and there’s a couple of alternatives out there and we’ve seen our customers over the past like four plus years, right? Explore all kinds of available tech out there, right? There’s Kata Containers which creates microVMs which probably works in your private cloud or on your bare metal Kubernetes clusters, but it’s not going to work in a public cloud with nested virtualization. Very problematic, right? But even in a private cloud, VMs have a lot of overhead. Yeah. Right. That’s a problem, right? It’s another issue. You you’re using a lot of capacity just to spin up these microVMs, right? and then you have other alternatives like gVisor, right, which is seccompp filtering, but you know, gVisor is a Google product, works great if you’re using GKEE, not so great any anywhere else, right? And then you have things like Sysbox, which using usernamespaces. Sysbox, you know, obviously pod of Docker now, right, through an acquisition, uh, is a product that is designed to run in Docker, not so much designed for Kubernetes, right? Like we’ve seen our customers struggle with any of these tech. There’s a couple more, right? Like our customers have done, you know, they’ve forked projects trying to make it work, right? And we’ve seen them struggle for years. So we’re like, why don’t we solve this for our customers, but also for anybody else who needs node level isolation. So we announced vNode. vNode is a combination uh of seccomp filtering uh and usernamespaces. And what we ultimately do is we create virtual nodes that are not virtual machines, right? On the host node. So you can even run a privileged container as root with host PID right you see the entire process chain on the container and you on the node and you feel like you complete admin you have an entire note

Itiel Shwartz: how do you do it it’s like Linux magics on top of the nodes like how does it work like like

Lukas Gentele: it’s a lot of Linux kernel magic right absolutely this is how it sounds like because in the end of the day let’s be honest when we’re running on AWS or GKE Okay, they’re not giving you like a real bare metal node, right? Like they they they have like the concept of some virtual machine on top of that virtual node. So like it does exist but it sounds very tricky to implement like it sounds like a it’s a lot of it’s also like

Itiel Shwartz: all of that like VM is a commercial product.

Lukas Gentele: Yeah, completely commercial. No open-source at all. This is the first time we’re doing completely commercial offering. So let’s see how that goes. But uh you know day one so it’s been about 24 hours since the announcement right and uh we’ve had a lot of requests come in uh about this project to the extent that I’m not sure if we can let everybody into like it’s a private beta right now right it’s uh yeah it’s not like a offer you can just like free trial download right it is very easy to set up it’s a it’s just a helm chart it sets up uh a DaemonSet in your cluster so it runs a lightweight container on each one of your nodes And that container is going to talk it’s going to register as a runtime. Y so you see a RuntimeClass in Kubernetes and every pod you’re starting with that runtime class is going to be packaged into a virtual node, right? So we’re creating when when containerd from the kubelet, you know, gets the instruction, hey, spin up this pod, we actually create that sandbox environment and then place the pod inside of it, right, to isolate it. And when you’re using it with vos, that’s even better because you don’t need a separate vnode for each pod. this you know the overhead is small but there’s still a little overhead with with vCluster and vnode combined you’re actually going to get a vnode for all the pods from the same vCluster right running on the same like uh on the same physical node exactly so you don’t you have less overhead for for running vnode combination with vCluster that’s really where the where the magic of the two products but you could use vnode just regular namespaces as well like normally okay no sounds like very interesting and I also So like it’s an interesting direction going from only open-source to commercial like I think it makes sense by the way like right like Commodore is a commercial product even we have some open-source it’s not our core I was always afraid of like fighting myself like it’s it sounds so hard to find that balance because you’re actually competing with yourself no matter how good you are it’s like you’re your own worst enemy and we see it for HashiCorp that you know like they they they built a big a very big company but they couldn’t like explode and compared to the usage of terraform they didn’t really tap into the potential right like I mean

Lukas Gentele: in retrostackt you know it’s it’s always obviously we want to learn from mistakes of the past right it’s it’s always easy retrostacktively to say oh hash should have done you know like it’s easy for for us no I have no idea I’m not sure what’s the best way like like I feel it’s not really you know it’s not really decisive and HashiCorp didn’t want to create any commercial, right? Like it was pod of like their DNA.

Lukas Gentele: Yeah. Yeah. For better or for worse, overall great product, great company, right? But it’s like when you when you look at the way I’m building the company, my team is building company with me, right? We are all open-source nerds in one way, but we also see that you know maybe the first generation of open-source companies a lot of that has led to rellicensing and you know Elastic right a lot right so we want to be much more we don’t want to end up doing that right so we believe there needs to be a commercial product and we believe that there needs to be a justification beyond just you know support and services, right? For people to say, “I’m buying this product,” right? and for us, you know, we’re being very transparent.

Itiel Shwartz: No, I think like the transparency is like the number one key here. And I think by the way, like this is one of the reason people were angry on Elastic, right, and Redis like the lack of transparency. Yeah. Okay. Like I think we’re almost done. Any prediction like where are we going as an industry? What’s the future have in store for us? Yeah.

Lukas Gentele: This is maybe a little self- serving, but I think virtualization everywhere like everything is going virtual pod virtual.

Itiel Shwartz: I was actually going to say the opposite. I was going to say we’re going to see a lot more VMs disappear.

Lukas Gentele: I think virtualization in the traditional VM sets, right?

Itiel Shwartz: Yeah, I agree.

Lukas Gentele: Is going to disappear to a large extent because when you look at something like vNode that allows you to have lightweight virtualization without actual VM.

Itiel Shwartz: Yeah, I agree with you.

Lukas Gentele: And you know, Broadcom probably helps a little bit as well with hiking the prices, right? we’ve seen two areas where there’s a lot of traction right now. One is GPUs obviously because people don’t want VMs there at all over and then we have uh people that run VMs right now and want to move off of VMs, right? VM doesn’t offer you any big advantage if you’re like Kubernetes all in. Like who cares if the underlying is really like it’s not that big as it used to be. Yeah. Yeah. We announced a case study I think uh maybe two weeks ago with Australian company called Barton Broadband. They had 200 VMs on top of their, you know, bare-metal nodes, right? And the reason why they span up these VMs is that’s how they did Kubernetes, right? They needed the control plane node and then three worker nodes, right? Like this kind of stuff. They got rid of they went from 200 VMs, they got rid of all of VMs from 200 VMs to zero. They freed up a thousand CPU cores just for the overhead of the VMs and one terabyte of memory. freed up immediately by going from virtualized nodes, right, like VMs to bare metal nodes. They spun up one Kubernetes cluster with all of their bare metal nodes and then they run virtual clusters on top, right? And the only missing piece in such an architecture is okay, if we need stricter multi-tenancy, how do we do it on the node level? That’s that’s why we went down on that level as well.

Itiel Shwartz: Okay. So, it’s a great it’s a great case study.

Lukas Gentele: Okay, sounds great.

Itiel Shwartz: So pleasure pleasure having you Lukas and good luck. We’ll we’ll follow up on like the VM vNode project.

Lukas Gentele: Awesome. Thank you. Thank you.

[Music] Kubernetes for Humans.

This is an AI generated transcript of the conversation