#045 – Beyond Cluster Creation: Mastering Multi-Cluster Kubernetes with Gianluca Mardente (Cisco)

Itiel Shwartz: Hello everyone and welcome to another episodes of the Kubernetes for humans podcast. My name is Itiel Shwartz and today we have at the show Gianluca. Gianluca h can you please introduce yourself?

Gianluca Mardente: If I’m look after everybody calls me like Gianluca. um and uh I’m a principal engineer at uh Cisco Systems and on the side I do maintain few open-source project around Kubernetes or which I’ve been working on for the last like three years now.

Itiel Shwartz: Yeah. And at least like one of them is like super exciting. But before we like deep dive into what you’re currently doing, do you mind like sharing a bit on your history? Like how did you start developing? What led you to tech? And then maybe a bit on like when did you first discovered about Kubernetes?

Gianluca Mardente: Yeah. So I was doing my PhD in Italy when uh I got an internship at Cisco. so I was supposed to be nine months in California and it turned out to be like 19 years. so now I’m back to Italy after 19 years. But um

Itiel Shwartz: where are you why are you back to Italy after 19 years?

Gianluca Mardente: you know we got two kids uh I became a citizen uh and my kids are uh with dual citizenship as well. So we decided to give Italy Europe a try. You know we thought that uh we don’t want to be like in our 70s and uh think what would have happened if we had moved back to Europe. So we give it a try and you know uh so far so good but we can always go back now. So that’s made the easy uh is easier. Yeah.

Itiel Shwartz: Okay. Sorry. I I stopped you. So you started like at the Cisco like in the like in the right.

Gianluca Mardente: Yeah. So I started like in uh so I went Cisco 2004 and then uh you know um I left Cisco and joined Cisco back a couple of times. I like

Itiel Shwartz: like what made you leave and what made you come back like it is uh the last uh

Gianluca Mardente: the last part the last time because I want to start working in Kubernetes uh it was 2019 and I had started Kubernetes on my own but I really want to learn it. So I said you know like the easiest way to learn it is if you go to people and work with people who have master like four years. So I joined Tigera and uh the company behind Calico. So I was I learned a lot. It’s an amazing team. It’s an amazing company, fantastic culture. and then um another project started at Cisco which was around Kubernetes creating like Kubernetes clusters on-prem on demand and um yeah because it was starting from scratch and they were looking for people who want like to contribute. says okay I’ll sounds like a right time for me to come back

Itiel Shwartz: so maybe before maybe even a bit before that do you mind sharing a bit like why do Cisco even care about Kubernetes and he said that you know he joined Cisco because they have like a very like strong people but you know like for me Cisco is in the end of the day like like it’s not that Kubernetes is the first team that comes to mind right I’m putting aside like all of the Splunk maybe now and Okay, now Cisco is huge, right? Yeah, Cilium is also quite new, right? Like it’s not really like Cisco and Cilium is very network oriented. So I’m saying okay like Cisco is doing network, they bought Cilium make complete sense. Why does like like uh Cisco care about Kubernetes? What’s the agenda? What are they trying to achieve? And yeah, like what what what give us like some context and like some background on that area.

Gianluca Mardente: Yeah, unfortunately it didn’t uh it didn’t last long. it uh this project like started because the idea was you know Cisco has uh servers and uh Cisco um you know as networking like and the idea was to build uh to manage uh Kubernetes clusters on-prem because you know like it was a tendency already that we saw like years back that uh people start considering the cost of the cloud and considering like moving back on-prem and uh I was you know I think considering what has happened like to VMware with Broadcom now it probably would have been like a great opportunity for Cisco but you know the project didn’t go far away um we worked like for it on on the project for a couple of years but then restructured and uh the project was canceled so I don’t think at the moment like Cisco has anything that big around Kubernetes you know Cisco is big and there may be like different business unit that may have some project around Kubernetes So um I might be wrong at least a view that timing um it’s not uh it uses Kubernetes because you know like it’s becoming like it it is the um um platform that uh everybody use it. So it has applications and solutions around Kubernetes but it’s not really um much involved now like in um creating those Kubernetes clusters on-prem as the project was supposed to to achieve. uh and um and when the project was crown essential was when I decided like to um I saw something that was missing uh in the project and I saw something that I didn’t find even like outside and that’s when I start like my open-source project because I did I do really like Kubernetes I love it I love the idea behind it and um

Itiel Shwartz: so so share share a bit about more about that like what why how uh like the project

Gianluca Mardente: oh the project is because we were using um Cluster API which is great to create um uh Kubernetes clusters on demand. So to create, upgrade or delete a cluster. So we did develop our uh Cluster API infrastructure provider. And then in this project we were managing many things. We’re managing the CNI, we’re managing the storage, we’re managing like the ingress. and uh one thing I noticed is that everybody was reinventing the wheel in the sense we had like different controllers and all those controllers were writing the same code to deploy something in a newly created cluster or upgrade something that is supposed to to go to a production cluster or to staging clusters and uh Cluster API is great but essentially it really stops at the cluster creation. They have something to deploy you know um resources of a Kubernetes cluster but it’s very limited and uh and I like to to do something there because I like the idea of that you programmatically say what you want and then uh um either creating a cluster but you know the same concepts apply also now what those are all the add-ons and applications that I want to be deployed on a newly created cluster it shouldn’t yeah should concept is the same and so I started working like cons because one of the things also that I noticed is that when you start managing more than an endful number of clusters you have more admins that are involved because you know like Kubernetes is so vast that nobody has knowledge of all the add-ons and all the application needs to be deployed. So very likely you have an admin which is an expert in uh monitoring and observability. There’s an admin who’s really good at admission policies and uh and the idea should be that those ad means can they can create a set of policy needs to go to production cluster and set of policy needs to go to staging cluster and set of policy needs to go to testing clusters and then when a new cluster is created the only contract that should be between those admins and admin that is creating the cluster should be you know if you’re creating a cluster and the scope of this cluster is production just put this label and because you put this label all the policy needs to go to the production cluster which have been prepared by different domain boom get deployed and um uh it should be like know

Itiel Shwartz: let me stop you for a second because I think I got you but let’s let’s just do like a recap for for our listeners what you’re saying is first of all Cluster API is great like before Cluster API even creation of cluster was very like uh scarce and sparse and Cluster API makes it easy to have some recipe to create the cluster. But once you have a lot of clusters then you found that they are not like the same like like they’re drifting or there is lack of standards and you want not only the cluster you want to be able to install in the same process in the same way uh the same things basically and to make sure that they work which is quite similar like if I take you back like 10 15 years ago a bit like I don’t know like docker or applications in general you want to have a cluster recipe, right? Like a production cluster recipe. This is a production cluster. It should have I don’t know like Fluentd and it would have maybe Datadog and some Cilium or whatever. And this is the cluster and if a new team wants another cluster, all they need to do is to say, hey, please provide me like a label production cluster and everything will magically works, right? Like that’s did I got it right?

Gianluca Mardente: Yeah, exactly. Yeah, you need consistency in creating those clusters and uh you need to be really fast creating those clusters. so you don’t want essentially uh an admin to start doing kubectl apply or anything. you just should be able to create programmatically a cluster and you can do with Cluster API or you can do with you know whatever tools you decide to use to create the cluster and then essentially if you have a management cluster uh you can simply say okay I’m creating this new cluster and this is the label for this cluster and because this cluster this set of labels now this is the Cilium configuration this is the Fluentd configuration this is the Prometheus configuration this is the Kyverno configuration and everything and those are the set of applications which uh you want to be running in a production cluster and boom they are up and running and it is consistent because you can reproduce you can do the same things every time you create the same cluster as long as you apply the same set of policies the same set of add-ons and applications gets deployed and and also you know once uh as a day two operation if you realize that okay I have uh four production clusters and I won’t upgrade Prometheus because there is a series that has been fixed in the latest release and I won’t like upgrade to the latest release. There is a single place where you say okay this is the configuration for the production clusters and this is the version of Prometheus you change in one place and boom gets upgraded in all the other places. Of course you know um there must be other considerations. You might want to roll out the upgrade like few cluster at a time but the idea should be that there is a single place where you go and change the configuration for production clusters and all your production cluster gets upgraded.

Itiel Shwartz: No, like it sounds sounds super interesting and makes sense. Maybe share with me a bit on the alternative like if I’m not using your project what do I do I have hundreds of cluster or maybe like a dozens right like I’m a small company h what are the alternative is Crossplane an alternative would you say terraform maybe is an alternative if not using you what should they use and how and why um I mean uh you could Crossplane could be like an alternative many uses a GitOps approach uh I do like GitOps approach. I don’t think too much what I’m doing like is an an alternative to GitOps. So the way I see is that with GitOps it’s it’s fantastic if you have uh um but it it is it becomes tough like for day two operation. Let’s let me put this way like it’s you can do like you can manage uh a fleet of clusters with a group approach. But uh um let me take one example like to explain why I think like it’s it’s difficult for the two operation. Let’s say that you create like a set of production cluster. You create like 10 production cluster. We are all the same where you deploy like a bunch of applications, right? And and now for whatever reason you decide that you want to upgrade only one application only in one of those production clusters. So you know you want your cluster to become like slightly different, right? the idea that uh um the way uh you can do with with with Sveltos for instance is that you can Sveltos is a concept of tier. So you can introduce like another application which has a tier which is higher than you know the configuration that you already have. And so what you can say is like okay this application is is more important. It’s matching only one cluster out of all my production clusters and it’s going to upgrade only this application. So what happens is that the application gets up upgraded like only in that specific cluster but the beauty is you’re not changing the rest of the configuration. So for sure all your other production cluster do not get affected. If you go with a GitOps approach unless you had this in mind right now it becomes difficult to take the cluster out or change the configuration just for that cluster. And what I’m scared of is that you know like if you want to change just a small thing in a a subset of your cluster whatever you do you must have something and guarantee that nothing is going to affect the other cluster and of course if you’re going to change the configuration you know the risk is there. So it I I find that there should be a solution that uh allows you to to do those day two operation like easily and we guarantee that nothing is going to happen to the other clusters. another thing which I think it’s uh it’s it’s useful is that um sometimes you may need uh um to you may want to use your management cluster as sort of a database in the sense I’ve created uh uh different secrets in this prov in this management cluster which contains different credentials and I want to deploy this application in all my production clusters and this application requires some credentials but the credentials that should go to cluster one should be different than the credential that go to cluster two because I’m giving cluster one to Gianluca and giving cluster two to ETL there so I want to add like different credentials right so so the way you can do with is that you can create your configuration is like okay this this application needs to be deployed in all production clusters but you express as a template then you say if you’re deploying on host one you should get the credentials from the secret in your management cluster. If you’re deploying this application on cluster two, get the credentials from the secret two which is in the management cluster. So now at runtime what happens is that it’s going to deploy this application on cluster one. is going to get the secret which is in the management cluster which contains the credential for Jooka and so the application gets deployed with Gianluca’s credentials and when it goes to cluster two it gets a different secrets from the management cluster which contains the Itiel’s credentials and now the same application exactly the same application from the same template gets deployed on cluster two but with a slightly different configuration with the credentials that allow pretty much you know with different credentials. So the this is another things that you cannot do like with uh uh with a GitOps approach because in a GitOps approach you have everything like in in in the repo or another example would be if I want to pipe if I very likely I want to pipe like multiple tools Crossplane for instance is great if I want to create resources outside the cluster if I want to create uh a Google storage bucket I will use Crossplane right because it’s very easy to to say what I want now assume that I want to deploy uh a pod in every production cluster an application in every production cluster which is supposed to fetch some data and push those data to a storage bucket right so every time I deploy this application I also want to create a storage bucket so if um you know I want to create a different storage bucket for every cluster so what I can do uh if I know that the management cluster can be used as a database now I can say okay create instruct Crossplanelane I create a single configuration that says instruct Crossplanelane to create a storage bucket. I’ll wait for Crossplanelane to create the storage bucket. When Crossplanelane is done upgrading creating the storage bucket, Crossplanelane is going to store some information in the status of the Crossplanelane bucket resources that says like this is the URL to connect to the storage bucket. Now I can get that URL and then deploy the instantiate my pod and deploying cluster one. And so now I created automatically a bucket for cluster one and I deployed a pod in cluster one which is collecting some information and uploading this information to bucket one. When I need the same application to be deployed in cluster two. It is essentially the same template because now it gets again instructs Crossplanelane to create another storage bucket. It gets this URL from the storage bucket. It instantiate the templates deploy to cluster two and now the pod in cluster two collects the data and boom it uploads the data to bucket two which has been created by Crossplane. So if you want to do like this piping of multiple open-source because you know it’s there are so many great open-source that you want to like likely tie all those things together. having a solutions that where you can chain events and you can use template to fetch um fetch data from the management cluster and use template instantiate those templates with those data and deploy to the various cluster. Now you’re reducing the amount of configuration that you’re creating because you’re creating a single configuration which boom gets deployed like to various cluster with uh different data which are coming at runtime from the management cluster and maybe those data in the management cluster were created by other open-source tool at runtime. So you can do like this training.

Itiel Shwartz: I think what you’re describing is you know becoming more and more like painful as customers have a lot of clusters to manage and I think like the problem is indeed there h you didn’t talk about it too much but so what is the best alternative if any like in the market open-source not open-source uh

Gianluca Mardente: I mean you have to go with uh I would say that there are solutions like the I don’t know if the Rancher Fleet is still there but you may want to look at a solution that uh it’s actually considering those clusters not as individual cluster but somehow like as interconnected clusters because if you’re looking at those clusters as independent clusters you may have a solution that allows you to consistently create a deploy application on those clusters but then you’re missing a part of um of things that you can do because Another example just to make it clear for instance uh this was one example that someone brought up like can I do I’m creating like an internal developer platform can I use to do this and their use case was uh I want uh to have a cluster where I have my Postgres databases and uh I want my application using those databases to be like in other clusters right because I want to use default domain and so I have like different clusters and by by database in a different cluster. The idea u the request was okay so when I create a cluster I simply want to label this cluster with the label possess required and because I added this label on the cluster now I want Sveltos to detect that this cluster requires a Postgres database uh in use CloudNativePG to create a Postgres database in a separate cluster now fetch the credentials of this cluster expose this database with a load balancer service so get the load balancer port 80 collect the credentials port and boom send this information to the application in the cluster one that needs to access this database and now I can automate all those things right it’s you simply put a label and everything is done

Itiel Shwartz: it’s like it sounds like a magical plumbing like idea right like both creation of the resources and connecting it to like the cross cluster I think we are like pretty much close to like ending the episode like there’s one thing that I really like to ask is what’s your prediction like where are we going as an industry inside the Kubernetes ecosystem as in your project like if I’m going to talk with you in two years what’s going to be different or five years you know whenever we want to talk

Gianluca Mardente: uh my hope like I’m I’m a lazy engineer in the sense that when I do something I want to do it once and I don’t want to touch it again so my my hope is that the amount of configuration that we have to create to manage a fleet of ephemeral clusters is going to reduce the very minimum and there are tools that are going to automate everything. Ideally what I want is that I come and say uh if X happens I want Y to happen and I express in English and then there are um LLMs that are able like to translate this in configuration either is Sveltos or whatever like other solutions that is going to be there. So translate in that one put the right configuration and boom like anything happens and uh you know we are free to enjoy our weekends nights and we don’t have to worry about like when something happens something needs to be done.

Itiel Shwartz: No it sounds like a great future. So with that I think we will close today episode I will share like the project Sveltos project in the footnote of the episode. H pleasure having you.

Gianluca Mardente: Thank you so much for having me. Thank you. There was money.

[Music] Kubernetes for Humans.

This is an AI generated transcript of the conversation