#044 – Scaling Platforms and Pioneering AI Agents with Hasith Kalpage (Outshift by Cisco)

Itiel Shwartz: Hello everyone and welcome to another episodes of Kubernetes for human podcast. My name is Itiel Shwartz and today we have at the show Hasith. Hasith can you please introduce yourself?

Hasith Kalpage: Hi, nice to meet you guys. I’m Hasith Kalpage and um I run uh platform engineering for uh Outshift by Cisco which is Cisco’s uh incubation engine.

Itiel Shwartz: that’s cool. Tell us a bit about your background like what did you do prior to being in Cisco and then we’ll talk about why does Cisco have incubation center and what do you guys do but let’s start at the beginning.

Hasith Kalpage: Yeah, I mean uh in a funny way uh so I uh started my uh career in in electronics. I did some internsHasiths in uh NXP Semiconductors but then Cisco was my first job uh uh as a graduate. It was actually a Norwegian startup called Tandberg which Cisco acquired uh about six seven months into the job and then I have since uh stayed at Cisco. obviously done a lot of different roles at Cisco. initially uh uh in quality assurance then um you know was you know always aspiring to be a software engineer. So did some of that. then I actually uh got into the got into platform domain with my uh quality assurance uh hat because uh that was an area where you had a lot of um uh issues you know instability and and what happens is you know when something goes wrong in the platform layer things usually cause like major problems you know things blow up if you don’t get the things in the platform layer right. So that was uh kind of you know the what nudged me towards getting involved in uh platform uh many platforms I think I’ve been involved in probably about nine platform efforts um and I’m and very much so you know

Itiel Shwartz: you know you said you’ve been in like nine platform and platform is a big word right like it means different things to different people and I’m sure that even inside Cisco when you say platform and maybe someone else say platform it means like completely two different things. So like what is a platform like you know when you say platform what do you mean and what maybe like makes like a platform more problematic or you know more error-prone compared to to another.

Hasith Kalpage: So I mean uh I would call platform uh kind of like the underlying layer uh in on which uh you know applications and other things can be realized. so uh before uh 2015 in my career I was involved in uh what you call on-premises type of platform uh settings where you did like systems engineering and uh funny enough there was a platform engineering back in the day which meant slightly different things and and then uh in 2015 I got involved in uh you know cloud platform and you know delivering SaaS um and uh what was the next question uh

Itiel Shwartz: what’s uh like I I wanted to understand from you like what is a good platform and maybe maybe what is like a problematic platform but you know let’s go back to 20 like 2015 right yeah

Hasith Kalpage: okay yeah yeah so 2015 um I uh so we initially had a charter around uh getting some of our media services in in a uh you know from on-prem move to cloud so I was involved with uh that for one year and And uh one of the best things happened actually they you know canned the entire project and said uh now repeat it in this different other cloud back end to be honest like it’s a it’s a very good opportunity because uh you get to do the same thing better the next time around. So you know in uh on-premises like you know we basically got like you know something that was taking you know you know you were you were like you know release cycles are like 12 months 9 months uh and you know you know doing builds and things it was like a week’s effort we got u in my first effort in 2015 uh got uh you know CI/CD part to for to about two weeks then in the next effort was like okay let’s push this and really improve and try to do it in like uh 3 hours. So that was kind of interesting uh going in that path you know doing the same thing uh with you know slightly different uh setting uh so being able to kind of you know cut that uh two weeks into three hours uh improving the process uh so this is still with u nothing involving containers it is very much still uh you know you go on VMs and you know doing things uh on top of it uh OpenStack bit of AWS um and then

Itiel Shwartz: however you went on the Docker journey afterwards.

Hasith Kalpage: for very much uh uh adopting uh containers you know.

Itiel Shwartz: So, so when uh when like did you guys or you like started to work with Kubernetes, right? Like I’m I’m not sure C Cisco was like an early adopter, late adopter like it wasn’t

Hasith Kalpage: I mean there has been pockets of playing going on. So so one of the interesting is um like 2016 2018 I was running this uh uh media platform which is on containers with a completely bespoke um orchestration system and it it was like centrally driven. you had like state machines and things that was like centrally managed. This thing was like doing like about 4,000 machines which was interesting.

Itiel Shwartz: yeah, it’s like a nice scale. Yeah.

Hasith Kalpage: Correct. Right. And then uh uh the interesting thing was um uh Kubernetes like we had a problem of uh repeating environments and uh there was an effort that happened called SparkX back in the day and internal effort and uh basically the entire kind of Webex uh messaging environment was set up on uh a Kubernetes setup. you know some folks a small team did this for about u uh you know I think they they spent couple of months to do this and it was like available and this was kind of a almost like a you know uh big New Year presentation this happened in January uh 2019 and I was like super excited and uh essentially took that on you know set up Kubernetes ourselves and we got it u running in a integration environment and then primarily because we uh when we you know the kind of the feature we were testing back then. We took down integration for like few hundred uh you know not few hundred actually you know well you know many many hundreds uh engineers by kind of u uh stressing some test and things which kind of like brought integration apart and then everybody was stuck and that was a interesting situation. So to avoid that we kind of set that up and then uh we had it like you know uh you know very much for internal use case in uh 20 2018 most of 2018 actually and then uh um interestingly then in 2019 I got um you know Webex had some Webex had a big incident where security was a concern and you know how you know quickly we could bring things up was a concern. So I got an opportunity in 2019 to kind of like uh think about doing the platform using Kubernetes and uh other technologies and this kind of went on for uh few months and and really and and in funny enough in March 2020 we didn’t have any production clusters. So we were kind of getting ready with this platform in uh 20 you know February March 2020 and then you know COVID and the COVID response happened. So during that time only we kind of really uh drove this transformation um uh and I remember my uh uh boss at the time was uh telling me Hasith uh don’t try anything new just uh you know keep the lights off right I was like okay don’t yeah don’t drop the ball uh but then I was like you know we don’t want to kind of like go back so let’s you know really uh introduce Kubernetes cloud native mostly because it was the kind of like you needed agility right so if you really look at the COVID uh response and everything that was going on it’s like so many unpredictable variables right in front of that what you want in your tool chest is being able to scale fast and do things and and the what was impressive was we uh you know no Kubernetes production clusters uh February 2020 may uh uh 2020 we had something like uh 50 56 or something and uh quite a bit and like a major transformation done that would have been like impossible. So that was like the kind of like a it was crazy but it was like a very good opportunity to get Kubernetes adopted and everybody excited then so that was like yeah

Itiel Shwartz: so you know like it’s a it’s a super interesting story because Webex in the end of the day it’s an enterprise product right like enterprise product done by enterprise sold to enterprise like it’s not like the you know like I would have guessed that it is hard to change the platform right like in the end of the day you have some like nice or very nice like money maker there is I would guess like legacy code legacy things that you have done like in any any enterprise product right that like accumulated over the years how much time did it took you guys like doing this migration weren’t there like a big opposition somewhere in in inside of Webex like telling you no I see it like you’re doing it wrong let’s do something else so so it took about 3 years to get like uh most of webex basically uh you know cloud native and adopted into Kubernetes. but the interesting thing was that um uh the first few months uh it was really uh like a very interesting moment because I had like a control over that stack and we were very much u uh you know uh very fluid. So in order to you know uh make capacity you know room for like you know uh infrastructure and other cases like you know you had kind of autonomy to do that which was uh quite nice uh in some ways like you know uh it’s a very unique opportunity like in in in your career like

Itiel Shwartz: co because uh uh I remember like um you know you know I’ll give an example right you know on Friday we were taking like you It was like 2,2,500 nodes in US East for this particular back end. Right. Friday uh because we had not done the subnets big enough. Friday we would uh destroy everything, redo the subnets and and bring bring everything up and and if you don’t have it for Monday, we would be in a in a pickup basically. So like and those type of things happened uh you know it’s almost like at my level of discretion obviously you can’t muck it up and but then you know those type of decisions could happen uh you know provided you know take ownersHasith and do things you know properly. so it was like a very unique opportunity because today you can’t think of uh you know anybody being allowed to kind of like do something like that because um it’ll be like uh or you know it’s like you know you need to get GM’s GM to sign off on on that type of a situation. So it doesn’t so no like challenges like bring opportunities I think like and it’s interesting like co when co started like like I just started like Komodor with with Ben so there wasn’t like any production so I must admit I didn’t feel like a a huge a huge change and and you know like you took the decision it worked like is there any story that you can share that wasn’t that happy from that time from migration to like Kubernetes any like big challenge or problem that you encountered like I guess there must be okay let me let me tell you like one story right I think um uh so we were like very aggressive during COVID time so one was uh so we had a need to get an amount of things like done in AWS uh sort of like you know Friday the decision was made uh and we were trying to get it done by Monday obviously it didn’t need to happen but this is like you know 500 nodes uh five uh five or six clusters right uh so that was the first time we were trying to do within AWS right and funny thing is so we were obviously you know based on some of the work that has happened you know new Terraform uh new Ansible playbooks uh getting things up and running like working I mean so like crazy others I think uh and then uh I think the core team. We were like uh on Saturday like up for like you know 36 hours at a stretch and uh and then the funny thing is okay we have everything like working built up but uh rookie error in terms of uh the disk sizes so default disk sizes so they weren’t set up so after about like uh so you know we are trying testing out you know taking some you know uh traffic to validate and what not so it’s like uh things are filling up and uh you know having issues, right? So, that was like, you know, and then, you know, the team was tired. Obviously, it was a very aggressive thing to do, but then we had options and then, you know, I pulled the plug around uh uh you know uh you know to you know before us woke up uh it was like you know actually no um this is you know we couldn’t do this like use this and then it wasn’t the end of the world. we had other options but that was like a thing we kind of you know uh but was very aggressive but then backed off but the good thing was this right once you live through something like that then uh everybody rallied on right so it’s almost like uh not everybody was involved in the weekend so and then everybody else on the team they also you know jumped on to improve it and then because we had you know kind of like lived through that pain it was like we need this to be successful right and then really pushing for it and then we took another week to kind of like improve all the playbooks like like the issues we were having and then uh we tried it next Monday and it it all like you know worked out. So it’s like a kind of like in some ways uh it was uh crazy thing to try and attempt and do but it’s almost like it was one of those uh uh deciding moments. uh you need to kind of like sometimes you know push and try to do it uh and then even like you know and then also call it out right when it’s not working you know it’s like okay it’s like yeah I think people like you know people who haven’t done like ops in like a higher level in terms of like environment or production and scale don’t understand how many different things can go wrong when you are trying to do it especially if you are trying to be aggressive especially if if you are doing it for the first time like there are so many moving parts that need to to coordinate together in order to make it work. So you know I like the story you know sounds super super cool and and interesting and very you know like novel and I understand how something like that even if sometimes it’s not successful does push you to the limit push the team to the limit and then like you you know become stronger basically and better and better in what you do. So it’s it’s a very cool story. So so what happened after like Webex like right like you are in Webex you moved to Kubernetes everyone was happy you guys won right? Oh um

Hasith Kalpage: I mean the main thing is like uh I think the convergence was like very uh uh it was like a very challenging right so because you know I actually took ownersHasith of a lot of things a lot of legacy things lot of uh tooling and other thing and and the like very much I took ownersHasith to transform so it’s like really um you know this is you know uh you know killing a lot of things but then actually you know making sure that you have the buy in of the people and actually get them to understand why adopt cloud native why is it better why the tech is better why is it good for their careers and then really help everybody kind of like go from A to B you know it’s like a every team uh every kind of uh you need to figure out you know how do you get everybody from A to B in order to make this transformation happen so so spent you know couple of years there uh in that and then uh I think you know when when things are mature and uh I think you know I kind of like knew when when it was right time to move on and then uh you know that that’s the time I uh you know decide to you know do do other things uh we went and went to the CTO organization for a couple months and then the Outshift opportunity came out so so I went uh you know wanted to do that so so so tell us a bit about you know about Outshift like I guess most people don’t really know Outshift what do you guys do what do you do in Outshift and share a bit about like you know what’s the opportunity that you got back then

Hasith Kalpage: yeah so so uh so interesting thing about Outshift is Outshift is uh Cisco’s u uh incubation engine so uh our chart is to really uh try to innovate new ventures and you know both technology the business uh see how it fits you know the market, see how it fits Cisco and then really um jet you know uh product market fit and then scale it up you know hand over to Cisco to scale it up right so that’s kind of our high level charter uh and uh in Outshift uh uh so I got the opportunity towards u uh end of uh 2023 uh very much to uh you know lead all of platform engineering so everything so whether it’s you know infrastructure cloud strategy so all the things around platform engineering and I also got the CISO angle as well so uh kind of like the security aspect of it as well so the empowerment and the stick that that’s also a nice combination to have um and and because of that I you know decide to uh take that um role uh the team was you know primarily working in an SRE fashion uh so very much because you are you know a lot of ventures you know you know pulled in 15 20 different directions. So very much I uh rebooted things to okay we need to change from SRE to platform engineering think like a you know platform engineering team uh made some you know process improvements and then then also with the new innovation and things happening also wanted to very much uh be on the chair of okay how is uh AI going to um you know impact this uh uh domain and very much be in the driving seat to kind of like okay how Can we use AI to solve some of the problems we have? One of the fascinating things is I’ve you know met so many people both you know teams I have deals with dealt with and in the industry as well and it’s hard being a DevOps SRE you know a platform person has become super super hard just because of all the complexity cognitive load and you know you don’t have work life balance you could you know depending on the situation it’s crisis if you’re not there I mean something is blocked and so there’s a lot of like stress that’s uh associated with this right and uh and then you know it’s like easily I mean for cognitive load it’s like uh you know you’re not going to know the system right you know it’s like you need to be comfortable I’m not going to be able to understand all the things or even have a picture and then like you know being able to navigate it is the you know your your skill right if you can’t do that then you can’t be successful as a SRE or a DevOps right um so yeah so That’s the uh kind of a nutshell and

Itiel Shwartz: so so we you know like like maybe share a bit about like in day we are saying like Outshift here right but Outshift is something that is quite distributed right I’ll be I’ll give like a full disclosure Outshift is like a Komodor customer right so I know a bit about about how you guys work but it’s like it’s not like I don’t know like webex has like one main product I know like webex the chat I guess and everyone are aligned on top of that. you guys are much more like an incubation center right like a lot of different teams different KPI different speed like how is it different and you know coming back to the beginning you talked about your experience on building platform how do you build a platform for like such a diverse uh like setup

Hasith Kalpage: yeah I mean the out shift the interesting thing was like it’s the uh you know you have a lot of creativity and innovation right and Then uh so when I when I look at looked at it on platform the thing is you know guys why are you guys doing Kafka in five different ways right you know it’s like a you know it’s like it’s a wrong thing to innovate on right so it’s like a so it’s that’s where the uh really the platform engineering aspect comes in we really want to you know the way to go really fast and accelerate is okay what are the common building blocks what do we need to put in the foundation you do that really well at the platform engineering domain And then you know uh everybody could spend their creativity in the right place about you know how do you advance the technologies, how do you push the boundaries, how do you make the business logic better and you know those type of things without you know while you know leveraging uh you know core capabilities because otherwise what happens is you know you know in normal any situation everybody likes to do it uh their own way and then you know it’s you know fully done

Itiel Shwartz: One of like you know even if we’ll take this Kafka as an example like the problem which exists in a lot of enterprises on one hand you want your teams to be self-sufficient right you have like a really good developer is a a Kafka expert and he says like I can PC with it I don’t know like in a day and in a week everything will work and it doesn’t really want to be bounded to the enterprise Kafka whatever that is might maybe like more bloated and has some security teams and whatever like there’s always this question of how much as a platform right like you give away to your customers compared to um are you forcing them like currently like in our chief if for example I want to use something that is already offered to me by the main team am I forced to use your Kafka in this example or

Hasith Kalpage: I mean we are we are trying to so like kind of like I think when I stepped into the role it was interesting like we had more cloud accounts then we had people now like you know you know it’s like the ratio the number of cloud accounts only goes up. It’s like it’s only go you never delete the you never delete it. Yeah. Yes. So I mean uh when when you kind of uh you know then you need I mean it’s it’s great like you know if you’re when you’re uh you know you know autonomy and like okay you know it’s good that you know we are able to facilitate that type of things but but at the same time uh you know currently you know we you know put a IDP like you know stand like okay three accounts here like so you can you know uh follow these things you know and then you’re kind of like you know empowering them because your you know second you create a new cloud account or like you know anything that you create yes you have autonomy to do this one you then you need the um expertise to understand some of those aspects and you also you know create a lot of you know day two and other problems as well even if you’re going fast right so when you uh platform you know make make it a platform and kind of like give everybody a kind of a common foundation uh they could one go faster and also uh kind of like cross the silos a little bit because what happens is like when you give everybody autonomy uh completely they all are very different right how you do it then then they can’t even like you know collaborate uh and kind of like you know leverage each other’s innovation I mean I’ll take a simple example of get repositories or something right like uh uh you know why you know some companies have really gone down the path of monorepo it’s like it’s repo company. Yeah.

Hasith Kalpage: So, so like uh it helps, right? it helps like people people like I feel like don’t like underestimate like how much this like decision can help or harm like the company like even and you are like git in the end of the day like you have three repos or you have one repo that encapsulate all the three of them. Why is there such a bigger difference? But but there is like there is again for better or worse there is.

Itiel Shwartz: I said we don’t have a lot of time and I know that you have a lot of thoughts on maybe like the future and where our platform going right like we have the current state kubernetes is stabilized right like it won the battle everyone is building something on top of kubernetes but you know maybe share with me like in couple of years from now what do you think will be the biggest changes in platform or like where are we going at

Hasith Kalpage: so I can really uh you know see AI being a solution uh and and also being a big part of uh the platform side and the reason is uh simply because um uh things have gotten so complex that um uh it’s really like you know in some ways uh for us you know you know as you know platform engineers it’s hard to kind of u you know be very effective at our jobs because of all the complexity. So that’s that’s like that’s one aspect of it. And then the other problem is uh uh trying to be like platform engineering we’ve you know effectively put a bottleneck right. So uh and and that way yes while we are doing all of this we are slowing things down deliberately. So now the really the solution to this is actually you know AI I’m kind of a believer that AI has the solution. In fact uh we have been uh uh doing an effort called JARVIS internally uh and agentic platform engineering really looking at you know how far we can uh push the boundaries here and and the results have been like uh been phenomenal like particularly with a lot of uh knowledge side of things because most of the time you know people ask you a lot of go ahead.

Itiel Shwartz: No, no, no. Just because you said the results are like very good. How do you measure the result? Right? Like and I felt that you are going to answer something like that. But you have your bot, it does things. How do I know if it was like a epic success or I don’t know like a failure or something in between, right? Like uh what’s the KPIs or measurement?

Hasith Kalpage: Yeah, I mean uh right now we are really looking at the usage metrics and kind of having like very effective feedback loops around uh whether you know it was able to help or not. And then like from a team point of view uh what we are seeing is like um it’s a lot of like um uh you know operations work that uh and also like you know answering questions and handholding that AI could really front and help with us because the thing is um like I mean I I don’t know I mean how many times I have done this like uh you know over my years you know sometime I’ve given the answer and it’s in the same like I’m repeating the answer in the same chat like you know after some time like you know it’s like so it’s like that’s not a good use of a uh you know sres or engineer’s time and you know it’s not it not doesn’t give you job satisfaction right so putting um you know AI to intercept these things and like being helpful and you know pointing the documentation and having you know those type of things and as long as you can do it with you know high degree of accuracy that’s going to really give uh uh you know SREs and platform engineers more kind of time to do you know automation and other interesting things right so that’s really you know where like these things can help and then the other thing is uh fundamentally nothing has changed here like with platform engineering you’re trying to do self-service right so the power is like once you put like uh uh you know something like LangGraph and like a ReAct type of a loop on top of that it actually makes the whole uh experience a lot more powerful and uh you know beyond uh uh you know what you put something people could type into a form uh you know putting uh some of those elements it can be very powerful. So one thing we have done there is we are uh using uh uh you know reasoning uh for our sort of self self service tooling along with some uh dynamic um aspects of forms. That way you know people can actually you know it makes the makes it makes them easier to use versus like you know even like okay here are the forms and like you know just go fill it right because uh you know the big thing I think AI has changed is like u you know one of the fundamental things is we’ve changed the interface right it used to be you had to know how to code you had to like you know you had to be very technical so we kind of like simplified all of this into natural language age which is like the like it’s the interface change literally right so that’s pretty powerful uh

Itiel Shwartz: like I’ll ask one followup because you said like another word like you know you talked about like API like measurement or success now you said the word like precision like how do you know your like that you’re precise right like it’s really easy to a you know you’ll ask me a question how do I don’t know like connect to Kafka from my local host and like agent can lie they will tell you Oh, we need a VPN whatever whatever but in reality like there’s nothing to do with VPN and so on. So how do you keep track on the on on like the accuracy?

Hasith Kalpage: So, so couple of things right I think uh like main thing is like yeah the problem big problem with the LLMs are like they are like very confident and they’ll they’ll tell you like stuff uh uh you know not good things in a in a and very confident right and um I think the main thing is it’s really almost like especially when we have like you know more complex you know ReAct type of loops uh the scoping the problem and to be more precise is like uh those type of things has helped like so it’s really you know how do you uh uh kind of you know uh create your you know um uh graph you know embeddings and things like you know in such a way that the you kind of you know scope the problem to be small right in some ways like uh we’ve actually experimented this uh using like very big models sometime produces more inaccurate answers so it’s really good when you want to kind of like um reason something complex, but then if you’re trying to get to like a precise technical answer, it’s actually better to use like the smaller models that are like less likely to hallucinate and go wrong, right? And then you get like, you know, high accuracy. So yeah, it’s not it’s there are challenges, right? But I think that’s the Yeah.

Itiel Shwartz: well we just like uh I’m just like releasing now like a comparison that I did of you know DeepSeek is like super trendy like we’re talking now in February 2025. DeepSeek is deep right like this is how you pronounce is is super trendy. So I tried it and I got one very like shitty response and another hallucination. And I’m using like the same prompts the same data. It’s like I did DeepSeekek versus Claude versus Llama like Meta. Yeah. And like DeepSeek was very bad like it illusionate without even like thinking about it. So it does like make me think because I usually because in commod we are very like kubernetes you need a lot of context. So small models are a bit problematic. But now I do wonder if I’ll take like a smaller model will I get like a more accurate result for for some of those things. So uh just share.

Hasith Kalpage: Mhm. Yeah. So, so we did some uh code generation but for that we are actually uh using a combined approach. We are using like uh LLMs and symbolic AI kind of in in conjunction uh using uh yeah uh so that that that kind of uh is working well. so we can we can do like um it’s it’s better like you know uh better than 0–1 accuracy we have and we are improving it with you know the data and other things we have. So, so that’s been kind of an interesting experiment like uh but yeah the space is crazy and the velocity of things like that are moving is even like faster. so I said it like any final words like something that you want to like say to our listeners audience.

Hasith Kalpage: No I mean the other big thing is it’s really the uh AI agents like talking. I know I know this is some you know people freak out about it but I am like genuinely excited because what you have is you know uh pockets of like expertise that are going to be like uh you know what that we are going to see as agents and then data right because fundamentally like one of the big problems is you know data is not like usable everywhere and it’s like it’s going to be in pockets and so we are going to have like you know the fact that you know there will be a lot of these uh machine you know agent agents who are going to be experts uh on certain domains with data and then being able to talk talking to each other. Those type of things I think is going to be uh quite quite fascinating um kind of like you know improving uh you know what type of solutions can be changed you know solved and whatnot. So this is where like I’m like super excited about uh JARVIS and Gloria talking and yeah so we’ll see uh I think it’s

Itiel Shwartz: I think like you know it’s like one of the things that people are like now talking both around agent agent communication but will in the end there be like one super huge model right that will do like a lot of things or a lot of dedicated models that will like interact with one another and like one model will be able to utilize other models because again the problem is like a lot of the time elucination And like too much context is bad context. And this like specialty of having something that is very good on like code creation frees the other agent that calls it like to know nothing about code creation cuz you know he got it covered a bit like microservices but in like a it’s like micro agents. I don’t know like maybe I think I think it’s going to be micro-agents like this is literally we’re just going to we need to to trademark it if you are listening. I think.

Itiel Shwartz: Okay, I see it. I think that with that we will finish the episode and yeah, thank you very much for being here. It’s been a pleasure.

Hasith Kalpage: Yeah, great. Thank you. yeah, thank you for having me. Yes.

[Music] Kubernetes for Humans.

This is an AI generated transcript of the conversation