#052 – The ‘Short Long Path’: Mastering Abstraction, Culture, and Kubernetes Scale with Shemer Mashiach (Playtika) & Scott Rosenberg (TeraSky)

Itiel Shwartz: Hello everyone and welcome to another episodes of Kubernetes for humans podcast. My name is Etiel Schwarz and today with me in the show I have two guests both Scott and Sheammer. Please introduce yourself.

Itiel Shwartz: Scott.

Scott Rosenberg: Awesome. So great to be here. Uh my name is uh Scott Rosenberg. I’m the lead architect in the CTO Office at TeraSky, a global systems integrator and uh happy to be here with uh Sheammer.

Shemer Mashiach: Hi, I’m Sheammer. I am the lead director of architecture and IT platform in Platica. Um happy to be with you.

Itiel Shwartz: Okay, that’s like a good start. If like one of you can like share a bit about his experience and his background like where did you start? what led you guys uh to the Kubernetes space and maybe also how did you meet? So uh each of you can start.

Shemer Mashiach: Uh so I will take it. So basically uh well I’ve been in this industry for the last 30 years. Uh got to Kubernetes uh since I don’t know 2016 uh long time ago for uh like everybody else from the lab of the technology and then to see what we can do with it. Um and I’ve been working with Scott for more than I can actually remember or should uh which was uh a very, very, very, very productive and happy journey for now. Um that’s been

Itiel Shwartz: yeah and uh you know like you have to give like more background sh so a bit more about like what is it that you currently do and maybe like how did you get like 30 years is indeed like quite a lot of experience everything let’s talk about like the last five years at least and like if you can like share a bit more in details like what is it that you actually do or responsible

Shemer Mashiach: so basically my role is divided into several pillars one of them is architecture And the second one is everything that relates to engineering and platforms. when it comes to architecturing and um deciding or choosing the best solution for in this case PLA which I’ve been working for the last eight years uh the role is to find the best solution for the best uh either budget if we want to buy something or if we’re going to the build plane is tend and pick something that is open source with the least amount of effort to put into in order to make it work while getting the best results from this project/fro. Um the Kubernetes realm in Platica have been with us for a very very very long time since uh it was version I don’t even remember what was the version it’s like

Scott Rosenberg: 1.5

Shemer Mashiach: five or 1.6 six. Yeah. Um and um we started this process where we would need to do everything inside because there were none um any solutions out there in the market and later on recently in the past few years when we understood that you know the market has matured and got to the point where we can actually buy something we shifted this um paradigm and bought what we’re going to discuss on this H [snorts] podcast um and this is basically the main description of my role. It’s like to get from this side to that side or by the way we also had other opportunities when we moved from something we bought into an open source solution that we understood that they are better at this point in time.

Itiel Shwartz: Okay. Now that that makes more sense. Maybe if you can like just in like two sentences and then I give Scott to talk Platica a bit about the company like I guess some of our listeners are not unaware of like the company even it’s quite famous in Israel we have some

Shemer Mashiach: outside of so is a gaming company we are um currently running more than 25 different games that are running on mobile basically uh we are the largest company for those uh casual social games on in the world. Um very complex orchestration, very complex architecture, a lot of infrastructure. We are running on-prem solely on-prem. We are not using any uh cloud solution. 95 of our traffic is located in the US. So basically we’re running in a collocated cluster um data centers in United States. 3,000 people globally um almost all over the world mainly Israel uh Ukraine and Poland but there are a lot of offices everywhere um very professional people very professional technological teams um that is very very very interesting and challenging to work with and that’s it

Itiel Shwartz: okay so I think like that’s a good intro Scott, now now it’s your turn.

Scott Rosenberg: Yeah. Yeah. So, I Sheammer said he’s been in the industry for 30 years. I’ve been in this world for 30 years. Um, but you know, [laughter] um, you know, I’ve been, uh, I grew up more in the traditional VMware on-prem world um, back starting in around like 2014. Um, and as of 2018, uh, I’ve been at TeraSky. So for the past uh you know seven and a half years or so um and my first introduction to Kubernetes was back in the days of VMware PKS um when they released their first Kubernetes offering I was very involved in the VMware ecosystem started with that loved the Kubernetes part didn’t like the product part and from there you know moved on into uh really just loving Kubernetes uh have been a contributor to Kubernetes for the past seven or eight year I basically like seven years or so um since the beginning uh was finding bugs and just contributing back um and uh that’s grown into a lot of the CNCF ecosystem around backstage Crossplane Kubernetes uh cube apps back in the day carell um and have really just you know dug into the Kubernetes world and over the past two or three years or so I have been really focusing in on platform engineering as well. Um, where I believe that Kubernetes is a key pillar point of that, but it’s really what’s interesting me a lot more these days is the layers above Kubernetes rather than Kubernetes itself, which has been basically democratized. Uh, go to any, you know, little tiny mom and pop shop cloud and they have a Kubernetes offering, right? Kubernetes is a few binaries that run and it’s complex and I don’t recommend anyone do Kubernetes the hard way unless you want to suffer but you know everything above it is where it’s really interesting um and that’s where I’m spending most of my time these days and helping customers um you know build those solutions out and I’ve been working with Sheammer and the Platica team for quite a few years um going down this journey together.

Itiel Shwartz: No, that’s a great intro. Maybe if you can like so why like TeraSky like why uh professional services and not joining a company and like if you can share like in a word about what do you guys do?

Scott Rosenberg: Yeah. So TeraSky is a global systems uh solution integrator, right? Many companies are systems integrators. We’re the same acronym but a solution integrator and it really comes down to the philosophy of the company. We started back in 1985. Uh the company has gone through multiple iterations and has grown but started really from the let’s say on-prem data centers backup data protection stoRAGe into the vSphere world into cloud and DevOps and all of that. Um, and we’re now a global company, an Israeli company with presence in New York, London, the Baltics, Israel. Um, and when I, you know, started out looking for where I wanted to work, I really wanted a systems or solution integrator. And I think the reason that it catches my eye is that I have two key ideals in my life, in my technical world, that I want to always follow. Number one is always being 100% technically honest. Um, and that is very difficult a lot of times when at a vendor where your product always has to be the best in a specific area, right? Like there’s this kind of balance that you have to play back and forth. And being at a systems integrator, I kind of have more of a freedom to choose what I want. Um, which is really interesting. The other thing that I really like working at a systems integrator is that I just get exposed to so many different unique um implementations and questions and this and my toolbox is just so much larger that I can play with. Right? When you work at a vendor, it’s much more um you know tied to what’s in our tool set. What can I offer you for my tool set? But if the customer has a challenge in stoRAGe and I’m a networking company, the there’s not much that I can offer there. I don’t like that situation of I don’t have anything to offer. Um and working at a systems integrator just gives me that more freedom to kind of widen um that you know area where I can try and offer something to my customers and look at a full solution and not just a system. Um right I think that I the plaint the project with you pla I have Sheammer I think that there may be three or four products involved in what we have right now and about 40 open-source projects involved in it and building that huge large ecosystem is where I find interest.

Shemer Mashiach: Yep.

Itiel Shwartz: Okay. So now now now it’s a good segue to like what is it that you guys are working on shimmer like you said that your responsibility is like trying to find the best new technologies or maybe like the adoption for like the tech like the Platica in general. So if you can like share a bit about like your guys like journey what did you work on how did it start and what do you currently do? Uh so

Shemer Mashiach: so we started this like journey of developing our own internal infrastructure or platform um seven years ago something like that uh eight when we started it there was nothing actually or tangible that we can use in the market. So we started like building our own. It’s not literally building our own. It was open-source Kubespray baseline uh as the infrastructure and then on top of it a lot of customizations and scripts that were created in order to support what we need. Now um back then it was an excellent solution very uh technologically complex but very let’s say mature and workable solution for pla. You need to understand that the size of Platica even when we started running Kubernetes is huge and we could not find any let’s say vendor grade or production grade solution so we needed to build our own um now as

Itiel Shwartz: maybe like let me stop you here even I feel there’s a lot of the story so you know you guys had already like a running infrastructure right a running like games

Shemer Mashiach: yes

Itiel Shwartz: everything was okay right like you’re making money, everyone is happy. But something did bother you, right? Like to go into this like journey. Like what were the reasons of you know like saying we need to like to search for like something right? Like what led you to know that what you currently have is not good enough. So at some point in time several years ago we started to notice that there is let’s say a tension between the things that you want to develop and need to develop in order to make your current platform more robust, more mature, more features and this goes towards first of all priorities, budgeting people that needs to support all of these. Um and at the same time you start to see that the industry is catching up with you know us and at some point in time when you actually understand that you know Platica makes its money from games not from developing Kubernetes platforms. So uh once these you know truth becames to be a reality in the management level and downwards then we started this journey of okay let’s see what’s out there and see how it can fit to our needs and at that time um TeraSky and uh Scott came and started to work with there were several iterations of trying to find the solution um

Shemer Mashiach: what we currently have in production or running in production today was not the original tested

Scott Rosenberg: right

Shemer Mashiach: it was a very long journey with several let’s say challenges and difficulties where we actually again because of our size and because of our implementation and part of it I would say it’s even a cultural issue uh we could not get it worked at that time um And mainly the solution came from uh V uh VMware like TKG uh that you know that that Scott tried to uh work with that and implement here. You can provide more information obviously. Uh but I would say that the game changer or the mind shift came where management understood that we need like to actually go to a product that comes from the industry. Um there was a very long process of understanding what we need. Uh and when we chose in this case Spectro Cloud or pallet from Spectro Cloud uh then with the backup of our management this process became much more let’s say constructive and we started the process. Scott you can add.

Itiel Shwartz: Yeah. Yeah. Scott give your your point of view and then I’ll a couple of followup.

Scott Rosenberg: Yeah. I mean I think like there’s and I completely agree with what Sheammer was saying. There were a lot of you know technical decisions that I think you know Platica has always been a you know a leader uh always going ahead of the curve in many times in terms of technology right from the first adopters of the whole spring ecosystem right a very heavy Java company uh immediately moved into service discovery what everyone talks about today in the world of service mesh they were doing it before service mesh was a term um using things like Eureka right? Um they were doing all of these things way ahead of the curve and what that ended up bringing is huge challenges though because when you’re ahead of the curve um what ends up happening is that the curve grows in the same direction right we need service discovery industry agrees what did they not agree on the implementation of Eureka they said no it needs to be done at a different level well now when you want to modernize a platform the biggest challenge is okay but we’ve been building something in complete opposite implementation of where the industry is and certain things it’s easy to move between or it’s you know doable with x amount of people moving between service discovery systems or having them understand one another and coexist is a hell I don’t wish on the worst of my enemies right there are so many things that come with this complexity of moving over time and play skill just so people realize they ran out of IPv4 IP P space because they were doing routable pods.

Scott Rosenberg: Um, right. Like that’s where a lot of this came in, right? And I think that the biggest cha change and the reason that we’ve actually been successful over the last year and a half together in this, you know, real monumental change I think from many perspectives at Platica was actually one key thing which is that till that point it was about, you know, kind of taking out the fire, right? There was a fire lit. We’re out of IPv4 space. How do we deal with the issue of routable pods? Ooh, there’s an issue over here of service discovery. How do we make it work over here? How do we replace your Kubernetes with a product? The way that it was actually done this time is we had dozens and dozens of hours of interviews with every single persona that touches the platform from product management to developers to SREs to DevOps to operations to management everyone to understand what are your challenges what works what doesn’t work and we tried to look at it from a holistic platform view um instead of looking it from a product view from a feature view how do we look at this from a more holistic view. And then once we did that and we had management together with us on this what we were able to do is say okay yeah it’s not going to be a easy to move to a cluster API based mechanism from a you know imute from mutable nodes where we do in place upgrades that’s never an easy change for someone but when we look at the bigger picture of where do we want to go what is this going to enable us then the pain from a specific feature can be swallowed, right? And I think that we failed too many times on these deep technical decisions on one feature instead of looking at what the full platform gives us. And that’s I think the change that, you know, together in this partnership that we’ve built um has enabled for Platica to really build out this mature platform because we looked at it from that more holistic view.

Itiel Shwartz: No, I think I think like that’s that’s like a very interesting point of view and I feel that it encapsulate like you mentioned like the pain right of like small decision or like a tactical decision compared to the more like strategic value that something like that can can provide. So maybe if you can guys like take me to the start or maybe like to a pivotal moment of like what were the main challenges in the beginning or again like it doesn’t have to be the beginning but a project like that in that scale in this like bleeding edge as you said right like there was no technology maybe if you can share a bit about the challenges and how did you overcome it and maybe like challenges how did you overcome it and maybe like as an additional point what what would you do differently or if some of our listeners are now moving to Kubernetes. What are like the main like points that you guys learn? Uh so

Shemer Mashiach: challenges in what challenges in the current implementation project?

Itiel Shwartz: No, in the past like if you want you can also like every challenge is a good challenge for me or for the listeners. I’m more interested in like the migration part but

Shemer Mashiach: so okay so I can I can say

Scott Rosenberg: I would say just on that for one second Sheammer I think that there’s two separate issues here that have to be talked about the challenges right number one was the technological challenges. Now technological challenges at the giants at the unicorns like Platica may be relevant to different companies at different levels. I think the other one that is extremely challenging is the people culture thing that I think we also need to cover because that’s where most of the challenges end up happening. Right. Technological challenges a few lines of code can fix.

Shemer Mashiach: Yes.

Scott Rosenberg: Right. So you can choose choose the challenge.

Shemer Mashiach: Go ahead. But no, it’s not just you cannot. But this is exactly the point. You cannot choose. You need to understand that again particularly for us, it’s it it might be different in another company, but particularly for us, it’s it’s more of a psychological and uh internal process that this company needed to be in in order to start this process. Now once this mind shift started and was delivered from bottom down then it was very uh let’s say not very but relatively easy to start the migration because first of all Platica was already uh based on everything is gated everything is uh u infrastructure as code so there is no let’s say special development implementation that needs to be done. Uh we changed the architecture a bit in order to implement pallet but generally speaking the places of our engineer needs to you know just migrate the same files in a different format because pallet works a bit differently was not the main challenge. Again the main like the number one challenge was the people. But as we started to work in this project and people start to see the value of this platform then you started to see the embracement of this uh you know first of all starting to work. It’s much easier because now everything is declarative even in a better or easier way that it was before. uh we also needed to create uh let’s say not create but use as we said before uh an open-source solution which in this case was KCL in order to uh rapid everything for the end users uh which basically also provided uh a lot of let’s say interest to our DevOps uh people uh because it’s not just you know just copy the YAML deploy it copy the YAML and deploy it you also need to create some kind of a wrapper

Scott Rosenberg: which like to laugh about because the name is we’ll tell you shortly about it.

Shemer Mashiach: Yeah. That uh we ended up with the same place where we started. It’s like a circle but um I would say that besides of besides these the other challenge which again it’s a mitigated one was bugs and things that we found in the product itself because you know although it is a vendor and it’s an enterprise and everything is good you know bugs happenings and feature requests are lacking so we are working constantly with Spectro Cloud in order to fix it. Um and again most of these issues relates to size. It’s because of the Sheammer size and the huge infrastructure requirements that comes from Platica. We manage to let’s say I wouldn’t say break but we managed to harm even this solution the fixed part of it. Everything else will be fixed obviously soon. But it’s also interesting to see how you know this will impact eventually on the end product that other people can uh get benefits from because of our experience with large infrastructure.

Scott Rosenberg: Yeah. Uh I think that one of the key things here you know is that the migration for play on one hand from the technical challenges may have been one of the more difficult ones because of again Eureka routable pods dot dot all of these challenges that existed. However, there was one huge thing that I have to give, you know, huge kudos to the Platica team originally when they built this first platform was, you know, forget the actual implementation of um, you know, what you know this deployment mechanism of things into Kubernetes was that they had already built an abstraction layer. Now that abstraction layer meant that people had a YYAML file. It was not a Kubernetes YYAML file. It was a YYAML file that went in the end to a Jenkins pipeline that went and did 35 million different things that ended up deploying to Kubernetes. But the fact that there was the abstraction layer and the user didn’t deal with ingress. They dealt with what is the host name that I am supposed to expose on things were intentbased and not let’s say the declarative nature of the specific KRM resource in Kubernetes. What that meant was that moving between different solutions was actually a lot easier. And I think that’s a huge point to anyone building things in Kubernetes. That’s one of the reasons we went with Crossplane here very heavily in this project going with Crossplane was because in Crossplane we can build these abstractions that let the user be intentbased, not need to care about the underlying resources. And creating that abstraction layer allows us then to play around with the backend fields because one important thing to know there’s R&D at Platica right the development teams of the studios what interest do they have in moving between a Kubernetes distribution A and B the answer is none and in the end when things when push comes to shove feature makes me money moving to a different Kubernetes cluster doesn’t make me money and that’s true in every company on planet earth and therefore finding the ways to do these things even if it’s you know um there’s a famous line of my monities of the Rambam that says to take the short long path right or the long short path and what this ends up being is that the long path right of the migration we didn’t take the short path but that long path ends up being the short path because in the end our end users didn’t experience experience anything except better service, right? But they didn’t need to change their abstraction layer. And I think that’s a key point for anyone doing this is abstraction, abstraction, abstraction because if you build the right abstraction layer, you’re able to make the technical decisions in the back end and make those changes. As difficult as it may be for the DevOps SR platform teams, that is doable. You don’t want to impact your enduser that’s actually consuming the platform. Um this was a major key for the decision of going into this project because eventually everything is measured in the migration plane because to install something or set it up like a clean slate very easy it’s like you know we install it one day everything is up and running but migrating from current infrastructure to new infrastructure

Shemer Mashiach: yes I can tell you that if it was not that let’s say easy that fast it would being a very very very big challenge to this project

Scott Rosenberg: right there’s a whole here where like you guys use cloud stack now right which is an abstraction layer above multipleers bare metal all of this as a mechanism to again we’re talking about abstractions right there may be decisions that change something on what infrastructure I’m using we’re using uh linbit purios here for a stoRAGe solution which abstracts away am I doing what type of stoRAGe I have under the hood here, Crossplane abstraction layer, right? And always creating those abstraction layers at the right places is what’s allowing us this flexibility to fail fast and move forwards and try something else in the back end and find the right solution. And what Sheammer mentioned about the name, you know, as he as we said before, the first attempt at trying to let’s say bring in a product of Kubernetes by Platica was PKS from VMware. uh there were multiple I believe doubledigit numbers of versions uh that had the internal name in VMware with Platica in it because of actually fixing issues to try and make it work at their scale. Um and as I came in to review you know some of the work that the SRE team did building out this Kubernetes deployment abstraction above Spectro Cloud I come in and the name of the service that they create is the Platica Kubernetes service and it’s back to being PKS. So finally we can see that PKS actually succeeded. It just took about six or seven years.

Itiel Shwartz: I think your point here is like super super interesting. So just to make sure that I understand. So to recap to my listeners, you’re saying we already we were in a huge scale. We already had some very decent abstraction and that allow us as a company to move faster to experiment better because the end user didn’t really know necessarily that this change happened or it allow us to minimize the impact of the change thanks to our like abstraction layer. Is that correct?

Shemer Mashiach: Yes. But maybe keep in mind at the same time okay and it’s very important to take this as a you know lesson learned abstractions are excellent but don’t abstract too much it’s also something that needs to be put on the table. So the main like the main guidelines in in in this case is to keep let’s say a single level of abstraction like one abstraction layer on top and that’s it. We did it with stoRAGe. We did it with uh uh hypervisors. We did it with Kubernetes uh agnostic deployment to any environment not just on-prem because in this solution we can deploy the same cluster on any Kubernetes vendor out there can be AWS it can be you know anything else. Uh so and that’s it. We are not creating abstraction on top of obstruction on top of obstruction and it’s because people tend to you know we as engineers it’s very nice I would say to create some kind of yes I want one button I will click it like somewhere up there and everything will work. No, it’s okay to to say that a solution might be uh a composition if we are taking Crossplane. So it’s a composition of multiple steps and it’s fine. It doesn’t have to be fully fully fully abstracted.

Scott Rosenberg: And I think also the key with that is one of the challenges if we say about the initial abstraction was abstractions over time uh get very leaky. Yeah. Because in the end there were leaked things of in the attempt of bringing in ISTTSTO a few years ago where certain ISTTE fields are now exposed in your abstraction because how did you let the person decide if they wanted or didn’t wantto depending on which service discovery they were using in their environment. So you end up leaking things. I think the key when it comes to an abstraction or anything in platform engineering is that 8020 rule and that’s really what we’ve tried to follow here. We can’t build an abstraction that is going to cover 100% of all use cases. How do you build an abstraction layer that can cover the 80% the 90% of use cases which takes 10% of the time? The other 10% takes 90% of the time and typically the ROI on that is negative, right? It’s really bad. Um, so

Itiel Shwartz: I think no, I think like that’s a great point and that that was also like what I was trying to better understand like I think a lot of the like companies that are trying to migrate to Kubernetes are over like abstracting themsel but like they’re trying to reach that like perfect abstraction state and in in reality it cost them quite a lot of resources like every abstraction we leaking them especially Kubernetes that is an ecosystem is moving fast so we need to update your abstraction maybe And it it is like a a hard process but it sounds like you guys were already in a great state and you kept on investing in that which is like great skill and like something that we see in common or a lot of companies are pretty much like are like failing due to over abstraction or under abstraction like both can be like demolishing like the state we don’t have

Scott Rosenberg: I think on that I think on that just one point which is like a perfect example right why does someone want to be in kubernnees Ubernetes right at the beginning of you know if we go back even just two years everyone said I have to be in Kubernetes why because I have to be in Kubernetes right it was the buzzword it’s like every app needs AI today and yes the reason is because otherwise you’re not going to get VC funding right and whatever but like the the truth of the matter is is that Kubernetes is right for certain applications and it’s wrong for other types of applications and I think that one of the perfect examples of this is though where does Crossplane fit in within platea for example Initially in this project we said great all data in Kubernetes. Now certain data sources do not have good database operators in Kubernetes simply they don’t. Um and what do we do now? Well the benefits of Kubernetes are not the things running Kubernetes. A container orchestrator is the least interesting part of Kubernetes on planet earth. The interesting thing about Kubernetes is the operational model. It’s the consistent reconciliation. And it’s the control loop. It’s the single API with a declarative API and all of that. And that’s where today all data services are controlled from Kubernetes. That is the line in play. Now does that mean that it is running as pods inside of Kubernetes? Not necessarily. Some things are some things are running outside but managed through Crossplane. So building that abstraction layer so I have that single platform API does not mean that I have to make those hard decisions on where something runs. I can make the right technical decision of a runtime separated from my control loop. And that’s something that has only been possible in Kubernetes for the last few years with things like Crossplane now crow coming around right for people that want to try that. Um I’m a Crossplane guy. Um but like I think that’s a key thing also don’t think that don’t try to force everything into Kubernetes as a runtime but strongly consider it as a control plane.

Shemer Mashiach: Yeah, it will also give you like Crossplane in this in this example or this concept of abstraction will also give you the ability to choose different deployment models for different environments because while we need certain capabilities in production which I don’t really care in lower environment I can decide that you know even as Scott mentioned we are talking about a single database in this case that you know that doesn’t fit to production workload in Kubernetes. So in production we deploy it outside as you know bare metal servers but on lower environment since we don’t really care about anything it’s not about the latency it’s not about you know keeping the data safe then we deploy it inside Kubernetes everything controlled by the same system eventually this is the most important part

Itiel Shwartz: I think that like just just to wrap it up and like let asking you guys the final question even I’m like a big crossben fan and condor has like an open source around visualizing Crossplane and we have like LLM for Crossplane. I do want to give like a like a small point of warning here. We did see a lot of companies much smaller than the plate scale that tried forest plane and it does have a learning curve. It’s not like you start with and everything is great. Just to say it like I see great implementation of Crossplane for large organization. If you guys are like five DevOps and I don’t know like you don’t really have a lot of scale and so on, I would put Crossplane as a secondary like phase. Again, I love Crossplane. I think it’s like one, if not the most interesting project. But just a a word of caution for everyone that listen and it’s like, “Yeah, I should use Crossplane because it offer like a great abstraction there.”

Shemer Mashiach: So, so first of all, let me just say that Platica engineers and DevOps people, we have a lot like really a lot of experience in in this, you know, world like coming from years of experience. And by the way, it’s not just Kubernetes, it’s like multiple technologies that we are managing. So obviously, it’s much easier for us to onboard into something like that. Uh but yes, you need to have a good people in order to run those projects. Anyhow, by the way, whether you are 20 DevOps engineers or two.

Itiel Shwartz: Okay. So with that, maybe I’ll ask you guys like the last question. Usually I ask about like your predictions but this time I’ll ask a more deliberate question that I’m starting to ask which is like Kubernetes and LLM or like using AIS AI ops like are you guys doing that do you have any prediction where is the industry going and yeah like that’s

Shemer Mashiach: so we are doing it facto it’s uh mainly between let’s say agents that run from carer cloud or agents that runs on agents infrastructure. In this case we are using solo in Platica the open source one h to start implementing this and it’s working relatively well those agents are connected to multiple data sources mainly elastic open search victoria metric all of those you know um the intend during this uh coming year is to extend it more towards um CI/CD, pull request agents, you know, all of those things that will make our developer life easier. Um, this is also something that I’m working with Scott together and um, it’s it’s been a very, you know, very good collaboration because um, as Scott said, coming from outside, I’ve been doing this work for 20 years before I got to Plata. The reason I’m doing predica is because to do this in predica it’s like pica for me every day it’s a new working day a new working place I’m doing a lot of projects so it’s very challenging and interesting but the fact that I have Scott to talk to consult work together on solutions even work together for other customers fight with him shout it’s always a a to the experience and from my perspective at least it always provide a good result even if you are not agreeing or even if the projects eventually doesn’t see the light of day the this loop amazing from my side

Itiel Shwartz: no that’s a great ending Scott maybe for you like last sentence

Scott Rosenberg: yeah I would say I think that you know we’re out of the disillusionment phase a bit from AI agents as well um where AI agents have their place. People think though just like they thought Kubernetes was going to solve everything, people think AI is going to solve everything. And the answer is no, it won’t. It will continue to hallucinate. It will continue to give you wrong information. It will continue to do all of this because that is the nature of these non-deterministic LLMs. They get better and then they get worse. And people say that 5.2 2 is the best uh GPT version and it hallucinates more than 5.1 and then they release codecs which is worse at code than 5.2 and right like all of these things make no sense on planet earth to us because they don’t work the way that humans do. Now I think that bringing Kubernetes is the number one runtime for AI agents and for these things because what is an AI agent? It is an application. What else is it? An application that exposes an HTTP endpoint that things connect to. In the end, that is what it is. And that is why Kubernetes is a great application runtime. Therefore, it is a great agentic runtime. What I think we’re going to see, and this is where I think we’re moving in 2025, is the idea of RAG based on Kubernetes. The current situations of Kubernetes understanding are horrible because they bloat up context. When I try to solve Kubernetes with an MCP and it now needs to go and find all my CRD schemas, I will explode my context with only an eighth of the CRDs that I have in a typical cluster by a customer with the amount of CRDs that exist today. And therefore we’re seeing more and more interesting things with Quadrant building RAG systems around Kubernetes. I think we’re going to see more in that area um where we’re going to see this agentic RAG type idea that both AWS and Azure are bringing out there into the market and some open source things. That’s what I’m really interesting in because we’re dealing with logs. Logs are too much data a lot of times for context, but it’s one of the greatest things that agents are good at doing. understanding the stack traits that a human is bad at. Um, understanding what CRD schema, it’s great if I give it one, but for it to find the right one is terrible. And that’s where I think these RAG based systems, which has kind of become less sexy since MCP came to the picture, I think is coming back and we’re trying to find now as an industry where all these pieces fit together, which is extremely interesting to me.

Shemer Mashiach: I can tell you that, from our side by the way the way that we are instructing our people to work with in this case cursor workload is on top of the rules that we provide you know as package to those employees we are mainly instructing them to ask a very particular and narrow question in order to avoid the bloat contact I mean nobody goes to okay I have a problem in Kubernetes find it it will never work really it’s like No, but

Itiel Shwartz: I agree. I agree. And just to give like a short plug here like Commodore does try to solve some of it using Claudia which is our own like AI for Kubernetes. But I think that it is one like context context engineering understanding how can I reduce the context in a way that provides enough context but not too much context so it will start to hallucinate is is one of the biggest challenges. Okay. So we we already like ran out of time but it was such an interesting episode. But I think it’s worth it. Uh, thanks a lot, Scott Sheammer. And good luck building the next generation of PK Platica service PKS. Sorry. Sorry. Awesome. Goodbye.

Scott Rosenberg: Thank you. Great being here.

Shemer Mashiach: Cheers.

[Music] Kubernetes for Humans.

This is an AI generated transcript of the conversation