#060 – Beyond ELK: Elastic’s 10-Year Evolution, Open-Source Licensing, and the AI Frontier with Philipp Krenn (Elastic)

Itiel Shwartz: Hello everyone and welcome to another episode of the Kubernetes for humans podcast. Today I have in the show Philipp. Philipp, do you want to introduce yourself?

Philipp Krenn: Yes, hey,

Philipp Krenn: Thanks for having me. I’m Philipp. I’m I work for Elastic. I feel like in the Kubernetes ecosystem many people still know us as ELK. And I’ve been surprisingly almost I’ve been at Elastic for 10 years at this point. so I’ve I’ve seen a a lot of iterations and and things we have done and I’ve been kind of like in different ecosystems over that time because we do a lot of observability but I also spend a lot of time on on search nowadays and the AI world like everybody else. so that’s what I’m doing and today I’m in Vienna where I’m from but normally I’m based in San Francisco. but I came a bit closer to you in time zones which is a little easier.

Itiel Shwartz: Mhm. Ah, that’s cool. That’s That’s super cool. so 10 years in Elastic. Maybe walk us through I don’t know like Elastic the Share a bit about your history, right? Like what led you to Elastic and maybe also it would be interesting to hear your experience first handed from like Elastic growing like you said from like a Kibana dashboard to to what it is today.

Philipp Krenn: Yeah, so I I was a the small startup at at my university back in the day though. They did well. I think they reached 100 plus people and were sold to American companies so they did well. I didn’t kill them. but there I had the opportunity to try out lots of data stores and we were using back in the day like 15 years ago it was the NoSQL days and we were using almost any NoSQL data store that was out there. We had Redis and MongoDB and Elasticsearch of course and we used MySQL and ActiveMQ and we I had a full circus of tools that I was using. And then I started doing more meetups and conferences around that. And at some point I felt like that was maybe too much for my role because I was basically the main person managing the data stores and doing all the devops work. I did AWS for the company, but I was the only person and there were only one of the co-founders and me we were the only ones on on the pager. even when we were like becoming a bigger and more relevant company and then I decided that maybe it’s time to to do this more professionally that the conference world. And that’s how I came to Elastic because I knew somebody from from meetups and then I saw that they were looking for somebody to do conferences and other things for Elastic and that’s how I ended up there. it started a long time ago and now 10 years later I’m still there. I feel like things of course change. 10 years ago Elastic was like 300 people, now it’s 4,000. in the early days it was mostly conferences or I feel like I always say the world before COVID and after. Before COVID like it was like way more conferences still or at least for me I was traveling 200 days a year and it was just out there telling everybody what we’re doing and what we can do. and then during COVID I stepped a little more back. Now I have a larger team and I I always jokingly say I spend 80% of my time management and 80% IC work. though a lot of it is is also internal facing like I get to review a lot of things so I would always jokingly say that you know the movie Ratatouille when they try to poison the rats at the beginning and there’s the rat that can smell very well and I feel like that’s sometimes my role in the company that people hold up a piece of content to me basically and say like is this good for developers? Or you spend so much time with the community Like, is this what we should be saying? And then it’s like in the movie it was this smell test is like is something poisonous or not? For us it’s not poisonous, but it’s more about like is this a good fit or not? The main problem though is it’s not like our content necessarily is poisonous, but it’s often like no, we should do something better. But then it creates a lot of work to actually say how it should be better and what needs changing. So those are the things where I spend a lot of my time today.

Itiel Shwartz: So but you know, like you’re talking about today, you guys are huge, you guys are big. Walk us through like 10 years ago, 300 people company very like open source like mindset, right? And maybe share a bit about like how Elastic started, how did it like evaluated did the evolution? And maybe like you can also talk a bit about the license changes or or we shouldn’t talk about it. yeah, so so it’s up to you.

Philipp Krenn: Yeah, so I at this point I think 16 years ago Shay started Elasticsearch and it Elasticsearch wasn’t even the first implementation, it was basically the third implementation. Three’s maybe the charm, I don’t know. There was Compass 1 and Compass 2 and then the third implementation I didn’t call it Compass 3, but Elasticsearch and that one stuck around. Mhm. And at my my previous company actually we were using Compass 2 in another project. so that was my very first experience with a search related project in that area. And so he started Elasticsearch and then over time there were these other components that were started by the community that joined I don’t know, the Elastic portfolio or family, which was mostly Kibana and Logstash at first and later on Beats. and I feel like Elasticsearch or then the lore of like how Elasticsearch started was that Shay’s wife was a chef and she had a lot of recipes and she needed to search her recipes. And he wanted to build a system to help her search her recipes. She’s still waiting for that recipe search today. he got a bit sidetracked starting a company and actually doing that. but that’s how Elasticsearch as a search engine started. And then it turned out that logs were also kind of a search problem because it was about like what errors did we have in the last hour or like how many errors did we have? or what is the average latency? All of those are kind of like search and retrieval problems that you store a lot of data and then you need to find something relevant in that. And then with the combination of Kibana for the visualization and Logstash, that just naturally grew and I’m I’m not even sure we had a plan to be a logging solution or observability solution early on, but it almost happened and it just worked too well not to do it. And then we started going down that path and I feel like at first it was almost accidental, but now today of course it’s a more intentional area. We’re also one of the biggest I think we’re always in the top three of OpenTelemetry contributors at this point. so I feel like the observability story or observability has just shifted from like Logstash had its place and still has its place for many of our users, but OpenTelemetry for example is is a very big part or how I feel like forward-looking almost everybody looks at observability. and that’s why we’re heavy contributors, but also heavy users of that. So we have a a managed OTLP endpoint in our solutions nowadays that you could just use and then you just point your applications at that and you can still pull and parse your log lines, but you don’t necessarily have to if you just integrate into OpenTelemetry. So there there have been a lot of the changes over the years. Like initially I feel like we were small and we were just doing stuff that kept working. nowadays that we’re larger, I feel like it’s more intentional efforts to some degree that observability is an official solution besides search and security what we do as a company. but that we have kind of grown into that. And then yeah, as you mentioned there we’ve had that license change. I was I’m always tempted to say like unfortunately involved in that because it was a kind of like a painful experience for everybody involved. But the thing turns out that if you have a successful open source project, there is a cloud provider that likes to take your work and sell it to others. and the problem is I think providing something as a service is is fine, but like if the ones doing the work don’t make any money out of that, but somebody else does, it kind of like strangles the product in the long run. So we tried to force them into their own path and we picked our own. so we have made the decision through the licensing. By the way, the while a lot of people know that we re-licensed to back then SSPL and Elastic License 2.0, a bit over a year ago we added the AGPL as a license option back. So Elasticsearch is open source again, or you can get at least large part of that source code under the AGPL license again. Again, that big cloud provider that does not want to touch AGPL normally. so that gives you a a safe enough path of of where you can go. and yeah, that we did the license the first license change was in ’21. and then yeah, a little while ago and oh, maybe maybe I think it’s almost two years ago at this point that we did the AGPL option. time is passing so quickly. but we have been open source for two years again.

Itiel Shwartz: Okay. No, that that’s cool. And I know there were like a lot of back and forth and like there was the ready story and like like I feel now that I see more and more like this elastic license in a lot of like open source projects.

Philipp Krenn: Yeah, it’s I feel like it’s complicated because like everybody likes open source, but at the end of the day you also want to have like a sustainable project. I agree. The sustainability means that I also want to get paid at the end of the day and most of us need to get paid and if you are a big cloud provider, then you have a very large distribution channel and then your world looks a bit different, but if you are an open source project, your distribution channel is more complicated or you need to find a final solution. So yeah, I think we but also MongoDB, Redis, HashiCorp, we’ve all It’s It’s fascinating by the way, I think that every one of these solutions has kind of like a slightly different approach and everybody had a slightly different problem they tried to solve for. Who were the competitors or what was the problem in the space? That’s why everybody’s solution or approach looks slightly different. but the direction of what everybody was trying to do is the same that everybody tried to find sustainable business model for it or for their project.

Itiel Shwartz: Yeah. Maybe now let’s talk a bit about like you know, licensing is indeed like a sexy topic, but let’s talk a bit about like the real the real sexy thing in our day and day and age, which is agents agentic and so on. Give us your take, the elastic take. Do they like compete? Do you guys love agents? Like what’s your take as a you know, someone who lives the like the ratatouille of developers maybe and like what’s the elastic take on that?

Philipp Krenn: Yes, I think if you don’t love agents, then your CTO or whoever makes sure that you will love your agents at some point. I feel like there’s a big drive. And while people individually maybe have like mixed feelings about like code generation and everything, so from our perspective where agents mostly come in and I’ll focus on the observability side, but it will apply to security to a large degree as well as that ideally you can get a bit away from like doing boring work and get to the more interesting work faster. so the I think the scenario that we like to think of is like in the middle of the night and while you wake up and maybe you make your your first coffee of the day then to investigate, in the background the agents hopefully start preparing the case already. That you know, when there is an outage, you normally you get an alert and then you would go to the alert and then you start pulling, I don’t know, you open a dashboard or you start searching with a query to find what what happened. And ideally an agent can start doing that work for you. So by the time your coffee is done and at your laptop, it actually shows you a bit more of like what is actually going on that you don’t just start clicking around in dashboards, especially if you don’t know a system that well. but then it can actually do some of that background check already and that the agent then can give you like a three-sentence summary of like what is going on and it will tell you I don’t know, we have a a latency spike on the load balancer and like some connections are timing out and then it will potentially figure out like it is because this one component has more load or has a higher error rate or whatever. And then it might even know like, “Oh, this was recently deployed and this part of the system changed.” Any of these are not super interesting investigations on their own. and I don’t think we would be able to replace humans completely, but some of that background and preparation work just to let you pick up the interesting tasks and then figure out what to do. maybe even have automatic remediation depending on how much you trust the agents that you have built around it. but to give you that tooling, that is the idea and we have now built something into Kibana that is we call workflows, where based on specific things, either as a cron job or based on an alert or an anomaly, you can actually take automatic actions as well. So, you could generate specific reports with that or there are other reports that are built into the solution. But you could build your own research reports. You could build your own remediations or or rules around it. It could include something that you have an error rate that is higher than something and you could automatically update your status page because we all know like people often forget to update the status page or it’s like very delayed and then everybody complains that you’re not transparent or it’s not visible what is going on with your system. So, there I think there is a lot of things where agents can actually do very interesting and helpful things, especially when you wake up in the middle of the night and you’re not fully awake yet and they they just help you do all the common steps that you should do and also give you some some options and just make your work easier. I don’t think we’ll completely replace the humans but agents actually hopefully allow you get rid of some of the boring and repetitive work and just do more of that for you.

Itiel Shwartz: No, then that that makes sense and let’s talk a bit about like business models, right? For Elastic. Like what’s Is it good? Like every day is a good days for you guys? Is it a bad day, right? Like everyone is asking themself, are we using AI, right? Are we enabling AI? What’s like what’s the Elastic take on that?

Philipp Krenn: I mean, I feel like right right is a is a very fleeting term or I think there’s this funny quote about people saying like, “Oh, I’m currently unemployed and I’m so glad that I’m unemployed. Otherwise, I wouldn’t be able to keep up with all the AI changes anymore. So, I think it’s important to keep in mind that it’s a it’s a process. It will just keep changing over time. and whatever we do today will evolve. and I don’t think anybody has the final answer yet. And so, the right amount of AI, we’ll have to see. or I don’t want to be too evasive here. but, I feel like there’s a lot of like Twitter is very polarized today and everybody is saying like this is the only way to do something and you need to do this, otherwise you’re doing it wrong. And I’m I’m not sure I subscribe to that. I think skills, for example, we use skills a lot, both to to do certain things or fix things in a certain way. there’s a lot of things that you can codify or help LLMs to do in a certain way that is like your business standard or company approach. from content creation, but you can also have skills about like researching an any issue or just to start a new project or how to instrument your applications. there is a lot to that, but it’s again, it’s an evolving thing of like how to do that. Sometimes I wonder like skills are like these very free-flowing text, which feels very, I don’t know, chaotic to me. Like when I think like code is very deterministic and that’s nice. skills are often like yes, you put something there and then it works or it doesn’t work and then you need to have evaluations how well it works. I feel like we’re still learning a lot of the things. But, when I look at like 3 years ago when you started using ChatGPT, maybe, the evolution of how far we have come and how much the systems have improved, but also how much better the integrations got. It is fascinating and like we made a lot of progress all over the place. So, I think there is it’s a process. we’re not at the end yet. And I feel like a lot of the products are also still working on integrating or fixing the AI but we are working on that and I think we have a lot of ideas. we’re also doing a lot of prototypes internally and our SREs use it internally a lot as well. but everybody is still trying to figure out like what will the end result be and I don’t think we are we are quite there yet. We’re still in this growth phase of like where agents will take us observability.

Itiel Shwartz: No, that that’s that’s quite a cool like take. How maybe like maybe like so how do you envision like Elastic a couple of years from now like in this like new world? Do you think the observability which was the I feel once it was like the main core and now I also use by the way like Elastic is a vector database for some of my agents, right? Like it’s also this capability. And you guys also have this security arm, right? So it’s like security, observability, normal database. Like what do you think is is the core or is it all of it or maybe like all those different like capabilities or pillars that you guys have?

Philipp Krenn: Yeah. I mean we want to grow in all of the areas and I think that where you see some of that strength, for example, is if you have like a knowledge base or you have your past GitHub issues, if we can pull that kind of like into the same engine as your observability data. And then, for example, when an agent does this investigation, you have a single system where you say like this is the error rate or the latency rate that we have, but it can also search for like past issues or it could search the commits that you have. And then figure out like why did you have a certain change? so there is definitely a strength to having multiple of these signals or data types combined. And then security is becoming a fascinating field with all the supply chain attacks. Like I feel like every other week there is a new bad npm package somewhere or or there’s some vulnerability that somebody finds. So, I don’t I think there is Well, as a company, the approach is very much like it is a strength that you can combine all of this into one engine and platform. so, I don’t think anyone of those is going away and but you are correct, I think the first few years of Elasticsearch was very much like search. And then it was elk with logging. And then it kind of like I want to say it almost exploded because then like vector search and AI one big pillar today that is not going anywhere and then observability grew up from logging to more open telemetry and being all the signals and then we added security. and I think also from an implementation point of view for us, the integration of like security and observability is like you have a large amount of data and you want to find what is relevant and it depends a bit on what is relevant and how you define relevance for the specific use case, but there’s a a lot of shared tooling under all of that because for security, you might look for a specific hash or action or chain of actions that a user or account is taking. but a lot of that is kind of similar to how an observability system collects errors and then you know have root cause analysis of like why something is failing or why something is slow. so, on the tooling side, there’s a lot shared under that hood even if the solution at the top looks a bit different. And so, I don’t think any of the solutions is going away. for us, it’s really a a strength that you can have that observability and security data and then have like the AI search world and can combine all of that make a richer experience.

Itiel Shwartz: Okay. No, that that that makes like total total sense. I think with that maybe we will close like today episode, but before that, maybe final remarks, thoughts on the future. Where are we going? What’s going to happen? What do you want to say?

Philipp Krenn: Ah, well, so I think that the future is is fascinating. I we’re at this very interesting point point where all the LLM providers are trying to go public and we’ll see how how cost will, for example, evolve. I feel like nobody’s quite sure like has it been heavily subsidized? Will things become cheaper, faster, more expensive? what will the actual usage look like and will there be a divide in the board like how how expensive will be LLMs be or junior developers suddenly become a thing again because the LLMs become more expensive? so I think it’s a fascinating world and everybody is trying to hedge their their bets to some degree. Like you want to be ready for a a full AI world but maybe this is also not happening. So it’s a fascinating time and every time I’m in in in the Bay Area, I feel like everybody’s very rushed to find figure out the next step and where to go. so it definitely it’s not like a relaxed time in tech, I feel like, but it’s very it’s a very interesting time even though if many people feel very pressured or rushed because everybody feels like it’s we need to find the right thing now or we’re we’re history. which is definitely exciting. So it’s not a boring time. I think that’s great.

Itiel Shwartz: Okay, that’s super cool. And with that, we will close today’s episode. I wanted to thank you Philipp. I know that my dual like this call was rescheduled a couple of times. So I’m super happy that we were able to get you and in all in good vibes. so thanks a lot. Thanks for having me. Yeah, bye-bye. Girl. Go to the left. Go to the

[Music] Kubernetes for Humans.

This is an AI generated transcript of the conversation