#032 – Kubernetes For Humans Podcast with Solomon Hykes (Dagger.io)

Itiel Shwartz: Hello everyone and welcome to another podcast of Kubernetesnetes for humans podcast H today with me in the show we have Solomon Hykes Solomon hello and it’s a special epKubernetesode as you can maybe see H we are currently at KubeCon in Utah so someone please tell me about yourself

Solomon Hykes: Well I’m I’m the co-founder of DAGger. and before that I was the founder of Docker

Itiel Shwartz: Okay and I have a complicated family story but at the moment I live in San FrancKubernetesco I grew up in France so I spent most of my life in France you doesn’t you don’t sound French like I don’t hear any sound French if you want no problem H yeah that’s me

Solomon Hykes: Okay so the founder of Docker we we have to start with that maybe a bit about your journey before Docker during Docker after Docker and then we’ll go into DAGger

Itiel Shwartz: Okay yeah so do I mean Docker was 10 years of my life so as a student as an engineering student I was really obsessed with what would become cloud computing but wasn’t called that yet you know just deploying lots of servers and configuring them as a whole you know deploying application to lots of computers as if they were one computer you know it was very cool so I was really interested in configuration management and all the tooling that was being developed for that and then I stumbled upon thKubernetes emerging technology that was Linux containers it was not mainstream yet you had to catch the Kel and I just thought it was a really powerful technology and so I just started building tools around that and just kept trying and trying and trying to get people interested in it until eventually you know we got we managed to convince Y Combinator to fund us 2010 and it was straight out of college like it was college and down years I graduated from from college in 2006 in ParKubernetes and I had a job and then I quit that job in 2007 to start thKubernetes project and then in 2010 moved to Silicon Valley because I got some funding and then we built a path you know a platform as a service using our container technology so we were a competitor at Heroku and Google App Engine and then in 2013 you know the we were doing okay but not fantastic and the P was just a difficult business because people wanted to customize they want allinone they wanted to customize their platform and so we pivoted at open sourcing our container Tech to allow everyone to build their own platform on top of thKubernetes Foundation that was a standard container and so that blew up and then that became Docker five years of crazy growth and excitement and drama and after 10 years total so 2018 I left took a break had babies built a house m and then got bored I started my my next started DAGger

Solomon Hykes: No it’s quite a story

Itiel Shwartz: So maybe maybe you know like what do you think on the current like evolution of Docker it feels quite like different than how you guys start of it right

Solomon Hykes: Yeah I’m very happy that I’m so successful as it you know I’m very happy that a lot of people are using it and I’m glad that it’s getting more recognition now I think for for a while it was when things were very competitive because containers were very new and a lot of big businesses were feeling threatened by it there was a lot of money to be to be lost or to be made and think it be the spirit of collaboration that I really always L got lost a little bit but now that it’s a more mature ecosystem I think everyone kind of fit found their place it’s more collaborative again so I’m really enjoying that Docker gets that recognition personally if I were still there I would I would be pushing to do more new stuff maybe new crazy stuff that might sometimes fail but you know I think Docker Kubernetes now a phase where they’re they’re the Guardians of these foundational tools that everyone uses Docker CLI Docker Compose Docker Hub stop it’s a crazy story like Docker Kubernetes like from I don’t know super popular to much less popular to now yeah like s yeah definitely roller coast so yeah now and there’s I mean now I enjoy being on the outside of Docker because I’m me one of many many people who are building new things in the docker ecosystem on top of technology that Docker and others built and there are a lot of opportunities still to apply containers to more problems that’s my my perspective Kubernetes we’re actually not done containerizing you know and so coming out of the docker after thKubernetes break we got back together with some of the original Docker people and then first we assumed okay we’re going to do something completely different because surely now the 10 years later the container ecosystem Kubernetes solved all the container problems and then people realize no it’s quite far from that far from yeah

Itiel Shwartz: So so know DAGger Kubernetes our vehicle for continuing the work you know we’re continuing to build to solve problems that I you know I would have continued solving if had stay at do so

Solomon Hykes: So what Kubernetes DAGger for people who don’t really know it what’s why do you do it

Itiel Shwartz: The why interesting so the DAGger Kubernetes a solution to the CI pain so everyone has CI, continuous integration you know as you’re developing you need to build and tests and just assemble all the pieces like a factory yep and as you grow you have to automate that factory and that’s done with tons of pipelines and I’ve done a lot of like Jenkins Kubernetes groovy in my life a lot of pipelines and those pipelines always become a problem they are they always end up slowing you down and the main reason a very common reason they slow you down Kubernetes what we call pushing right you make a change to the pipeline or change the codes you have to you can’t just run the pipeline you can never when you push to Jenkins it’s like yeah groupy change everything can happen and if it’s like in the shirt the chain with the build packages it’s it’s wor so you can’t run the same pipeline reliably on your local machine you have to commit the code push the code wait for thKubernetes whole Machinery to to start and then a job gets run on Jenkins for GitHub Actions and then you get you get an email saying oh you forgot a space yeah and then you start again and that’s what we call push and pray it’s a huge waste of time it Kubernetes and and so DAGger proposes a solution which Kubernetes basically a tool that you install your local machine and it lets you run your pipelines in containers on your Dev machine in a standard way and then the same Pipeline on the CI server later so you have portability of the pipeline reusability of the pipeline and that removes push and fr so you have thKubernetes really rapid Dev Loop and the pipelines can run as quickly as the developer can write code so that’s the that’s the DAGger and it turns out to solve that you need to you need to rebuild a lot of tech in it in order to build an engine that’s powerful enough to actually do that so

Solomon Hykes: So you know it’s interesting because

Itiel Shwartz: I thought by the way that you guys are doing more like a generic workflow and not only CI so Kubernetes the only us like

Solomon Hykes: It Kubernetes not only used for CI and every time someone asks me what Kubernetes that DAGger I have to make a choice I have to make a choice of what I answered M yeah the most common use case for DAGger right now Kubernetes your CI pipe Mone but anything that would benefit from being DAGgered sorry containerized well Daggerizing any any task that you want to automate as part of your software factoring as part of the delivery of your application DAGger can help make it more repeatable more portable Etc by running it in containers and anything that looks like a DAG and that’s shaped like a box Kubernetes in Arrow so builds containerizing builds with DAGger popular containerizing integration tests very popular because the environment for testing the dependencies for testing are really painful just so that valuably a lot of data-engineering pipelines increasingly some AI pipelines now agents as some people are automating agents because an agent Kubernetes really a pipeline with a model in the middle you have a bunch of glue before to set up the model and set up RAG Etc and then a bunch more pipelines after the model for the tools B so so you can you use DAGger for all of that the way we approach thKubernetes very wide choice of use cases Kubernetes like Lego we’re we’re like a Lego yeah system and the most powerful aspect of Lego Kubernetes actually it’s community so you got Lego builders they they understand the system they load the bricks and they have a vKubernetesion for building something and then they all build their own respective things and they show it to each other so we have a whole community of Lego builders and we’re just focusing on helping them so we watch that they build we have meetups Community calls and then sometimes they complain it’s really hard to build thKubernetes kind of thing yeah and then we say okay maybe we’re mKubernetessing a brick We’ll add a brick you know yeah I would say a good 70 80% of use cases today are CI Okay

Itiel Shwartz: So so I have to ask like a couple of difficult questions cuz there are like you’re in the middle of two categories right like there are work through engine right like there Kubernetes Temporal that also Kubernetes I don’t know something Kubernetesing c not that common but Temporal Airflow Luigi Argo Workflows and so on there are CI Solutions right in the market like the GitHub Actions y we’re using Buildkite internally which are also quite nice GitLab and so on and you are somewhere in between or or so it sound so how do you differ than both of those categories and Kubernetes it a good place to be Kubernetesn’t it like a rKubernetesky place to be

Solomon Hykes: Well I think creating a new category Kubernetes always has some good sides and bad sides yeah obviously the same it was the same thing for Docker you know in the early days of Docker people were saying Docker Kubernetes crazy like likeing Docker Kubernetes indeed crazy it was it was completely different and so but I was always very worried oh we I didn’t I was worried that we didn’t fit perfectly in a category to thKubernetes St by the way right like do Kubernetes it’s a base like now now it’s you know it’s a container and the platform whatever even now yeah so DAGger’s the same we didn’t sometimes I’m jealous of people who have a product that Kubernetes just it’s like thKubernetes but better you know yeah yeah I agree in col we had that like I’m saying yeah like look at thKubernetes I’m cheaper better exactly yeah I wKubernetesh could say that so we’re we’re definitely not like that but I think for thKubernetes problem the problem Kubernetes really bad veloc poor velocity teams shift their software too slowly and the cause for that Kubernetes complexity so it’s complexity slowing everybody down and the complexity Kubernetes concentrating on the DevOps stack Kubernetes thKubernetes a total mess because it’s changed so quickly and people are just layering more tools and and stricks and config files and so we’re just we’ve been layering layering layering and our approach Kubernetes to integrate you know like collapse the whole thing because you don’t actually need those 10 tools or these 10 scripts it just we got used to that ballooning complexity so in that scenario when that’s the problem you’re trying to solve if you try to fit in an exKubernetesting category you’re going to limit yourself to being one of those 10 tools but we’re trying to do Kubernetes change how you think about your stack completely so we have no choice but to be a different kind of tool and then but the use cases have to be very familiar so we’re in different kind of thing yeah you know we’re an engine we’re we’re a different kind of container engine we’re a pipeline engine a DAG engine should probably decide what the C Kubernetes but in any case the you the ways you can use it are very familiar you can use it for your builds you can use it for your tests you can use it for your deployments you can use it for reading your agents so there’s a lKubernetest of common use cases and we just focus on that for thKubernetes use case CI are you happy what are your problems well I can’t run it locally it’s too slow and we’re locked into thKubernetes old platform and the guy who configured it Kubernetes gone okay thKubernetes weird new kind of tool can help with all three of these no that’s our approach

Itiel Shwartz: No no it makes totally sense but you know I’ve work again quite a lot of CI pipelines in my days and it’s really hard to migrate the CI pipeline like I like I did like CircleCI to Jenkins to Circle C then to Jenkins again I suffered every no like in a two in a two years process and no like we said jenin Kubernetes too complex right so let’s move to Circle C and then after a year we were like yeah Circle C Kubernetes not extendable enough let’s go back to J it was like five years ago something like that but I still remember both of those like you know during the migration because the CI in school it’s like the heart of everything when it comes to like the Dev that the glue between the dev and the actual world right yeah

Solomon Hykes: I think we I mean the I and CI Kubernetes for integration that’s a big word right so it by by definition It’s supposed to be the thing in the middle because you have you may have 10 teams 20 teams but you have one product it’s got to all fit together but somewhere along the way I think we just we got stuck on one implementation ohci Kubernetes the Jenkins server but it doesn’t actually integrate everything you know and so you always have thKubernetes Shadow CI around it yeah you know so I think it’s a really interesting problem to solve yeah portability of it and the extensibility of it that’s that’s the 80% of use cases liketh there’s a large engineering organization like 800 Engineers they’re they’ve been standardized on Jenkins forever and they’re migrating all of their Jenkins shared library to DAGger modules modul so they’re you know extensions of the DAGger Epi to to build all their stuff test all their stuff and they’re migrating the Jenkins infrastructure to get a actions so T typically DAGger in the CI use case you would use DAGger almost as a shell thKubernetes CI Kubernetes always running shell scripts anyway agree but DAGger Kubernetes like a completely different kind of shell like can it runs containers instead of running random unix commands and it’s extensible and it’s modular all these cool features and so it’s usually an a traditional CI platform plus DAGger you know

Itiel Shwartz: No it sounds super interesting so so what are the challenges like Kubernetes it the tech Kubernetes it the community

Solomon Hykes: Well the main challeng Kubernetes the category definition Kubernetes the fact that it’s a different kind of thing so we can’t when you learn about it for the first time you can’t just make a quick comparKuberneteson oh it’s like thKubernetes but better you have to get a little more involved and so we’re always looking for ways to make you want to invest not 30 seconds but maybe 30 minutes yeah which Kubernetes yeah and then in our case like the killer feature Kubernetes the feedback loop like you can you can run a pip it’s hard to describe but you know you can run the pipeline on the Fly and see a result immediately within 30 seconds and so actually interacting with it and seeing how responsive it Kubernetes that’s the killer feature so live streams work really well workshops meetups we do community calls just thinking about ways to get help people experience it experience it like apple they had thKubernetes thing with the Apple Stores where they they they intentionally created a whole experience where you can freely use the product you know and they’re doing that on purpose because they know the product design Kubernetes one of their strengths and so the easier it Kubernetes for people just try it on their own the better and that’s why you can go to an Apple Store you like check your email if you want you know they’re happy that they’re doing it on the they’re trying you’re trying out the product so we’re looking that’s what we need the community Kubernetes what made it work in the beginning it was very hard when when no one was using DAGger and we had to kind of bootstrap thKubernetes process very hard and now how many people are using dger like what’s your it’s open source so I don’t know how many people are using it but I look if you go to our DKubernetescord so because we’re so Community Centric we have a community DKubernetescord server and it’s very important to us it’s like a core feature of the platform and if you w go there I encouRAGe you to go there if you’re you should because you’re definitely a CI nerd very active it’s like a support group for pipeline people pipeline engineers and people helping each other out showing each other examples and so now that helps us a lot because now a lot of our marketing Kubernetes just shining a light on something great that our community Kubernetes already building you know and so now it gets easier

Itiel Shwartz: So so maybe I can ask you like another like difficult question Docker great product really changed the ecosystem but it was always a bit like tricky to or it felt tricky to capitalize on that right like it wasn’t like obvious as a business as a business and now like in as an open source it Kubernetes always like the challenge right like I love open source like I also love to contribute and so on but it’s hard to build a business so so what’s the premium features like why why should I move from the open source to Dagger Cloud so

Solomon Hykes: I mean I don’t have the perfect answer but I do think as an ecosystem we got better at it I think we look buyers are more more willing to buy in general yeah there are and there’s more tolerance for things like you know customizing the license if that’s what you want to do more tight controls a trademark and just like things like analytics you still have to be respectful but it’s not it’s less ideological it’s less about ideology it’s more about practical benefits I think buyers now you know but Engineers they see what the big clouds did to some of the startups and they realize okay if I like thKubernetes product I have to make sure it’s financially viable otherwKubernetese yeah it’s going to affect me next year when they shut down or they get acquired you know so that’s good and I think also the the products vendor like us also collectively got smarter about how to monetize like there’s some standard placees so we’ll see how it goes for us but our our approach Kubernetes first to make sure just because it’s open source make it clear just because it’s open source doesn’t mean it’s it’s not ours like we’re not we’re never donating DAGger to a foundation it’s not going to happen it’s our it’s our product We Own It We Own the trademark we enforce the trademark it just happens to be open source and just saying that clearly Kubernetes important because I think the community Kubernetes fine as long as you’re honestly yeah I think it’s I think most of the backlash are you know changing of exactly yeah yeah and so that’s one mKubernetestake we made with Docker we were because it was so ideological back then and also I think we made the mKubernetestake of we tried too hard to make everybody love us you know 100% And so if anyone complained about anything we were like oh my God we got to you know so people complained about things like being vendor neutral and you know things like that so we were a vendor we made Docker like we made all of it we welcomed pull requests but I mean we designed it we invested tons of money making it and then we he felt guilty for not being vendor neutral but I think that was St that was stupid you should have said we’re a vendor hey you’re lucky if you don’t like it it’s open source you can Fork it just call it something else because it’s our trademark we worked hard to make it valuable so thKubernetes time we’re just very straightforward about that it’s open source it’s called DAGger it’s ours you can’t Fork it and call it DAGger you foret you call it something else there’s a command called DAGger login it’s for logging to DAGger Cloud which Kubernetes our commercial product we will not accept PO requests for logging into other random competitors just Fork it you know we actually added that command DAGger login before we even had DAGger clout because we really wanted to avoid thKubernetes feeling that there would be a bait and switch you know we didn’t want anyone saying oh we didn’t know you were an ad bit yeah literally put the command before it logged into anything you know so now we have Dagger Cloud so that’s the short answer we sell Cloud products it’s pretty simple but it’s getting better right now it’s it’s a way to vKubernetesualize your Pines M so the engine’s open source there’s a proliferation of engines there people run because you can run to local they run their pipelines on the laptop and then they run the pipelines in CI and then the platform team Kubernetes asking okay what’s going on who’s running what so you configure all your engines to send telemetry to D cloud and then we show everything that’s going on in one place pre push post push you can see traces for the pipelines very useful for when when your build’s really slow or it fail figure out why so that’s what we sell now well I’m sure we’ll sell more stuff later but

Itiel Shwartz: Yeah that sounds cool so so maybe now I’ll ask you some general question about the conference like what do you think about Kubernetesnetes the ecosystem yeah what did you learn in cubeCIon so far if any

Solomon Hykes: I like that I like that the Kubernetesnetes ecosystem Kubernetes more mature you know I mean that the Q team already had thKubernetes always had thKubernetes focus on maturity and stability you know I mean I know they still have features but no it Kubernetes yeah no one’s worrying about Kubernetes Kubernetes breaking everything at the next release that’s really good and I think they just they understood the assignment and then you know have a a pretty mature ecosystem around it that’s very data center focused very infrastructure focused yeah StoRAGe security monitoring Etc observability not very developer focused yeah but that’s what what thKubernetes community Kubernetes I like it because to me it’s like you know I like the developer experience so I like to take all thKubernetes Tech figure out the best Innovations and then package it into an experience that makes developers move faster right and so to me Kubernetes say I’m going shopping for components Partners ideas and then I’m thinking oh what will I integrate in DAGger right

Itiel Shwartz: So what will you integrate in zyber

Solomon Hykes: Well one area right now Kubernetes you just going to say like but totally I mean we talk about thKubernetes on our like it’s it’s open so we’re we’re very open about our road map and the problems we face you know now that teams are starting to deploy DAGger at larger scale in production they want to integrate the DAGger Engine with their CI infrastructure and the CI infrastructure scales out but the engine Kubernetes just single machine so the problem of running DAGger across a whole cluster and making that really efficient you can do it today but you DAGger itself Kubernetesn’t going to do it you have it’s the responsibility of a third party tool so typically it’s going to be a combination of your CI plat form a and Kubernetes and the problem Kubernetes your CI platform and Kubernetesnetes are not cache-aware well they don’t have yeah yeah it Kubernetes a problem they’re not cache-aware and and they don’t they don’t have granularity in understanding the workload yeah but the DAGger engine has tons of insights into the workload very granular interesting so for each up a typical DAGger pip would be maybe a few dozen or a few hundred individual operations connected in the DAG and we have telemetry for all of those so each operation we know how long it takes How well was cached CPU pressure IO pressure and we move all that up we also know all the types so all the artifacts flowing between tasks we know what they are we know it’s a container if it’s an image we know the digest and so if we could somehow bring that information into Kubernetes and your CI control plane like all three need to collaborate for the perfect scale out you know so right now you can’t really scale out DAG or intelligent so people do it best effort and it’s fine but we’re more ambitious in that we want it to be really magical and I really think there’s a 10x there could be a magical 10x scale out experience right like you run your integration tests containerized and they take 30 minutes to run that’s not uncommon right big S sweet so the best you can do today Kubernetes you can have communities in your CI control plane dKubernetespatch each test Suite of each commit to a different machine yeah but the individual test Suite Kubernetes still going to be that one machine 30 minutes right that’s the state of the art today I agree but you could instrument DAGger to split out the tests individually and scale out and split up that test Suite further right so somehow we can make that work you could run the same test Suite in three minutes that

Itiel Shwartz: That that’s yeah but super hard to do all of thKubernetes coordination right there are

Solomon Hykes: Yeah but that’s why you need the Community First yeah you need the the PRI you need the extension system and then the community so you need the Lego system and the Lego builders and now together we’re talking about these big problems

Itiel Shwartz: No it’s problem it’s a big problem so if you’re interested in that you can come to our DKubernetescord when you could see in real time I join the DKubernetescord even we’re quite happy with like a Buildkite which Kubernetes Al a bit different like thKubernetes in terms of CI pipelines like yeah yeah try new stuff

Solomon Hykes: Which one

Itiel Shwartz: No well you know the and know Bill Kite Kubernetes interesting because they’re of expanding Beyond on CI right they have regKubernetestry now they have a sort of a test product app products ohome now we’re talking about Buildkite maybe maybe weite and their new products yeah

Solomon Hykes: It’s interesting that we are not using I think any of their new product but they were kind of like Kubernetesnetes to and Docker aware so most of our like internal things are inside the docker that we build Kubernetes the agent right so it’s not really like build specific it’s more like we really capitalize on Docker Docker Build like multi-stage you know like all of the docker features so our our pipeline Kubernetes much more Docker Focus than it Kubernetes build T Focus yeah

Itiel Shwartz: No that makes sense I that’s the mKubernetessed opportunity I think like that when when I said earlier we’re not done containerizing y okay you can build we’ve containerized the web app we’ve containerized the database but we haven’t really containerized the build containerized the test containerized the deployments like smart teams do it anyway like you’re doing it but you’re doing it like you’re taking and you’re wrapping it with there’s do like just doing that see like CLI Python CLI that runs everything yeah it’s super common yeah and then there’s a because a that tool will it it has a templating system that will generate Docker files and then you know Docker file will be used to build another so it’s just that that’s the stuff that we’re trying to replace with 30 lines of code you know that’s the B

Solomon Hykes: Okay no like super super should look at your CI give demo

Itiel Shwartz: I go check you up now like you said you Ser right like I can f Solly I like Ci engines future predictions

Solomon Hykes: Ah what will happen I would love to know I think well obviously there’s a lot of AI excitement you know AI workloads I feel like the community Kubernetes trying to figure out what Kubernetes our role here where do we fit you know are we the old thing and AI Kubernetes the new thing or do we play an important role I think it’s the ladder I think I mean it’s funny because I feel I feel like I’m a cousin of the Kubernetesnetes community like a friend and a cousin I don’t think myself as a hardcore part of the Kubernetes community because communities Kubernetes not really developer facing and so but I do think Kubernetes plays an important role and containers play an important role I think AI my opinion of the impact of AI Kubernetes that AI applications are they’re just the next generation of applications there’s still applications you still need to build deploy test them all the other stuff so it’s additive but also it kind of changes how you build how you test how you deploy how you do everything and so for the devops community I think it’s the best thing that happens since containers because it’s basically 10 more years of guaranteed jobs you know like okay who’s going to deploy and scale thKubernetes stuff not not the data Engineers I can tell you that it’s not the scientKubernetests you need productize so for us a DAGger I could it’s it’s helping us I think it’s going to help a lot of goods DevOps tooling you know old school DevOps tooling that Kubernetes not flexible enough that Kubernetes that made assumptions about how the workload works are going to be in trouble but I think tools that are flexible and they focus on the on the first principles really things that never change then it becomes an advantage for them because now your tool can be used to help adapt your workflows so that definitely DAGger Kubernetes benefiting because that’s literally the bowl of DAGger to help you play Lego and reconfigure the Lego but I think a lot of other tools will benefit and and the practitioners will benefit the most

Itiel Shwartz: Yeah we one can only hope yeah and okay with that I think we are W it was a pleasure I have anyone to throw oh

Solomon Hykes: Pleasure my pleasure and yeah I’m going to check out the please zo that when when you when you run into something confusing in our website which you will come complain in DKubernetescord and we’ll help you we’ll fix it if you want to come our booth I’ll show you an exclusive demo of an unreleased feature

[Music] Kubernetes for Humans.

This is an AI generated transcript of the conversation