#034 – Infrastructure Automation & the Future of Ops with Cory O’Daniel (Massdriver)

Itiel Shwartz: Hello everyone and welcome to another episodes of kubernetes for human podcast today with me in the show we have Corey Cory can you introduce yourself please

Cory O’Daniel: Yeah I’m Cory O’Daniel I’m the CEO and co-founder of Massdriver we infrastructure automation tool and I’m also the host of the Platform Engineering Podcast

Itiel Shwartz: That’s great it’s like a crossover like episode

Cory O’Daniel: Yeah I was I was going to say when are when are you coming on when are you guys coming on the platform today

Itiel Shwartz: We will come we will come and I’ll come with the I just bought a new microphone so if people hear and first time like they hear me clearly it’s thanks to another another like person that recommended me this microphone and but Cory like CEO of of like infrastructure company automation right and tell us a bit about your background like I guess that you didn’t start this is like your first your first step right so yeah share what’s your story

Cory O’Daniel: Yeah I’ve been I’ve been in the space for quite a while so I’m I’m a software engineer the frequent joke is the e in my title and apologies to PCO I’m I’m in our office today so I’m not of my my good my good audio but yeah I’m software engineer the jok is are on the oppos like the E and my title is Chief Elixir Officer so we program a lot in Elixir and Golang but I’ve been you know working in Cloud operation since 2005 and before that I was working in data centers my master’s degree in healthcare information system so late 90s Y2K era around like the era HIPAA of being rolled out in the US was kind of when I was getting my feet wet in operations and so I moved to California in 2005 I think 20 I’m not really good with time I think it was about 2005 to join a startup i’ worked for like a bunch of big companies and support go to California work in a startup wear board shorts to work like all the cool stuff and my very first job I also kind of switch from like the off side to like the development side and so I’m in the office one day and my boss comes by and he’s like you’ve worked in a data center before right and I’m like yep it’s cool and then he just like walks away and I was like oh right that was an interesting conversation and like two hours later an email goes out to the entire operations team and the entire software development team it’s like 30 or 40 people and he’s like hey our lease is expiring on our data center and Amazon I don’t know if anybody’s familiar with Amazon they’re a book company that they’re selling these like Cloud computers I think is I think I don’t think he said VM I’m pretty sure he said Cloud computer or something like said something really goofy and he’s like and Cory’s going to lead the migration since he’s worked in software and operations and so I’m just like oh like there’s like 4 I have no idea I’m leading this thing I’m like I guess I’m guess I’m leading a data center migration to AWS so U I just kind of got like thrown into the cloud unexpectedly in my very first role and then kind of got pigeon hold into like a devops engineer or the next people can’t really see it here but there’s a lot of air Quil going around both yeah yeah both around like the the cloud and devops and so I completely understand yeah go carry on and so yes I’ve been in this air quotes devops role for most of my life as a matter of fact yesterday as a CEO I was doing Fluentd configuration that’s that’s what you do as a CEO people you’re you’re still dealing with logs but yeah so from there I started my first startup in 2008 it was a smashing success by that I mean it went out of business and then yeah you proceeded to join a few different startups started a few of my own for a couple of years and then in 201 2016 or 2017 I can’t really recall exactly when it was i’ got this like Niche expertise which is erlang and elixir like yErlang beam VM and kubernetes and like there was a point in time where if you search for like kubernetes Erlang or kubernetes Elixir like I just showed up as like the first search result like that that it was just me there and I managed the kubernetes client for Elixir I built the extension framework so you can I built the operator framework for kubernetes in Elixir so you can actually extend kubernetes with Elixir so I was like the thing that showed up and Google had just secured this contract for a really big cloud migration nine physical data centers to the Google Cloud and the company was Erlang based and they were planning to move from VMS and

Itiel Shwartz: You know who is irland based like you know think I think a lot of my listeners don’t really know erand because you know it is like a niche HIPAAster language that was invented by B I don’t know like in the 80s or something like that something so you know like you do need to give us like one minute on Erlang who why WhatsApp was written in Erlang right like it’s like a telecom hardcore program know like you tell it cuz I’m not a after seeing the low World in Erlang I was like I’m never going to be like an Erlang developer it’s not for me and it was like one of the mind [expletive] sessions that I had in my in my like I think all of my career H so yeah if you can like two three minutes on Erlang

Cory O’Daniel: Yeah and I got I got introduced I got introduced early in like 2010 in a very weird database that company I worked at like buman house databas and was really Erlang but Erlang you know if if anybody who listening has used Heroku you’ve actually run on Erlang as well there’s a good amount of heroku’s like routing layer that’s built in Erlang and so Erlang was built by Ericsson old cell phone company in the 90s you may you heard of Ericsson in the US at least I don’t know if they’re still big elsewhere but it’s a it’s a functional programming language but it’s it’s a bit more than a language so there’s this there’s this protocol OTP somebody’s gonna be mad like totally messed that up I’m not give it acronym but the idea is it’s functional program language has its own VM and it is designed to be extremely concurrent and fault tolerant right if you think about where the phone system is right like you need to be able to you can’t call everybody on the planet and be like hey we’re taking the phones down for 30 minutes while we like upgrade the software so Erlang itself you can actually Network the nodes pretty transparently you can update it’s a compiled language you can update the bytecode live so you can actually like stuff in you can do remote procedure calls like between the programming language itself and there’s even like built in SQL servers k key value stores so you can actually have an application that runs and I’ve got a couple different blog posts and talks on on Earl Elixir as well they’re they’re very similar Erlang Elixir the whole beam there’s a bunch of language in the BEAM ecosystem but I wrote a blog post a few years ago called from C# to Elixir that talks through how we went from Lambda to to Elixir and how it saved a ton of money and I got another talk called oh [expletive] the beam and kubernetes better together and so that one’s actually pretty interesting if you’re familiar with kubernetes I kind of talk through the fault tolerance and the features of this programming language that overlap with kubernetes and so this is one of the hard Parts when you’re coming into kubernetes ter M language you’re like where does where does The Fault tolerance and like Dev Ops of this Pro programming language is extremely powerful end and where does operations begin right so you can take if you look at companies like WhatsApp like they’re they’re famous for like they had like three VMS that all of WhatsApp was running on and it was running on these insane Erlang nodes and you can have you databases in memory that are moving around the cluster so it’s it’s a pretty wild language Erlang itself like people see it and they they think it’s hard I think B [expletive] on a lot because of the syntax erlay sorry Elixir and gleam are two other languages that run on top of the same VM which is called the beam so if you want that like functionality you can look towards Elixir and gleam but it’s really funny I feel like when people start playing with Elixir they’re like oh what’s this llang thing underneath they’re interoperable you can call erlang libraries from Elixir and then you start to play with Erlang and you’re like oh it’s this is pretty cool as well the syntax is super bizarre but it’s one of those things like you get used to it in like a day or so I think the concepts coming from you know procedural or object-oriented languages of like the bigger hurl for a lot of people that are coming out of universities where they haven’t traditionally seen a lot of functional programming but yeah so I was like this top search result and so Google which is really really funny like somebody was working on this project at Google like Googled for berlang kubernetes and like my ass just shows up first result and so they somebody emailed me and they’re like hey we’re migrating this company nobody like on this Professional Services team at Google has any experience with Erlang like whatsoever there was at the time some like real hard core interoperability issues between earing and kubernetes and so they’re like do you want to come in and help us do this migration and I was like yes so I got to join Google for a while and that’s kind of where the idea for Massdriver came from so I got to go in I worked with this amazing team there’s like 40 operations engineers in this company that we helped migrate there was about 12 people on the Google team that was doing the migration it it was really interesting is like the operations people that were in these data centers they weren’t software developers they weren’t in this world of like devops where you like do a little bit of cloud automation like they managed switches and Linux and cables like they did like the hardware work right but now all of a sudden they were in the cloud like they’re all junior Terraform engineers overnight right like they their experience still mattered but like their job was drastically different and when I left that contract you know I felt I felt almost like good luck everybody like like R just changed the world over like six months and they were like enjoy the cloud and kubernetes and the contractors were all gone and it was just up to them to kind of figure it out and so seeing this constant struggle of companies from startups to these large scale organizations like you don’t quite have a lot of cloud operations knowledge globally like for all these software teams that are out there running on the cloud today like that the number of people with Cloud operations and administrative skills is declining year over-year for the past six years so the idea was start to build some software that helps operations engineers and developer engineers kind of scale themselves L them like package up a lot of their practices and policies into a diagram and made it easy for their teams have of draw stuff

Itiel Shwartz: So yeah and so here I am today

Cory O’Daniel: So started Massdriver about three years ago now we just did A8 million seed round last year right before the entire world imploded and so now we’re just trying to you know get to that billion dollar IPO on the sky yeah no sounds sounds non-trivial but super it’s not or it is super ambitious it’s hard as [expletive]

Itiel Shwartz: So so maybe you know tell us a bit cuz I think I’m not sure if I completely understand that like that that what do you guys do like you know I’ve done Ops all my life right like I use Datadog I use Grafana I use Prometheus on one hand I use Chef I used Ansible I used whatever on the other side I use Kubernetes where do you guys fit in like when do I need you and like what’s the trend right like

Cory O’Daniel: Yeah yeah so we’re we’re on on the F on the surface we’re an infrastructure automation platform so it it feels like something like Terraform Cloud actually doesn’t feel like it behaves like a terraform cloud or a Spacelift or an env0 but the thing is a bit different about our platform is it’s visual it’s all based on infrastructure code but it’s visual for your users so the idea is your operations Engineers are building terraform modules CloudFormation Helm like they’re doing all this infrastructure is code stuff but what’s funny is like as developers start using the cloud like devel like that that infrastructure code in the operation stuff starts to become developers concerns right I build this module maybe it deploys EKS or maybe it deploys Postgres do I want my developers who are building business value to go learn terraform and monitoring and metrics and all the stuff to run RDS or do I want them to say I need Postgres 16 and then let them get back to building features right like that’s what most companies want they want their Developers sHIPAAping value but we have to build scalable compliant secure things right when you’re looking at something like RDS there’s thing there’s configuration of Postgres that developers care about and then there’s configuration that developers don’t give a [expletive] about right like how many zones is it running in how many replicas does it have developer doesn’t really care as long as it works and doesn’t page him at 2 am right but well your operations team absolutely cares about how many zones it in like what’s our Disaster Recovery plan what is backup look like does this thing make sense cost wise versus like running RDS non-Aurora or running our own like VM with postrest on it right like there’s a bunch of concerns that are Ops related a bunch of concerns that are Dev related and they’re all in the same API import right and so you’re looking at something like terraform there a bunch of variables that developers care about there’s a bunch of variables that Ops care about but now it’s on the in the plate of whoever’s looking at that module do I as a developer I’m like [expletive] like how many zones do I need let we go ask the op people well like what’s our practice for this okay great I did that now let me copy the module so my apps get a repo let me copy the CI/CD pipelines and let me copy the check off and the policy is code like you’re doing all this [expletive] just to get Postgres 16 where locally as a developer you’re like R install Postgres and you’re done right like your world in Dev is very different than your world in PR and so what the platform is is your operations Engineers take infrastructure’s code terraform OpenTofu or one of the companies behind OpenTofu Helm whatever tool it is they write the code like they do but we have this tool that packs their infrastructure code and all of your policies together and makes this unit that we call a bundle and now your engineers when they go into max driver when they need stuff they just drag and drop they literally diagram their infrastructure and as they’re diagramming it it’s provisioning and building the terraform workspaces and all that stuff behind the scenes doing all the automation it’s building these ephemeral cic pipelines that run like all of your policy code and so now as a developer I can literally draw what I need for a new service I can set just the fields I care about and as the Ops person I know I get you selfservice but I didn’t have to worry about picking the wrong Zone config or the wrong encryption for RDS or the wrong backup rotation like I can enforce all of that and I can understand what you did right like I’m like oh [expletive] like something’s down at 2 am I can literally go in and I can see your infrastructure live I can make a configuration I can roll back changes from this diagram and it’s all real infrastructures code and you know GitOps behind the scenes

Itiel Shwartz: That sounds ambitious and interesting but let me let me challenge you right like and I’m sure thatle like you know Backstage where Backstage sorry and backstage one of its promises it’s around the templating around the empowerment around like this service catalog that will enable your organization to take those devops practices and make them into backstage like template and then that’s all we need so if I’m using in like backstage do I need you are you fighting are you friend maybe and and also what’s your take either way on the backstage ecosystem which is becoming more and more like U common and and popular in the industry

Cory O’Daniel: Yeah I mean backstage is an interesting one I’m it’s a it’s a tool built by a company for a scale of platform engineering that most people will never be at right and like the tools like Foundation was like hey we can’t find [expletive] right and they’re kind of backing it into like oh it can manage the right but the reality is like a lot of these operations Engineers there’s still a lot of people that do operations they need to manage the cloud but they’re still not what You’ consider like a traditional software developer so all of a sudden heating on their plate like hey learn how to do TypeScript learn how to do this right like the whole idea with Massdriver was no one has to learn anything new you write your terraform you publish it to Massdriver mass bundle publish it to Massdriver your developers don’t have to learn terraform they don’t have to learn about your CI policy code they literally just draw [expletive] like nobody has to learn anything new but when you look at Backstage like there’s the Pains of rolling out backstage to the point that there’s like 30 [expletive] clones up it now right and then you have to learn something new the operations team can’t use the infrastructure’s code tooling that they want right so I think that we’re solving kind of different problems but I also don’t I’ve honestly never really seen the value of something like backstage now if it’s okay if it’s the thing that will now manage my infrastructure okay now now you’re starting to talk something I think is interesting and valuable if you can manage infrastructure if my developer can make a new microservice and the database and be able to understand how that stuff works together and maybe I can get in there and see it but if people have to learn new things we don’t have enough time to like learn another tool like we have so much stuff do like developers you only 40 hours a week you got to build features and operations teams I don’t know we’re all [expletive] [expletive] underwater anyway right like find the an Ops person who’s not drowning in debt while also trying to serve like 10 new business work streams at the same time right so our B goal is like we don’t want either of these teams to incur more debt because they have to stop and learn how our thing works and so our goal is always been like we don’t want to increase any sort of learning curve on either of these teams so I mean we’ve gone head-to-head with like oh should we host backstage ourself or not funny enough like we do have I know one customer uses driver for all their infrastructure and then they run Backstage on top of their clusters which I think is funny I know that we also have we have a couple of customers that do that with people that we would consider I’m not going to drop any names but people that we would consider competitors they use us for all the infrastructure management and then essentially the kubernetes stuff like all the kubernetes Clusters they have that company running it’s CL Su you know what I’m talking about so yeah so I mean I think it’s I think it’s tool I think if it fits your team that’s great but I think the more important thing like looking at the ecosystem there’s a lot of Taco Tero motivation out there Spacelift env0 RNs like we kind of fit in that space there’s all the IDPs that are out there right whether it’s portal or platform like we kind of fit in that space too I think the more important thing is none of these tools Massdriver included are the solution for everybody right and like and I think that’s what’s really key is there’s a lot of these tools especially when you start getting open source space they try to do everything under the sun and it gets to the point where it’s like it does so much that you don’t even know what to do with it right and I think that backstage had a very good place like a year ago but now it’s starting to get to that place where it’s like what is this what is this thing like is it that is it this thing that runs my like deploys my infrastructure is it doing a typescript or do I make a typescript module that calls terraform and I still do it with terraform right like I think it’s gotten to a place where it can over complicate and since it’s like the leader kind of in like the open Community it’s like oh that’s the thing we should use but the reality is is like M dri might be the better fit for you spacelift might be the better fit for you it really comes down to what your DevOps maturity level is what your team makeup is and how you even Define de boops right like

Itiel Shwartz: No I think all all are like you know you hit it in the head like you know the complexity the tool sprawl the no one it all on one hand and you know a lot of people will will say that you know you’re saying that backstage was developed by Spotify which is true to their use case you know kubernetes and Google is even more into the extreme in a way right like taking this huge huge huge behem they they develop internally to run your like for micros service doesn’t necessarily makes a lot of sense and you know I’m I’m a big Kubernetes Advocate but but let me ask you like another question because I think it’s very interesting talking about those underlying tools the support the infrastructure what’s your take on Crossplane like I see a lot of my customers that are telling me that they are now thinking about migrating to Crossplane mainly because it is much more kubernetes native it allows you all of the state con control plane so what’s your take on that should they move to Crossplane

Cory O’Daniel: I don’t know so I actually I have a talk from hold I’m I’m gonna google it really quick to see what I don’t I don’t know if it predates Crossplane but I gave a talk it’s called command Deering kubernetes and I gave it at the 2019 so I don’t think I don’t think I think Crossplane existed before 2019 but I hadn’t heard of it and so this talk was when I first started building The kubernetes Elixir framework so you could actually like write operators and schedulers using Elixir instead of Golang and in that talk I pretty much show off like a homegrown Crossplane so I’m showing you like oh you can actually like you can build these operators and now I can have like a Postgres config that will actually provision on RDS alongside my Helm chart right and so like I was definitely interested in the idea at at at one point in time but I think where it kind of jumps the shark is if you’re a company that is 100% bought in on running your applications on kubernetes crossan’s probably fantastic and we support cross win and nobody uses it on as we support it right but I think the more important thing the thing that kind of like gets to like chicken and egg with with it is like when you look at companies that that don’t use kubernetes across the board it’s like I don’t think it makes sense with them at all it’s if I’ve got a couple of things in kubernetes and then I’ve got a whole Suite of landas and step functions and I got some I got some sap stuff running on some VMS if I have to write kubernetes manifest to make these things that oh I need to write a kubernetes manifest to make a VM that’s not in kubernetes or I have to write a kubernetes Manifesto deploy my Lambda like I feel like we’re really Jumping the Shark there right but if it’s like hey all of our stuff we’re we’re on open fast like all of our workloads stay in kubernetes the entire team is just eating and breathing kubernetes great but if that’s not the case it’s like okay well maybe that team that’s doing a bunch of Lambda stuff maybe they should use in Serverless Framework right like that that’s the real rub is like I feel like we just do this time and time again like with teams it’s like I actually give a really good example of this we we say like oh my team’s like hey I we are using this tool and this company will use it be damned whether it fits or not and we had this problem in platform engineering too it’s like oh we should build a platform it’s like no like there should be multiple platforms you’ve got multiple teams that do things very differently right if you come in you’re like hey everybody just Port here that’s [expletive] it right or or everybody is going to use Dapr or whatever and that’s it I’ll tell you what you have a bunch of pissed off Engineers you might have some real great dead for some of them you have some real bad deback for us them like your teams are different organizations are different right we already talked about that right Orcs are different there’s gonna be different tools but you get into the teams my front end team my front end team doesn’t use MTH driver funny they use Rell and I’m never going to take it away from maybe one day we’ll make a CDN integration so they can deploy to S3 but they like for cell great everything else a Massdriver runs through Massdriver on AWS right okay that’s cool look at a bigger organization they’ve got a front-end team they got a backend team they probably got multiple of these but let’s just say they have three teams front end team backend team and a data team you gonna Jam them all in the backstage tell you what that that frontend team probably wants to use a Vercel or a Flightcontrol or something like that that data team probably wants to use some new fancy data tool but you’re jamming them all into this one thing and where I saw this like happen where it was just like like the shark was jumped and then the shark jumped the person jumping the shark and it was it was a stunt show that company that helped do that migration they decided to like we’re using terraform if you do anything in Google Cloud you have to use terraform I’m working with a data team one day and they’re working on a big fery table and they’re like this thing is really hard to do in terraform and I’m like oh yeah let me let me help you guys out I’ve been using this terraform thing forever I look what they do and they literally they had a null resource in terraform they they packaged the gcloud CLI command and they were just using the gcloud CLI command they were just executing it with terraform they were literally doing it in spite of the Ops Team they’re like they said I have to use terraform we don’t want to use terraform we just like using the gcloud thing we’ll just write a terraform module that wraps the gcloud gcloud CLI command and manage BigQuery tables and I’m like oh like like that’s that I feel like that’s kind of like where we get a lot of times is like you can’t just come in and say hey everybody has to use this thing and our philosophy with Massdriver is like platforms as cattle like in Massdriver I can build a platform for a specific team using terraform modules in a matter of minutes oh you want to do stuff on Lambda great I throw down some infrastructure you your team has your infrastructure platform for Lambda great you can clone it you can make as many environments as you want that team’s happy oh my gosh this team needs to run microservices design a platform that’s kubernetes based assign it to them they can clone it make staging prod Us West EU whatever ever data team wants to use I don’t know maybe data team wants to use some new hot platform as a service for data gray throw in the terraform module build them put the provider in there give it to them like now this whole team has the same UI but different platforms they have the same tool but different IC mechanisms different clouds that it runs on you haven’t forced anybody to start to use some technology that impacts the way that their application RS right and I think that’s the hard part with a lot of what we do is it’s so much easier for op schem of three people to say we’re just going to use C A but when you have a team of 150 developers doing things very different ways because you’ve been in business for seven or eight years and you got debt Services all over the place like falls apart it gets it gets hard to move that initiative forward

Itiel Shwartz: Okay I don’t think that we have a lot of time so maybe I’ll do like a bit like a wrapping question of asking you and I think you mention it quite quite a lot here but what’s the future like for Mass Drive in particular but for the industry as a whole like three years from now we’ll be talking what’s going to happen what’s going to be different then today like Trends cultural shift whatever you like like shoot

Cory O’Daniel: Yeah I think I think that we have gotten to a point where everything is way more complicated than it leads to be right like there is you know like you it’s funny like when you look at the cloud AWS never had the goal of being symol or easy they had the goal of being Caper they want to sell whatever type of compute to you that you need right and so when you sit down and look at AWS and it’s like we see this all the time C customers are moving from like Heroku to AWS the first time they’re like oh there’s 200 AWS services like yeah just use a VM bro like if you’ve been on Heroku just deploy one VM like just start there throw an autoscaling group on it like don’t get lost in the flurry of 200 services but there’s so many companies I’ve seen like they come to AWS and like oh I have to use ECS or and they they just they get way in over their head on the amount of like Cloud sprawl day one so I think you’re seeing this a bit with like repatriation to data centers like you’re seeing people start to talk about this you saw the 37signals move of a Basecamp and and hey I think we’re going to see a Resurgence of Simplicity and I think we’re starting to see that there is like 15 platforms of service now there’s a platform is a service from is every way that you want to run now I think this is exciting I think PaaS platforms super fun and it’s not threat to us because as long as it could be automated with terraform or IAC like it still works a Massdriver but the thing that’s concerning about this is we aren’t making enough Ops people still and AI is not catching up to Ops at all if anybody’s like wait what the what the [expletive] do you mean by that we don’t train AI on Ops we train AI on really shitty terraform code on medium articles we don’t train it on production data we don’t train it on your traffic patterns we don’t train it on your architecture we train it on snapshots of little pieces of configuration so AI is not going to really help the general company it might make an operations person more efficient right and so as we start to reach for these simpler systems people going back to PaaS platforms maybe people using simpler things like Lightsail I don’t know if anybody actually uses it but let’s just pretend right what I’m concerned with is what happens to security at that point in time because a lot of these PaaS platforms do not do private networking they don’t do peering right and you’re starting to see these people reaching for Simplicity big companies starting to use Vercel huge companies starting to use Supabase these are both fantastic products but when my back end traverses the public internet to go to a database as a service and then I’ve got Redis as a service over here and I’ve got Kafka as a service over here from three different companies and I’m just constantly traversing the public internet we’re going to start seeing more breaches start seeing more latency issues I think

Itiel Shwartz: It’s interesting like I to be honest I’m not sure you know how how are we going to to to solve it because I agree with with the problem I don’t know what will be the solution but I agree that we’re much like the complexity can’t really like we can go down this route like something need to change okay I think with that we pretty much finished the episode and it were like a bit behind schedule why should or like how can I contact you or M driver should they go to the site is there a free trial how can I you know you want me over how can I use you

Cory O’Daniel: Yeah I’m Cory O’Daniel no e pretty much everywhere LinkedIn GitHub Twitter Etc Massdriver is Massdriver. Cloud we do have a free trial so get on there hop on it play with it and yeah I mean if you’re if you’re a startup grounder like starting to fund raise and like need intros or want to like get some like look at your deck like feel free to reach out I got a ton of help getting to where Massdriver is today so I like kind of pay that forward so happy to chat about startup life if you’re somebody you’re stuck with an Ops problem shoot me put me email I might know the answer but yeah cor Daniel everywhere I’m cor. Cloud and yeah also you can find me at platformengineeringpodcast.com

Itiel Shwartz: So maybe I will come as well and then we’ll do like the complete

Cory O’Daniel: Heck yeah full circle do it

Itiel Shwartz: Yeah yeah so thanks a lot hry yeah that W rather me

[Music] Kubernetes for Humans.

This is an AI generated transcript of the conversation