#056 – Cloud Contradictions and Cautionary Tales with Corey Quinn (The Duckbill Group)

Itiel Shwartz: Hello everyone and welcome to another episode of the Kubernetes for humans podcast. Today with me in the show I have a very special guest. Corey, do you want to introduce yourself?

Corey Quinn: Oh, by all means. I’m Corey Quinn. I make fun of large-scale cloud infrastructure companies. Mostly due to my own personality defects. I write the last week in AWS newsletter and accompanying podcast and I’m at Duckbill where we fix the horrifying AWS bill for very large entities.

Corey Quinn: Thank you for having me. I love the sound of my own voice. May as well dive into indulging it.

Itiel Shwartz: Yes and I started the off record saying that I’m I’m a huge fan and like like following you for for years. I’ll be honest. I thought that you are only write about AWS, not on other vendors as well or or like it’s mainly AWS.

Corey Quinn: It’s I focus is primarily AWS, but if we take a look at the entire tech industry as a whole. This tech itself far too seriously and I do intend to change that in the long term. I’ve done a fair bit of work with Google, with Oracle, a bit with Azure in years past. Mostly I tend to pay attention to what the customers are paying attention to.

Itiel Shwartz: Okay, that makes sense. So that makes sense. Corey, like I feel that we dive into the specific. Maybe if you can in a couple of words share about your own journey. Like how did you start? What happened and yeah, first experience with AWS?

Corey Quinn: Most people ask that question through a lens of it wanting to be an inspirational thing that other people can draft along and follow behind. In my case it’s much more of a cautionary tale. I’m one of the best in the world at getting myself fired inexplicably from random jobs based on nothing other than my personality. So ultimately starting my own thing was really the only path that was left to me at a certain point.

Itiel Shwartz: You are the only one that was like that was okay with hiring you. Like that’s what you are saying basically.

Corey Quinn: Well, part in seriousness part of it was that I the things that make me a good consultant are the things that make me a terrible employee cuz once the big interesting problem is done, the firefighting is gone. Well, I want to do something else. I don’t want to sit there and do the same thing day after day. I was lucky in that back in my SRE days when I was running infrastructure at varying scales, it was it was before Kubernetes really broke out, which means that I didn’t have to gaze into that abyss for a long time. It was wonderful. Now of course all my clients all use it and I have a spare Kubernetes cluster in the sewing room as one does and it it taught me a lot specifically that I never want to go back to that world because now I’d have to touch Kubernetes with my hands and we’re not sure how that spreads.

Itiel Shwartz: So so you don’t like Kubernetes that much, but you know like you are like internet like celeb right in at least in our community. So how did it start? Like I understand that you are fired from a lot of places, so you went to do consultancy. That makes sense. But what made you know start being so successful over Twitter, which is now X? Like what happened there or why do you think you are so successful?

Corey Quinn: I think it has a lot to do with the idea that most people can get fired and they have that that gremlin over their shoulder watching to make sure that they don’t wind up saying something career limiting. I just don’t really combine that with a I have a strong ethos against punching down. I’m not here to make individuals feel terrible, which is how I have avoided some of the worst excesses of being a Twitter gremlin. It’s I say the things a lot of folks viscerally feel and I think that resonates at least historically. I mean I talk smack about Kubernetes. I talk smack about everything, but it’s it’s there for a reason. People generally don’t deploy things just because it’s trendy. These days that’s AI. There are usually reasons behind why people are doing the things that they’re doing and it solves a problem for them. That doesn’t mean we can’t make fun of it. It doesn’t mean we can’t enjoy a an esprit de corps of having to deal with the misery that is production in other people’s scale. But that’s how I see it.

Itiel Shwartz: Well, that that that’s a good answer and I think the fact that you are indeed saying a lot of the things that other people think but don’t have the stage or the they have too much you know things holding them back for saying them.

Corey Quinn: I’m probably off base. No one would listen to me or care what I have to say.

Itiel Shwartz: Mhm. Mhm. I think you are hitting some really really important points and and maybe to to derive the conversation a bit, you talk about hyper-driven development and agents, right? I know that a lot of what you do or a lot of what you’re writing is around cost aspects of running cloud, running production environment, running AWS. And as like someone who is also managing or using quite a lot of AWS Bedrock and agents in production maybe you can share a bit on the landscape of agent FinOps if that’s a thing. And like maybe like share a bit about agents in general. Like what do you see? What do you think? Where are we going or

Corey Quinn: I want to call out though that my version of production is very different than other people’s version of production. At Duckbill we’re building a product called Skyway and the engineering team has the good sense not to let me touch that directly for reasons that should be blindingly obvious. But that does mean that I get to spend a lot of time building weird things. Mostly arc lark. In fact, I keep a running collection of them on the web at shitposting.ai. Yes, that’s real. And it it lets me build weird one-off things that I can throw over the wall see what resonates, what’s fun, what’s enjoyable. And that in turn leads me to I get to dip my toes in the water without the pain of having to run this at scale for a long period of time. How is other people’s requirements? Now, have I learned a lot from doing this? Absolutely. Is this a way to run I don’t know your bank? Absolutely not. But not every environment needs to be treated with the utmost seriousness of if one of my dumb shitpost projects stops working in the middle of the night, nothing needs to wake me up for this. It’s just not that important. It’s not revenue bearing. I’m not eroding customer trust. Our actual SaaS product, yeah, that’s a bit of a different story, which is again why I don’t get to touch that stuff.

Itiel Shwartz: Mhm. Now that that that is indeed interesting. Like your take. What what sites do you have? Maybe give us a couple of like those projects.

Corey Quinn: Oh, I have my own Mac I have my own Mac app now where it tracks the current status of CI/CD builds that hangs out in your menu bar. It’s called deploybar.app. It’s a free app, but it makes fun of you. If you want it to stop, that’ll be $5 a month. It’s it’s a different go-to-market approach that honestly I just find amusing. I have but hasn’t quite launched yet. I’m putting the finishing touches on it, but I have chatjippity.com, g i p p e t y.com where it’s terrific. It’s just like ChatGPT in that it refuses to do anything useful citing the grounds of AI safety because why wouldn’t it? I’ve done a number of other things here and there. That’s a good question. It all runs together past a certain point. When agents are building stuff, you don’t have the same closeness to it. So instead it’s just throwing stuff over a wall and well, it’s the internet’s problem now. There’s always my joke ratsissimo, which is an AI-powered singing rat trap that whenever it catches a rat, it’ll perform a victory aria at the top of its lungs at 3:00 in the morning. It’s powered by RatGPT technology. It’s backed by Y Exterminator and it’s great. The website itself at shitposting.ai has a bunch of songs that it actually will sing and frankly, they’re bangers. Mhm.

Itiel Shwartz: I hope you host this site on top of Kubernetes.

Corey Quinn: Surprisingly most of this stuff tends to be just static sites that I’m throwing over the wall. A bit of it lives in Vercel, but some of these names are backed by my home Kubernetes cluster just because I the thing I built the thing I may as well use it for something. Nothing loaded. Okay. It’s a it’s a bunch of Raspberry Pis. What am I going to do with that if it starts overheating? But yeah, a bunch of my internal tools lives on that. It’s it’s great from an AI perspective just to have a place where step one, build something horrible and stuff it into a container. Step two, where do I chuck that over the fence so that it can run somewhere where I don’t have to worry about things like security or getting access to it. With the combination of Tailscale and somewhat security policies it’s not as big of a concern. It’s it’s a great garden to start playing with these things in. Would I run something there that had access to customer data? Absolutely not, but I will use it for stuff that amuses me.

Itiel Shwartz: Well, that’s super cool. And by the way, I’ve done something similar, maybe less funny but one of the problems that I had when I started to use AI was that I had a lot of like you know projects, right? Some of them were good, some of them were shitty, but I wanted to host all of them in a single place. Like a single site that will host all of them. So I built myself also something quite similar. Even I host everything with Cloudflare like Cloudflare Workers or something like that. And then every time I do push, it deploys it automatically to production cuz like you said, I don’t really care. But it makes everything much more faster.

Corey Quinn: Oh yeah, everywhere I need to be building these things. I have a there’s a there’s a service that runs in my kubernetes cluster. That’s the Narrative Oracle because every commit message contains conspiracy theories about the code because then why wouldn’t it? And this lets it track the larger narrative Arc that it’s building towards. So these aren’t just disconnected commits. They lead to a film noir style level of conspiracy nuttiness.

Itiel Shwartz: Okay Corey hard to beat that but I do want to hear you know, we’re talking a lot about agent. Give us your take on the current state of agents and the future state of agents maybe like where are we going? Where are we now? And like is everything going to be agent like no agent base in two years. This podcast is going to be ETL agent. Talking with Corey agent. So so what does the future as you spoke?

Corey Quinn: Hopefully it’s going to be some sort of agent that listens to it because one thing there are many is that if we couldn’t be bothered to spend the time to create the thing, I don’t think it’s a reasonable to expect that anyone else will care enough to listen to it. It’s So now right now I will say that AI across the board agentic and otherwise is a really neat tool and we are in this industry-wide state where we are currently trying to find the right nail that we can use this hammer on and it’s being shoved into everything kind of like the early days of cloud and I think companies are getting it wrong when they’re marketing about AI. AI is how you do a thing but the customer doesn’t care how you achieve an outcome. They care about the outcome. Instead of talking about how talk about what and that seems to be missing across the board right now. In fact, we’re seeing companies take amazing brands and more or less let them on fire light on a fire just so that they can talk about oh this is AI powered like it checks a marketing box. I don’t know. But it’s frustrating.

Itiel Shwartz: Can Corey can you give us a couple of those brands? you know, you said that there are a couple of brands that took their basically product and you know, shifted towards like this AI facade.

Corey Quinn: Sure in a year or two. Of course and they’re being in the punching up perspective. Oracle has now become instead of the traditional thing that they did which was a law firm that also makes databases. Now they’re entirely doing an AI data center build out company spiel. AWS has repositioned everything that they’re doing through a lens of generative AI which I think is a giant mistake. Microsoft or microslop is how it’s coming across now shoving Copilot into literally everything that holds still long enough. Notepad.exe for God’s sake. It’s they are they are effectively lighting fire to any form of brand Goodwill that they have to replace it with something that is non-deterministic and further non-differentiated. If I can use a the same agent to deliver the same outcome in Google Docs or Microsoft Word or yes apparently notepad now then you’re just keeping up with the Joneses but it’s something that I don’t hear customers asking for. It’s all it’s all push. It is not told.

Itiel Shwartz: So we are in the fact that everyone is doing these three brands and I wanted to ask you and then they have their like shareholders right like that’s their goal in a way to look sexy. Don’t you think it’s part of that? Of this like rebranding?

Corey Quinn: Absolutely. I think customers are rational companies are rational actors. They’re not doing things that generally don’t make sense. Whenever we see that with there’s clearly contacts that we lack and don’t have. But I miss I guess I’m too much of an of an innocent in this sense. I miss the olden days when Amazon used to say oh we’re willing to be misunderstood for long periods of time. AWS for its first decade was something the market didn’t really understand why they would waste their time and energy on it until they started breaking out its revenues and they realized how wildly profitable it was. It’s they they would do things that did weren’t going to pay off in the next 90 days. They got out away from the quarter to quarter thinking and right now they have to quick hurry up and be just like everyone else. I understand the incentives. I just think they’re crappy.

Itiel Shwartz: Okay that makes sense but so what do you think they should do if you are like you know who is now running AWS?

Corey Quinn: The problem is is that AWS was so quick to jump on the narrative of we’re the best at AI when it was very clear that they were well behind in AI to the point where they wound up setting fire to a lot of the earned trust that they had. Now some of their AI offerings are legitimately decent but okay that’s the that’s a hard question. You don’t necessarily want customers asking them. Okay, you’re saying it’s good now. Last time you said this it was complete garbage. Are you telling the truth now? -huh. That’s the space they find themselves in. They have eroded an awful lot of credibility along the way and I don’t know if they can get that back with just a product cycle or two.

Itiel Shwartz: You think so?

Corey Quinn: I don’t know. AWS in the days launched a lot of shitty product right like I’ve been AWS user for the last 12 years I think. Oh absolutely. The Look across the board at the things AWS is great at. It is it is tier one infrastructure. It is building out the building blocks that everyone else builds on top of. They’re the next generation equivalent to the tier one backbone providers. If NTT goes down, the entire internet’s having a crappy day but most people don’t know who the hell they are. Everyone builds on top of them and the margin grows on top of them and that’s AWS’s fate. Amazon does not know how to build something further up the stack. and I think that they’re letting their focus that they’re the stories they’re telling accrue around these things that they’re not good at. It though they’re barely going to mention EC2 but it’s the largest service they have by a lot. It’s what everyone cares about. It’s what everyone uses but it’s not interesting or fun to talk about virtual machines in the same way as it is about agentic AI. So everything you’re doing everything gets the attention is shining a spotlight specifically on the things they’re not great at. I don’t know how that benefits them in the long term.

Itiel Shwartz: I think like you know, it’s a good question and I and where I use like AWS internally and and GKE as well and a bit of Azure a bit a bit of Azure. I think as crappy as AWS AI offering, I do think it’s better than the rest. Like I think if you’re an enterprise right like all the like not an enterprise but a company that wants to run a reliable AI like what would you choose? Okay Bedrock is terrible.

Corey Quinn: Perfect as far as being a uniform to run on. I mean it’s late

Itiel Shwartz: Yeah it’s slow right it goes down a lot but

Corey Quinn: I was talking to a number of companies that whenever whatever they’ll wind up doing inference against some of the cloud models, they will build in a router that okay if suddenly we see latency spike again on Bedrock as happens all the time, we can seamlessly pivot to using other providers to fulfill that because an inference API calls an inference API calls not that differentiated. And yeah that’s terrific. You move look higher up the back their Q Developer developer tooling which is basically Amazon Basics cursor. No thanks. I don’t see people using that by a as you enterprises sure because it comes along with a deal and get to get a really beneficial proceed approach but the folks who are saying that Q Developer’s amazing are generally the folks that haven’t had a lot of exposure what else the market is doing.

Itiel Shwartz: Oh yeah those I agree. I think I talked only about yeah I talked only about Bedrock which we are using. You know, it it’s not that good. That being said, I think it’s better than any other single provider. Like I don’t know of like a better uptime solution to run on Anthropic right like we’re using solid quite a lot. Oh

Corey Quinn: And trop sales goes down as well and that’s why the answer has been you wind up keeping an eye on the telemetry coming out of your of your AI API calls and you pivot in real time to Vertex AI or Bedrock or to sort of or OpenRouter or

Itiel Shwartz: OpenRouter allows to do it like quite easily but but it is like for us for example for my compliance reasons, I can’t really leave AWS. As shitty as as Bedrock will be, I will use Bedrock cuz I find to my

Corey Quinn: But do you think that AWS wants the narrative to be that the people who are all in locked in on Bedrock are just the people that can’t because of compliance reasons?

Itiel Shwartz: I mean they they they don’t want they don’t want to arm twist people into using it. They they legitimately want this to be the best answer. It’s just all right then.

Corey Quinn: I but I didn’t hear you disagree.

Itiel Shwartz: I think it is the best answer. I think the rest are shittier compared to that. All of them are not that good. Like none of them has the reliability and promises of like EC2 right or S3 or SQS or like those RDS like all of these like available AWS services but

Corey Quinn: Or I have it that goes down we’re all having a bad day.

Itiel Shwartz: I don’t remember when I am had issues. I’ll be honest like a lot of times to be honest.

Corey Quinn: It it never it never has failed open and that’s the important part.

Itiel Shwartz: Yeah like I don’t remember that. But but interesting take. What do you think about like the other companies like you know, like your take on Google and Microsoft agentic offering? Like are they in a better position in a worse position?

Corey Quinn: They’re in a better position if for no other reason than that they move up the stack more effectively. Every office worker out there is a potential customer for Copilot because you’re using Microsoft software day in day out. Google has that similar surface area with Google Docs. They have it with Gmail. Amazon does not have anything in that space. So yeah if you’re talking about a go-to-market motion for distribution, Amazon’s coming from very far behind. What the one exception that they’ve had historically has been connect which is what they used to run their call centers. And that tends to be fascinating just from the perspective of it’s the one breakout success that’s a little bit above the plumbing it into the porcelain. But it’s not it’s still not nearly as well known industry wide.

Itiel Shwartz: You know, we mentioned the name Anthropic a couple of times already. Right? And and like you know, it seems like they are going after everyone. Do you think that like I and I have a lot of conversation with other CTOs or other people is Anthropic going to be like the AWS Bedrock that in structure that other players are going to build their solution on top of. So, maybe give us your take on Anthropic in general as you like punching up and also like where are they heading toward?

Corey Quinn: Anthropic, OpenAI, and the others have the same problem in that they have made wide ranging very large commitments against future revenue that they’re depending on the depending the life of the company that that revenue is going to materialize. And what that looks like is going to be very interesting. The the harnesses around these foundation models are effectively the differentiation to some extent. The foundation models themselves are really not as differentiated as you might expect. If suddenly due to some contractual reason or compliance reason you’re only allowed to use OpenAI’s models and you’re not allowed to use Anthropic’s anymore most people will be able to retool within a week. It it they there’s not a lock in. There is not a story of differentiation. So, they’re going to have a serious problem with that as as they continue to be neck and neck. one milestone that seems to be being removed from OpenAI’s approach is serverless inference was only allowed to be done on Azure. And as a result, we aren’t seeing OpenAI penetration into the market the same way that we have Anthropic with respect to business. Sure, the consumer side ChatGPT versus Claude, there’s no contest there’s no contest there. Everyone every customer knows ChatGPT. Like my my parents know about that. Claude is not on their radar. But that tends to be a relatively low dollar market. It tends to be very fickle and they tend to be hard to serve. when you move up the stack to business you start to get a little bit better alignment around these things. But Anthropic is setting fire to their own reputation on this all the time with weird tests around pricing changes about suddenly you’re not able to wind up using as much of a of your plan as you thought you were. This just spies that level of uncertainty. Rightfully so. as a business owner, what you want to see is that tomorrow doesn’t look that meaningfully different than today with respect to your financials.

Itiel Shwartz: So, who do you think is doing it the best like this AI revolution? Like you know, we mentioned like a lot of big players. Do you think anyone is exceptionally well in in this new era earlier?

Corey Quinn: I think one that’s doing reasonably well is Harvey. they are effectively a wrapper around a lot of these foundation models specifically aimed at one vertical, legal folk. And I’ve talked to a number of attorneys who’ve been using it who say terrific things. Okay, could I The stories they’re telling, could I get Claude to do that or Claude Code? Sure. But it’s unreasonable to expect most folks to be able to go into a command line and play those games. Is there a story where suddenly Claude Code gets a Claude Code work gets better at this and starts disrupting them? Absolutely there is. But for for lawyers that is a terrific vertical to explore. Intercom is doing a terrific job with their own models as well as when with thin. And they have completely pivoted their business around this, which is great because human contact for support does seem like something likely to be disrupted. They’re doing super well with it, too. a number of observability companies are using AI to solve a very real problem, which is I don’t know what the hell your custom language for querying your data structure is supposed to look like but now I can ask the robot to do it in plain English and I don’t have to go down your path. That’s a huge

Itiel Shwartz: It is. It was as someone who learned a lot of shitty query languages, it is a huge win.

Corey Quinn: Yeah, but whether these things worth multi-trillion dollar valuations? I don’t know. I have some pushback on that.

Itiel Shwartz: E e we have like Cursor as maybe like the biggest example of a company does it actually worth what it is supposedly worth, which is

Corey Quinn: Cursor a lot but we’ve been in a revolution lately where we are starting to switch over to Claude Code. And I won’t and I won’t lie about this. A big reason is suddenly they’re doing a bunch of work with xAI. And when it comes to down to it to be direct I am very hard pressed to name a company I trust less than xAI. I tend to agree with you. I tend to agree You’ll notice API changes. shitposting as a primary business strategy is kind of funny with me being independent and shitposting for fun. But when you’re a public company, that doesn’t work nearly as well, which they’re desperately trying to become. it feels like it’s a shell game. I think that their governance is a disaster. -huh. Good luck. That’s not something that I would want to stake the fortunes of my business on. But maybe I’m too conservative like that.

Itiel Shwartz: No, I don’t I don’t think you are or if you are, I think a lot of other people are probably even more than that. But the but the examples that you gave are quite cool. So, you are saying the ones that are actually successful and it ties to the beginning of the your take on agents are the vertical ones that know how to provide really meaningful value to their customers by abstracting the AI in a way like

Corey Quinn: Correct. It’s about talking to a vertical they know how to speak to. A counterpoint to this that has been a disaster is everyone building around a feature that the foundation models themselves do not have today. the day that OpenAI and Anthropic suddenly gained the ability to work on PDF files put a bunch of companies under. Yes, clearly that was always going to be the case. Focus on a vertical of customers, not a specific technological capability. One of those moats is a lot more durable than the other.

Itiel Shwartz: No, that makes sense. I will say though there’s there’s no better way I have ever found though to burn tokens in any tool I use other than throwing Kubernetes output at it.

Corey Quinn: Oh my god.

Itiel Shwartz: I think there are better ways. I think there are better ways. But Corey, like we are almost running out of time. maybe you want to give like final words or like things that we should know about you about the company about the future or anything else.

Corey Quinn: Something that I that I’ve been focusing on for a while has been the cost of this. And people are getting that wrong. Right now when you’re in POC stage, the cost does not really matter nor should it outside of insane boundary points. If it works, terrific. There’s always time to optimize later. If it doesn’t, you’re going to shut the whole thing down anyway, so why waste your time and energy optimizing for it now? But you should be making decisions along the way so that when the time comes assuming success you have a line of sight to being able to understand what the cost per transaction is. That’s something people are getting wrong.

Itiel Shwartz: A very cool take. I can tell you that in Commodore we started like that and in a year we went to it’s negligible to what the [expletive] is happening here. And then in a week and then in a week we were able to go back to normal. But yeah, when it does spike, it spikes very fast. so Corey, I think with that I we will wrap this episode. It was a super cool episode. Thank you for having me. It is always a pleasure.

[Music] Kubernetes for Humans.

This is an AI generated transcript of the conversation