Video: Team Komodor Does Klustered with David Flannagan (AKA Rawkode)

An elite DevOps team from Komodor takes on the Klustered challenge; can they fix a maliciously broken Kubernetes cluster using only the Komodor platform? Let’s find out!

Watch Komodor’s Co-Founding CTO, Itiel Shwartz, and two engineers – Guy Menahem and Nir Shtein leverage the Continuous Kubernetes Reliability Platform that they’ve built to showcase how fast, effortless, and even fun, troubleshooting can be!

Below is an auto-generated transcript of the video:

0:00 so starting from left to right starting with you guy could you please say hello and introduce yourself and share a little bit more if you wish yeah hey

0:07 everyone I’m guy I’m a solution AR commodor I’m here for the last two years

0:12 very excited to join cluster cool everyone my name is

0:19 newstein I’m social engineer I came week after guy and

0:25 it and hey everyone my name is Schwarz I’m the CTO of commodor watch and

0:30 clustered happy to be here all right thank you all very much so this is a

0:35 special edition of clustered and I have broken the cluster personally myself with a handful of braks to hopefully

0:42 reveal and show us the power of commodore as a tool for understanding and debugging problems within your

0:47 customer so with that being said I will now let guy share his screen and we will begin the debugging process best of luck

0:54 team commodor thank you let’s let’s start cool there you go so we got our

1:00 cluster with a condor install so I will consider this cluster

1:07 fixed if we can port forward and visit the duple website that is your

1:13 mission okay sounds easy okay easy

1:18 right you know the reason is it’s scheduling yeah

1:24 so we can go into like the service in Commodore and see the timeline right like how did the service change over

1:30 time and you can see that it’s currently broken and you can try and understand why is it broken so why is it broken I

1:37 see there is a deploy like in 15 25 open

1:43 so do you mean like oh to zoom in okay there you go I think it’s a great story

1:49 of what happened yeah and we can see there was a change in GitHub yeah there was a like an awesome

1:57 update update was it an awesome upate we will

2:03 not and here David remove some environment variables I think they’re

2:08 crucial and change the CL even worse cool so let’s take a look about

2:14 the problem let’s see if they are like connected to one another oh so we definitely can see that the volume the

2:21 PVC not found it’s definitely the problem with the p let’s try to do the roll back right yeah let’s cck R back

2:29 you don’t have permission to do it because it’s your cluster

2:35 David it’s also a getup’s cluster so the roll back wouldn’t actually work it would be overwritten a few minutes later

2:41 however the first you’ve discovered uh essentially the second problem there is

2:48 something that’s more current with this deployment but you yes we will have to fix this too so let’s you mentioned the

2:55 rate at the start somebody said scheduling right let’s let’s look at the pots guy

3:02 the pods yeah let’s look

3:08 atending other than the previous thing let SCH not ready yeah let’s go back we

3:15 have information in here by the way like on the first scheduling of the event would be much easier to see from

3:22here ah I think that one node is not available

3:28no maybe let’s check the pods list first yes let’s try check on the resources

3:35node and see that there is a node that is not ready and schedule disabled maybe

3:41let’s try to unone it it’s C condone so yeah I don’t think we will

3:47have it maybe let’s go to the terminal try to fix it from the terminal you add yourself as a yeah as a

3:56user try to add my my person

4:01do you want you to I okay so we’re doing like a switch just because of security

4:08permissions and because David created the cluster and the account basically H then we need to give like to add another

4:16team member to the commod account so David if you can invite Neil it can be

4:23great yes so let’s go to the notes let’s take

4:28the action and home

4:37unle and also let’s do the wall everything at the same time let’s

4:44do so now nearly like rolling back the service oh maybe the first deploy was

4:51not in Commodore that’s why I didn’t say

4:58that try yeah no

5:04fory yeah we can also take the changes from the GitHub and we can also take the

5:10CH from commod right andth yeah from GitHub I think the service was deleted and then reinitiated

5:17right like it’s generation one it basically means that like David played with the service apparently then he

5:25deleted the deployment then he recreated the deployment with the fil yeah with the failed configuration and that is the

5:32reason that the came fall back because for us it’s a new kind of service B that

5:39it has a different unique ID and this is the first generation of the new Pro

5:45deployment so we need to wall out this workload yeah is the notes okay now

5:53let’s check it yeah they are okay do you like check it on your screen or no no I’m asking I I can no it’s still not

6:01ready ready why is that and contain network is not

6:11[Music]

6:16ready it looks like more like a like a3s issue maybe yeah we we need

6:24the N plugin to be ready can we check maybe on the service as what is

6:30configure do the netor plug in maybe let’s try to take a look in the

6:46Y the network unavailable okay so what’s the reason CN is not

6:58initialized it’s k3s right no these are bare metal

7:04cube admin clusters bare metal

7:11here

7:16um maybe those are the things this is a 48 core 64 gig ram bare

7:24metal machine okay okay so you can have some fun with it right

7:32okay so let’s recap where we are right now using commodor we explored the broken service we identified two bugs

7:38one is that my awesome update in git which you were able to visualize and see right away uh potentially broke the PVC

7:45claim name which we’re going to come back to I would assume I also highlighted that the cluster couldn’t

7:52schedule or pod and you went to the node dashboard and identified that the node

7:57was cordoned and you were able to un coordinate directly from commodor moving us past the scheduling problem however

8:05we now have the node being not ready because of a potential issue with the cni networking

8:12plug-in yeah yeah like we can see that there are like I don’t know

8:18like four different plugins that are installed CSI plugins that are install

8:24and C we looking for cni not CS sorry sorry sorry

8:29maybe should I describe maybe the node what sorry

8:35describe it looks like that we have the celium operator installed yeah uh in in this cluster yeah it might

8:44be with the operator yeah there is maybe the crds ofum like there is the operator maybe

8:52the helm yeah it’s like using hel oh this fail

8:59fail deploy in here fail deploy yeah yeah so we can see there is Agent true

9:06agent not ready as well minimum replica

9:11unavailable yeah but it’s just just the operator itself

9:19on the deploy let take a

9:24look deployment version one

9:30there is a spec Affinity of like label match label ium

9:36operator it’s the P template that is UN unmatch the deployment do you think like

9:42the relevant part in here maybe oh it’s it’s funny it’s running

9:48and ready but it’s like the node is not ready it’s

9:55always fun watching people f a broken cluster

10:02no maybe like look at the hel dashb no like in the hel dashboard we can see

10:08like the current isum like we can see quite a lot on this like what annotation does this the

10:15cler not no I think the not found yeah I just found it

10:24exactly maybe let’s check if there is there is the clust the wall and cl

10:30wall binding in the cluster do do you mean like those like resources which are

10:36not exist M I think we need to create something I’m not sure maybe let’s check

10:41the log of the which is running no I think it’s like one The annotation right like he doesn’t find The annotation on

10:47the Node this why doesn’t inst it on it’s running on the

10:54no so this may be a little bit harder to debug because I think I found a bug and

10:59commodor but try comparing the values from the release three to release

11:04two okay obum yeah you it

11:13okay so there are changes but they don’t actually show up here yeah maybe met

11:20[Music] changes we have only the three version

11:27we don’t have the second no we do have it we do of the operator

11:34ah in the hand does only show changes doesn’t show anything no do two

11:41and then compare with rision two it’s two

11:47compared with division three

11:53no yeah I don’t know why it’s not showing the change for me show the change great then

12:00manifest then compare with version two here when you do here’s the changes you

12:05deleted the service account and all of those do guy I will

12:14do need to do the don’t have permission to I just

12:19perform but well maybe it’s a permission thing

12:25yeah I think the Watcher doesn’t have a permission maybe for that m possible yeah let’s see if also here it

12:32doesn’t have secret FS let’s do also W back to the we can’t

12:39we can’t we can’t our own agent we need the access to the to the class yeah so

12:47we will use it so do all to our agent and then we’ll do to to the seni okay

12:53soorry we sh my screen I’ll stop here yes so we found out that we are missing permission inside Commodore and

13:00it was installed without the possibility of like a roll

13:15back

13:27okay that’s

13:33it I to

13:39just okay you okay yeah that’s that’s

13:44it

13:55okay cool cool cool okay now let’s go back and check if the not is ready now

14:02yeah he is is ready okay and now let’s

14:07check out our so before we continue the upgrad to the Commodore I did in

14:13commodor because it turned on dashboard but I see that it moved the secret access which is probably why the values

14:19didn’t show yeah reason okay I just wanted to make sure I understood what

14:25happened there okay cool and so now the node is ready let’s go back to Services

14:32only thing remaining is the verion for the okay so we have a working node and

14:39you fix the deploy nice work what we yeah now we need to roll it back so what

14:44we we can’t it back because what so we need to edit like let’s edit it yeah I

14:49think that I need to show because remember this is a get UPS P Lanes so you might want me just to push a fix if

14:54you can tell me what you want that fix to be so be reverse yeah let’s just Che the

15:02latest oh I don’t know how to fix it I mean I I just did aw some updates you don’t need to tell me how to fix

15:09it so please your bed Cod yeah so yeah

15:18let’s just get check out to the like this revision if nothing else change in

15:24between then that’s probably the easiest solution you check out with the ref for the

15:30change are you doing the I have pushed an update to

15:36get I’m sharing I’m sharing I’m sharing do you have like a pipeline that

15:42know how to like it automatically deployed yes flux CD is running in the cluster it will detect this change and

15:48it will push it out we can speed up the process and I will do so now just so it’s a bit quicker

15:53yeah so what we can see is that see in near there like the the PVC

16:01change and we got some Environ variables which can be missing and what David

16:07changeed yeah it’s only the P so maybe maybe we still miss those yeah so let’s

16:15maybe start from the let’s wait for the roller to happen

16:21yeah we should see it in commod once the happen you can take look on the walk SPS

16:27to see this still yeah yeah but it’s the previous one yeah it wasn’t

16:34any so we looking for the new

16:41one what so I that push the update however our get offs pipeline is broken due to

16:47the fourth break in the cluster so good luck so there’s another break maybe I’ll

16:55go right like a let yeah let's check Aro is there Aro

17:02flux sorry Source control notification all of them look

17:09healthy what do we check sorry

17:14yeah but maybe it’s misconfigured or something like that seems like thex is

17:20working fine let’s check maybe logs of one of the workflows the controller or some other service The Source controller

17:27like the log message look

17:33good maybe is it updated by Source controller

17:40or I think there is still problem with the like one of the parts are unhealthy

17:46in the sour control yeah the C operator is

17:51pending scheduled because it didn’t match part Infinity Wes if you go to the

17:58walk on the

18:03white click on the the operator okay it’s just because when you did the roll back I set the replicas to one because

18:11we were a single node cluster so you can ignore that pending pod no take the first no he saying like

18:19it’s it’s not theity no no it’s like you said like in the logs of the source control yeah yeah it was

18:26there lo there’s like message s artifa foric let me go back and then garbage

18:33collected one artifact why did Garbage collected it and then a lot of changes but why did the garbage

18:39collected one artifact maybe it’s related to that I don’t know

18:46yeah Chang like this is the change this is

18:52what you mean right again yeah and then like one

18:57afterward remove typo in PVC name yeah this is the

19:05commit like d [Music]

19:10question yeah but what does it mean let’s see if we got any warnings in

19:15here or you can do like maybe

19:24like so what happen is one point in the it it find out that there there was a

19:30change M but for some reason the garbage

19:36collected it we need to change something in FL

19:44yeah let’s check the configuration maybe it’s something about this configuration

19:50CH yeah this by way in the customized controller it always failed the FL CD

19:57name is changed from system to and what is the name in The

20:07Log saw that yeah yeah okay so your rollback for cium actually fixed this

20:13problem but there’s a 10 minute sync time on the customization so I’ve just encouraged it to run again

20:21so so we don’t need to do anything as long as this customization runs no it’s

20:27still failing it’s is in networking and cluster is not working yeah I don’t know if your RB back for celum fixed the

20:34problem I think the RO of C didn’t no like there if you look at the logs of the customized controller there are

20:40really bad logs there and it says that it failed on like HTTP faed call in web let me just show

20:50that everyone can see yeah you consideration fail after second has the

20:55cnpg service who is it name

21:07rout the cpg thing is I think the

21:20network what is this service the cpg yeah there is like one thing here I’m

21:27looking at logs of the cpg is it a p it’s there is a pod but

21:35like the latest message is like periodic PLS certificate

21:41maintenance which I don’t really

21:51know e on this series what was the in it doesn’t likeed like with the

21:58relevant service basically yeah so let me give you context on that selum break right because you did a rule back but

22:04you didn’t really identify what the problem was and what changed and uh I don’t want is to debug

22:11something that you can’t have visibility into right now because of that secret values thing so in theum health chart

22:18what I did was disable the agent which is definitely rolled back because we can see the agent is now deployed next to the operator however I also disabled the

22:26ebpf cube proxy replacement and you may notice there’s no Cube proxy in this cluster so in the interest of not

22:34debugging something that we’re not entirely sure if it’s been fixed or not I’m going to redeploy celium right now and assume the r back hopefully fixed it

22:41properly and if we still have an issue then I’m debugging with you because I’m not really sure what the problem will be

22:46after that let let’s no maybe

22:55worse it’s not that okay so my my update for celium has

23:02triggered a redeploy of celium so the config map definitely changed so we may

23:08be moving in to a better

23:19situation yeah maybe delete the latest cium operator oh who can delete

23:25things delete the celium operator

23:32okay the previous one yeah

23:38the the operator wait a sec the one that is

23:43pending no not the one that is the other one what will happen yeah so go to the C operator to

23:51the are you sure yeah I’m going to delete the

23:56oldum oh that’s a bold move I like it yeah

24:02yeah we’re not playing around here you know so now the the new version is

24:09running and should or we won’t have anyone there

24:14rning right it seem like it’s face SCH hey that

24:20worked did it work yeah yeah well we had no doubts

24:26about

24:32seems like the new of theum doesn’t I think that’s okay because he

24:37has like two replica but now like it’s a new one that is running great so

24:44now I can scare it to one

24:49keep issues you know I’m scaling the the c one no no I

24:56think the no I think now it’s okay now let read the the logs of the flux thingy

25:01there the customized one I think right let’s The Source I think the C oh the Dr

25:08is just sinking it is yeah you see it’sing wall yeah let see

25:15it when a doubt delete stum operators fixes

25:21everything oh now it’s healthy look on the Yeahs and I do like

25:27a say okay let me share my screen and I’ll

25:33test the website for you right moment

25:38yeah and you understand like all you get is like druple working that’s

25:44like that’s like the best scenario Drupal is running we have a

25:49problem with our database configuration but maybe we don’t need it so an interest of testing we can go port

25:56forward not

26:03do also

26:09have okay so it’s almost working let’s see if we can actually open it in a

26:22Brer don’t be too happy

26:30you try to save it now he’s going to try to use the

26:35database so this shouldn’t actually be needed but the net script is unable to run for the same reason that this

26:42command will fail oh no our duple instance is unable

26:49to communicate with the postgress database back over to you and this is the last

26:54break because maybe the enir right it’s going to time out it cannot post

27:00dle cannot speak to post G there we go temp failure DNS

27:05resolution yeah back back to you last break there we go so it cannot resolve

27:12the database and okay so let’s check the

27:19events of the

27:24everything elction Network policy maybe you did a lot of network policy

27:31changes indeed why did you do it the event you can see policy changes

27:42and less Network policy [Music] change

27:48was I

27:54scraping so we saw that there are a lot of network policy changes and it look like someone changed

28:01the Untitled policy yeah there was a policy that prevent us for executing

28:09request the cluster there is a policy type of igress so let’s try to take on

28:15action and I mean what I love about comar here right is this the vent log as a gold mine of information and you can see this

28:22network policy was created in the last 24 hours it’s obviously well intended but you know mistakes are easy to make

28:28in kubernetes very easy

28:34then all right if you can stop sharing your screen I will give the application another spin I think we should be

28:40sitting pretty now Cas I still have my portf running if we remove the install script

28:48yeah we’re holding the view and if we make

28:56sure okay it completed 16 seconds ago the

29:03database is now running oh I shouldn’t have to do this

29:10but we run through it anyway that’s

29:16it woo well done you fix all the brakes on the cluster and duple is now working

29:23as [Music] intended

29:29so you know a small recap and then I’ll e get back up day right but that’s was a whole lot of fun for me right um I

29:36actually found it really difficult to break the developer the consumer API of

29:42kubernetes in a way that commodor couldn’t show right up front what the problem was with the GE integration the

29:49diffs the helm charts the node information even revealing all the labels and annotations everything was

29:55just there in front of me and I think that’s just superow for people that have to operate kubernetes so I’ll thank you

30:00all for your work it made it harder to break but I hope you enjoyed each of the breaks that were presented to you and uh

30:06yeah any final remarks from anyone no it was super

30:13fun