Exit Code 127: Fixing File or Directory Not Found in Kubernetes

What Is Exit Code 127? 

Exit Code 127 is a standard error message that originates from Unix or Linux-based systems, and is commonly seen in Kubernetes environments. This exit code is a part of the system’s way of communicating that a particular command it tried to execute could not be found. It’s essentially the system’s way of saying, “I tried to run this, but I couldn’t find what you were asking me to run.”

Exit codes are a typical way for computer systems to convey the status of a process or command, with zero generally representing success and any non-zero value indicating some sort of error or issue. Exit Code 127, specifically, is part of a range of codes (126 to 165) used to signal specific runtime errors in Linux/Unix-based systems.

Common Causes of Exit Code 127 in Kubernetes 

While the generic explanation for Exit Code 127 is that a command could not be found, in a complex environment like Kubernetes, there could be several underlying reasons for this issue:

Incorrect Entrypoint or Command in Dockerfile or Pod Specification

One of the most common reasons for Exit Code 127 in Kubernetes is an incorrect entrypoint or command in the Dockerfile or pod specification. This error occurs when you specify a command or entrypoint that doesn’t exist or is not executable within the container.

For instance, you may have specified a shell script as your entrypoint, but the script doesn’t exist at the specified location within the container. Alternatively, the script might exist but is not marked as executable. Both these scenarios would lead to Exit Code 127.

Missing Dependencies in the Container

Another frequent cause of Exit Code 127 is missing dependencies within the container. This issue occurs when your application or script depends on certain libraries or software that are not available in the container.

For example, your application might be written in Python and require certain Python packages to run. If these packages are not installed in the container, the Python interpreter would be unable to find them, leading to Exit Code 127.

Wrong Image Tag or Corrupt Image

It’s also possible to encounter Exit Code 127 if you’re using a wrong image tag or a corrupt image. Kubernetes uses Docker images to create containers for your pods. If the image specified in the pod specification is incorrect or corrupt, Kubernetes would be unable to create the container and execute the commands, resulting in Exit Code 127.

Issues with Volume Mounts

Lastly, issues with volume mounts can also lead to Exit Code 127. In Kubernetes, you can mount volumes to containers in your pods to provide persistent storage or to share data between containers. If there’s a problem with these mounts, such as a wrong mount path or permissions issue, it could prevent your application from accessing required files or directories, leading to Exit Code 127.

Tips from the expert

Itiel Shwartz

Co-Founder & CTO

Itiel is the CTO and co-founder of Komodor. He’s a big believer in dev empowerment and moving fast, has worked at eBay, Forter and Rookout (as the founding engineer). Itiel is a backend and infra developer turned “DevOps”, an avid public speaker that loves talking about things such as cloud infrastructure, Kubernetes, Python, observability, and R&D culture.

In my experience, here are tips that can help you better handle Exit Code 127 in Kubernetes:

Verify Command and Entrypoint in Dockerfile

Ensure the command or entrypoint specified in your Dockerfile exists and is executable. Use absolute paths and verify the presence of necessary binaries within the image.

Test Locally Before Deployment

Run your Docker image locally using docker run to ensure all dependencies are correctly installed and entrypoints work as expected. This catches issues early before deploying to Kubernetes.

Check Image Tags and Versions

Verify that the correct image tags and versions are used in your Kubernetes manifests. Avoid using latest tag in production to ensure consistency and predictability.

Ensure Required Dependencies Are Installed

Confirm that all necessary libraries, binaries, and scripts are included in the Docker image. Use multi-stage builds to ensure dependencies are correctly added and minimize image size.

Use Init Containers for Setup Tasks

Employ init containers to perform setup tasks such as installing dependencies or creating directories. This ensures the main application container runs in a prepared environment.

How to Diagnose Exit Code 127 in Kubernetes 

Now that we’ve explored the common causes of Exit Code 127, let’s look at how to diagnose this error when it occurs.

Check the Kubernetes Pod Logs

One of the first steps in diagnosing Exit Code 127 is checking the logs for the affected pod. This can be done using the kubectl logs command, which displays the logs for a specific pod. These logs often contain valuable information about what went wrong, including error messages from your application or script.

Inspect the Pod Description

Another useful step in diagnosing Exit Code 127 is inspecting the description of the affected pod. This can be done using the kubectl describe pod command, which provides detailed information about the pod, including its current state, recent events, and any error messages.

Verify the Dockerfile and Build Process

It’s crucial to verify your Dockerfile and build process when diagnosing Exit Code 127. This involves checking your Dockerfile for any mistakes or omissions, and ensuring that your build process correctly builds the Docker image and pushes it to the image registry.

Check the Pod’s Configuration

It is important to examine the pod’s configuration. This involves a thorough check on the pod’s specifications, such as the command line arguments and environment variables, to ensure they are correctly defined. The configuration could be wrongly set, thus leading to Exit Code 127. A detailed examination of the pod’s logs could shed more light on why the application inside the pod is not executing as expected.

Test the Docker Image Locally

The next step is to test the Docker image locally. This is a crucial step as it helps you verify if the problem lies in the image itself. By running the image on your local machine, you can determine whether the application starts as expected. If it doesn’t, then there’s a good chance that the image is the problem.

Check Volumes and ConfigMaps

Another essential check involves examining volumes and ConfigMaps. These are critical components of your container runtime environment and can cause Exit Code 127 if not correctly configured. This check involves ensuring that the volumes are correctly mounted and that the ConfigMaps are properly defined and accessible.

Related content: Read our guide to exit code 1 and exit code 137

How to Fix Exit Code 127 

Once you’ve diagnosed the root cause of Exit Code 127, the next step is to fix it. This section details practical steps on how to address this error.

Correct the Entrypoint or Command in Dockerfile or Pod Specification

The first fix involves correcting the entrypoint or command in your Dockerfile or pod specification. This step involves ensuring that the application’s binary path is correctly specified and that the commands are executable.

For example, if you are using a Dockerfile and your ENTRYPOINT instruction looks like this:

ENTRYPOINT ["./app"]

And if ./app isn’t a valid command within your container, you will have to correct it to the right path. It might be that your application binary is in /app/bin directory and the corrected ENTRYPOINT would look like:

ENTRYPOINT ["/app/bin/app"]

Add Missing Dependencies in the Container

Exit Code 127 could also be a result of missing dependencies in the container. If this is the case, the solution is to add the missing dependencies in the container image or pod specification. 

Suppose your Python application requires the requests library, you would ensure it is installed during image build by adding this in your Dockerfile:

RUN pip install requests

Correct the Docker Image Tag or Build a New Image

Another solution could be correcting the Docker image tag or building a new image. If the image is the problem, correcting the image tag could resolve the issue. Alternatively, building a new image often helps, especially if the initial image was faulty.

To correct the Docker image tag in the Kubernetes pod specification, you might update from:

spec:
  containers:
  - name: my-app
    image: my-app:v0.1

To:

spec:
  containers:
  - name: my-app
    image: my-app:v0.2

To build a new Docker image, you can use this command.

docker build -t my-app:v0.3 .

Resolve Volume Mount Issues

If the problem lies in the volumes, resolving mount issues could fix Exit Code 127. This involves ensuring that the volumes are correctly mounted and that they are accessible by the container. This could involve changing the mount path or adjusting the permissions.

For example, you could change the volume mount path from:

volumes:
- name: app-volume
  hostPath:
    path: /data/my-app
containers:
- name: my-app
  image: my-app:v0.2
  volumeMounts:
  - mountPath: /app
    name: app-volume

To:

volumes:
- name: app-volume
  hostPath:
    path: /data/my-app
containers:
- name: my-app
  image: my-app:v0.2
  volumeMounts:
  - mountPath: /data
    name: app-volume

Use an Init Container

Lastly, using an init container could also resolve Exit Code 127. Init containers are specialized containers that run before application containers and can be used to set up the environment for your application container. This could involve installing necessary software, setting up configuration files, or doing anything else that’s necessary to prepare the environment for your application.

Here’s an example of an init container that creates a necessary directory:

spec:
  initContainers:
  - name: init-myservice
    image: busybox
    command: ['sh', '-c', 'mkdir -p /app/data']
  containers:
  - name: my-app
    image: my-app:v0.2

Note: initContainers was introduced in Kubernetes version 1.8. If you are running an earlier version, you might receive a strict decoding error. To resolve this, update your Kubernetes version.

Kubernetes Troubleshooting With Komodor

Kubernetes is a complex system, and often, something will go wrong, simply because it can. In situations like this, you’ll likely begin the troubleshooting process by reverting to some of the above kubectl commands to try and determine the root cause. This process, however, can often run out of hand and turn into a stressful, ineffective, and time-consuming task.

This is the reason why we created Komodor, a tool that helps dev and ops teams stop wasting their precious time looking for needles in (hay)stacks every time things go wrong.

Acting as a single source of truth (SSOT) for all of your k8s troubleshooting needs, Komodor offers:

• Change intelligence: Every issue is a result of a change. Within seconds we can help you understand exactly who did what and when.
• In-depth visibility: A complete activity timeline, showing all code and config changes, deployments, alerts, code diffs, pod logs and etc. All within one pane of glass with easy drill-down options.
• Insights into service dependencies: An easy way to understand cross-service changes and visualize their ripple effects across your entire system.
• Seamless notifications: Direct integration with your existing communication channels (e.g., Slack) so you’ll have all the information you need, when you need it.

If you are interested in checking out Komodor, use this link to sign up for a Free Trial.