Hold on!

Before you go, why not take Komodor for a spin? Simplify Kubernetes troubleshooting in 5 minutes.

Try Komodor for Free *No credit card required.
Komodor-platform
This website uses cookies. By continuing to browse, you agree to our Privacy Policy.

How to fix ErrImagePull and ImagePullBackoff

4754 Views

What is ErrImagePull or ImagePullBackOff

Kubernetes pods sometimes experience issues when trying to pull container images from a container registry. If an error occurs, the pod goes into the ImagePullBackOff state.

When a Kubernetes cluster creates a new deployment, or updates an existing deployment, it typically needs to pull an image. This is done by the kubelet process on each worker node. For the kubelet to successfully pull the images, they need to be accessible from all nodes in the cluster that match the scheduling request.

The ImagePullBackOff error occurs when the image path is incorrect, the network fails, or the kubelet does not succeed in authenticating with the container registry. Kubernetes initially throws the ErrImagePull error, and then after retrying a few times, “pulls back” and schedules another download attempt. For each unsuccessful attempt, the delay increases exponentially, up to a maximum of 5 minutes.

To identify the ImagePullBackOff error: run the kubectl get pods command. The pod status will show the error like so:

NAME        READY    STATUS              RESTARTS    AGE
my-pod-1    0/1      ImagePullBackOff    0           2m34s

We’ll provide best practices for diagnosing and resolving simple cases of these errors, but more complex cases will require advanced diagnosis and troubleshooting, which is beyond the scope of this article.

ImagePullBackOff and ErrImagePull Errors: Common Causes

Cause Resolution
Pod specification provides the wrong repository name Edit pod specification and provide the correct registry
Pod specification provides the wrong image name, or an unqualified image name Edit pod specification and provide the correct image name
Pod specification provides an invalid tag, or fails to provide a tag Edit pod specification and provide the correct tag. If the image does not have a latest tag, you must provide a valid tag
Container registry is not accessible Restore network connection and allow the pod to retry pulling the image
The pod does not have the appropriate credentials to access the image Add a Secret with the appropriate credentials and reference it in the pod specification

Take Komodor for a Spin!

Get the context you need to troubleshoot efficiently and independently

ImagePullBackOff and ErrImagePull Errors: Diagnosis and Resolution

As mentioned, an ImagePullBackOff is the result of repeat ErrImagePull errors, meaning the kubelet tried to pull a container image several times and failed. This indicates a persistent problem that needs to be addressed.

Step 1: Gather information

Run kubectl describe pod [name] and save the content to a text file for future reference:

kubectl describe pod [name] /tmp/troubleshooting_describe_pod.txt

Step 2: Examine Events section in describe output

Check the Events section of the describe pod text file, and look for one of the following messages:

  • Repository ... does not exist or no pull access
  • Manifest ... not found
  • authorization failed

The image below shows examples of how each of these messages appears in the Events output.

authorization-failed-imagepullbackoff

Step 3: Troubleshoot

If the error is Repository ... does not exist or no pull access:

  • This means that the repository specified in the pod does not exist in the Docker registry the cluster is using
  • By default, images are pulled from Docker Hub, but your cluster may be using one or more private registries
  • The error may occur because the pod does not specify the correct repository name, or does not specify the correct fully qualified image name (e.g. username/imagename)
  • To resolve it, double check the pod specification and ensure that the repository and image are specified correctly.
  • If this still doesn’t work, there may be a network issue preventing access to the container registry. Look in the describe pod text file to obtain the hostname of the Kubernetes node. Log into the node and try to download the image manually.

If the error is Manifest ... not found:

  • This means that the specific version of the requested image was not found.
  • If you specified a tag, this means the tag was incorrect. Double-check that the tag in the pod specification is correct, and exists in the repository. Keep in mind that tags in the repo may have changed.
  • If you did not specify a tag, check if the image has a latest tag. Images that do not have a latest tag will not be returned, if you do not specify a valid tag. In this case, you must specify a valid tag.

If the error is authorization failed:

  • The issue is that the container registry, or the specific image you requested, cannot be accessed using the credentials you provided.
  • To resolve this, create a Secret with the appropriate credentials, and reference the Secret in the pod specification.
  • If you already have a Secret with credentials, ensure those credentials have permission to access the required image, or grant access in the container repository.

As above, please note that this procedure will only resolve the most common causes of ImagePullBackOff. If one of the quick fixes above did not work, you’ll need to undertake a more complex, non-linear diagnosis procedure to identify which parts of the Kubernetes environment contribute to the problem and resolve them.

Also, in many cases, after this error is resolved, when you run kubectl get pods again, you’ll discover another issue preventing the pod from functioning properly. If so, you’ll need to troubleshoot the new issue.

Solving Kubernetes Errors Once and for All With Komodor

The troubleshooting process in Kubernetes is complex and, without the right tools, can be stressful, ineffective and time-consuming. Some best practices can help minimize the chances of things breaking down, but eventually something will go wrong—simply because it can.

This is the reason why we created Komodor, a tool that helps dev and ops teams stop wasting their precious time looking for needles in (hay)stacks every time things go wrong.

Acting as a single source of truth (SSOT) for all of your k8s troubleshooting needs, Komodor offers:

  • Change intelligence: Every issue is a result of a change. Within seconds we can help you understand exactly who did what and when.
  • In-depth visibility: A complete activity timeline, showing all code and config changes, deployments, alerts, code diffs, pod logs and etc. All within one pane of glass with easy drill-down options.
  • Insights into service dependencies: An easy way to understand cross-service changes and visualize their ripple effects across your entire system.
  • Seamless notifications: Direct integration with your existing communication channels (e.g., Slack) so you’ll have all the information you need, when you need it.

If you are interested in checking out Komodor, use this link to sign up for a Free Trial.

Related Articles

Latest Blogs

Komodor Closes $42M Series B Led by Tiger Global

Komodor Closes $42M Series B Led by Tiger Global

Troubleshooting in Kubernetes: The Shift-Left Approach

Troubleshooting in Kubernetes: The Shift-Left Approach

In this blog post, we will discuss a new paradigm for making Kubernetes easier to troubleshoot: the shift-left approach....

ValidKube Update: Adding Polaris to Auto-Audit K8s YAMLs

ValidKube Update: Adding Polaris to Auto-Audit K8s YAMLs

We are expanding ValidKube’s capabilities with the inclusion of Polaris - a cool OS project by our good friends at Fairwinds!...