Issue with pulling ManageIQ images from Docker Hub

Hello,

I want to deploy ManageIQ on Kubernetes, I am unable to pull the latest docker images from Docker Hub.
I noticed a difference between some older and recent images.
Older images are “clickable” and have a defined size. Newer Images are not “clickable” for me. All “unclickable” images are affected.

It seems like the tag was created, but has no content, if that makes sense.


https://hub.docker.com/r/manageiq/manageiq-operator/tags?page=1&ordering=last_updated

All “blue” (href’ed) images can be pulled and all “black” (no href) images cannot. (see screenshot above)

Our current setup:

Kubernetes cluster → Jfrog Artifactory (Docker registry mirror) → Docker Hub
It is not possible for us to pull these directly from Docker Hub

Did anyone experience something similar? What are the differences between the “blue” and “black” images?

Best,
Paul

Good question. I’m not sure what’s different about the tags without links. I was able to docker pull and run them though. Which docker tags are you unable to access?

joerafaniello@Joes-MBP-2 manageiq % docker pull manageiq/manageiq-operator:latest
latest: Pulling from manageiq/manageiq-operator
ea7568f095c4: Pull complete
f00d7ab0b218: Pull complete
e9dc18d523da: Pull complete
081611e5b35f: Pull complete
be4cd5ed400f: Pull complete
32dd2f107706: Pull complete
7917baa30e82: Pull complete
Digest: sha256:3c6df3806c83cfc015503ba5e07da2469d88f612f6876781097f0bbad87698da
Status: Downloaded newer image for manageiq/manageiq-operator:latest
docker.io/manageiq/manageiq-operator:latest

joerafaniello@Joes-MBP-2 manageiq % docker pull manageiq/manageiq-operator:latest-lasker
latest-lasker: Pulling from manageiq/manageiq-operator
ea7568f095c4: Already exists
f00d7ab0b218: Already exists
802098d51f6c: Pull complete
0d2cee4464b8: Pull complete
ebce2e873b7c: Pull complete
df811297f964: Pull complete
7729e0751e37: Pull complete
Digest: sha256:a63510997bfa0e08f363d3482dfa1364991c957e139119ade426bf9523262b52
Status: Downloaded newer image for manageiq/manageiq-operator:latest-lasker
docker.io/manageiq/manageiq-operator:latest-lasker

joerafaniello@Joes-MBP-2 manageiq % docker pull manageiq/manageiq-operator:lasker-1
lasker-1: Pulling from manageiq/manageiq-operator
7782f100031b: Pull complete
399e5d2c71d8: Pull complete
e4fec16804ff: Pull complete
60a667c70530: Pull complete
c2947807bbab: Pull complete
79028b2038e7: Pull complete
095d8122d390: Pull complete
Digest: sha256:4c7ea40ab1fcd503a8a5ff289af76aa7c3636212222dd6ea94a62ebcb00de957
Status: Downloaded newer image for manageiq/manageiq-operator:lasker-1
docker.io/manageiq/manageiq-operator:lasker-1

@bdunne can correct me if I’m wrong, but in the past, we used to use docker hub itself to do our automated builds but more recently builds we build ourselves and push to docker hub. This could possibly explain that difference, but I’m not sure exactly when we started doing that.

That being said, even though they are not clickable online, you should be able to docker pull them as @jrafanie has done.

Note that in some distributions of Kubernetes you may have to configure docker.io as an allowed registry before it allows you to pull from there.

@jrafanie the problematic tags are lasker-1 or latest-lasker for example (all images marked black in the screenshot). Working tags are for example kasparov-1 (all images marked blue in the screenshot).
@Fryguy I was also able to pull the images with docker, but not with crictl (no docker installed, only containerd). docker.io is allowed as registry.

root@kubemaster01:~# crictl -D pull manageiq/manageiq-operator:kasparov-1
DEBU[0000] get image connection
DEBU[0000] connect using endpoint 'unix:///run/containerd/containerd.sock' with '10s' timeout
DEBU[0000] connected successfully using endpoint: unix:///run/containerd/containerd.sock
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:manageiq/manageiq-operator:kasparov-1,Annotations:map[string]string{},},Auth:nil,SandboxConfig:nil,}
DEBU[0001] PullImageResponse: &PullImageResponse{ImageRef:sha256:a2b5201b4d8e10299a29f3034b3099aeb97b3104705a239668710c1e515c5f82,}
Image is up to date for sha256:a2b5201b4d8e10299a29f3034b3099aeb97b3104705a239668710c1e515c5f82


root@kubemaster01:~# crictl -D pull manageiq/manageiq-operator:latest-lasker
DEBU[0000] get image connection
DEBU[0000] connect using endpoint 'unix:///run/containerd/containerd.sock' with '10s' timeout
DEBU[0000] connected successfully using endpoint: unix:///run/containerd/containerd.sock
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:manageiq/manageiq-operator:latest-lasker,Annotations:map[string]string{},},Auth:nil,SandboxConfig:nil,}
DEBU[0010] PullImageResponse: nil
FATA[0010] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/manageiq/manageiq-operator:latest-lasker": failed to extract layer sha256:785573c4b94554cc3ba023f2648ba476bc5c33a05344c341f78bb384fb4b9d58: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount295522308: archive/tar: invalid tar header: unknown

root@kubemaster01:~# crictl -D pull manageiq/manageiq-operator:lasker-1
DEBU[0000] get image connection
DEBU[0000] connect using endpoint 'unix:///run/containerd/containerd.sock' with '10s' timeout
DEBU[0000] connected successfully using endpoint: unix:///run/containerd/containerd.sock
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:manageiq/manageiq-operator:lasker-1,Annotations:map[string]string{},},Auth:nil,SandboxConfig:nil,}
DEBU[0013] PullImageResponse: nil
FATA[0013] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/manageiq/manageiq-operator:lasker-1": failed to extract layer sha256:bc7bdf0ec1b9b8dbc50dbd3fea337c2b251f1da258e1cffda61d157d3a86066f: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount836777242: archive/tar: invalid tar header: unknown

For testing purposes I built the images with the build script in the manageiq-pods repository and pushed them to my docker account. These images are clickable online and I can pull them from docker hub with docker and crictl.
Not sure what this script does differently compared to the way you do it.

This is starting to sound like a bug somewhere - can you open an issue on github.com/manageiq/manageiq-pods/issues, please?

We created following issue.