What is the GitLab "Digest" on the container registry next to the images, why is different for two duplicated images

Question

The GitLab container registry has a bunch of Digests, you can see them here,

Normally, I would expect them to represent the image's own digest, however, they don't, when I look at them here locally after pulling these images down I see the following with podman images

REPOSITORY    TAG               IMAGE ID      CREATED        SIZE
myImage       latest            f07e84515821  3 months ago   1.91 GB
myImage       pre-kaniko        f07e84515821  3 months ago   1.91 GB

Shocked, I re-ran podman pull on both of these images, and nothing changed, locally they have the same digest after I run podman pull..

What I would expect is both of the Digests in the Container registry to match my own image which is "f07e84515821" however, in GitLab both of the digests are different. Even worse, the sizes are different. So I'm confused. What is a GitLab Digest?

Answer 1

If podman is doing the same thing as docker, the image ID is the config digest from the image. The registry is showing the manifest digest. There are multiple digests for an image, the registry is a content addressable store and images are effectively a Merkle tree.

The root node for that Merkle tree is the manifest. There are two types, a manifest list (for multi-platform images) and an image manifest (for a single platform). The manifest list, as its name suggests, contains a list of pointers to manifest (by digest). And the image manifest contains pointers to the config blob and an array of pointers to layers. Putting that all together with a docker example, you have:

$ docker pull nginx
Using default tag: latest
latest: Pulling from library/nginx
f1f26f570256: Pull complete
7f7f30930c6b: Pull complete
2836b727df80: Pull complete
e1eeb0f1c06b: Pull complete
86b2457cc2b0: Pull complete
9862f2ee2e8c: Pull complete
Digest: sha256:2ab30d6ac53580a6db8b657abf0f68d75360ff5cc1670a85acb5bd85ba1b19c0
Status: Downloaded newer image for nginx:latest
docker.io/library/nginx:latest
$ regctl manifest get nginx --format body | sha256sum
2ab30d6ac53580a6db8b657abf0f68d75360ff5cc1670a85acb5bd85ba1b19c0  -
$ regctl manifest get nginx --format '{{jsonPretty .}}'
{
  "manifests": [
    {
      "digest": "sha256:bfb112db4075460ec042ce13e0b9c3ebd982f93ae0be155496d050bb70006750",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      },
      "size": 1570
    },
    {
      "digest": "sha256:227794033aebbcf39a8d11a99a2b42e945a3c00cde978804082f9d05153e8192",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v5"
      },
      "size": 1570
    },
    {
      "digest": "sha256:43a1d47f229b9b9f20f0e414ccce8bbe8f3d8e2c11e3e222ef81d7b0fe7f3eb7",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v7"
      },
      "size": 1570
    },
    {
      "digest": "sha256:3be40d1de9db30fdd9004193c2b3af9d31e4a09f43b88f52f1f67860f7db4cb2",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm64",
        "os": "linux",
        "variant": "v8"
      },
      "size": 1570
    },
    {
      "digest": "sha256:77df563c3da838275fc3253099a272aa385f1797ab458661c35ad325be4fecb0",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "386",
        "os": "linux"
      },
      "size": 1570
    },
    {
      "digest": "sha256:917c49246f0ec7a1ee82fb7bcf3e9cbe199d851b33cd98f97a19c951b05b8051",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "mips64le",
        "os": "linux"
      },
      "size": 1570
    },
    {
      "digest": "sha256:1ff30bac28af0a75228c65450a41045323a1f21ffd1de5f21733a403f08ca183",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "ppc64le",
        "os": "linux"
      },
      "size": 1570
    },
    {
      "digest": "sha256:30350eec6d1283dda69de3addac1a546a1b7435cd7519bc509b375679b744e12",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "s390x",
        "os": "linux"
      },
      "size": 1570
    }
  ],
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "schemaVersion": 2
}
$ regctl manifest get nginx@sha256:bfb112db4075460ec042ce13e0b9c3ebd982f93ae0be155496d050bb70006750 --format '{{jsonPretty .}}'
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 7916,
    "digest": "sha256:080ed0ed8312deca92e9a769b518cdfa20f5278359bd156f3469dd8fa532db6b"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 31411405,
      "digest": "sha256:f1f26f5702560b7e591bef5c4d840f76a232bf13fd5aefc4e22077a1ae4440c7"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 25583851,
      "digest": "sha256:7f7f30930c6b1fa9e421ba5d234c3030a838740a22a42899d3df5f87e00ea94f"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 626,
      "digest": "sha256:2836b727df80c28853d6c505a2c3a5959316e48b1cff42d98e70cb905b166c82"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 956,
      "digest": "sha256:e1eeb0f1c06b25695a5b9df587edf4bf12a5af9432696811dd8d5fcfd01d7949"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 772,
      "digest": "sha256:86b2457cc2b0d68200061e3420623c010de5e6fb184e18328a46ef22dbba490a"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 1404,
      "digest": "sha256:9862f2ee2e8cd9dab487d7dc2152a3f76cb503772dfb8e830973264340d6233e"
    }
  ]
}
$ docker image inspect nginx --format '{{.ID}}'
sha256:080ed0ed8312deca92e9a769b518cdfa20f5278359bd156f3469dd8fa532db6b

For more details on the structure of these various objects that make up the image, see the OCI image-spec.

Answer 2

When I expand both of these I get a bit more information, the "Manifest Digests" are different.

Not exactly sure why they would both have different manifests.