Can not authenticate to DockerHub docker.io with ctr; works fine with crictl

[emerson]

Authentication works on crictl, when I use --creds

$ sudo crictl pull --creds "evancarroll:$TOKEN" docker.io/alpine:3

But when I try the same command ctr, I get an error:

$ sudo ctr images pull --user "evancarroll:$TOKEN" docker.io/alpine:3
docker.io/alpine:3: resolving
INFO[0000] trying next host error="pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed" host=registry-1.docker.io
ctr: failed to resolve reference "docker.io/alpine:3": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

---

You can see the --http-dump --http-trace,

$ sudo ctr images pull --http-dump --http-trace --user "evancarroll:$TOKEN" docker.io/alpine:3
INFO[0000] HEAD /v2/alpine/manifests/3 HTTP/1.1
INFO[0000] Host: registry-1.docker.io
INFO[0000] Accept: application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*
INFO[0000] User-Agent: containerd/v1.5.11-k3s2
INFO[0000]
docker.io/alpine:3: resolving      |--------------------------------------|
elapsed: 0.1 s      total:   0.0 B (0.0 B/s)
INFO[0000] HTTP/1.1 401 Unauthorized
INFO[0000] Content-Length: 149
INFO[0000] Content-Type: application/json
INFO[0000] Date: Wed, 15 Jun 2022 16:00:59 GMT
INFO[0000] Docker-Distribution-Api-Version: registry/2.0
INFO[0000] Strict-Transport-Security: max-age=31536000
INFO[0000] Www-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:alpine:pull"
INFO[0000]
INFO[0000] POST /token HTTP/1.1
INFO[0000] Host: auth.docker.io
INFO[0000] Content-Type: application/x-www-form-urlencoded; charset=utf-8
docker.io/alpine:3: resolving      |--------------------------------------|
elapsed: 0.2 s      total:   0.0 B (0.0 B/s)
INFO[0000] client_id=containerd-client&grant_type=password&password=HIDDEN&scope=repository%3Aalpine%3Apull&service=registry.docker.io&username=evancarrollHTTP/1.1 200 OK
INFO[0000] Transfer-Encoding: chunked
INFO[0000] Content-Type: application/json; charset=utf-8
INFO[0000] Date: Wed, 15 Jun 2022 16:00:59 GMT
INFO[0000] Strict-Transport-Security: max-age=31536000
INFO[0000] X-Trace-Id: 5da4dacdeb208ead79e15a59f83499b2
INFO[0000]
INFO[0000] 833
INFO[0000] {"access_token":"HIDDEN","scope":"","expires_in":300,"issued_at":"2022-06-15T16:00:59.464114622Z"}
INFO[0000]
INFO[0000] 0
INFO[0000]
INFO[0000] HEAD /v2/alpine/manifests/3 HTTP/1.1
INFO[0000] Host: registry-1.docker.io
INFO[0000] Accept: application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*
INFO[0000] Authorization: Bearer HIDDEN
docker.io/alpine:3: resolving      |--------------------------------------|
elapsed: 0.3 s      total:   0.0 B (0.0 B/s)
INFO[0000] HTTP/1.1 401 Unauthorized
INFO[0000] Content-Length: 149
INFO[0000] Content-Type: application/json
INFO[0000] Date: Wed, 15 Jun 2022 16:00:59 GMT
INFO[0000] Docker-Distribution-Api-Version: registry/2.0
INFO[0000] Strict-Transport-Security: max-age=31536000
INFO[0000] Www-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:alpine:pull",error="insufficient_scope"
INFO[0000]
INFO[0000] trying next host error="pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed" host=registry-1.docker.io
ctr: failed to resolve reference "docker.io/alpine:3": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

[esther]

docker.io/library/$IMAGE:$TAG

The problem here is simple the right syntax for ctr isn't docker.io/alpine:latest but docker.io/library/alpine:latest that should now work with --creds "evancarroll:$TOKEN" too.

ctr images pull docker.io/library/alpine:latest

IRC logs

Thanks goes out to larks and ada on IRC #docker for helping out with this,

larsks> EvanCarroll: you probably need to include the tag there (ctr images pull docker.io/library/alpine:latest). The /library is there because that's the actual path to the image repository. There's some magic somewhere that transforms docker.io/alpine into docker.io/library/alpine; I don't know if that's client side or server side; ada will know much more about that than I do.

And thanks to https://github.com/distribution/distribution/blob/main/reference/normalize.go#L106 , and clarifying

library/ is the namespace for all the top-level images on dockerhub with docker, that namespace is implied if you leave off the registry & namespace part of the tag : docker pull alpine == ctr images pull docker.io/library/alpine