Buildkit failing arm builds with missing binaries even though they are in $PATH

[shizuka]

A few of our developers recently got some M1 machines, I want to provide them with ARM compatible images and have set my sights on Buildkit.

I set up a podTemplate in Jenkins and run the build with this command

buildctl build --opt platform=linux/arm64 --frontend=dockerfile.v0 --local context=. --opt build-arg:HTTP_PROXY=http://proxy-here:8080 --opt build-arg:HTTPS_PROXY=http://proxy-here:8080 --local dockerfile=. --output type=image,push=true,\\"name=\$BUILD_TAG,\$SHORT_TAG,\$MINOR_TAG,\$FULL_TAG\\"

When running for linux/amd64 everything goes as planned, however linux/arm64 will fail with a variety of errors depending on the Dockerfile, for instance:

RUN update-ca-certificates:
# Error while loading /usr/bin/run-parts: No such file or directory

I have checked that run-parts exits and that it is linked to to musl with the right architecture.

RUN which run-parts
# /bin/run-parts
RUN ldd which run-parts
/lib/ld-musl-aarch64.so.1 (0x5500000000)
libc.musl-aarch64.so.1 => /lib/ld-musl-aarch64.so.1 (0x5500000000)
RUN file which run-parts
/bin/run-parts: symbolic link to /bin/busybox
RUN file which busybox
/bin/busybox: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-aarch64.so.1, stripped
RUN ldd which busybox
/lib/ld-musl-aarch64.so.1 (0x5500000000)
libc.musl-aarch64.so.1 => /lib/ld-musl-aarch64.so.1 (0x5500000000)
RUN file which update-ca-certificates
/usr/sbin/update-ca-certificates: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-aarch64.so.1, stripped

The error seems to indicate that it is not running with the correct $PATH, but then again, which seems to have no issue finding the commands, /lib/ld-musl-aarch64.so.1 also definitely exitsts.

The error will happen to all sorts of otherwise pedestrian commands ls, addgroup (but not deluser).

I have tried linking files from /bin/ into /usr/bin/ but that quickly becomes a game of whack a mole and seems too much of a hack for me.

The base image for the container is php:7.4.28-fpm-alpine

My Google-fu completely fails me here, what am I doing wrong?

[elysha]

In my case the issue was that QEMU wasn't installed and thus unavailable for emulation, apparently buildkit can get through a large amount of steps before needing emulation.

I added an init container to the pod template which installs QEMU emulators as required:

spec:
    initContainers:
    - name: prepare
      image: tonistiigi/binfmt:latest
      args: ["--install", "arm64"]
      securityContext:
          privileged: true