Buildkit failing arm builds with missing binaries even though they are in $PATH

  • A few of our developers recently got some M1 machines, I want to provide them with ARM compatible images and have set my sights on Buildkit.

    I set up a podTemplate in Jenkins and run the build with this command

    buildctl build --opt platform=linux/arm64 --frontend=dockerfile.v0 --local context=. --opt build-arg:HTTP_PROXY=http://proxy-here:8080 --opt build-arg:HTTPS_PROXY=http://proxy-here:8080 --local dockerfile=. --output type=image,push=true,\\"name=\$BUILD_TAG,\$SHORT_TAG,\$MINOR_TAG,\$FULL_TAG\\"

    When running for linux/amd64 everything goes as planned, however linux/arm64 will fail with a variety of errors depending on the Dockerfile, for instance:

    RUN update-ca-certificates:
    # Error while loading /usr/bin/run-parts: No such file or directory

    I have checked that run-parts exits and that it is linked to to musl with the right architecture.

    RUN which run-parts
    # /bin/run-parts

    RUN ldd which run-parts

    /lib/ (0x5500000000) => /lib/ (0x5500000000)

    RUN file which run-parts

    /bin/run-parts: symbolic link to /bin/busybox

    RUN file which busybox

    /bin/busybox: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/, stripped

    RUN ldd which busybox

    /lib/ (0x5500000000) => /lib/ (0x5500000000)

    RUN file which update-ca-certificates

    /usr/sbin/update-ca-certificates: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/, stripped

    The error seems to indicate that it is not running with the correct $PATH, but then again, which seems to have no issue finding the commands, /lib/ also definitely exitsts.

    The error will happen to all sorts of otherwise pedestrian commands ls, addgroup (but not deluser).

    I have tried linking files from /bin/ into /usr/bin/ but that quickly becomes a game of whack a mole and seems too much of a hack for me.

    The base image for the container is php:7.4.28-fpm-alpine

    My Google-fu completely fails me here, what am I doing wrong?

  • In my case the issue was that QEMU wasn't installed and thus unavailable for emulation, apparently buildkit can get through a large amount of steps before needing emulation.

    I added an init container to the pod template which installs QEMU emulators as required:

        - name: prepare
          image: tonistiigi/binfmt:latest
          args: ["--install", "arm64"]
              privileged: true

Suggested Topics