Why does stripping executables in Docker add ridiculous layer memory overhead?

[chanisef]

On https://github.com/T145/black-mirror/blob/master/Dockerfile#L55 , I ran the following command to reduce executable sizes:

find -P -O3 /usr/bin/ /usr/local/bin -type f -not -name strip -and -not -name dbus-daemon -execdir strip -v --strip-unneeded '{}' \;

And its size jumped up from ~779.53 to ~986.55MB!

As an attempt to bypass this caveat I created an intermediate layer to copy the changes over from, like so:

FROM base as stripped
RUN find -P -O3 /usr/bin/ /usr/local/bin -type f -not -name strip -and -not -name dbus-daemon -execdir strip -v --strip-unneeded '{}' ;
FROM base
COPY --from=stripped /usr/bin/ /usr/bin/

COPY --from=stripped /usr/local/bin/ /usr/local/bin/

However the resulting image size did not change. Also note that the base image has other programs installed on it, so simply using another Debian distribution as the intermediate layer wouldn't cover stripping each program on the base image.

Why is this large size difference happening? Is there a way to strip executables in Docker at all without having this happen?

[Anderson]

Each directive in your Dockerfile adds another layer to the image. So anything you do -- removing files, stripping binaries, etc -- is only going to increase the size of the image.

It looks like you're trying to overcome this issue by using a multi-stage build, but that's not doing you any good: those two COPY directives are introducing effectively the same changes introduced by the find command in the previous stage.

The way to solve this is by discarding the old layers, generally by creating a new image that reflects the state of the top layer only. This is called "squashing" the image, and there are various ways of doing this. Here's one mechanism that works. For this example, I'm using this Dockerfile (based on your linked example) to build squashtest:base:

FROM docker.io/parrotsec/core:base-lts-amd64
RUN export DEBIAN_FRONTEND=noninteractive && 

apt-get -q -y update --no-allow-insecure-repositories 

&& apt-get -y upgrade --with-new-pkgs 

&& apt-get -y install --no-install-recommends 

aria2=1.35.0-3 

apparmor=2.13.6-10 

apparmor-utils=2.13.6-10 

auditd=1:3.0-2 

curl 

debsums=3.0.2 

gawk=1:5.1.0-1 

git  

iprange=1.0.4+ds-2 

jq=1.6-2.1 

libdata-validate-domain-perl=0.10-1.1 

libdata-validate-ip-perl=0.30-1 

libnet-idn-encode-perl=2.500-1+b2 

libnet-libidn-perl=0.12.ds-3+b3 

libregexp-common-perl=2017060201-1 

libtext-trim-perl=1.04-1 

libtry-tiny-perl=0.30-1 

localepurge=0.7.3.10 

locales 

miller=5.10.0-1 

moreutils=0.65-1 

p7zip-full=16.02+dfsg-8 

pandoc=2.9.2.1-1+b1 

preload=0.6.4-5+b1 

python3-pip=20.3.4-4+deb11u1 

rkhunter=1.4.6-9 

symlinks=1.4-4 

&& apt-get install -y --no-install-recommends --reinstall ca-certificates=* 

&& apt-get -y autoremove 

&& apt-get -y clean 

&& rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* 

&& rm -f /var/cache/ldconfig/aux-cache 

&& find -P -O3 /var/log -depth -type f -print0 | xargs -0 truncate -s 0 

&& localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8 

&& localepurge 

&& symlinks -rd / 

&& apt-get -y purge --auto-remove localepurge symlinks 

&& find -P -O3 /etc/ /usr/ -type d -empty -delete

1. Build the base image.

   docker build -t squashtest:base -f Dockerfile.base .
   

   This produces the following:

   $ docker image ls squashtest:base
   REPOSITORY   TAG       IMAGE ID       CREATED             SIZE
   squashtest   base      58dff2c40a28   About an hour ago   786MB
   

2. Build a new image squashtest:stripped with stripped binaries using this Dockerfile:

   FROM squashtest:base
   RUN find -P -O3 /usr/bin/ /usr/local/bin 
   
   -type f -not -name strip -and -not -name dbus-daemon 
   
   -execdir strip -v --strip-unneeded '{}' ; || :
   
   

   Which produces:

   $ docker image ls squashtest:stripped
   REPOSITORY   TAG        IMAGE ID       CREATED             SIZE
   squashtest   stripped   42aa25ebc0c7   About an hour ago   997MB
   

   At this point, the image consists of the following layers:

   $ docker image inspect squashtest:stripped | jq '.[0].RootFS'
   {
     "Type": "layers",
     "Layers": [
       "sha256:7e203d602b1c20e9cf0b06b3dd3383eb36bc2b25f6e8064d9c81326dfdc67143",
       "sha256:1fc5866a0b6b7a23a246acfd46b4c513b4a188d2db2d8a26191989a4a18c74d3",
       "sha256:cc3a9d1a7f9222eee31b688d887c79745e20389ecfe0fe208349c73cfd172b4a"
     ]
   }
   

3. We can collapse these into a single layer like this:

   docker run --rm squashtest:stripped \
     tar -C / -cf- --exclude=./dev --exclude=./sys \
     --exclude=./proc  . |
     docker import - squashtest:imported
   

   This produces:

   $ docker image ls squashtest:imported
   REPOSITORY   TAG        IMAGE ID       CREATED          SIZE
   squashtest   imported   6f036f16d477   46 seconds ago   626MB
   

   We've saved 160MB off the base image.

There are other ways to squash a Docker image; there are a number of tools on GitHub ( https://github.com/goldmann/docker-squash , https://github.com/jwilder/docker-squash , https://github.com/qwertycody/Bash_Docker_Squash ) that accomplish something similar. docker build https://docs.docker.com/engine/reference/commandline/build/ if you enable experimental features, but that doesn't appear to accomplish much when I try it.

I would argue that for the 160MB we've managed to save here the effort isn't worth it. Unless you're running Docker in an extremely constrained environment, that's going to be nothing but a drop in the bucket (for reference, that's about the size of /bin/ls).

In fact, the strip operation in your Dockerfile is mostly pointless: distributions generally strip binaries by default; you can verify this by running file on all the binaries in /usr/bin and /bin on docker.io/parrotsec/core:base-lts-amd64:

$ docker run -it --rm docker.io/parrotsec/core:base-lts-amd64 bash
# apt -y install file
# file /bin/* /usr/bin/* | grep ELF  | grep -v stripped

That last command returns zero results: all the binaries have been stripped.