Natively Building Multi-Platform Docker Images

Without emulation, an application compiled for a processor using the ARM ISA will not run on a computer running the x86/x64 ISA or vice versa. Adding Docker into the mix can complicate this story even more because Docker images also need to be built for a specific instruction set.

Thankfully, there are solutions for compiling applications to specific ISAs and building multi-architecture Docker images. Emulation is the simplest option but is often not a good choice as application builds grow in complexity.

Here we will explore using Docker context for building single-platform images natively. Then, we will use Docker Buildx to create multi-platform images natively.

Docker Build Using Context

The simplest approach is to build a Docker image on a computer running the target ISA by creating and setting docker context to a target machine running the instruction set where your Docker containers will run.

# create an ssh context named amd64 via ssh, with user01 on a computer named minilx
docker context create amd64 --docker host=ssh://user01@minilx
# set context to the remote host running Docker
docker context use amd64

If you are prompted for a password, you will not be able to switch context to the remote machine via ssh without creating an ssh key pair. See Docker context how to use specific ssh key for how to resolve this.

Create and Build an Application

Next let's create a dotnet web application, add a Docker build definition (Dockerfile) and build and run the application via Docker. While this approach isn't specific to dotnet, Microsoft makes this straightforward with a concise, yet robust Docker multi-stage build definition. Create a Dockerfile for an ASP.NET Core details the approach I use here.

If you prefer cloning and running the sample without walking through the setup steps next, you can find the entire sample at https://github.com/TechScoped/samples-build-across-archs

# create a web app
dotnet new webapp -o AWebApp --no-https

Create a file named Dockerfile with this content and copy it to the AWebApp project directory.

FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /app
COPY *.csproj ./
RUN dotnet restore

COPY . .
RUN dotnet publish -c Release -o out

FROM mcr.microsoft.com/dotnet/aspnet:6.0
WORKDIR /app
COPY --from=build /app/out .
ENTRYPOINT ["dotnet", "AWebApp.dll"]

Create a file named .dockerignore with this content and copy it to the AWebApp project directory.

Dockerfile
appsettings.Development.json
bin/
obj/

Build and run the containerized application.

# build (and tag) a docker image for the app
docker build -t awebapp:1.0 .
# run the application in a docker container and expose the app over port 8080
docker run -d -p 8080:80 --name webapp awebapp:1.0

Browsing to the Application

Recall that we switched our Docker context to a computer host named minilx. Therefore, the application is hosted at http://minilx:8080.

Building on ARM

My local machine is running the ARM ISA so by switching Docker context back to default, I can compile the application for this instruction set.

docker context use default
docker build -t awebapp:1.0 .
docker run --rm -p 8080:80 --name webapp awebapp:1.0

You can now browse the application using http://localhost:8080.

At the terminal, press Ctrl + C to exit and remove the container.

Image Limitations

In each case, the application is compiled for the underlying processor instruction set of the target web server. To demonstrate this, I'll first create a repository named awebapp-arm in a Docker registry, tag the image built on a machine running an ARM processor and push the image to the repository.

# The awebapp-arm repository has already been created. See the text after this script for details.
docker tag awebapp:1.0 techscoped/awebapp-arm:1.0
docker push techscoped/awebapp-arm:1.0

I'll first run the application on an ARM (RISC-based) processor.

docker run --rm -p 8080:80 --name webapp techscoped/awebapp-arm:1.0

I can now browse to the application as I did before.

However, if I switch context to a computer running an AMD (CISC-based) processor and try to run the container, a platform incompatabiity warning mesage will appear and the application will fail to start.

docker context use amd64
docker run --rm -p 8080:80 --name webapp techscoped/awebapp-arm:1.0

WARNING: The requested images platform (linux/arm64/v8) does not match the detected host platform (linux/amd64) and no specific platform was requested standard_init_linux.go:228: exec user process caused: exec format error

In order to test, you will need a Docker registry. To use docker.io, see Docker Hub Quickstart.

Multi-Architecture Docker Images

In the event that you do need to run an application on either instruction set, you can take the context switch approach I outlined earlier. However, this isn't a great solution when you publish your Docker image and want to make the container runtime experience simple for users. A better option in this case is to create a multi-architecture image for your application. Microsoft (and many others) have now taken this approach. In fact, the two dotnet images used in this application's multi-stage build definition: dotnet/sdk:6.0 and dotnet/aspnet:6.0 are multi-architecture enabled. To see for yourself, use the Docker manifest experimental command to see the manifest of Microsoft dotnet sdk image.

docker manifest inspect mcr.microsoft.com/dotnet/sdk:latest

In the json manifest, you will see multiple objects in the manifests array with platform values, such as:

"platform": {
    "architecture": "amd64",
    "os": "linux"
}

and

"platform": {
    "architecture": "arm64",
    "os": "linux",
    "variant": "v8"
}

This is why we were able to run Docker build on ARM and AMD processors without making any build platform-specific updates to the build definition (Dockerfile).

Using Docker Buildx To Build Across Platform Formats

Docker Buildx is plugin by Docker for the Docker client that extends Docker build to include the features from Moby Buildkit. Note that the term "Platform Formats" is just another way to describe building images that work across processor instruction sets. As of this writing, Buildx is a Docker experimental feature so make sure to enable using expermemtal features in Docker unless you are on a version of Docker that contains Buildx as the default builder.

The Build multi-platform images section in the Docker Buildx post discusses different ways to build images for different platform formats: emulating an instruction set via QEMU, building on native nodes or using multi-stage docker files to cross-compile the application for different processor architectures. Docker further recommends that QEMU is the simplest option. However, they point-out that for more complex builds, using native nodes is a better option. Avoiding emulation improves build performance, see Faster Multi-Platform Builds: Dockerfile Cross-Compilation Guide for details, stability and compatability. The last option, using a stage in the Docker multi-stage build definition to cross-compile the application, increases the complexity of your build definition.

Next, let's demonstrate leveraging native processor instructions sets to create a multi-platform image.

# create a builder instance named awebapp-multi using the amd64 Docker context created earlier
docker buildx create --use --name awebapp-multi --node amd64
# append the local default context to the builder instance
docker buildx create --append --name awebapp-multi --node default

Listing the new builder instance should show something like this.

$ docker buildx ls
NAME/NODE        DRIVER/ENDPOINT             STATUS   PLATFORMS
awebapp-multi *  docker-container
  awebapp-multi0 amd64                       inactive
  awebapp-multi1 unix:///var/run/docker.sock inactive

Because I haven't yet run the build, the status is inactive. Let's run the build to create a multi-platform Docker image.

# build the multi-architecture image for amd and arm processors using native processor instruction sets
docker buildx build --platform linux/amd64,linux/arm64 -t techscoped/awebapp-multi:1.0 -o type=registry .

Build instance after running a successful build.

NAME/NODE         DRIVER/ENDPOINT              STATUS    PLATFORMS
awebapp-multi *   docker-container
  awebapp-multi0  amd64                        running   linux/amd64, linux/386
  awebapp-multi1  unix:///var/run/docker.sock  running   linux/arm64, linux/riscv64, linux/mips64

There is one observation worth mentioning here. If you have emulation (qemu) installed on a host and it's listed first in your builder instance, then buildx might use emulation to attempt the build. To avoid this, ensure that you don't have qemu installed or create and append native nodes first, as I did by adding the amd64 node before adding the default node in the prior example.

Here is the resulting Docker manifest (abreviated).

docker manifest inspect techscoped/awebapp-multi:1.0

{
   ...
   "manifests": [
      {
          ...
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
        ...
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      }
   ]
}

There is a lot more to Moby BuildKit brought to Docker via Buildx. This is an essential read about multi-platform Docker builds Faster Multi-Platform Builds: Dockerfile Cross-Compilation Guide

Conclusion

Using Docker context is an effective approach for fully compatible and performant Docker builds, resulting in single-platform Docker images. Similarly, using Docker Buildx with multiple hosts is a great strategy for fully compatible and performant Docker builds, resulting in multi-platform Docker images.