--- title: "Run a remote Docker builder with a warm BuildKit cache" description: "Point docker buildx on laptops and CI at one big exe.dev VM running BuildKit over SSH: a shared warm cache, GC limits, amd64 and arm64." url: https://exe-staging.dev/deploy/docker-builder category: "Developer tools" updated: 2026-11-04 --- # Run a remote Docker builder with a warm BuildKit cache One big VM with a warm build cache that every laptop and CI job builds on. - **Software:** BuildKit (https://docs.docker.com/build/buildkit/) - **Source:** https://github.com/moby/buildkit - **License:** Apache-2.0 - **Time:** 15 minutes - **Example VM name:** `my-builder` (yours will differ; on the VM, `hostname` prints it) - **Machine-readable:** https://exe-staging.dev/deploy/docker-builder/manifest.json - **File:** https://exe-staging.dev/deploy/docker-builder/builder-setup.sh ## What you get - BuildKit running as the buildkit systemd service, started at boot - A socket that only the buildkit group can use, with exedev in it - Garbage collection that keeps the cache under 60% of the disk, leaves 20% free, and keeps cache mounts for two weeks - Builds for linux/amd64 natively and linux/arm64 through BuildKit's own QEMU Running on the VM: - **BuildKit 0.34**: buildkitd from the official release, the buildkit systemd service, socket /run/buildkit/buildkitd.sock, data in `/var/lib/buildkit` - **SSH**: docker buildx's remote driver runs `buildctl dial-stdio` over exe.dev's SSH; no port is opened - **Garbage collection**: limits in /etc/buildkit/buildkitd.toml, at most 60% of the disk for cache and 20% always free How it is reached: - ssh://my-builder.exe.xyz (private): BuildKit, for docker buildx, over SSH with exe.dev keys (yours, your team's, CI's) ## Deploy with Shelley Open https://exe-staging.dev/new?prompt=Set%20up%20BuildKit%20on%20this%20VM%20by%20following%20the%20exe.dev%20recipe%20at%20https%3A%2F%2Fexe-staging.dev%2Fdeploy%2Fdocker-builder.md%20from%20top%20to%20bottom.%0A%0AThe%20recipe%27s%20example%20VM%20name%20is%20my-builder.%20This%20VM%27s%20name%20is%20the%20output%20of%20%60hostname%60%3A%20use%20it%20wherever%20the%20recipe%20writes%20my-builder.%20Skip%20the%20step%20that%20creates%20the%20VM%3B%20you%20are%20on%20it.%0A%0AThis%20recipe%20needs%20a%20100GB%20disk%2C%20and%20a%20new%20VM%27s%20may%20be%20smaller.%20Before%20you%20install%20anything%2C%20run%20%60df%20-h%20%2F%60.%20If%20its%20size%20is%20under%2090G%2C%20stop%20and%20ask%20me%20to%20grow%20the%20disk%2C%20either%20with%20this%20link%3A%20https%3A%2F%2Fexe.dev%2Fsuggest%3Fcommand%3Dresize%2520my-builder%2520--disk%253D100GB%20or%2C%20on%20my%20computer%2C%20with%20%60ssh%20exe.dev%20resize%20my-builder%20--disk%3D100GB%60.%20When%20I%20say%20it%27s%20done%2C%20run%20%60sudo%20resize2fs%20%22%24%28findmnt%20-no%20SOURCE%20%2F%29%22%60%2C%20check%20%60df%20-h%20%2F%60%20again%2C%20and%20carry%20on.%0A%0AYou%20are%20done%20when%20all%20of%20these%20are%20true.%20Check%20each%20one%20yourself%3A%0A-%20BuildKit%20running%20as%20the%20buildkit%20systemd%20service%2C%20started%20at%20boot%0A-%20A%20socket%20that%20only%20the%20buildkit%20group%20can%20use%2C%20with%20exedev%20in%20it%0A-%20Garbage%20collection%20that%20keeps%20the%20cache%20under%2060%25%20of%20the%20disk%2C%20leaves%2020%25%20free%2C%20and%20keeps%20cache%20mounts%20for%20two%20weeks%0A-%20Builds%20for%20linux%2Famd64%20natively%20and%20linux%2Farm64%20through%20BuildKit%27s%20own%20QEMU%0A%0ARead%20the%20whole%20recipe%2C%20but%20run%20only%20%22Set%20up%20the%20builder%20by%20hand%22%2C%0Askipping%20VM%20creation%3A%20fetch%20and%20run%20builder-setup.sh.%20Everything%20about%0Adocker%20buildx%2C%20~%2F.ssh%2Fconfig%2C%20CI%2C%20and%20sharing%20runs%20on%20the%20owner%27s%0Acomputer%20or%20is%20the%20owner%27s%20choice.%20Don%27t%20install%20or%20configure%20Docker%2C%0Adon%27t%20open%20BuildKit%20on%20TCP%2C%20and%20don%27t%20add%20users%20or%20SSH%20keys%20unless%20the%0Aowner%20asks.%20Verify%3A%20buildkit.service%20is%20enabled%20and%20active%3B%0A%60sudo%20-iu%20exedev%20buildctl%20debug%20workers%60%20lists%20one%20worker%3B%20and%0A%60sudo%20buildctl%20debug%20workers%20-v%60%20shows%20the%20five%20GC%20rules.%20Finish%20with%0Athis%20VM%27s%20CPUs%2C%20memory%2C%20and%20free%20disk%20%28nproc%2C%20free%20-h%2C%20df%20-h%20%2F%29%2C%20the%0A%60ssh%20exe.dev%20resize%60%20command%20from%20%22Size%20it%22%20with%20this%20VM%27s%20name%2C%20and%20the%0Acommands%20for%20the%20owner%27s%20computer.%0A%0AFinish%20with%20the%20URLs%20I%20should%20open%20and%20these%20commands%2C%20which%20only%20I%20can%20run%2C%20from%20my%20own%20computer%20%28with%20this%20VM%27s%20name%20in%20them%29%3A%0A%20%20docker%20buildx%20create%20--name%20exe%20--driver%20remote%20ssh%3A%2F%2Fmy-builder.exe.xyz%0A%20%20docker%20buildx%20inspect%20exe%20--bootstrap%0A to create a VM whose agent, Shelley, follows this recipe. From a terminal: ```sh curl -fsSL https://exe-staging.dev/deploy/docker-builder/prompt.txt | ssh exe.dev new --disk=100GB --prompt=/dev/stdin ``` It needs a 100 GB disk. The terminal command makes the VM that size; after the button, Shelley checks and, if the disk is smaller, gives you a link to grow it. Then, on your own computer: ```sh docker buildx create --name exe --driver remote ssh://my-builder.exe.xyz docker buildx inspect exe --bootstrap ``` Or follow the steps below by hand. Every `docker build` on a laptop or a fresh CI runner starts with whatever cache that machine happens to have, which for CI is usually none: pull the base images, install the packages, download the dependencies, compile everything. A remote builder moves that work to one machine whose cache everyone shares. This page runs [BuildKit](https://github.com/moby/buildkit), the engine behind `docker build`, on its own on an exe.dev VM, and points `docker buildx` on your laptop, your teammates' laptops, and your CI jobs at it over SSH. The first build of a commit fills the cache; every build after that, from anyone, starts from it. An exe.dev VM suits the job: it's always on, it can have the CPUs and disk your plan allows, and exe.dev's SSH already knows everyone's keys, so there's no port to open and no certificate to manage. Builds go to your registry with `--push`, or come back to your computer with `--load`. If you'd rather run the whole Docker Engine remotely (containers, Compose, and builds that leave images on the VM), see [Remote Docker host](/deploy/docker-host). If you want CI jobs themselves on exe.dev, see [GitHub Actions](/deploy/github-actions); the two combine. ## How it works - **buildkitd** runs on `my-builder` as a systemd service, listening only on a Unix socket, `/run/buildkit/buildkitd.sock`, that members of the `buildkit` group can use. - **Your docker CLI** has a builder that uses Buildx's [remote driver](https://docs.docker.com/build/builders/drivers/remote/) with an `ssh://my-builder.exe.xyz` endpoint. For each connection it runs `ssh my-builder.exe.xyz buildctl dial-stdio`, which joins SSH to the socket. BuildKit's gRPC travels inside your SSH session. - **The cache** lives in `/var/lib/buildkit` and belongs to the VM, not to any one client. BuildKit sends only the files your Dockerfile needs from your checkout, deduplicates identical steps across concurrent builds, and prunes old cache by the limits in `/etc/buildkit/buildkitd.toml`. Nobody gets Docker on the VM through this: BuildKit builds, and that's all it does. ## What it buys you We built [Gitea](https://github.com/go-gitea/gitea) v28.3.0 with its own [Dockerfile](https://github.com/go-gitea/gitea/blob/v28.3.0/Dockerfile): a pnpm frontend, a Go backend with `RUN --mount=type=cache` for the Go build cache, and an Alpine runtime image of 72 MB. The client was a fresh 2 vCPU / 4 GiB exe.dev sandbox, about the size of a hosted CI runner, with Ubuntu's Docker 29 and Buildx 0.30. The builder was an 8 vCPU / 16 GiB VM with a 100 GB disk, running BuildKit v0.34.2. Times are wall-clock, as the client saw them: | Build | On the client's own Docker | On my-builder | | --- | --- | --- | | First build, empty cache | 3 min 20 s | 1 min 27 s | | Again, nothing changed | 2.0 s | 1.6 s | | One Go file changed | 33.5 s | 24.4 s | | Fresh checkout of a new commit, as in a new CI job | 3 min 20 s (no cache) | 27.7 s | | Fresh checkout of a commit already built | 3 min 20 s (no cache) | 14.1 s | | `--load` the image into the client's Docker | | 7.5 s | The first row is the builder's raw speed: more cores, same work. The fourth row is what changes CI: a fresh runner has nothing cached, so every job is a first build, while on the builder it is a rebuild of what changed. The 14 to 28 seconds are uploading the 44 MB build context (3 seconds), rebuilding the frontend, and compiling the changed Go packages from the warm cache mount. Each fresh checkout's `.git` differs, which is why even an already-built commit recompiles the backend in Gitea's Dockerfile. ## Create the VM ### Size it Builds are bursty: a builder uses all its cores for a minute and then nothing for an hour. On exe.dev a VM's CPU and memory come from your plan's capacity, which your VMs share, so a big builder costs nothing extra while idle. Give it most of your plan, and disk for the cache. On your computer: ```bash ssh exe.dev new --name=my-builder --cpu=8 --memory=16GB --disk=100GB ``` VM names are global, so pick any free name and use it wherever this page says `my-builder`. To change the size later, even with a cache in it: ```bash ssh exe.dev resize my-builder --cpu=16 --memory=32GB --disk=200GB ``` Disks only grow, and are billed by what's used, not by their size. The GC limits below are percentages of the disk, so they grow with it. See [your plan's capacity](/docs/billing/personal), or use a [standalone VM](/docs/standalone-vms), billed by the second, for a builder bigger than your plan.
Set up the builder by hand On the VM (`ssh my-builder.exe.xyz`), check the disk first. If `df -h /` shows less than 90 GB, grow it with the `resize` command above and run `sudo resize2fs $(findmnt -no SOURCE /)`. Then run [builder-setup.sh](https://exe-staging.dev/deploy/docker-builder/builder-setup.sh). It downloads the latest [BuildKit release](https://github.com/moby/buildkit/releases), checks its SHA-256 against the one GitHub publishes, installs it in `/usr/local/bin`, creates the `buildkit` group with `exedev` in it, writes `/etc/buildkit/buildkitd.toml` (only if it doesn't exist), and enables the `buildkit` service: ```bash curl -fsSL https://exe-staging.dev/deploy/docker-builder/builder-setup.sh | bash ``` It ends with `BuildKit v0.34.2 running on /run/buildkit/buildkitd.sock` (or a newer version). To pin a version, run it with `BUILDKIT_VERSION=v0.34.2` before `bash`. Check the worker and its GC rules: ```bash sudo buildctl debug workers -v ``` It lists one worker with platforms `linux/amd64` (and its `v2`-`v4` variants), `network: host`, and five `GC Policy` rules. The VM's Docker Engine isn't used and stays as exeuntu ships it. That's the VM done; the rest happens on your computer.
## Connect docker buildx On your computer, which needs the `docker` CLI and the Buildx plugin (see [Only want the docker CLI?](/deploy/docker-host#get-the-docker-cli)), check that SSH reaches BuildKit: ```bash ssh my-builder.exe.xyz buildctl debug workers ``` Accept the host key if asked; its fingerprint is [exe.dev's](/docs/faq/host-key). You should see one worker and `linux/amd64`. Then create a builder named `exe` and start it: ```bash docker buildx create --name exe --driver remote ssh://my-builder.exe.xyz docker buildx inspect exe --bootstrap ``` `inspect` shows `Status: running`, `BuildKit version: v0.34.2`, and the GC rules. Build with it: ```bash docker buildx build --builder exe -t ghcr.io/you/app:dev --push . ``` To make it the default for `docker build` and `docker buildx build`, `docker buildx use exe`; `docker buildx use default` switches back. `BUILDX_BUILDER=exe` does the same for one shell. If `ssh my-builder.exe.xyz` doesn't connect from your network, use the [other form of the address](/docs/faq/ssh-destination): `ssh://vm+my-builder@vm.exe.xyz`. > [!TIP] > Each build opens a new SSH connection. Let them share one with > `ControlMaster` in `~/.ssh/config`, as in > [Make it fast](/deploy/docker-host#make-it-fast) on the Docker host page, > with `my-builder.exe.xyz` on the `Host` line. ### Get the image out The build runs on the VM, so its result is there too, in the cache. `docker buildx build` without an output says `No output specified with remote driver` and leaves it there. Say where it goes: - **`--push`** sends it straight from the VM to the registry in `-t`. Run `docker login` on your computer as usual; Buildx passes your credentials to the build when the registry asks, and they aren't stored on the VM. Pushes count toward your plan's outbound bandwidth. - **`--load`** sends the image back into your local Docker (Gitea's 72 MB took 7.5 s above). To load every build, create the builder with `--driver-opt default-load=true` (Buildx 0.14 or newer). - **`--output type=local,dest=./out`** copies the final stage's files to `./out`, for a binary or a static site. `.dockerignore` matters more with a remote builder: whatever your Dockerfile copies is uploaded, and leaving out `node_modules`, build output, and large data keeps the first upload of each checkout small. Later builds from the same directory send only what changed. ## Use it from CI A CI job needs three things: an SSH key that exe.dev accepts for `my-builder`, exe.dev's host key, and a Buildx builder with the same endpoint. Give CI its own key, scoped to this one VM and logging in as an unprivileged user that can build and do nothing else as root. **On the VM**, create the user: ```bash sudo useradd --create-home --groups buildkit ci ``` **On your computer**, make a key for CI and register it for this VM only, as that user: ```bash ssh-keygen -t ed25519 -N '' -C builder-ci -f builder-ci cat builder-ci.pub | ssh exe.dev ssh-key add --vm=my-builder --ssh-user=ci ssh -i builder-ci -o IdentitiesOnly=yes my-builder.exe.xyz 'id; buildctl debug workers' ``` The last line shows `uid=…(ci)` with the `buildkit` group, and the worker. The key can't reach any other VM or run `ssh exe.dev` commands; see [VM-scoped SSH keys](/docs/vm-scoped-ssh-keys). `ssh exe.dev ssh-key remove builder-ci` revokes it. Then the host key, the same for every exe.dev VM: ```bash ssh-keyscan -t rsa my-builder.exe.xyz | tee known_hosts | ssh-keygen -lf - ``` The fingerprint must be `SHA256:JJOP/lwiBGOMilfONPWZCXUrfK154cnJFXcqlsi6lPo`. ### GitHub Actions In the repository's **Settings → Secrets and variables → Actions**, add the contents of `builder-ci` as the secret `BUILDER_SSH_KEY`, and the line in `known_hosts` as the variable `BUILDER_KNOWN_HOSTS`. Then delete both files from your computer. A workflow that builds and pushes to the GitHub Container Registry: ```yaml name: image on: push permissions: contents: read packages: write jobs: image: runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 - name: SSH to the builder env: KEY: ${{ secrets.BUILDER_SSH_KEY }} KNOWN_HOSTS: ${{ vars.BUILDER_KNOWN_HOSTS }} run: | install -d -m 700 ~/.ssh printf '%s\n' "$KEY" > ~/.ssh/id_ed25519 chmod 600 ~/.ssh/id_ed25519 printf '%s\n' "$KNOWN_HOSTS" >> ~/.ssh/known_hosts - uses: docker/setup-buildx-action@v4 with: driver: remote endpoint: ssh://my-builder.exe.xyz - uses: docker/login-action@v4 with: registry: ghcr.io username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - uses: docker/build-push-action@v7 with: context: . push: true tags: ghcr.io/${{ github.repository }}:${{ github.sha }} ``` The job needs no Docker of its own and no `cache-from`/`cache-to`: the cache is on the builder. When the job ends, `setup-buildx-action` removes the builder from the runner; that touches nothing on the VM. The same steps work on jobs you run on exe.dev with [GitHub Actions](/deploy/github-actions) (`runs-on: [self-hosted, exe]`). Keep the key in GitHub's secrets there too, not on the base VM, which every job's VM is copied from. > [!WARNING] > Don't give this key to workflows that run on pull requests from forks. > Their code would build on your builder, next to your release builds; > see [Who to share it with](#who-to-share-it-with). ### Other CI Any CI that can run a shell step works the same way: write the key to `~/.ssh/id_ed25519` and the host key line to `~/.ssh/known_hosts`, then `docker buildx create --use --driver remote ssh://my-builder.exe.xyz` and `docker buildx build --push …`. GitLab CI, Buildkite, and Forgejo or Gitea Actions only need the `docker` CLI with Buildx in the job's image. ## Share it with your team Anyone who can SSH in as a member of the `buildkit` group can build. There are two ways to let people in. - **Give them their own access.** On your computer: ```bash ssh exe.dev share add my-builder alice@example.com --root ssh exe.dev share add my-builder team --root # everyone on your exe.dev team ``` They use their own exe.dev keys and run the same `docker buildx create` as you. [Root access](/docs/sharing) means a shell with `sudo` on the builder, so they can change it too. - **Give them build-only access.** Do as for CI above: a user on the VM in the `buildkit` group but not `sudo`, and their public key, added with `ssh-key add --vm=my-builder --ssh-user=`. They don't need an exe.dev account. The key is on your account, and `ssh exe.dev ssh-key remove` takes it away. BuildKit handles many builds at once. Two people building the same commit at the same time share the work, and a step one person has built is cached for everyone. Cores are shared, so a team that builds all day wants a bigger builder or a second one. ### Who to share it with A shared cache is shared trust. Everyone who can build can run any `RUN` step on the builder, as root inside BuildKit's sandbox, with the VM's network (BuildKit uses host networking here), and they can write to cache mounts that other builds then use: a malicious build can put a poisoned `~/.cache/go-build` or npm cache in front of your release build. So: - Share a builder with people and CI jobs you would trust with your release builds. Give untrusted code, such as pull requests from forks, a separate builder or GitHub's own runners. - Pass secrets with `--secret` and `RUN --mount=type=secret`; BuildKit doesn't keep them in the cache. Values in `ARG` and `ENV` do end up there, and in the image. - Keep the builder for building. Don't run other services on it that build steps could reach on `localhost`. ## SSH or tcp+mTLS BuildKit can also listen on TCP with mutual TLS, which is how most remote builder guides set it up. On exe.dev, SSH is the better default: | | SSH (this page) | tcp+mTLS | | --- | --- | --- | | Reachable from | anywhere `ssh` works | only a private network, such as Tailscale: exe.dev forwards no raw TCP from the internet, and VMs have no network between them | | Who can build | exe.dev keys: yours, shared users', VM-scoped keys | anyone holding a client certificate | | Revoking one client | `ssh-key remove` or `share remove` | a new CA and new certificates for every client | | On the VM | nothing listens on the network | a TCP port, with certificates to renew | | Per connection | an SSH handshake (keep one open with ControlMaster) | a TLS handshake | | Client needs | `ssh` | the three certificate files | Choose TCP when your clients are already on a [Tailscale](/docs/integrations-tailscale-wif) network with the builder and you'd rather hand out certificates than SSH keys, or when a client can't run `ssh`.
Add a tcp+mTLS listener This keeps the SSH socket and adds TCP port 1234. It assumes the builder is on your tailnet, with the name `my-builder` and the address `100.101.102.103`; use yours from `tailscale status`. On your computer, make a CA, a server certificate, and a client certificate with BuildKit's [create-certs](https://github.com/moby/buildkit/tree/master/examples/create-certs) bake file, which runs on the builder itself: ```bash SAN="my-builder 100.101.102.103" docker buildx bake --builder exe \ "https://github.com/moby/buildkit.git#v0.34.2:examples/create-certs" scp -r .certs/daemon my-builder.exe.xyz:buildkit-certs ``` It writes `.certs/daemon` and `.certs/client`, each with `ca.pem`, `cert.pem`, and `key.pem`, and throws the CA's key away, so new clients share the one client certificate. On the VM, install the server's half and add the listener: ```bash sudo install -d -m 700 /etc/buildkit/certs sudo install -m 600 ~/buildkit-certs/* /etc/buildkit/certs/ && rm -r ~/buildkit-certs sudo mkdir -p /etc/systemd/system/buildkit.service.d sudo tee /etc/systemd/system/buildkit.service.d/tcp.conf >/dev/null <<'EOF' [Service] ExecStart= ExecStart=/usr/local/bin/buildkitd --addr unix:///run/buildkit/buildkitd.sock --group buildkit \ --addr tcp://0.0.0.0:1234 --tlscacert /etc/buildkit/certs/ca.pem \ --tlscert /etc/buildkit/certs/cert.pem --tlskey /etc/buildkit/certs/key.pem EOF sudo systemctl daemon-reload && sudo systemctl restart buildkit ``` The Unix socket stays plain (the log says `TLS is disabled for unix:///run/buildkit/buildkitd.sock`), so SSH keeps working. On a client in the tailnet, with `.certs/client` copied to it: ```bash docker buildx create --name exe-tcp --driver remote \ --driver-opt cacert=$PWD/.certs/client/ca.pem,cert=$PWD/.certs/client/cert.pem,key=$PWD/.certs/client/key.pem,servername=my-builder \ tcp://100.101.102.103:1234 docker buildx inspect exe-tcp --bootstrap ``` In GitHub Actions, `setup-buildx-action` takes the three files from `BUILDER_NODE_0_AUTH_TLS_CACERT`, `_CERT`, and `_KEY`; see Docker's [TLS authentication](https://docs.docker.com/build/ci/github-actions/configure-builder/#tls-authentication). The runner has to be on the tailnet too.
## Multi-arch images exe.dev VMs are x86-64, so `linux/amd64` builds natively. BuildKit's release ships QEMU emulators (`buildkit-qemu-aarch64` and others), and BuildKit uses them by itself for any `RUN` step of another platform: ```bash docker buildx build --builder exe --platform linux/amd64,linux/arm64 \ -t ghcr.io/you/app:dev --push . ``` `docker buildx ls` lists only the amd64 platforms; arm64 works anyway. Emulation costs a lot for compiling and little for copying files and installing packages: | Adding linux/arm64 to the Gitea build on my-builder | Time | | --- | --- | | v28.3.0's Dockerfile: compiles Go for arm64 under QEMU | 10 min 34 s, 9 min 36 s of it in `make backend` (amd64 natively, empty cache: 55 s) | | `main`'s Dockerfile: compiles on amd64 with `GOARCH=arm64` | 34 s, 32 s of it compiling | So cross-compile what you can: build on `--platform=$BUILDPLATFORM` and set `GOOS`/`GOARCH` (or use [xx](https://github.com/tonistiigi/xx)) from `TARGETOS` and `TARGETARCH`, as Docker's [multi-platform guide](https://docs.docker.com/build/building/multi-platform/#cross-compilation) shows. For native arm64, Buildx can add a second node to the `exe` builder on an arm64 machine running BuildKit (`docker buildx create --append`), and builds each platform on its own node. If a step fails with `exec format error`, the image mixes binaries of several architectures and needs the emulators registered with the kernel. On the VM, `sudo apt-get install -y qemu-user-static` does that at every boot. ## Keep the disk in check BuildKit prunes its own cache: when it starts, and a minute after builds. `builder-setup.sh` wrote these rules, applied in order until the cache fits; percentages are of the VM's disk: 1. Build contexts and Git checkouts unused for 3 days, once they take over 5%. They're cheap to upload again. 2. Cache mounts (`RUN --mount=type=cache`) unused for 2 weeks, once over 20%. BuildKit's default drops them after 48 hours, which empties a team's Go or npm cache over a long weekend. 3. Anything unused for 30 days, once the cache is over 50%. 4. The least recently used cache, until it's under 60% of the disk and at least 20% of the disk is free, never going below 10%. See what the cache holds, from your computer: ```bash docker buildx du --builder exe | tail -3 ``` Prune by hand if you need room now: ```bash docker buildx prune --builder exe --filter until=72h --force # unused for 3 days docker buildx prune --builder exe --all --force # everything ``` To change the limits, edit `/etc/buildkit/buildkitd.toml` on the VM (Docker's [garbage collection guide](https://docs.docker.com/build/cache/garbage-collection/) explains each field), then `sudo systemctl restart buildkit` and check with `sudo buildctl debug workers -v`. A restart stops builds that are running, so do it when nobody is building. If the cache is always near its cap, grow the disk instead (`ssh exe.dev resize my-builder --disk=200GB`); the percentages follow. ## Update Run the setup script again on the VM when a new [BuildKit release](https://github.com/moby/buildkit/releases) is out, when nobody is building: ```bash curl -fsSL https://exe-staging.dev/deploy/docker-builder/builder-setup.sh | bash ``` It replaces the binaries and restarts the service, keeps `/etc/buildkit/buildkitd.toml` and the cache, and prints the new version. Clients don't need to change; Buildx and BuildKit agree on what they both support.
Something not working? Start from your computer with SSH alone: ```bash ssh my-builder.exe.xyz buildctl debug workers ``` - **`Permission denied (publickey)`**: SSH offers a key exe.dev doesn't know for this VM. See [which key to use](/docs/faq/ssh-key); in CI, check that the secret holds the whole private key, ending with a newline. - **`Host key verification failed`**: connect once and accept the key, or in CI check `BUILDER_KNOWN_HOSTS`. - **`buildctl: command not found`**: the setup script hasn't run on this VM, or failed. Run it again and read its output. - **`permission denied` on `/run/buildkit/buildkitd.sock`**: the user isn't in the `buildkit` group (`id` shows it). Add it with `sudo usermod -aG buildkit ` and reconnect; with ControlMaster, `ssh -O exit my-builder.exe.xyz` first. - **Builds hang after your laptop sleeps**: `ssh -O exit my-builder.exe.xyz`, then build again. - **`No output specified with remote driver`**: add `--push`, `--load`, or `--output`; see [Get the image out](#get-the-image-out). - **`no space left on device`**: `docker buildx du --builder exe`, prune, or grow the disk; see [Keep the disk in check](#keep-the-disk-in-check). - **`exec format error`** on a non-amd64 platform: see [Multi-arch images](#multi-arch-images). On the VM, `journalctl -u buildkit -n 50` shows BuildKit's log, `systemctl status buildkit` whether it's running, and `sudo buildctl debug workers -v` its platforms and GC rules.
## Frequently asked questions ### How is this different from using exe.dev as a remote Docker host? [Remote Docker host](/deploy/docker-host) points your whole `docker` CLI at the VM's Docker Engine: containers, Compose, and images all live there, and builds use Docker's built-in BuildKit. This page runs BuildKit on its own, for builds only: images go to a registry or back to you, the cache has its own limits, and many people and CI jobs can build on it without being able to run containers on the VM through Docker. Use the Docker host for one person's day-to-day Docker; use this when a team and CI share one build cache. ### How much faster is it? It depends on the Dockerfile. Building Gitea, a fresh 2 vCPU runner with its own Docker took 3 min 20 s every time, because it starts with an empty cache. The same build of a new commit, from a fresh checkout on that runner but built on an 8 vCPU builder with a warm cache, took 28 s. See [What it buys you](#what-it-buys-you). ### How much does it cost? The builder is an ordinary VM on your plan. Its CPU and memory come from your plan's shared capacity, and builds are bursty, so it rarely holds them for long; on Personal, a plan with 8 vCPU and 16 GB is $75 a month. Cache counts toward your plan's disk allowance, and image pushes toward its bandwidth. See [exe.dev pricing](https://exe.dev/pricing). ### Do I need Docker Desktop? No. You need the `docker` CLI with the Buildx plugin, 0.9 or newer for `ssh://` endpoints (we tested 0.30). Docker Desktop's CLI works too. Nothing runs on your computer but the CLI and `ssh`. ### Can it build arm64 images? Yes. exe.dev VMs are x86-64, and BuildKit's release includes QEMU emulators, so `--platform linux/amd64,linux/arm64` works with no setup. Emulation is slow for compiling; Dockerfiles that cross-compile with `$BUILDPLATFORM` avoid most of it. See [Multi-arch images](#multi-arch-images). ### Is the cache shared between projects and people? Yes, that's the point. Identical steps run once, even when two people build the same commit at the same moment. Cache mounts with the same target (say `/root/.cache/go-build`) are shared across projects too. It also means everyone who can build can affect what others' builds reuse; see [Who to share it with](#who-to-share-it-with). ### Does it work on Windows? Use the docker CLI in WSL. Windows' own OpenSSH also works with the `ssh://` endpoint. ## Related - [Use exe.dev as a remote Docker host](https://exe-staging.dev/deploy/docker-host.md): Keep the docker CLI. Send the builds, the containers, and the fan noise to a VM. - [Run GitHub Actions jobs on exe.dev, a fresh VM per job](https://exe-staging.dev/deploy/github-actions.md): Every job on a fresh copy of a warm VM, billed by the second. - [Gitea](https://exe-staging.dev/deploy/gitea.md): Self-hosted Git forge with exe.dev SSO - [Forgejo](https://exe-staging.dev/deploy/forgejo.md): Self-hosted Git forge with exe.dev SSO - [Host your Tangled repos on your own knot](https://exe-staging.dev/deploy/tangled.md): Your git repos on your own VM, with tangled.org on top. - [Sandboxes on exe.dev](https://exe-staging.dev/deploy/sandbox.md): A VM with its own CPU and memory, billed by the second while it runs.