Migrating from Docker Desktop to Colima

by

Docker Desktop now requires a paid subscription for professional use and is getting dropped by smaller companies. Rancher Desktop is one of the usual suggestions, but since it uses a tool called Colima under the hood, it’s a very viable alternative.

What is Colima?

Colima is an open-source tool that allows you to run Linux containers on macOS with minimal setup. It is designed to be a simple and Docker Desktop-like experience for macOS users, but it uses containerd and Lima under the hood. Lima is an open-source project that provides a Linux virtual machine on macOS, which is necessary because containers are a feature of the Linux kernel.

Colima makes it easy to start, stop, and manage the lifecycle of containers and Kubernetes clusters on a Mac, providing a convenient alternative for developers who prefer not to use Docker Desktop or who want more control over their container runtime environment.

Installation

Requirements:

  • MacOS with ARM64 chip
  • Homebrew installed

Remove Docker Desktop

Assuming you already uninstalled Docker Desktop, let’s also clear some cached Docker files by running the following script:

#!/bin/bash

# Files that must be deleted.
paths=(
        "~/Library/Cookies/com.docker.docker.binarycookies"
        "~/Library/Logs/Docker Desktop"
        "~/Library/Application Support/Docker Desktop"
        "~/Library/Caches/com.docker.docker"
        "~/Library/Group Containers/group.com.docker"
        "~/Library/Saved Application State/com.electron.docker-frontend.savedState"
        "/Library/PrivilegedHelperTools/com.docker.vmnetd"
        "/Library/LaunchDaemons/com.docker.vmnetd.plist"
        "/usr/local/lib/docker"
        "~/.docker"
)

# Loop to delete declared files.
for path in "${paths[@]}"; do
      eval rm -rf $path
      echo "Deleted: $path"
done

echo ""
echo "DONE."

After you execute this script, you can continue with the next steps.

Update/Clean up your Homebrew installation

Verifying if your Homebrew installation is current is a good idea. So first verify if you have any outdated packages:

$ brew outdated

If the previous command shows you any obsolete pachages, you can upgrade them with the following command:

$ brew upgrade

Finally, clean any cached file used by Homebrew in previous installations:

$ brew cleanup

Install the Docker CLI tools

brew install docker

Do not install the docker-compose plugin using the brew command because Docker plugins must be installed differently.

Try to execute the docker version command to verify the installation:

$ docker version
Client: Docker Engine - Community
 Version:           26.1.0
 API version:       1.43 (downgraded from 1.45)
 Go version:        go1.22.2
 Git commit:        9714adc6c7
 Built:             Mon Apr 22 17:00:04 2024
 OS/Arch:           darwin/arm64
 Context:           colima
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Notice that only the Docker client info is displayed. This is because we haven’t installed the Docker Engine yet.

Installing Colima Core dependencies

The first dependency is QEMU, an open-source software virtualization tool that performs hardware emulation. In the container space, QEMU is essential in running containers on hardware platforms other than the host. This is especially relevant in environments where containers run on multiple processor architectures. So you can install it using brew:

$ brew install qemu

The other dependency is Lima, a tool that allows us to run Linux virtual machines on macOS, acting as a container environment similar to Docker Desktop. It benefits developers working on macOS who need a Linux environment for developing, testing, or running containerized applications.

Lima will be installed as part of the Colima installation:

$ brew install colima

Let’s also install Docker Compose and Buildx:

brew install docker-buildx docker-compose
mkdir -p ~/.docker/cli-plugins
ln -sfn /opt/homebrew/opt/docker-buildx/bin/docker-buildx ~/.docker/cli-plugins/docker-buildx
ln -sfn /opt/homebrew/opt/docker-compose/bin/docker-compose ~/.docker/cli-plugins/docker-compose

Once we have everything installed we can get Colima started and test that Docker is running:

colima start
docker run --rm hello-world

You should get something like this:

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
478afc919002: Pull complete
Digest: sha256:a26bff933ddc26d5cdf7faa98b4ae1e3ec20c4985e6f87ac0973052224d24302
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (arm64v8)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Assigning more resources to the Colima VM

The default VM Colima creates has 2 vCPUs, 2GBs of RAM and 60GBs of disk storage. Let’s change that:

$ colima stop
colima start --cpu 2 --memory 8 --disk 20

Adjust the CPU, memory, and disk space values according to your needs.

That’s it, now we have Colima running with 2 vCPUs, 8GB RAM and a 20GB disk!

Some common issues

Permission errors when running containers

Colima is specifically designed to run containers directly on macOS, using the Docker daemon inside an Alpine and Lima-based virtual machine as stated earlier. The virtual machine is sort of an alpine-docker bundle for your OSX development environment.

However, you might face issues where mounting volumes doesn’t work as expected:

postgresql-colima-database-1  | chown: /var/lib/postgresql/data: Permission denied

When a Colima-based virtual machine is created, you can access the configuration file and examine the mountType used

# /Users/<yourUserName>/.colima/deafult/colima.yaml
# Default: virtiofs (for vz), sshfs (for qemu)
mountType: sshfs

The recommended and most stable option for QEMU type VMs is 9p, so we need to change that from sshfs.

It’s important to remember that Colima relies on Lima, a hypervisor that enables running Linux containers natively on macOS. When your Colima-based virtual machine is created, changing the colima.yaml file won’t have any effect because the virtual machine is already established.

The key is to modify the mounting type before creating the virtual machine by instructing Lima to override the mountType configuration when creating a new virtual machine. Here’s how:

Create a file named override.yaml under ~/.lima/_config/ with the following content:

mountType: 9p
mounts:
  - location: "/Users/<yourUserName>"
    writable: true
    9p:
      securityModel: mapped-xattr
      cache: mmap
  - location: "~"
    writable: true
    9p:
      securityModel: mapped-xattr
      cache: mmap
  - location: /tmp/colima
    writable: true
    9p:
      securityModel: mapped-xattr
      cache: mmap

If you already have a Colima-based virtual machine running, follow these steps:

  1. Stop Colima: colima stop
  2. Delete the existing virtual machine: colima delete
  3. Create a new virtual machine: colima start --cpu 2 --memory 8 --disk 20

With these adjustments in place you should be able to easily mount volumes to your containers without permission errors.

UDP forwarding to the host does not work

UDP support in the Docker ecosystem has always felt a bit like an afterthought…

With Colima, all the (TCP) ports used by your containers are forwarded to the host machine, so you can access them comfortably in localhost. That doesn’t work for UDP though. The reason is that Lima uses SSH to forward the ports, which doesn’t support UDP.

A way around this is to tell Colima to assign an IP address to the VM, and then use that:

colima stop
colima start --cpu 2 --memory 8 --disk 20 --network-address

We can now check with colima status:

INFO[0000] Colima is running using QEMU
INFO[0000] arch: aarch64
INFO[0000] runtime: docker
INFO[0000] mountType: 9p
INFO[0000] address: 192.168.106.2
INFO[0000] socket: unix:///Users/saghul/.Colima/default/docker.sock

The IP address we can use is 192.168.106.2 and won’t change, so that’s handy. Another potential solution would be to configure bridging and have the VM get a “real IP”, but honestly this Just Works (TM) so I didn’t bother.

Docker login

When attempting to use docker login I stumbled upon this error:

Error saving credentials: error storing credentials

This is likely caused by the fact we had Docker Desktop installed before.

To resolve, edit ~/.docker/config.json and remove the credsStore line, that should do the trick.

Buildx woes

Buildx is the new way to build Docker images using BuildKit. It allows one to build multi-architecture images easily. However, we may run into an issue where all amd64 images are producing an error at build time.

The problem seems to be some cgroups v2 shenanigans with the default Alpine image Colima uses, but thankfully there is a workaround by creating a custom Buildx builder with the rootless image:

docker buildx create --name mybuilder --driver-opt 'image=moby/buildkit:rootless' --bootstrap --use

Now we can build multi-arch images without issues.