Abusing the ceph orchestrator for other services

Today I learned I can completely abuse the ceph orchestrator to run all my (other) services in high availability mode. For many years my 30+ services all ran on a single host which goes down sometimes and which runs out of memory most of the day, even when I download more ram from the internet (zram).

Since I already run 30+ servers for my distributed ceph storage, 6 of them even have 64GB memory, I would like to swap out all of my services and keep them up and running, even when a single host dies.

Welcome to the stage: Ceph Custom Container Service

Ceph allows you to deploy custom docker images and you decide how many instances you want on which kind of hosts (I have mostly odroid-hc4/arm64 and a few odroid-h3-plus/x86_64)

Very crucial is that all hosts have the already highly available cephfs mounted to the same directory /mnt, so the docker container can obtain its configuration and data files from there. Because the shared cephfs is the same on all hosts, it doesnt matter where the service runs.

Example on how to run minecraft on ceph:

service_type: container
service_id: minecraft
placement:
    count: 1
    label: odroid-h3
spec:
  image: docker.io/library/openjdk:22
  entrypoint: /mcserver/start.sh
  uid: 1000
  gid: 1001
  args:
    - "--net=host"
    - "-p25565"
    - "-p8123"
    - "--workdir=/mcserver"
  ports:
    - 25565
    - 8123
  bind_mounts:
    - ['type=bind', 'source=/mnt/mcserver', 'destination=/mcserver']

One of the biggest problems that I was facing, is that ceph does not only not use docker and instead podman, but also they specify the command line argument --init which kills most docker images, that run their own init system inside the docker container. Running lidarr, sonarr, radarr and so on was not that easily possible.

My workaround: I am overridding the entrypoint to directly call the lidarr binary without going through the docker image’s own init system. This seems to be pretty stable!

Example yaml configuration for lidarr:

service_type: container
service_id: lidarr
placement:
    count: 1
    label: odroid-h3
spec:
  image: lscr.io/linuxserver/lidarr:latest
  entrypoint: /app/lidarr/bin/Lidarr
  uid: 1000
  gid: 1000
  envs:
    - TZ=Europe/Vienna
  args:
    - "--net=host"
    - "-p8686"
  ports:
    - 8686
  bind_mounts:
    - ['type=bind', 'source=/mnt/lidarr', 'destination=/config']
    - ['type=bind', 'source=/mnt/komposthaufen', 'destination=/komposthaufen']

This makes my ceph services list much more interesting 🙂