my container host at home uses quadlet to do container stuff directly from systemd using podman. pretty nice and simple. The only annoyance I have run into is getting it to pull conainers from a private registry without tls. This is not production in the sense that I serve a business from it, but it currently runs about 16 different local services I run at home.

It is exceedingly simple compared to anything having to do with k8s.

I'm a little surprised that no one has mentioned the most boring option possible: just run the containers.

A container is just a fancy Linux process. We have tools for running those - systemd, openRC, pick your poison.

I set this up at a previous job where the devs were all in on docker for development, but we were building a single mobile app backed so a cluster was overkill. It worked. It was boring.

Granted, inter-container networking won't work out of the box - that's probably the main affordance of compse. But honestly, if you have a dedicated machine anyway, I would consider just whacking -net=host on the run command.

It was my understanding that Docker changed its stance about production usage some years back.

https://docs.docker.com/compose/how-tos/production/

I've been using Docker Compose + a few systemd units for awhile now without any issues. Granted it's only a small app with a few hundred users.

microk8s, k3s, and “plain old” k8s if you’re looking for orchestration - otherwise some glue + docker-compose. i think those are the best supported options out there for running containers in production

why does it seem like the only option for deploying them that claims to be “production ready” is Kubernetes?

a few systems (like nomad) make this claim, but i’m suspect of anything non-k8s honestly. it just has so much market capture that it’s hard to justify picking anything else.

i think this might be similar to the reason linux is used in 99% of cases over arguably more boring systems like freebsd - compatibility, ease of googling for solutions, familiarity for new hires - in other words, market share.

choosing something like nomad means people will need to spin up on it - but tons of ppl are already familiar with k8s concepts.

k8s systems are definitely more complex in a general sense - so many moving parts! but i’d also argue that k8s-like systems are the boring choices due to the sheer mass of their userbase.

to frame this another way: if you were choosing a database for production, would you choose sqlite or postgres? postgres is certainly more complex - but its popularity kinda shoehorns it into that “boring” category.

I'm one of those NixOS people, so in the lab at the minute I'm using it's wrapper around oci-containers to do declarative management, in the past I've done these sorts of "big boat" arrangements with plain old docker-compose; so long as you're careful it's perfectly safe to use, it's tablesaw dangerous, not lathe dangerous.

podman-compose might be what I'd use today for small scale deployments in a non-NixOS environment; and I'd deploy K8s after that if only because it's what I know, but it is a lot.

I use nomad and it's pretty close to trivially easy. Definitely boring once running. The next person who asks me to open source the saltstates I use for my single-node nomads will make it happen.

This seems very boring to me and got into my list of solutions to consider for problems: https://github.com/containrrr/watchtower

This also seems interesting and there was a lobste.rs post about it recently https://kamal-deploy.org/ https://lobste.rs/s/sa7vbo/ruby_on_rails_37signals_future_web https://lobste.rs/s/02njfl/kamal_2_0_released https://lobste.rs/s/gax96g/supercharge_one_person_framework_with https://lobste.rs/s/zhzzjz/elixir_clustering_on_kamal_hetzner

I've used dokku for 10+ years in various projects. It's reliable and simple. (And yes, you can use it with pre-built images.)

https://dokku.com/

Docker compose, though Kubernetes is kinda starting to win a lot of the remaining mindshare.

Single host? Just use docker-compose. What makes you think it's not a good idea? It's extremely straightforward and works great.

I run 20+ containers on my home lab server. It used to be docker-compose, but it was never working properly. I had to ssh to restart a container once a month or something.

Nowadays it is systemd+podman. It took me a while to find a good unit file template, but I never had any problems with it since. The best part is that I can put mount dependencies in my unit files and things still work reliably if zfs pool failed to mount after power crash (the only real source of instability I’m having).

At my $workplace, we have an environment with a predictable workload running on plain old VMs. Each VM has a single Podman/Quadlet unit that manages the start of a stateless container. Manually like a sidecar inside the vm we also deploy in the same way a couple of containers: node_exporter (expose vm metrics for prometheus) and promtail (that sends journald logs to loki).

We follow a DNS naming convention with app.env.local for the application/service and machine names like env-app-01.domain and env-app-02.domain. Most of the VMs in use are part of a two-node setup.

Load balancing is handled by a plain old proxy/balancer, which was originally done with Apache's mod_proxy_balance and is now managed by HAProxy, as the container is also deployed via systemd/Quadlet on a dedicated VM.

We have two primary balancers: one for internal "horizontal" traffic (service to service) and another for external "vertical" traffic (exposed service etc). Third party paid Web App Firewall is in front of the external proxy.

As a "docker swarm single node" we have a Grafana/Prometheus/Loki stack.

Currently, there isn't a direct replacement for Docker Swarm in a single-node setup.

I'm planning to try "podman compose", which is a community tool that could help in this situation. However, I’ve read there may be issues with the new Aardvark network stack, which replaced the previous slirp4netns networking stack in Podman,

All the db are as-a-service from the infra provider.

For a small multi-node setup for on-premise deployment, my first choice would be using K3s especially if I wanted to manage the control plane, runner up for mixed on-premise and cloud: ECS Anywhere; In the cloud I would go to ECS since we are alredy on the aws bandwagon.

Deployments and updates are handled by Jenkins jobs, which typically execute an Ansible playbook.

I think this is all pretty boring and classic compared to kubernetes, and our IT people is happy because you can go to the vm, lookup the service name, find the logs all in a old fashioned way.

The only configuration for a new service besides creating the new vm is the proxy config, but pretty straightforward and fast to do.

I run 2 production systems, Incus for my personal setup and Nomad at $WORK. Both are great, though at $WORK, we do multi-node Nomad.

incus which does VM's and Docker. For instance, this is how I run Redis and what was once Firefox Send on my machine:

incus launch docker:redis docker-redis

incus launch gitlab:timvisee/send:latest docker-send -c environment.REDIS_HOST=$IPADDRHERE -c environment.FILE_DIR=. -c environment.DETECT_BASE_URL=true -c environment.CUSTOM_FOOTER_TEXT="admin email $MYEMAILADDRESS"

Be sure to read the docs on running docker images under incus, there is a little setup before the above commands will just work. Incus can expand to multi-node and grow as needed.

For Nomad it's 119 line file to run Send and 148 line file to run Redis. (That's including comments) a lot of it is boilerplate, the core of it looks like this(which also uses consul for service discovery):

        task "send" {
            # Use Docker to run the task.
            driver = "docker"

            # Configure Docker driver with the image
            config {
                #image = "mozilla/send:v3.0.21"
                # forked and maintained here: https://github.com/timvisee/send/blob/master/docs/docker.md
                image = "registry.gitlab.com/timvisee/send:v3.4.20"
                }
            meta {
            }
            env {
                REDIS_HOST="redis.service.consul"
                PORT=1443
                NODE_ENV="production"
                BASE_URL="https://send.$WORKDOMAIN"
                }
            service {
                name = "send"
                tags = ["send", "development"]
                port = "send"
                check {
                    type = "http"
                    port = "send"
                    interval = "10s"
                    timeout = "2s"
                    path = "/__heartbeat__"
                }
            }

Both work great and are very reliable, no complaints. If you may want to also run VM's then I'd recommend Incus over Nomad. Incus can go multi-node, but I have no experience with it.

I've found the OCI spec to be a source complexity and overhead that I could get rid of. I've essentially co-opted debian as a distribution platform for dependencies and I distribute my applications to VMs as tarballs, with contents that conforms to an activation script. Each VM runs a "root" installation of debian, with a second "sub" installation of debian using debootstrap. I develop my software against debian, sometimes against the debian OCI images. Out of that comes a list of debian package dependencies. The root debian installation has a script to merge the dependencies of all applications deployed and install those into the sub-installation. Then each application is run with bubblewrap, via systemd from the root-installation with a non-privleged user, with read-only bind-mounts to the main sub-installation directories like /usr and /lib. The purpose of that sub-installation is to facilitate multiple sub-installations, sometimes for bug hunting and sometimes for migrating to a new debian version. What I think makes this boring is how much it achieves process isolation without changing how I use the software (usually locally) where I don't require process isolation. It also effectively requires me to make sure the software can run without admin privileges.

Big caveat: This has not passed a significant security review yet.

Some things about Podman I have not seen written in other comments:

• Development is very active and recent versions add a lot of new features and fix bugs. For example, Quadlet (the feature that makes it possible to generate systemd units by writing files which define your resources with a syntax close to systemd itself) got new features in 5.3.0.
  • This means that you would rather use a Linux distribution that ships with the latest version of Podman - but be prepared that it could break if the version is not pinned and you don't test upgrades. Fedora ships with up-to-date versions, unlike with Ubuntu, for example.
• If you use Podman as root, you won't need to deal with iptables/netfilter, as published ports are bound to addresses just like any other process, unlike Docker which usually bypass your firewall rules. (Apparently Podman can use iptables under the hood in some situations, though.)
• Podman support rootless setups, with user and network namespaces. It means that you can create separate users for your apps and "root" in a container will have the UID/GID of your user on the host and other UID/GID are defined using unique ranges for each host users (in /etc/subuid and /etc/subgid). Though, as users can't directly bind to ports 80 and 443, you'll have to find a solution based on reverse proxy, iptables, CAP_NET_BIND_SERVICE or some magical networking thing (as root on the host you can theoretically access all user network namespaces...).
• You can use Docker Compose with Podman rootless just fine, but you need this:

  systemctl --user enable --now podman.socket
  export DOCKER_HOST="unix://${XDG_RUNTIME_DIR}/podman/podman.sock"
  

  I use this setup on my work laptop and I haven't had any issue. You could even use the Docker cli with this environment variable (I sometimes make the mistake as Docker got installed along with Docker Compose). Podman implements most of the Docker HTTP API for compatibility reasons but it has its own HTTP endpoints for the Podman cli.
• The default network created by Podman only supports IPv4, you need to create your own to support also IPv6.

I've used docker on single-node hosts for several years and employing different strategies. I've used compose, I've used Ansible, and I've used bash scripts. This has all been "boring" to me in the sense that there haven't been issues that I cannot attribute to something other than my misunderstanding of OCI containers and Docker specifically as I learned their ins and outs.

These days I'm transitioning to Podman for rootless containers and pods. As for managing the containers, I'm exploring either continuing to use Ubuntu for tighter integration with systemd using Quadlet, or Alpine Linux as a minimal host with services configured using OpenRC.

Do note that Docker did recommend against running compose in production at one point, but that's no longer the case. For most single node setups, compose is good enough unless you require complex deployment strategies, replication, etc.

There's also kamal, but I've not used it personally.

I would say incus. I've been very impressed with its support and you can even run docker within a container which is even cooler!

I use Portainer - in fact I just migrated 5 containers from one machine to a new one w/no drama.

I use Docker Swarm for single VM deployments with multiple containers, both at work and my home server. I've been very happy with it. It meets the boring description in that it doesn't have loads of features compared to other orchestrators. You basically get the same service, network, secrets model as compose. The reason reason for choosing it over others for me was because it's easy to pick up for people to know compose, and I couldn't expect everyone who'd be using it to know/learn something with a steep learning curve.

You can configure it using docker-compose.yml files (via the docker stack command-line) or manually with docker service xxx. It's easy to learn for people used to compose because most of the concepts map across. You have an scale up path to make your swarm cluster multi-node if you need more capacity, but single machine cluster works well.

I use Traefik to load balance to containers and terminate https. You configure labels on services to configure how services are exposed publicly by Traefik.

Over compose you gain 0-downtime deploys, docker's daemon-managed secrets, on-the-fly scale up/down of service instances (but not autoscaling, you have to tell it to change).

On the downside, the error reporting when it fails to deploy a service change is not very clear. Generally if you're using the cli to deploy a stack config (compose file) you don't get error messages there, just that it's rolled back. You have to docker service inspect xxx to find status info in the service data. This might happen if you have an auth failure pulling images, or an image has the wrong arch for example. Also if a container fails to start it can be difficult to debug, because the failed containers get cleaned up quite quickly by swarm. Finally, service containers have more restrictions on privileges than regular containers — you can't just give them host permissions for special cases that require it.

There's a site which has some decent info on configuring swarm deployments. Note that the author now suggests using Kubernetes because it's so widely adopted. OTOH, because swarm is very close to compose, if swarm was ever to be killed off, you could relatively easily use compose again. https://dockerswarm.rocks/

I have several personal softwares in a VPS with docker-compose in a Linode VPS. Everything worked quite smoothly and no issue. The only exposed container is a caddy instance that act as RP to the internal containers exposing web applications.

What are the corner cases that you are mentioning?

nixos provides a declarative interface: https://search.nixos.org/options?channel=unstable&show=virtualisation.oci-containers.containers&from=0&size=50&sort=relevance&type=packages&query=oci-containers.containers

I think you best option is Podman + Quadlet + their Ansible modules. It is not a drop in replacement for Docker. It works best if you update your app to use Systemd APIs like sdnotify and socket activation. The K8s manifest format can be read by podman without K8s, and you don’t even have to use that.

Running Compose mostly works. I would not recommend using their remote contexts; just copy the compose file onto the server and run it from there. Contexts have footguns where some commands still work from the server but others do not. And the environment variable resolution is very unintuitive. Secrets need to be on disk for Compose to read them. You need to write a systemd timer unit to cleanup old container images. Set the “name” explicitly in the compose file so that it doesn’t depend on the parent directory.

Zero downtime deploys are a challenge with either approach. Some options: Systemd socket activation (Podman only), nginx-proxy, Traefik (heed warnings about docker socket), DIY scripts.

Kamal is too new to be boring but solves a lot of this.

I am doing single node Kubernetes with microk8s

Hashicorp Nomad is quite nice. I've used it for several production environments at moderately high scale, and it also shrinks nicely to a single box.

I have some stuff that's been running in a high-traffic production environment with docker compose for years at this point. It is rock solid.

I have a production system that uses just docker with several containers setup on a couple VMs. Very basic containers. No docker compose but not for any good reason. It uses a Docker systemd unit. The containers restart on reboot. Watchtower for deploy of new images. A 100k or so users. These containers had slow changes of their data and fronted by a CDN. We had problems with iptables/netfilter. There are several different solutions to this. Very boring OTOH I do think of this as something I'll move off of in the future.

Is it minikube?

Selfhosting is a big topic at pico.sh since this is where we see ourselves fitting in the best. For our production systems, we use docker compose with great success.

This production system also leverages services that we self-host in our own homelabs using tuns.sh and for our static sites we lean heavily on pgs.sh. For any comms between services that live outside of our databases we lean heavily on pipe. Basically, we are trying to lean on managed services while still having all of our containers running on our own VMs with docker compose and basically making everything but running containers as simple as possible to manage.

Do you want to rearchitect when you need multinode? Boring is boring for as long as it works. Multinode isn't always about scale, it's often about reliability.

Do you currently control the VM configuration somehow? What you have is always more boring than something new.

Do you have any networking/proxy requirements, or do things just directly hit the port on the app? The most boring network and SSL certificates make the world more boring.

Do your apps have state? The most boring solution for state makes the world very boring.

People who are architecting your production environment with containers on a VM

Not exactly an answer to your question, but: at $DAYJOB we don't do this, and having experienced many production environments that do, in a variety of roles (from "keeping it running" to "explaining to the exec team why it's hard to keep running") I'm very happy we don't.

I've used Ansible to run a production cluster which ran multiple docker containers on multiple machines. Works, probably will use it again if needed, but not painless IMO. It's as "boring" as it gets: a Django app, so nginx fronting a bunch of sub containers all within a specific docker network on x machines, talking to a pg hosted on another machine, proxied to by pgbouncer deployed via similar setup.

The most I had to fight with were the weird oddities around "update" and "restart" semantics. There was no docker registry here, so I had to export and load the docker images onto each of those machines. Perhaps this entire ordeal was made slightly easier since we have a load balancer and firewalls pre-setup that helped me avoid fiddling with some security stuff at least.

I prefer not using the latest images, but actual releases instead. I currently use diun to notify me when a new image with the desired format (e.g. SemVer) is released. I'm mostly happy with this, but I'm wondering of others have similar but better solutions?

In some cases one can avoid containers altogether (kinda like Cloudflare does with V8 isolates). https://smallweb.run/ is a good example of this.

If you’re really dedicated to a Boring Technology approach, you should really consider whether you should be running the containers in a VM at all. I understand that there are situations where this is necessary, but in my experience people can believe that they need to control the underlying infrastructure in cases where in fact they do not. So, for the OP, and for posterity in this thread, do consider whether something like AWS Fargate is enough for you. The point of Boring Technology is that it really is boring. The point of Boring Technology is that (often) it will feel like de-skilling. So why not take a long hard look at your thoughts about your production system and think “perhaps a boring managed solution could be all I need”.

Dokku?