Upgrading your mastodon instance

As mastodon v3.3.0 has been released for over two months now, I figured it was time to upgrade my family instance from v3.2.1 to v3.3.0 (thereby skipping v3.2.2).

I wrote about hosting your own instance, via docker-compose, in a previous post. The past week I’ve been tinkering with mastodon development. Considering what a pain it has been to setup a local development environment, I remain convinced that a containerized setup is the way to go for hosting mastodon. Let’s get started!

Documentation

Before starting the upgrade, I would suggest going through the documentation on ‘Upgrading to a new release‘ and through the v3.3.0 release notes. Together with the notes here, you should be all set for a successful upgrade.

Database backup

Prior to attempting an upgrade, you should ensure you have a recent backup of the mastodon database. I used pg_dumpall inside the running postgres container (whose docker-compose service is named db) as follows:

docker-compose exec db /bin/bash
pg_dumpall -U mastodon -f mastodon_db_pg_dumpall_output_20210309.sql
tail mastodon_db_pg_dumpall_output_20210309.sql

If the backup was successful, tail should output:

--
-- PostgreSQL database dump complete
--
-- PostgreSQL database cluster dump complete

Make sure you store the resulting sql file in a location that is persisted on the host system, so that the file isn’t lost when the container is shutdown and removed. In the container, I moved the backup file to /var/lib/postgresql/data, which is mounted on the host at /home/mastodon/mastodon/postgres. On the host I moved the file to /home/mastodon.

Backup public/system folder

While not necessary, you might want to backup the files under mastodon/public/system, e.g. excluding the cache folder via

tar --exclude=public/system/cache -zcf mastodon_system_backup_20210309.tar.gz public/system/

Upgrading to new release

After completing the database backup, the steps necessary to upgrade to a new version are roughly as follows:

Update your local copy of mastodon to the latest release
Rebuild the container images
Apply pre-install migrations of the mastodon database
Shutdown containers running old release of mastodon
Clear mastodon cache and finish DB migration
Start newly built containers
Prune old containers

In shell commands, make sure you point your working directory to the root folder of the mastodon repo:

git fetch --tags  # fetch all tags from the remote, leaving the working directory intact
git stash # stash any changes made to the working directory (typically only docker-compose.yml file)
git checkout v3.3.0  # checkout working directory to the v3.3.0 release
git stash pop # restore any changes you made to the working directory (i.e. passwords in docker-compose.yml files, verify via git diff)
docker-compose build  # build the web, streaming and sidekiq services, they all share the same container image. This may take a while.
docker images  # you should see the freshly build image at the top, something like: tootsuite/mastodon   latest              aa82ba57a3f4        About a minute ago   1.98GB
docker-compose run --rm -e SKIP_POST_DEPLOYMENT_MIGRATIONS=true web rails db:migrate  # Run the pre-deployment database migrations
# You can check the return code via echo $?, if the rc is zero than the migration was successful
docker-compose down # shutdown all docker-compose services
docker-compose run --rm web bin/tootctl cache clear # clear cache
docker-compose run --rm web rails db:migrate # Run post-deployment database migrations, again verify that it succeeded via echo $?. 0 means success
docker-compose up -d  # create and start containers, detaching from their stdout
docker system prune -a # remove unused docker data (images, containers, volumes)

Verify that the containers are up and running via docker-compose ps and docker ps. You should see something like:

[mastodon@sarch mastodon]$ docker-compose ps
        Name                      Command                  State                     Ports
---------------------------------------------------------------------------------------------------------
mastodon_db_1          docker-entrypoint.sh postgres    Up (healthy)
mastodon_redis_1       docker-entrypoint.sh redis ...   Up (healthy)
mastodon_sidekiq_1     /tini -- bundle exec sidekiq     Up             3000/tcp, 4000/tcp
mastodon_streaming_1   /tini -- node ./streaming        Up (healthy)   3000/tcp, 127.0.0.1:4000->4000/tcp
mastodon_web_1         /tini -- bash -c rm -f /ma ...   Up (healthy)   127.0.0.1:3000->3000/tcp, 4000/tcp

[mastodon@sarch mastodon]$ docker ps
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS                   PORTS                                NAMES
3f7ea048f4f7        tootsuite/mastodon    "/tini -- bash -c 'r…"   4 minutes ago       Up 4 minutes (healthy)   127.0.0.1:3000->3000/tcp, 4000/tcp   mastodon_web_1
c673502e6afa        tootsuite/mastodon    "/tini -- node ./str…"   4 minutes ago       Up 4 minutes (healthy)   3000/tcp, 127.0.0.1:4000->4000/tcp   mastodon_streaming_1
1bba8a9e1934        tootsuite/mastodon    "/tini -- bundle exe…"   4 minutes ago       Up 4 minutes             3000/tcp, 4000/tcp                   mastodon_sidekiq_1
6c623213428d        redis:6.0-alpine      "docker-entrypoint.s…"   7 minutes ago       Up 7 minutes (healthy)                                        mastodon_redis_1
f65c1f822600        postgres:9.6-alpine   "docker-entrypoint.s…"   7 minutes ago       Up 7 minutes (healthy)                                        mastodon_db_1