Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 7 Next »

Better known as: Backups without Ansible.

We are considering getting rid of our Ansible-managed servers as we move to Heroku. This means that we can no longer rely on certain backup mechanisms which were performed by cron jobs installed by Ansible.Backups and Recovery contains a summary of our pre-Heroku backup infrastructure.

Original files and derivatives

These are stored in S3, are backed up within S3 by a process managed by AWS, and are pulled from S3 to Dubnium. No Ansible cron jobs are used in this workflow, and and there is thus no need to make any changes to our existing setup.

Database backups

A script on the production server, home/ubuntu/bin/postgres-backup.sh, performed the following tasks nightly:

  • pg_dump the production database to /backups/pgsql-backup/.

  • aws s3 sync the contents of that directory to s3://chf-hydra-backup/PGSql.

The above script will need to be discarded.

A second cronjob running on our dubnium backup server (/home/dsanford/bin/scihist-digicoll-backup.sh) then copies the S3 file to a local network storage mount (/media/SciHist_Digicoll_Backup). This then gets backed up to tape.

Heroku database backup commands

It’s easy to setup a regular database backup in Heroku, as follows:

heroku pg:backups:schedule DATABASE_URL --at '02:00 America/New_York'

You can check the metadata on the latest backups like this: heroku pg:backups
heroku pg:backups:download a006 will produce a “logical” database dump – a binary file – that can easily be converted to a “physical” (i.e. garden variety SQL file) dump as follows: pg_restore -f mydatabase.sql latest.dump.

Heroku retains daily backups for 7 days, and weekly backups for 4 weeks. (more details re: retention schedule)

Additional backup to S3

We intend to replace the Ansible-managed scriptpostgres-backup.sh with a rake task, rake scihist:copy_database_to_s3, run regularly on a one-off Heroku dyno. This downloads the production database to a temp file, then uploads that to s3, where it can wait to be harvested by the Dubnium script.

Given the size of the database in late 2020, the dump takes 13 seconds, and the upload another 13. The entire job (with the overhead of starting up the dyno and tearing it down) takes a bit under a minute.

  • No labels