Scihist_digicoll Backup and Recovery

Our Analysis and Motivations: How we thought through what backups to make, and why

The 2019 version of our digital collection application (scihist_digcoll) has slightly different preservation needs than the old Samvera application it was based on. However, our institutional goals remain the same. While we will be making technical changes to preservation methods, our backup and recovery goals are still linked to the following key points in order of importance:

...

Finally in cases of issues affecting all on-line storage systems, another copy of the data is held on our in-house storage system so that we can potentially recover data even in case of a full loss of AWS. This only holds the original files and all other aspects will need to be rebuild, a process that can take days in addition to the time taken to upload the files again. Using the local backup means recovery could take a week or more. This is not paid for/managed by our team as Institute IT handles these systems.

Technical Notes

As of March 2019, the Digital Collections has a small set of data to be handled for recovery.

...

For each of these, we have TWO levels of backups: 1. in an S3 bucket, 2. on local on-premises Institute storage, which is also backed up to tape. 

What Backups Exist

• postgres database
  • is backed up by a cronjob scihist-digicoll-backup.sh running on the database server, to S3 disk at: s3://chf-hydra-backup/PGSql/digcol_backup.sql
    • This S3 bucket is "versioned" and keeps 30 days worth of past versions. 
  • cronjob running on "dubnium" backup server then copies that nightly from that S3 location to local network storage mount. So the network storage mount will nightly ensure latest copy is at standard location, to be backed up to tape.  
• Original ingested assets (binary files)
  • use S3 replication rules to replicate to a bucket scihist-digicoll-production-originals-backup
    • This is intentionally in a different AWS region, US West (Oregon)
    • The replication rules do not replicate deletes, so deleted files should still exist in the backup bucket
    • Both the live production bucket and this backup bucket are versioned, and keep 30 days worth of past versions. We do not keep complete version history, the backups are intended mostly for handling corruption and disaster, not recovering from user or software error. Although 30 days of version history allows some limited recovery from software or user error. 
    • a cronjob scihist-digicoll-backup.sh running on "dubnium" backup server then uses an "rsync" like functionality to sync to a on-premises network mount, which is backed up to tape. 
    • In a disaster recovery scenario, the live app could temporarily be pointed to use bucket scihist-digicoll-production-originals-backup as it's source of files. However, cross-region data transfer will be more expensive than usual same-region.  You may want to turn off ingest to prevent new data from being written to the backup bucket. 
• Derivatives
  • Both the "ordinary" derivatives bucket and the "dzi" bucket (used for "deep zoom" viewer) use S3 replication rules to replicate to backup buckets in US West (Oregon). 
  • In a disaster recovery scenario, the live app could be switched over to use these buckets as a source of data, but cross-region data transfer will be more expensive than same-region. You may want to turn off ingest to prevent new data from being written to the backup bucket. 
  • All of these buckets also keep 30 days of version history. 
  • scihist-digicoll-production-dzi => scihist-digicoll-production-dzi-backup
  • scihist-digicol-production-originals => scihist-digicol-production-originals-backup

Recovery Overview

Minimal public recovery requires the following data:

...

In the case of smaller issues, like single file corruption or deletion, the simplest method for original files is to locate them on S3 and look at previous versions. We keep 30 days worth of versions so if the error was found within a month you should be able to revert back to an earlier file. For derivatives it is easier to simply regenerate them via the command line.

Recovery Notes

Single File Recovery

If a single file (or small number of files) is missing or corrupted, recovery should be handled at the individual level. For derivative images, simply regenerate them. For originals use the following recovery procedures
Steps:

1. Go to the Work with the damaged files
2. Select the Members tab and click on the file name
3. If the Fixity Check shows an error it should have a link to the file in S3, if not you will need to get the UUID. This can be found via the rails console or by stripping it out from the Download Original or Derivatives links.
4. Log onto AWS.
5. If the Fixity check shows an error, simply click on the link. In the S3 Web console select the Versions Show option. If the fixity check does not show an error or the link does not work:
   1. Log into S3 and go to the bucket scihist-digicoll-production-originals. In the S3 web console, select the Show button for versions.
   2. Search for the UUID in the prefix
   3. Select the UUID "Folder"
6. If the file has changed, been deleted, or corrupted within the last 30 days you should see prior versions. If the fixity check has a date and time for when the file changed, you can simply select all the newer versions by clicking the check box next to them and then use the Actions button to Delete them. The old version will become the current version. Run a fixity check to confirm the fix. If it was deleted, you may see a "Delete Marker." Simply delete it like a file and the old file become the current version.
7. If the file version you need is not available from version history, it may be available on the backup bucket. This is called scihist-digicoll-production-originals-backup and is in the US-WEST region. Deleted files should remain in the backup bucket forever. But changes are replicated to backup bucket, so a version prior to a change made more than 30 days ago will not generally be available on backup bucket either. 
8. Confirm the damaged file is in scihist-digicoll-production-originals-backup, you may either use the S3 web console and going to the bucket then searching for the UUID to confirm the file in in that "folder" or you may use the AWS CLI or some other tool to make a head request.

9. If the file is there, you may sync it to the the scihist-digicoll-production-originals bucket. Make sure that scihist_digicoll thinks there is a file there already. Syncing a file that is not in the postgres database will not add it to the application. You may use any preferred sync method, here is an example via the AWS CLI SDK. 

   Code Block
   language bash firstline 1 title Sync
   aws s3 sync s3://scihist-digicoll-production-originals-backup/asset/UUIDHERE s3://scihist-digicoll-production-originals/asset/UUIDHERE --source-region us-west-2 --region us-east-1

   This sample will move the file from backups to the originals production bucket, inside the asset key (part of our application) and to the UUID location. If the UUID key is missing this will make the needed key. If you're unsure of a command, the --dryrun option allows for a safe test.

10. Run the fixity check to confirm the file is fixed.

S3 Outage or temporary use of backups

1. If US-EAST-1 S3 is down, or we have some issue where our normal S3 buckets are missing or empty we can temporarily use the backup buckets as a source for files.
   1. Before this is done, staff users cannot add any new works or files. New works added will be missing when we stop using the backups. Editing metadata is fine.
   2. While we use the backup buckets, we will be charged for inter-region data transfer. This can quickly add up, so the duration of this switch should be kept to a minimum
2. Go to the ansible codebase (ansible-inventory).
3. In the roles/kithe/templates/local_env.yml.j2 file
   1. edit s3_bucket_originals, s3_bucket_derivatives, and s3_bucket_dzi to add -backup to the end of their name (ex. scihist-digicoll-production-originals-backup)
   2. edit the aws_region to us-west-2
      1. this might break access to other buckets that we don't have an alternate of in us-west-2, oh well this process is not completely polished. 
   3. (optional but probably) edit or add logins_disabled: true to lock staff users out to not be making changes. 
   4. and then commit the change and merge to master so that the edit is pushed out to all the servers – you may have to wait 10-15 minutes for the changes to be deployed. 
   5. NOTE: Staging, lacking buckets called eg "scihist-digicoll-staging-originals-backup" will be be broken if deployed with these. 
4. Email all users about the need to avoid adding new works or files.
5. If the entire US-EAST region is down, the on_demand_derivates and upload buckets may also be unable to be used. This would prevent uploads and pdf/zip generation. If this is expected to last a long time, you may wish to create those buckets in US-WEST-2 and point the local_env file to these buckets.

Full Recovery from S3

1. A full recovery from S3 rolls the entire system back to the latest backup. This is usually the prior business day. All changes between this period will be lost.
2. As noted in the prior documentation, this involves a postgres database change, reindex of solr, and then moving files from backup to production.
   1. Only the originals files are required, but it may be faster to copy derivative files rather than regenerate them. (This requires testing)

3. Currently you can, in ansible, run the playbook restore_kithe.yml which will automatically handle these steps.

   Code Block
   title Ansible Playbook
   ansible-playbook --ask-vault-pass restore_kithe.yml --private-key=~/.ssh/chf_prod.pem

   
   

4. Do not run this unless you are prepared to lose all changes made to the system during the past 24 hours.
5. If you cannot or do not want to use the playbook or it does not work, you may manually undertake the following steps. (logged in as ubuntu)
   1. Stop passenger on the web server, this should end connections to the postgres database. It also allows you to avoid honeybadger errors when the database is lost.

      1. sudo systemctl stop passenger

   2. On the database server, restart the postgres service. This will terminate any hanging connections.

      1. sudo systemctl restart postgresql.service

   3. Download to the database server the last postgres backup, found at the s3://chf-hydra-backup bucket under PGSql key as digcol_backup.sql
   4. On the database server drop the existing postgres digcol database.
      1. dropdb -U postgres digcol
-or-
      2. psql -U postgres


         1. DROP DATABASE digcol;

   5. Import the backup database with

      1. psql -U postgres < BACKUP_LOCATION

   6. You will then need to reindex solr, which can be done remotely and will run on jobs

      Code Block
      language ruby title Rake tasks
      bundle exec cap production invoke:rake TASK="scihist:solr:reindex scihist:solr:delete_orphans"

      
      

   7. You'll need to move over any original files that are missing with a S3 sync command

      1. (using AWS credentials that have necessary access; developer accounts such as jrochkind and eddie should). 

         Code Block
         language bash firstline 1 title Sync
         aws s3 sync s3://scihist-digicoll-production-originals-backup/ s3://scihist-digicoll-production-originals/ --source-region us-west-2 --region us-east-1

         
         

   8. Then you need to restore/sync derivatives too. You can do the same aws s3 sync for the derivative files/bucket. 

      1. Or you can run lazy creation scripts, but they may miss some derivatives recorded in the postgres DB but missing on S3. (Not sure if this is an issue for DZI)

      2. If you run the rake task, ssh into the jobs server and move to the current deployed application directory (/opt/scihist_digicoll/current).

      3. Switch to the application's user (digcol) and then run the commands. Since they take a long time to run, it is best left in a screen or tmux session.

         Code Block
         language ruby firstline 1 title Derivative creation
         ./bin/rake kithe:create_derivatives:lazy_defaults ./bin/rake scihist:lazy_create_dzi

         
         

   9. Start passenger on the web server again to bring the app back up!

      1. sudo systemctl start passenger

Full Recovery from Local Backups

This has not been tested. Instructions here are general guides but do not have the step by step quality of our other instructions. This is to be run when we have lost all access to data in S3. Note this will likely take a long time so should expectations should be set with the project owner and library on how to notify people of the outage.

...