...
- A full recovery from S3 rolls the entire system back to the latest backup. This is usually the prior business day. All changes between this period will be lost.
- As noted in the prior documentation, this involves a postgres database change, reindex of solr, and then moving files from backup to production.
- Only the originals files are required, but it may be faster to copy derivative files rather than regenerate them. (This requires testing)
Currently you can, in ansible, run the playbook
restore_kithe.yml
which will automatically handle these steps.Code Block title Ansible Playbook ansible-playbook --ask-vault-pass restore_kithe.yml --private-key=~/.ssh/chf_prod.pem
- This will ask if you are sure you want to run it before it starts. If you type Y it will start the restoration process and quickly lose the data. Do not run this unless you are prepared to lose all changes made to the system during the past 24 hours.
- If you cannot or do not want to use the playbook or it does not work, you may manually undertake the following steps.
- Stop passenger on the web server, this should end connections to the postgres database. It also allows you to avoid honeybadger errors when the database is lost.
passengersudo systemctl stop passenger
- On the database server, restart the postgres service. This will terminate any hanging connections.
sudo systemctl restart postgresql.service
- Download to the database server the last postgres backup, found at the s3://chf-hydra-backup bucket under PGSql key as digcol_backup.sql
- On the database server drop the existing postgres digcol database.
dropdb -U postgres digcol
-or-psql -U postgres
DROP DATABASE digcol;
dropdb digcol
Import the backup database with
psql -U postgres < BACKUP_LOCATION
You will then need to reindex solr, which can be done remotely and will run on jobs
Code Block language ruby title Rake tasks bundle exec cap production invoke:rake TASK="scihist:solr:reindex scihist:solr:delete_orphans"
You'll need to move over any original files that are missing with a S3 sync command
Code Block language bash firstline 1 title Sync aws s3 sync s3://scihist-digicoll-production-originals-backup/ s3://scihist-digicoll-production-originals/ --source-region us-west-2 --region us-east-1
Then either do the same for the derivative files or regenerate them with a rake task.
If you run the rake task, ssh into the jobs server and move to the current deployed application directory (/opt/scihist_digicoll/current).
Switch to the application's user (digcol) and then run the commands. Since they take a long time to run, it is best left in a screen or tmux session.
Code Block language ruby firstline 1 title Derivative creation ./bin/rake kithe:create_derivatives:lazy_defaults ./bin/rake scihist:lazy_create_dzi
- Stop passenger on the web server, this should end connections to the postgres database. It also allows you to avoid honeybadger errors when the database is lost.
...