Currently due to some issues with the S3 recovery we've got a bit of an issue with the Fedora restore so this information is current as of 7/5/16.
Backup
If something goes wrong, go to
EC2→Snapshots
Then select the snapshot(s) you want to restore (opt and fed_data) and you have a new opt and fedora setup disk.
Run Ansible and capistrano to build a fresh machine and when it comes up, turn it off. Detach the automatically generated /opt disk (/dev/xvdg or /dev/sdg by name).
Attach the restored opt and fedora disk.As of 7/21 a test using Staging data worked well with the S3 storage method and things are back in gear.
Recovery Options
Fedora:
S3:
Currently we are using the s3 sync tool (akin to rsync for S3) to pull over key fedora data into the chf-hydra-backup bucket. This is a slight misnomer as it handles backups for ArchivesSpace as well now, but Fedora data is pulled over into
FedoraBackup (contains all Fedora data)
TomcatConfig (contains the Tomcat configuration and levelDB)
Both sets are needed to do a full restore.
Note: As a reminder while S3's visual interface uses folders, those locations are actually just the first step in a path of individual block stored objects.
Download both sets of data to the new machine. You'll want to make sure FedoraBackup goes to a disk with enough space to handle all of our data.
Stop Tomcat
Replace the default fedora location (/opt/fedora-data) in a new Ansible machine with the copied data from FedoraBackup.
Then go to /var/lib/tomcat7/webapps/ and replace the fedora folder with the fedora folder downloaded from TomcatConfig
This will migrate the Fedora database. You will need to finish bringing over the Users and Minter state for a full restore.
Users:
Go to S3 and download the minter-state and postgres backup files.
Move Minter state to it's new location. (/var/sufia) Note: You may need to make /var/sufia with sudo mkdir /var/sufia if it does not exist yet.
Change ownership for the minter state to the owner (hydep) and the group (deploy)
...
If Tomcat is not stopped, stop Tomcat.
Restart the postgres service, this should remove the default connection to the Hydra database that Hydra has when running so you can change it.
In Postgres delete the automatically generated chf_hydra database
...
The postgres account password is in ansible-vault (groupvars/all)After that restart postgres,
You may now restart postgres and Tomcat.
Minter:
Go to S3 and download the minter-state backup files.
Move Minter state to it's new location. (/var/sufia) Note: You may need to make /var/sufia with sudo mkdir /var/sufia if it does not exist yet.
Change ownership for the minter state to the owner (hydep) and the group (deploy)
Command: sudo chmod -R hydep:deploy /var/sufia
Indexing:
Finally you will need to follow the instructions for Reindexing Solr in Hydra under Application Administration to index all the data so it shows up in Hydra.
Note: I found that a few files with versioning got missed on the first reindex so you may want to run it a second time if some data is missing in Hydra.
Last steps:
Finally restart Tomcat, and Apache.
Snapshots:
If something goes wrong, go to
EC2→Snapshots
Then select the snapshot(s) you want to restore (opt and fed_data) and you have a new opt and fedora setup disk. If the damage is just something local you can reattach these snapshots.
If a new machine is needed
Run Ansible and capistrano to build a fresh machine and when it comes up, turn it off. Detach the automatically generated /opt disk (/dev/xvdg or /dev/sdg by name).
Attach the restored opt and fedora disk.