Page Comparison

...

Currently we are using the s3 sync tool (akin to rsync for S3) to pull over key fedora data into the chf-hydra-backup bucket. This is a slight misnomer as it handles backups for ArchivesSpace as well now, but Fedora data is pulled over into

FedoraBackup (contains all Fedora data)

TomcatConfig PGSql (contains the Tomcat configuration and levelDBFedora Postgres database)

Both sets are needed to do a full restore.

Note: As a reminder while S3's visual interface uses folders, those locations are actually just the first step in a path of individual block stored objects, folders do not exist in S3.

Download both sets of data to the new machine. You'll want to make sure FedoraBackup goes to a disk with enough space to handle all of our data.

Stop Tomcat

...

1. Stop Tomcat
2. Download the Postgres database fcrepo_backup.sql to the Fedora machine.
3. Import the database psql fcrepo < fcrepo_backup.sql
4. Check that the user trilby has
5. Delete the existing folder(s) inside /opt/fedora-data (This step is not always required but makes it simpler)

...

2. Using screen or tmux start an aws s3 sync to copy all the data over in the FedoraBackup "folder" to /opt/fedora-data
3. Wait a while for all the data (>800 GB) to copy over.
4. Restart Tomcat
5. This will migrate the Fedora database

...

1. .

...

Users:

1. Go to S3 and download the postgres backup files.

...

1. Stop Apache
2. Restart the postgres service, this should remove the default connection to the Hydra database that Hydra has when running so you can change it.
3. In Postgres delete the automatically generated chf_hydra database
   1. Log in via

...

1. 1. psql -u postgres

   and run

   1. 1. 1. The postgres account password is in ansible-vault (groupvars/all)
      2. Run: DROP DATABASE chf_hydra;

   make a blank database chf_hydra

   1. 1. Run: CREATE DATABASE chf_hydra;
   2. Then import the downloaded database
      1. Either:
         1. pg_restore -d chf_hydra -U postgres chf_hydra.dump

   ...

   1. 1. 1. psql chf_hydra < chf_hydra_dump.sql
   2. Then set permissions

   ...

   1. 
      
      1. psql -U postgres

   Once logged in run

   1. 1. GRANT Create,Connect,Temporary ON DATABASE chf_hydra TO chf_pg_hydra;

   The postgres account password is in ansible-vault (groupvars/all)

   1. You may now restart postgres and

   ...

   1. Apache2

    

   Minter:

   The minter is now part of postgres, just restore the chf_hydra database for the users to app and the minter will be restored. 

   
   

   Redis

   Redis keeps a database in memory which handles , as far as we currently know (8/1/16), the transaction record data such as the history of edits on a record. It does not contain the actual data, simply the timeline of changes. Losing this causes the history of object edits to be lost, but the objects themselves will be fine.

   1. Copy over redis-dump/dump.rdb

   ...

   1. to App
   2. It must be changed to be owned by redis

   ...

   1. 1. sudo chmod -R redis:redis filename
   2. Then you will need to stop the redis server

   ...

   1. 1. sudo service redis-server stop

   ...

   2. Move redis-dump.rdb to /var/lib/redis/dump.rdb

   ...

   1. overwriting the existing file there called dump.rdb

   ...

   2. Restart redis

   ...

   1. 1. sudo service redis-server start

   ...

   2. When starting up redis will read the .rdb dump file and copy that data back to the in memory database.

   ...

   
   

   Indexing:

   The index is being backed up to speed up the time to recovery for DR or migrations.
   Finally you will need to follow the instructions for Reindexing Solr in Hydra under Application Administration to index all the data so it shows up in Hydra.

   Note: I found that a few files with versioning got missed on the first reindex so you may want to run it a second time if some data is missing in Hydra.

    

   Last steps:

   Finally restart Tomcat, and Apache.

    

    

   Snapshots:

   If something goes wrong, go to

   EC2→Snapshots

   Then select the snapshot(s) you want to restore (opt and fed_data) and you have a new opt and fedora setup disk. If the damage is just something local you can reattach these snapshots.

   If a new machine is needed

   Run Ansible and capistrano to build a fresh machine and when it comes up, turn it off. Detach the automatically generated /opt disk (/dev/xvdg or /dev/sdg by name).

   Attach the restored opt and fedora disk.If you cannot access it, a manual reindex can be done with the instructions in Application administration. This process takes at least one business day, so is not recommended versus rebuilding from the backup.

   1. In the chf-hydra-backup, pull down the solr-backup.tar.gz file under Solr to the Solr server.
   2. Extract the archive
   3. Use the solr restore commands at Application administration
       

   
   

   Last steps: