Server Migration (Obsolete)

Owned by Former user (Deleted)
Last updated: Jul 30, 2021 by Eddie Rubeiz
2 min read

When migrating servers (Production or Staging) follow these steps. In order to reduce costs, it is recommended to not use S3 backups or the public IP addresses unless it is noticeably easier. Instead use the internal IPs to transfer data (usually 172.xxx...) to send data back and forth avoids any fees for out of AWS/region transfer.

Some of this should be scriptable with variables, look into setting up a playbook.
A script for this was tested (Feb 2018) for syncing production and staging with fair success. It has not (2/28) yet been added to ansible.

Data required in migration:

Samvera PGSQL database (located on app): Containers user accounts, minter, and search history

Redis DB (located on app): Stores work history on objects

Fedora PGSQL database (located on fedora): Contains RDF data about objects

Fedora binary files (located on fedora): Binary resources uploaded to disk.

Data useful for migration:

Solr index (located on solr): Can be recreated, but using a backup cuts time from hours of indexing to minutes to restore.

Data off servers:

Deep Zoom Images (S3): Will not need to be moved if migration is within the same tier of service (new Production machines) but if you want to sync data to staging, they will need to be copied to Staging's bucket.

Derived Images (3): Will not need to be moved if migration is within the same tier of service (new Production machines) but if you want to sync data to staging, they will need to be copied to Staging's bucket.

Both of those can be, if needed, recreated. This does take time and should be avoided.

Note: There are millions of DZI tiles. Copying or loading all them is expensive. Try and avoid doing it often.

  1. Backups on S3 will be from the night before. If doing this in a morning before others work you'll be current. Otherwise you may wish to check the time and see if a new backup is needed.
  2. Migrating can be done while people are using the site, but staff should not be making changes. Otherwise you will need to restore data a second time due to changes.
  3. The quickest order to get machines up is solr, app, and fedora. With derived images and DZI buckets, the lack of Fedora only impacts the ability to edit/ingest objects and download tiffs.
  4. In addition to data, you will need to adjust 3 other things
    1. If you are making new boxes for use, copy over the SSL certificate.
    2. As with part a, once the boxes are ready you will need to migrate the Elastic IPs over.
    3. Either update the internal IPs in ansible so they can be deployed, or assign the same internal IPs to the new machines. You will need to decommission the old machines this way. The preferred option is to deploy the changed addresses with ansible.

See steps on Backups and Recovery (Historical notes) for how each bit of data should be restored.