General outline

Move over minter
Fedora Export - see below
migrate postgres
Fedora Import - see below
run (currently nonexistent) verification job
migrate dump.rdb
Reindex solr

Spin up machine
1. Run ansible scripts
Creating Drives
1. In the AWS visual interface, go to EC2
2. Go to Volumes
3. Select Create Volumes
4. Make two volumes with the following features:
  1. General Purpose SSD
  2. 150 GB
  3. Availability Zone b
5. Once each one is made, select it and under Actions choose Attach Volume. Type the name or id of the machine and attach the volume.
6. ssh into the box
7. sudo fdisk -l
  1. You should see /dev/vxdg and /dev/xvdh
  2. If not, check if the volumes are attached
8. Create the filesystem for each disk
  1. sudo mkfs.xfs /dev/xvdg
  2. sudo mkfs.xfs /dev/xvdh
9. Mount each disk
  1. sudo mount /dev/xvdg /opt/fedora-data
  2. sudo mount /dev/xvdh /opt/sufia-project/releases/XXXX/tmp
10. Edit the fstab file to retain these mounts
  1. sudo vi /etc/fstab
  2. /dev/xvdg /opt/fedora-data xfs defaults 0 0
  3. /dev/xvdh /opt/sufia-project/releases/XXXX/tmp xfs defaults 0 0
11. Change the owner of the two mount locations
  1. sudo chown -R tomcat7:tomcat7 /opt/fedora-data
  2. sudo chown -R hydep:deploy /opt/sufia-project/releases/XXXX/tmp
Deploy Sufia
Solr
1. If this is the first time sufia has been deployed, Solr now runs outside of tomcat and needs to be restarted after deployment.
  1. sudo service solr restart
Spin up machine
1. Run ansible scripts
2. Box Build changes
3. The current build scripts in Ansible have problems with a few changes we need.
Activate maintenance mode on old server
Export Fedora data (in sufia 6 instance)
1. Run audit script (takes 4 or 5 mins)
  1. RAILS_ENV=production bundle exec sufia_survey -v
2. Make sure you have the latest deployment
3. Make sure tmp/export is empty
4. Run json export (takes < 10 mins)
  1. $ RAILS_ENV=production bundle exec sufia_export --models GenericFile=Chf::Export::GenericFileConverter,Collection=Chf::Export::CollectionConverter
5. Open up fedora port to the other server so it can grab the binaries
6. Change all the 127.0.0.1 URIs to reflect internal IPs, e.g.
  1. $ find tmp/export -type f -name "*.json" -print0 | xargs -0 sed -i "s/127\.0\.0\.1/[internal_ip_of_prod]/g"
  2. The internal IP of prod is: 172.31.48.168
  3. The internal IP of staging is: 172.31.58.101
7. Move the resulting directory full of exported data from tmp/export to the new server's tmp/import (or wherever desired; this can be provided to the import script)
  1. $ cd tmp; tar -czf json_export_201611141510.tgz export
8. Then from your own machine:
  1. $ scp -3 -i ~/.ssh/test.pem hydep@staging:/opt/sufia-project/current/tmp/json_export_201612141435.tgz hydep@new_box_ip:~/.

Migrate postgres

Run the following to generate the export.

pg_dump -U postgres chf_hydra -Fp > chf_hydra_dump.sql

Copy the file to the new machine

scp -3 -i ~/.ssh/test.pem ubuntu@production_ip:~/chf_hydra_dump.sql ubuntu@new_box_ip:~

Import the file

psql -U postgres chf_hydra < chf_hydra_dump.sql

Deploy chf-sufia to new server
Create Drives
1. In the AWS visual interface, go to EC2
2. Go to Volumes
3. Select Create Volumes
4. Make

sure the volume is

1. two volumes with the following features:
  1. General Purpose SSD
  2. 150 GB
  3. Availability Zone b
Create 2 of these
1. Once each one is made, select it and under Actions choose Attach Volume. Type the name or id of the machine and attach the volume.
2. ssh into the box

Run

1. sudo fdisk -l
  1. You should see /dev/vxdg and /dev/xvdh
  2. If not, check if the volumes are attached
2. Create the filesystem for each disk
  1. sudo mkfs.xfs /dev/xvdg
  2. sudo mkfs.xfs /dev/xvdh
Mount each disk
1. sudo mount /dev/xvdg /opt/fedora-data
2. sudo mount /dev/xvdh /opt/sufia-project/releases/XXXX/tmp
1. Edit the fstab file to retain these mounts
  1. sudo vi /etc/fstab
  2. /dev/xvdg /opt/fedora-data xfs defaults 0 0
  3. /dev/xvdh /opt/sufia-project/releases/XXXX/tmp xfs defaults 0 0
2. mount the disks
  1. sudo mount -a
3. Change the owner of the two mount locations

sudo

1. 1. sudo chown -R tomcat7:tomcat7 /opt/fedora-data
  2. sudo chown -R hydep:deploy /opt/sufia-project/releases/XXXX/tmp

Deploy Sufia

Restart Solr
1. If this is the first time sufia has been deployed, Solr now runs outside of tomcat and needs to be restarted after deployment.
  1. sudo service solr restart
Ensure apache is off on new server
1. We don't want anyone doing stuff before we're ready.

Activate maintenance mode on old server

Fedora export

In sufia 6 instance:

Run audit script

RAILS_ENV=production bundle exec sufia_survey -v

Run json export

$ RAILS_ENV=production bundle exec sufia_export --models GenericFile=Chf::Export::GenericFileConverter,Collection=Chf::Export::CollectionConverter

Open up fedora port to the other server so it can grab the binaries
Change all the 127.0.0.1 URIs to reflect actual host, e.g.

$ find tmp/export -type f -name "*.json" -print0 | xargs -0 sed -i "s/127\.0\.0\.1/staging.hydra.chemheritage.org/g"

Move the resulting directory full of exported data from tmp/export to the new server's tmp/import (or wherever desired; this can be provided to the import script)

$ cd tmp; tar -czf json_export_201611141510.tgz export

Then from your own machine:

$ scp staging:/opt/sufia-project/current/tmp/json_export_201611141510.tgz new_box_ip:/opt/sufia-project/current/tmp/.

Fedora import

On sufia 7 instance:

Mount the /dev/xvdh drive on the tmp directory in sufia (/opt/sufia-project/releases/XXXXX/tmp)
Change the owner of the tmp directory
- sudo chown hydep:deploy /opt/sufia-project/releases/XXXXX/tmp
unpack the exported json files

...

Restart Tomcat on new server
1. sudo service tomcat7 restart
Move over minter statefile
1. On Production
  1. sudo cp /var/sufia/minter-state ~
  2. sudo chown ubuntu:ubuntu minter-state
2. Then copy the file
  1. ```
  scp -3 -i ~/.ssh/test.pem ubuntu@production_ip:~/minter-state ubuntu@new_box_ip:~
```
3. On New Box
  1. sudo mv minter-state /var/sufia
  2. sudo chown hydep:deploy /var/sufia/minter-state
Import Fedora data (in sufia 7 instance)
1. Start a screen or tmux session
2. Become hydep
3. Unpack the exported json files
  1. cd /opt/sufia-project/current/tmp/
  2. cp ~/json_export_201612141435.tgz .
  3. tar

...

1. 1. -

...

1. 1. xzf json_export_

...

1. 1. 201612141435.tgz
  2. mv

...

1. 1. export

...

1. 1. import
2. configure sufia6_user and sufia6_password in config/application
3. run the import
  1. $

...

1. 1. RAILS_ENV=production

...

1. 1. bundle

...

1. 1. exec

...

1. 1. sufia_import

...

1. 1. -d

...

1. 1. tmp/import

...

1. 1. --json_mapping

...

1. 1. Chf::Import::GenericFileTranslator=generic_file_,Sufia::Import::CollectionTranslator=collection_

You can use the little bash script I wrote to create batches of files if you want. It's at /opt/sufia-project/batch_imports.sh

...

1. 1. $ time RAILS_ENV=production

...

1. 1. bundle

...

1. 1. exec

...

1. 1. sufia_import

...

1. 1. -d

...

1. 1. tmp/import --json_mapping

...

1. 1. Chf::Import::GenericFileTranslator=generic_file_,Sufia::Import::CollectionTranslator=collection_ &>> import.log

run verification job

Currently the job itself is hung up in a continuous integration mess / awaiting code review. Here's how to do it manually

$ bin/rails c prodction

validator = Sufia::Migration::Validation::Service.new
validator.call
Sufia::Migration::Survey::Item.migration_statuses.keys.each { |st| puts "#{st}: #{Sufia::Migration::Survey::Item.send(st).count}" }
[:missing, :wrong_type].each do |status|
  puts "#{status} ids:"
  Sufia::Migration::Survey::Item.send(status).each do |obj|
    puts " #{obj.object_

...

Postgres export/Import

...

1. 1. ```
  id}"
    end
  end
```
migrate dump.rdb
Reindex solr

Downsizing

Spin up machine
1. Use ansible playbook, set groupvars/all to have
  1. ec2_instance_type: m4.large
On the migration and downsize machine, stop tomcat.
1. sudo service tomcat7 stop
Install mdadm
1. sudo apt-get install mdadm
Create two new disks for the new machine
1. In the AWS visual interface, go to EC2
2. Go to Volumes
3. Select Create Volumes
4. Make two volumes with the following features:
  1. Magnetic
  2. 1 TB
  3. Availability Zone b
5. Once each one is made, select it and under Actions choose Attach Volume. Type the name or id of the machine and attach the volume.
6. ssh into the box
7. sudo fdisk -l
  1. You should see /dev/vxdg and /dev/xvdh
  2. If not, check if the volumes are attached
8. Create the filesystem for each disk
  1. sudo mkfs.xfs /dev/xvdg
  2. sudo mkfs.xfs /dev/xvdh
Build a RAID 1 array with the two disks
1. mdadm --create --verbose /dev/md0 --level=mirror --raid-devices=2 /dev/xvdg /dev/xvdh
Mount the array
Migrate postgres from old machine to new machine
1. Run the following to generate the export.
  1. ```
  pg_dump -U postgres chf_hydra -Fp > chf_hydra_dump.sql
```

On Migration

...

pg_dump -U postgres fcrepo -Fp > fcrepo_dump.sql

Copy the files to the new machine
1. ```
scp -3 -i ~/
```

...

1. 1. ```
  .ssh/
```

...

1. 1. ```
  test.pem 
```

...

1. 1. ```
  ubuntu@pbig:~/chf_hydra_dump.sql 
```

...

1. 1. ```
  ubuntu@small:~
```
2. ```
scp -3 -i ~/.ssh/test.pem ubuntu@pbig:~/fcrepo_dump.sql ubuntu@small:~
```
2. Drop the current chf_hydra and fcrepo
  1. psql -U postgres
    1. drop database chf_hydra;
    2. drop database fcrepo;
3. Import the files
  1. ```
  psql -U postgres
```

...

1. 1. ```
   chf_hydra < chf_hydra
```

...

```
_dump.sql
```

psql -U postgres fcrepo < fcrepo_dump.sql

Grant database permissions
1. GRANT

...

1. 1. Create,Connect,Temporary

...

1. 1. ON

...

1. 1. DATABASE

...

1. 1. chf_hydra

...

1. 1. TO

...

1. 1. chf_pg_hydra

...

1. 1. ;
  2. GRANT ALL privileges ON DATABASE fcrepo to trilby;
Move the minter statefile from old machine to new machine
1. On Bigbox
  1. sudo cp /var/sufia/minter-state ~
  2. sudo chown ubuntu:ubuntu minter-state
2. Then copy the file
  1. ```
  scp -3 -i ~/.ssh/test.pem ubuntu@big_ip:~/minter-state ubuntu@small_box_ip:~
```
3. On Small Box
  1. sudo mv minter-state /var/sufia
  2. sudo chown hydep:deploy /var/sufia/minter-state
Move the derivative files from the old machine to the new machine
Move dump.rdb from old machine to new machine
Detach the fedora drive on the old machine
Stop tomcat
Attach the fedora drive on the new machine
1. See visual guide
Mount the fedora drive to /mnt
1. sudo mount /dev/xvd? /mnt
Copy the data from the fedora drive to the RAID array.
1. sudo cp -ar /mnt/* /opt/fedora-data/
2. sudo chown -R tomcat7:tomcat7 /opt/fedora-data/*
Backup Solr and move the backup to the new machine
1. curl 'http://localhost:8983/solr/collection1/replication?command=backup&name=migration&location=/solr-backup'
Restore Solr on the new machine
1. curl 'http://localhost:8983/solr/collection1/replication?command=restore&name=migration&location=/solr-backup'
Go to Production and copy SSL certs and keyfiles
Set up SSL redirection

How to check the statefile

...

Stop Tomcat and Solr
Remove all the folders in /opt/fedora-data.
Remove all the files in /var/solr/data/collection1/data/index/, and /var/sufia/derivatives
Remove all the upload files in the tmp directory of the version of sufia used.
Enter into postgres (psql -U postgres)
Drop the fcrepo database (DROP DATABASE fcrepo;)
Build a new fcrepo database (CREATE DATABASE fcrepo;)
Grant the fcrepo user (currently tribly until we get a better user) all privileges on fcrepo. (grant all privileges on fcrepo to tribly;)
Restart tomcat and solrsol

Page Comparison

Versions Compared

Old Version 32

New Version Current

Key

Table of Contents

General outline

Spin up machine

Creating Drives

Deploy Sufia

Solr

Activate maintenance mode on old server

Export Fedora data (in sufia 6 instance)

Migrate postgres

Deploy chf-sufia to new server

Create Drives

Deploy Sufia

Restart Solr

Ensure apache is off on new server

Activate maintenance mode on old server

Fedora export

Fedora import

Restart Tomcat on new server

Move over minter statefile

Import Fedora data (in sufia 7 instance)

run verification job

Postgres export/Import

migrate dump.rdb

Reindex solr

Downsizing

Spin up machine

On the migration and downsize machine, stop tomcat.

Install mdadm

Create two new disks for the new machine

Build a RAID 1 array with the two disks

Mount the array

Migrate postgres from old machine to new machine

On Migration

Move the minter statefile from old machine to new machine

Move the derivative files from the old machine to the new machine

Move dump.rdb from old machine to new machine

Detach the fedora drive on the old machine

Stop tomcat

Attach the fedora drive on the new machine

Mount the fedora drive to /mnt

Copy the data from the fedora drive to the RAID array.

Backup Solr and move the backup to the new machine

Restore Solr on the new machine

Go to Production and copy SSL certs and keyfiles

Set up SSL redirection

How to check the statefile