General outline
- Spin up machine
a. If mounting drives for fedora-data or for the tmp directory for migration, make sure to change the owner to tomcat7 (sudo chown tomcat7:tomcat7 folder) for fedora-data and hydep:deploy for the tmp directory (sudo chown hydep:deploy folder) - Deploy Sufia
- Ensure apache is off
- Activate maintenance mode on old server
- Move over minter
- Fedora Export - see below
- migrate postgres
- Fedora Import - see below
- run (currently nonexistent) verification job
- migrate dump.rdb
- Reindex solr
Box Build changes
The current build scripts in Ansible have problems with two changes we're making to the new boxes.
- Migration from Postgresql 9.3 to 9.5
- Moving to Fedora 4.7
Postgres
- Remove the old version of postgres:
- sudo apt-get purge postgresql*
- Create the file /etc/apt/sources.list.d/pgdg.list
- Add the line deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main to that
- Add the repo key with this command
- wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
- sudo apt-get update
- sudo apt-get install postgresql-9.5
- Log in as postgres and enter psql
- sudo su postgres
- psql
- In Postgres create the chf_hydra and fcrepo databases
- CREATE DATABASE chf_hydra;
- CREATE DATABASE fcrepo;
- In Postgres create the needed users
- CREATE USER chf_pg_hydra WITH PASSWORD 'SEE ANSIBLE GROUPVARS/ALL';
- CREATE USER trilby WITH PASSWORD 'porkpie2';
- Grant each user access to their table
- GRANT Create,Connect,Temporary ON DATABASE chf_hydra TO chf_pg_hydra;
- GRANT All Privileges ON DATABASE fcrepo TO trilby;
- Set the user password for postgres
- \password postgres
- Enter password from groupvars/all
- sudo nano /etc/postgresql/9.5/main/pg_hba.conf
- Change the sections under "Database administrative login by Unix domain socket"
- peer should be set to md5 for
- local all postgres
- local all all
- host all all
- peer should be set to md5 for
- Restart postgres, try to log in with
- psql -U postgres
- If it lets you use the password, it works!
Fedora
- Stop Tomcat
- sudo service tomcat7 stop
- Replace the current /etc/default/tomcat7 with
# A backup of the original file with addition options is at /etc/default/tomcat7.bak
TOMCAT7_USER=tomcat7
TOMCAT7_GROUP=tomcat7
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
#Postgres
JAVA_OPTS="-Dfcrepo.home=/opt/fedora-data -Dfcrepo.modeshape.configuration=classpath:/config/jdbc-postgresql/repository.json -Dfcrepo.postgresql.username=trilby -Dfcrepo.postgresql.password=porkpie2 -Dfcrepo.postgresql.host=localhost -Dfcrepo.postgresql.port=5432 -Djava.awt.headless=true -XX:+UseG1GC -XX:+UseCompressedOops -XX:-UseLargePagesIndividualAllocation -XX:MaxPermSize=128M -Xms512m -Xmx4096m -Djava.util.logging.config.file=/etc/tomcat7/logging.properties -server"
- Start Tomcat
- sudo service tomcat7 start
Fedora export
In sufia 6 instance:
- Run audit script
RAILS_ENV=production bundle exec sufia_survey -v
- Run json export
...
Spin up machine
Run ansible scripts
- Make sure groupvars/all has
ec2_instance_type: c4.2xlarge
Activate maintenance mode on old server
Export Fedora data (in sufia 6 instance)
Run audit script (takes 4 or 5 mins)
RAILS_ENV=production bundle exec sufia_survey -v
Make sure you have the latest deployment
- Make sure tmp/export is empty
Run json export (takes < 10 mins)
$ RAILS_ENV=production bundle exec sufia_export --models GenericFile=Chf::Export::GenericFileConverter,Collection=Chf::Export::CollectionConverter
Open up fedora port to the other server so it can grab the binaries
Change all the 127.0.0.1 URIs to reflect
...
internal IPs, e.g.
$
...
find
...
tmp/export
...
-type
...
f
...
-name
...
"*.json"
...
-print0
...
|
...
xargs
...
-0
...
sed
...
-i
...
"s/127\.0\.0\.1/[internal_ip_of_prod]/g"
- The internal IP of prod is: 172.31.48.168
- The internal IP of staging is: 172.
...
- 31.
...
- 58.101
- Move the resulting directory full of exported data from tmp/export to the new server's tmp/import (or wherever desired; this can be provided to the import script)
- $
...
- cd
...
- tmp;
...
- tar
...
- -czf
...
- json_export_201611141510.tgz
...
- export
- Then from your own machine:
- $
...
- scp -3 -i ~/.ssh/test.pem hydep@staging:/opt/sufia-project/current/tmp/json_export_
...
Fedora import
On sufia 7 instance:
- Mount the /dev/xvdh drive on the
- unpack the exported json files
...
- 201612141435.tgz hydep@new_box_ip:~/.
Migrate postgres
- Run the following to generate the export.
pg_dump -U postgres chf_hydra -Fp > chf_hydra_dump.sql
Copy the file to the new machine
scp -3 -i ~/.ssh/test.pem ubuntu@production_ip:~/chf_hydra_dump.sql ubuntu@new_box_ip:~
Import the file
psql -U postgres chf_hydra < chf_hydra_dump.sql
- Run the following to generate the export.
Deploy chf-sufia to new server
Create Drives
In the AWS visual interface, go to EC2
Go to Volumes
Select Create Volumes
Make two volumes with the following features:
General Purpose SSD
150 GB
Availability Zone b
Once each one is made, select it and under Actions choose Attach Volume. Type the name or id of the machine and attach the volume.
ssh into the box
sudo fdisk -l
You should see /dev/vxdg and /dev/xvdh
If not, check if the volumes are attached
Create the filesystem for each disk
sudo mkfs.xfs /dev/xvdg
sudo mkfs.xfs /dev/xvdh
Edit the fstab file to retain these mounts
sudo vi /etc/fstab
/dev/xvdg /opt/fedora-data xfs defaults 0 0
/dev/xvdh /opt/sufia-project/releases/XXXX/tmp xfs defaults 0 0
- mount the disks
- sudo mount -a
Change the owner of the two mount locations
sudo chown -R tomcat7:tomcat7 /opt/fedora-data
sudo chown -R hydep:deploy /opt/sufia-project/releases/XXXX/tmp
Restart Solr
If this is the first time sufia has been deployed, Solr now runs outside of tomcat and needs to be restarted after deployment.
sudo service solr restart
Ensure apache is off on new server
We don't want anyone doing stuff before we're ready.
Restart Tomcat on new server
- sudo service tomcat7 restart
Move over minter statefile
- On Production
- sudo cp /var/sufia/minter-state ~
- sudo chown ubuntu:ubuntu minter-state
- Then copy the file
scp -3 -i ~/.ssh/test.pem ubuntu@production_ip:~/minter-state ubuntu@new_box_ip:~
- On New Box
- sudo mv minter-state /var/sufia
- sudo chown hydep:deploy /var/sufia/minter-state
- On Production
Import Fedora data (in sufia 7 instance)
Start a screen or tmux session
Become hydep
Unpack the exported json files
cd /opt/sufia-project/current/tmp/
...
cp ~/json_export_201612141435.tgz .
tar -xzf json_export_
...
201612141435.tgz
mv
...
export
...
import
configure sufia6_user and sufia6_password in config/application
run the import
- $
...
- RAILS_ENV=production
...
- bundle
...
- exec
...
- sufia_import
...
- -d
...
- tmp/import
...
- --json_mapping
...
- Chf::Import::GenericFileTranslator=generic_file_
...
- You can use the little bash script I wrote to create batches of files if you want. It's at /opt/sufia-project/batch_imports.sh
...
- ,Sufia::Import::CollectionTranslator=collection_
- $ time RAILS_ENV=production
...
- bundle
...
- exec
...
- sufia_import
...
- -d
...
- tmp/import
...
- --json_mapping
...
- Chf::Import::GenericFileTranslator=generic_file_,Sufia::Import::CollectionTranslator=collection_ &>> import.log
run verification job
- Currently the job itself is hung up in a continuous integration mess / awaiting code review. Here's how to do it manually
$ bin/rails c prodction
validator = Sufia::Migration::Validation::Service.new
validator.call
Sufia::Migration::Survey::Item.migration_statuses.keys.each { |st| puts "#{st}: #{Sufia::Migration::Survey::Item.send(st).count}" }
[:missing, :wrong_type].each do |status|
puts "#{status} ids:"
Sufia::Migration::Survey::Item.send(status).each do |obj|
puts " #{obj.object_
- Currently the job itself is hung up in a continuous integration mess / awaiting code review. Here's how to do it manually
...
Postgres export/Import
...
id}"
end
end
migrate dump.rdb
Reindex solr
Downsizing
Spin up machine
- Use ansible playbook, set groupvars/all to have
ec2_instance_type: m4.large
- Use ansible playbook, set groupvars/all to have
On the migration and downsize machine, stop tomcat.
- sudo service tomcat7 stop
Install mdadm
- sudo apt-get install mdadm
Create two new disks for the new machine
In the AWS visual interface, go to EC2
Go to Volumes
Select Create Volumes
Make two volumes with the following features:
Magnetic
1 TB
Availability Zone b
Once each one is made, select it and under Actions choose Attach Volume. Type the name or id of the machine and attach the volume.
ssh into the box
sudo fdisk -l
You should see /dev/vxdg and /dev/xvdh
If not, check if the volumes are attached
Create the filesystem for each disk
sudo mkfs.xfs /dev/xvdg
sudo mkfs.xfs /dev/xvdh
Build a RAID 1 array with the two disks
- mdadm --create --verbose /dev/md0 --level=mirror --raid-devices=2 /dev/xvdg /dev/xvdh
Mount the array
Migrate postgres from old machine to new machine
- Run the following to generate the export.
pg_dump -U postgres chf_hydra -Fp > chf_hydra_dump.sql
- Run the following to generate the export.
On Migration
...
pg_dump -U postgres fcrepo -Fp > fcrepo_dump.sql
Copy the files to the new machine
scp -3 -i ~/
...
.ssh/
...
test.pem
...
ubuntu@pbig:~/chf_hydra_dump.sql
...
ubuntu@small:~
scp -3 -i ~/.ssh/test.pem ubuntu@pbig:~/fcrepo_dump.sql ubuntu@small:~
- Drop the current chf_hydra and fcrepo
- psql -U postgres
- drop database chf_hydra;
- drop database fcrepo;
- psql -U postgres
Import the files
psql -U postgres
...
chf_hydra < chf_hydra
...
_dump.sql
psql -U postgres fcrepo < fcrepo_dump.sql
- Grant database permissions
- GRANT
...
- Create,Connect,Temporary
...
- ON
...
- DATABASE
...
- chf_hydra
...
- TO
...
- chf_pg_hydra
...
- ;
- GRANT ALL privileges ON DATABASE fcrepo to trilby;
Move the minter statefile from old machine to new machine
- On Bigbox
- sudo cp /var/sufia/minter-state ~
- sudo chown ubuntu:ubuntu minter-state
- Then copy the file
scp -3 -i ~/.ssh/test.pem ubuntu@big_ip:~/minter-state ubuntu@small_box_ip:~
- On Small Box
- sudo mv minter-state /var/sufia
- sudo chown hydep:deploy /var/sufia/minter-state
- On Bigbox
Move the derivative files from the old machine to the new machine
Move dump.rdb from old machine to new machine
Detach the fedora drive on the old machine
Stop tomcat
Attach the fedora drive on the new machine
- See visual guide
Mount the fedora drive to /mnt
- sudo mount /dev/xvd? /mnt
Copy the data from the fedora drive to the RAID array.
- sudo cp -ar /mnt/* /opt/fedora-data/
- sudo chown -R tomcat7:tomcat7 /opt/fedora-data/*
Backup Solr and move the backup to the new machine
Restore Solr on the new machine
Go to Production and copy SSL certs and keyfiles
Set up SSL redirection
How to check the statefile
...
- Stop Tomcat and Solr
- Remove all the folders in /opt/fedora-data.
- Remove all the files in /var/solr/data/collection1/data/index/, and /var/sufia/derivatives
- Remove all the upload files in the tmp directory of the version of sufia used.
- Enter into postgres (psql -U postgres)
- Drop the fcrepo database (DROP DATABASE fcrepo;)
- Build a new fcrepo database (CREATE DATABASE fcrepo;)
- Grant the fcrepo user (currently tribly until we get a better user) all privileges on fcrepo. (grant all privileges on fcrepo to tribly;)
- Restart tomcat and solrsol