...
Log into the web server as digcol
user. You can find current web server IPs with ./bin/cap production list_ec2_servers
from an app checkout.
Get good status info on passenger workers:
$ PASSENGER_INSTANCE_REGISTRY_DIR=/opt/scihist_digicoll/shared passenger-status
Restart application without restarting apache
This will reload config files.
...
passenger-config can do some other interesting things as well, such as system-metrics
Resque admin panel
If a file doesn't get characterized correctly the first thing to do is check the resque admin panel. There you can view failures and restart jobs. If you are logged in as an admin user you can view the admin panel at `digital.chemheritage.org/admin/queues`
What version of the app is deployed?
$ cat /opt/sufia-project/current/REVISION
Restart passenger
$ passenger-config restart-app
Reindex all of solr:
Check README for scihist_digicoll
Rebuild Solr with 0 Downtime tips:
For scihist_digicoll, we can easily build and swap in a new Solr server. This will result in downtime until the index is remade. While reindexing takes only a minute or two, the server changes being applied to jobs and web can take a while, so there may be many minutes between when one of them is connected to the new Solr server and the other does not. During that time, we can't reindex.
...
CORENAME is scihist_digicoll
Location is build by ansible for backups, /backups/solr-backup
BACKUPNAME can be anything you like
On the old Solr machines run
Logged in as ubuntu
curl 'http://localhost:8983/solr/CORENAME/replication?command=backup&name=BACKUPNAME&location=/backups/solr-backup'
...
Commit the changes to the staging branch so you can easily merge a PR from staging to master to swap the IP address without waiting for staging to update.
On the new Solr machine
Logged in as ubuntu
Extra the tar tar xzf solr-backup.tar.gz
to the /backups/solr-backup spot (or anywhere as long as the Solr user can access it)
...
Once the servers are switched, run a reindex to catch any changes made during that time.
Clearing out the tmp directory (removes everything older than 8 days.)
This is invoked by a cron job on app prod, but just in case...
...