The content on this page is obsolete. For now, striking it out.

Passenger (web worker) administration

Log into the web server as digcol user. You can find current web server IPs with ./bin/cap production list_ec2_servers from an app checkout. 

Get good status info on passenger workers:

$ PASSENGER_INSTANCE_REGISTRY_DIR=/opt/scihist_digicoll/shared passenger-status

Restart application without restarting apache

This will reload config files.

...

passenger-config can do some other interesting things as well, such as system-metrics

Resque admin panel

If a file doesn't get characterized correctly the first thing to do is check the resque admin panel. There you can view failures and restart jobs. If you are logged in as an admin user you can view the admin panel at `digital.chemheritage.org/admin/queues`

What version of the app is deployed?

$ cat /opt/sufia-project/current/REVISION 

Restart passenger

$ passenger-config restart-app

Reindex all of solr:

Check README for scihist_digicoll

Rebuild Solr with 0 Downtime tips:

For scihist_digicoll, we can easily build and swap in a new Solr server. This will result in downtime until the index is remade. While reindexing takes only a minute or two, the server changes being applied to jobs and web can take a while, so there may be many minutes between when one of them is connected to the new Solr server and the other does not.  During that time, we can't reindex.

...

CORENAME is scihist_digicoll
Location is build by ansible for backups, /backups/solr-backup
BACKUPNAME can be anything you like

On the old Solr machines run

Logged in as ubuntu

curl 'http://localhost:8983/solr/CORENAME/replication?command=backup&name=BACKUPNAME&location=/backups/solr-backup'

...

Commit the changes to the staging branch so you can easily merge a PR from staging to master to swap the IP address without waiting for staging to update.

On the new Solr machine

Logged in as ubuntu

Extra the tar tar xzf solr-backup.tar.gz to the /backups/solr-backup spot (or anywhere as long as the Solr user can access it)

...

Once the servers are switched, run a reindex to catch any changes made during that time.

Clearing out the tmp directory (removes everything older than 8 days.)

This is invoked by a cron job on app prod, but just in case...

...