...

$ passenger-config restart-app

Reindex all of solr:

# ssh to server as deploy user

$ cd /opt/sufia-project/current

$ RAILS_ENV=production bundle exec rake chf:reindex

Or, try using cap remote rake:  `cap (production|staging) invoke:rake TASK=chf:reindex`

Note: If Reindexing due to a server move, import the Postgres database of users prior to reindexing. Otherwise you will need to reindex again once the users have been moved over.

Reindex just works in solr:

# ssh to jobs server (either jobs-prod or jobs-stage)
$ cd /opt/sufia-project/current
$ RAILS_ENV=production bundle exec rake chf:reindex_works

Or, try using cap remote rake:  `cap (production|staging) invoke:rake TASK=chf:reindex_works`

Note: reindex_works task only works when you already have a complete solr index, unlike the much much slower full reindex, which can be run with an empty index to begin with.

Note: make sure to use either "screen" or "nohup", so if/when you get disconnected from terminal on jobs, it’s still running.

Delete all the data

(Don't do this on prod!)

Optional: stop apache or use capistrano's maintenance mode

Shut down tomcat and solr

rm -rf /opt/fedora-data/*

rm -rf /opt/solr/collection1/data/* # solr 4

rm -rf /var/solr/data/collection1/data/* # solr 5

If using Sufia 7 also run

psql -U trilby -d fcrepo -c 'DELETE FROM modeshape_repository'

The temporary testing password for trilby is porkpie2

Delete database stuff (notifications, mostly)

(you'll need the password. it's in the ansible vault.)

psql -U chf_pg_hydra -d chf_hydra

delete from mailboxer_receipts where created_at < '2015-11-9';

delete from mailboxer_notifications where created_at < '2015-11-9';

delete from mailboxer_conversations where created_at < '2015-11-9';

delete from trophies where created_at < '2015-11-9';

Turn tomcat back on (and apache if needed)

Inspect stuff

Note when using the rails console to look at actual live production data it's possible to change and delete things! Please be very careful before submitting commands if you are working with live data. Consider a dry-run on the staging server before doing anything on the production box.

$ bundle exec rails c[onsole] production

# Or if you use my dev box, mess around on a development instance with just $bundle exec rails c

# Get a count of users

> User.all.count

# List all users (you can also work with users directly in pgsql)

> User.find_each { |u| p u.email } 

# Get a count of files

> GenericFile.all.count

# Inspect a file

> f = GenericFile.find(id='3b5918567')

> f.depositor

...

State File

The state file (formerly in /tmp) has been moved to /var/sufia. It is currently being backed up nightly. It must be included in any server migrations to avoid generating errors when uploading (a new state file may try to use an already used fedora ID). 

Rights statements

It's useful to periodically check that all publicly available works have rights statements. As of Summer 2018 this was in fact true, but if you want to quickly check for the ID's of any public works that still need rights statements, log onto jobs_stage or jobs_production, open a console (see "Inspect Stuff" above) and paste the following directly into the console:

GenericWork.search_in_batches('read_access_group_ssim'=>'public') do |group|
  group.each do |gw|
    if gw["rights_tesim"] == nil || gw["rights_tesim"].count == 0
      puts gw["id"]
    end #if
  end #group.each
end #search_in_batches

Adding and removing items from large collections

This is a known bug as of Summer 2018. See https://github.com/sciencehistory/chf-sufia/issues/1068 for more details about this. We seldom need this functionality, but if you do, here's how to do it in the Rails console.

Assuming the work you want to add or remove has ID work_id and the collection you want to add it to or remove it from has id collection_id,

Removing:

the_collection = Collection.find(collection_id)
the_collection.members.delete(GenericWork.find(work_id))
the_collection.save

Adding:

the_collection = Collection.find(collection_id)
the_collection.members.push(GenericWork.find(work_id))
the_collection.save

For large collections, expect these operations to take five to ten minutes and place considerable load on the server.

Regenerating derivatives on a work

Log into the jobs server (prod or staging, depending on the situation.)

$ sudo su
$ cd /opt/sufia-project/current
$ WORK_IDS=x346d4165,5425k9692 RAILS_ENV=production bundle exec rake chf:derivatives:s3:create

Regenerating derivatives on a single fileset

$ sudo su
$ cd /opt/sufia-project/current
$ bundle exec rails c production
irb(main) > file_set = FileSet.find("z029p569t")
irb(main) > file_set_id = file_set.files[0].id
irb(main) > CHF::CreateDerivativesOnS3Service.new(file_set, file_set_id).call

Creating, cropping, and uploading collection thumbnails

Instructions for doing this are on github Check README for scihist_digicoll

Rebuild Solr with 0 Downtime tips:

For scihist_digicoll, we can easily build and swap in a new Solr server. This will result in downtime until the index is remade. While reindexing takes only a minute or two, the server changes being applied to jobs and web can take a while so there may be many minutes between when one of them has the new Solr server and the other does not during which time we can't reindex.

To minimize downtimes during Solr changes, the preferred method is to take a backup of the old Solr version (if it can be used with the new Solr version, test first) and then restore that backup on the new Solr server so that public users will always get content.

Core Name is scihist_digicoll
Location is build by ansible for backups, /backups/solr-backup
Backup Name can be anything you like

On the old Solr machines run
curl 'http://localhost:8983/solr/CORENAME/replication?command=backup&name=BACKUPNAME&location=/backups /solr-backup'
sudo tar czf /backups/solr-backup/solr-backup.tar.gz /backups/solr-backup/snapshot.BACKUPNAME

Then move/copy the backup tar to new server via whatever method you care to use

On the new Solr machine

Extra the tar to the /backups/solr-backup spot (or anywhere as long as the Solr user can access it)
Make sure all files are owned by Solr
Run
curl 'http://localhost:8983/solr/CORENAME/replication?command=restore&name=BACKUPNAME&location=/PATH'

Now the new machine has a recent backup and when you update the server IP address users will always get search results. Staff who have recently added or edited items may notice that they look off if it took place after the backup.

Once the servers are switched, run a reindex to catch any changes made during that time.

Clearing out the tmp directory (removes everything older than 8 days.)

...