Site's search is broken

This covers common troubleshooting for when the site cannot be reached.

Step-by-step guide

If the search works but shows no items, it may have lost the index. You can restore from the prior day's backup for production (see Backups and Recovery (Historical notes)). Otherwise you will need to reindex.

If the search is broken or throws out an error:

1. Check that the solr box is running by going to the AWS console.
   1. If it is down or missing, a new box will be needed.
      1. If it is down, just turn it back on by selecting it and going to Actions→Instance State→ started
      2. If it is missing, rebuild a solr box using the Ansible instructions.
         1. Update the sufia code with the new internal IP address of Solr
         2. Reindex Sufia using the instructions there
2. If the Solr box is active, make sure it can be reached by the Application box
   1. Check to see that both Solr and App are in the right internal-security group (app or prod)
      1. If not, go to ansible and edit the staging or prod variables to put the right group in
      2. Run the ansible update-box.yml script for app and staging
      3. If they are still not showing the right group, check the ansible log for information.
   2. Check that the internal-security group has port 8983 listed by going to the AWS console and checking in EC2→ Security Groups
      
      1. If it does not, click on the group.
      2. Select the inbound tab on the bottom, then edit.
      3. Add a port 8983 opening for members of the same group (by listing the group in for IP address)
3. If Solr is up and the security group settings seem fine, check if Solr itself is active.
   1. ssh into the server
   2. Check that solr is running with sudo service solr status
   3. If it returns a no, run sudo service solr start to start solr again
4. If all the above are true and the search still throws an error, check the Semvera code.