ArchivesSpace is a server whose main purpose is to host a software program also named… ArchivesSpace. The program is “an open source archives information management application for managing and providing web access to archives, manuscripts and digital objects”. The server also hosts a few auxiliary programs who take the output from ArchivesSpace and convert it into various other formats, which are then made available via an Apache webserver on the same machine.
...
The current production version of Aspace is 2.7.1 .
Terminal access: ssh -i /path/to/production/pem_file.pem ubuntu@50.16.132.240
The ubuntu user owns all the admin scripts.
The relevant Ansible role is: /roles/archivesspace/ in the ansible-inventory codebase.
...
Restarting the server to fix Tomcat memory leak
Note: as of August 2020, the below procedure has been rendered obsolete. We are now simply restarting the server with We restart the ArchivesSpace program (not the server) using a cronjob that runs /opt/archivesspace/archivesspace.sh restart every Sunday at 2 am, which appears to solve the problem before it occurs.
ArchivesSpace has a memory leak that causes it to use more CPU time than it should. This will slowly drain all the burst credits, at which point the server slows down.
Another clue: if you go to the AWS console for the server, under the Monitoring tab, if the CPU Utilization graph shows anything over about 15 %.
Procedure:
...
Contact all the main users listed above (especially Kent), and make sure they’re not actively working on the server.
Once given the go-ahead:
...
Log in to the server.
...
Throughout the process, keep in mind you can run sudo systemctl status archivesspace for the daemon status at any point. If it’s running, you’ll see a variation on: [...]Active: active (running) since Tue [...]
...
/opt/archivesspace/archivesspace.sh statusshould give you the status of the program. (“ArchivesSpace is running as (PID: 7483)”).
...
Run top in a separate window to monitor the CPU usage. The goal is to see a dramatic reduction in usage after this process.
...
sudo systemctl stop archivesspace
...
sudo systemctl start archivesspace (You may have to run this two or three times – the start script is finicky)
...
/opt/archivesspace/archivesspace.sh restart
...
If all else fails, you can also go into the AWS console and reboot the EC2 instance.
...
Once everything is properly restarted:
the https://archives.sciencehistory.org/ front-end is available again
After a few minutes, you should see the CPU use go down dramatically in
top.The AWS monitoring graph for CPU Utilization graph should drop. (see figure below.)
Once you’re done, notify all involved that the server is available again.
...
. This prevents a chronic memory leak from eating up all the CPU credits for the machine.
Export
The ArchivesSpace EADs are harvested by:
...