...
A quick cost analysis has restoration costing $30-35 dollars, this is as of 6/11/2018 with approximately 1 TB of data. Approximately 66% of the cost was due to inter-region transfer fees (moving data from US-WEST to US-EAST). The rest is standard LIST, GET, and related fees.
...
Scihist_digicoll Backup and Recovery
Digital Collection Recovery Overview:
...
Provide backups of all original files to they can be recovered in case of natural disaster, data corruption, user error, or malicious computer activity.
Allow the Institute to recover from data loss or outages in a reasonable rate of time.
Adhere to OAIS model rules when possible.
The following classifications of recovery may prove helpful
Partial public, partial staff recovery: The public has access to limited functionality, staff has access to a limited set of functions but certain functions may not be used. This is considered an incomplete recovery.
Full public, partial staff recovery: The public can use the site normally, staff has access to a limited set of functions but certain functions may not be used. This is also an incomplete recovery but time sensitivity is reduced as public users are not impacted. Staff recovery times should be minimized, but public use takes priority.
Full public, full staff recovery: A total recovery.
Then there are levels of disaster/data loss
Minimal Data Loss: In this case a small subset of data is lost, general site functionality is unaffected.
Minimal Data Inaccessibility: A small subset of data is temporarily inaccessible, general site functionality is unaffected.
Major Data Inaccessibility: Data is not lost permanently, but our ability to access the data is compromised. In this case general site functionality is affected.
Major Data Loss: A large amount of data is lost, and general site functionality is impacted.
Generally speaking public recovery is the higher initial priority since it impacts more people, contacting staff about outages is easy, and public outages affect the perception of the institute.
Inaccessibility is also different from data loss, though they share certain characteristics. In both cases a solution is to have additional copies of data, but for inaccessibility it is so they can be used as a temporary source of data until the outage is resolved. Handling data inaccessibility requires that the secondary source of data be similarly structured to minimize the time to switch over. With data loss the backups can be in any format as the intent is that a new source of data will be built from the backups.
Our data can be broken into two categories, one is data that is potentially irrecoverable. This includes our original binary files (images, audio, other) and the metadata about them (a SQL database). The other data is restorable but needed for normal site operation but takes significant time to restore, such as the derived download files and viewer files. The second set of data may be worth backing up to shorten recovery times for public users when data is lost.
As an estimate, our cost to hold extra backups for our current scihistcoll staging environment costs less than $2 a month out of a total $70.07 spent on data storage inclusive of data transfer and storage. While a production environment will have slightly higher cost ratios, it should not be massively higher. Thus by spending an additional 2-3% cost on S3, we can mount a full public recovery in an afternoon from a massive failure of our entire infrastructure. While we currently are not backing up our viewer tiles, an examination of our old application shows the cost for production averages around 5 dollars for storage. Adding a second copy of the viewer files should roughly double the cost, with a slight reduction for less use, so will add another 5 dollars to the cost, so for about $7-12 dollars a month we can be widely covered for data inaccessibility or other failures of S3 in a specific region. While it is hard to get specific details, there have been multiple outages or issues in a region whose duration lasted over an hour, and at least one major outage in the last two years lasting around 6 hours. Assuming about 8 hours of problems every two years, we can estimate that a rough cost of $36/hour of outage spent to avoid being down. Shorter outages may not be worth the difficulty of switching over.
In cases of small scale data loss, such as corrupted files or user error, the application will be working fine but a limited set of data will have a problem. In these cases we can locate the problematic data and use a backup copy to restore any damaged original files or use versioning to restore an earlier version of the file. Derived files can either be regenerated or copied from backups as well. This is the most common expected use case .at requires only that we keep versions of our original files and backups of files and
In cases of broad AWS failure or regional disasters, it is possible that much or data loss most or all of the data is rendered unavailable. In these cases we will suffer a loss of service until we can recover the data and apply new servers to run the software. This can be thought of as two recoveries, one is to get the digital collection site back as soon as possible for the public and the second is to restore all functionality. Getting the site back for the public is our primary concern, so as noted above for outages we have a few methods to speed up recovery at an additional cost to our backup costs. Both the derivative and original files are backed up to a region on the West Coast in S3 like our actual use files. This means we can in case of a disaster point our application to the backup S3 storage and use it in productionWe can recover public access by using these backup files directly while we spend more time working on a full recovery for staff functionality. Due to current setups we would not want staff to add new works, but this allows us to rapidly restore public facing access to our site should the normal data sources be unavailable. A longer process allows us to restore the data back to the original locations while leaving public access up, once the data is restored to its original place full staff functionality will likewise be restored.
In Finally in cases of issues affecting all on-line storage systems, another copy of the data is held on our in-house storage system so that we can potentially recover data even in case of a full loss of AWS. This only holds the original files and all other aspects will need to be rebuild, a process that can take days in addition to the time taken to upload the files again. Using the local backup means recovery could take a week or more.
The files which are considered key, the original item files and the postgres database storing their metdata are the only files which require backup. The other files being saved, derivative and index files, are only saved in order to reduce our proposed downtime during different accidents. When we look at recovery it will be useful to distinguish between full and partial recoveries. The following classifications may prove helpful
Partial public, partial staff recovery: The public has access to limited functionality as above, staff has access to a limited set of functions but certain functions may not be used.
Full public, partial staff recovery: The public can use the site normally, staff has access to a limited set of functions but certain functions may not be used.
Full public, full staff recovery: A total recovery.
Generally speaking public recovery is the higher initial priority since it impacts more people, contacting staff about outages is easy, and public outages affect the perception of the institute. That is why we’ve taken on the extra cost of storing non-needed files to speed up public recoveryThis is not paid for/managed by our team as Institute IT handles these systems.
Technical Notes
Kithe currently (March 2019) has a small set of data to be handled for recovery.
A postgres database which contains user data and item metadata
Original binary files
Derivative Files
The first two are the ones that are needed the derivatives are merely backed up because the cost to back them up is low relative to the amount of time saved on a recpvery recovery by having a copy ready.
- The postgres database is to be backed up to S3
...
- , with a version history of the last 30 versions of the file representing a month of backups roughly.
- The binary files are replicated via S3 replication to a second location in US-WEST rather than US-EAST in case of outages. When we actually switch over, these will also be backed over to local on-site storage. These files are also versioned and prior versions are held for 30 days before being cleared away to reduce storage costs. This offers a month period to revert a file back if something is damaged.
- The derivative files will also be replicated via S3 replication to a US-WEST location. They can also be regenerated by the application though this takes
...
- days to complete for all files. Replication requires versioning, so this is enabled but unlikely to be used.
Minimal public recovery requires the folllowing following data
Postgres database
Original binaries for downloading tiffs
Derivative files
If we have an AWS outage affecting our region the fastest recovery options are to (if needed) rebuild the servers in another region and edit the local_env.yml file in either ansible or by hand to point to the backup S3 buckets for the original and derivative files. The postgres database will need to be downloaded from S3 and installed onto the new kithe machine if there is oneservers if we needed to use new servers. After that point all current public facing aspects will be restored. Since the backup buckets do not sync with the original data buckets staff should not upload new files though they can edit metadata on existing works. Once the original S3 buckets have had service restored or their data copied back, set the application to use the original buckets with local_env.yml and users can now add items,. The postgres database may need to be copied back to the original server(s) or region if a new server setup was used.
In the case of smaller issues, like single file corruption or deletion the simplest method for original files is to local them on S3 and look at previous versions. We keep 30 days worth of versions so if the error was found within a month you should be able to revert back to an earlier file. For derivatives it is easier to simply regenerate them via the command line.