Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. If US-EAST-1 S3 is down, or we have some issue where our normal S3 buckets are missing or empty we can temporarily use the backup buckets as a source for files.
    1. Before this is done, staff users cannot add any new works or files. New works added will be missing when we stop using the backups. Editing metadata is fine.
    2. While we use the backup buckets, we will be charged for inter-region data transfer. This can quickly add up, so the duration of this switch should be kept to a minimum
  2. Go to the ansible codebase (ansible-inventory).
  3. In the roles/kithe/templates/local_env.yml.j2 file
    1. edit s3_bucket_originals, s3_bucket_derivatives, and s3_bucket_dzi to add -backup to the end of their name (ex. scihist-digicoll-production-originals-backup)
    2. edit the aws_region to us-west-2
      1. this might break access to other buckets that we don't have an alternate of in us-west-2, oh well this process is not completely polished. 
    3. (optional but probably) edit or add logins_disabled: true to lock staff users out to not be making changes. 
    4. and then commit the change and merge to master so that the edit is pushed out to all the
    servers. Staging, lacking these backup buckets will be missing all files.
    1. servers – you may have to wait 10-15 minutes for the changes to be deployed. 
    2. NOTE: Staging, lacking buckets called eg "scihist-digicoll-staging-originals-backup" will be be broken if deployed with these. 
  4. Email all users about the need to avoid adding new works or files.
  5. If the entire US-EAST region is down, the on_demand_derivates and upload buckets may also be unable to be used. This would prevent uploads and pdf/zip generation. If this is expected to last a long time, you may wish to create those buckets in US-WEST-2 and point the local_env file to these buckets.

...