Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Current »

Our recipe for rebuilding the production scihist_digicoll web and jobs servers from scratch. We may not need to save this recipe.

0. Preliminaries

Some shell substitutions to help avoid typos and make the commands easier to read.

PRIVATE_KEY="~/.ssh/chf_prod.pem"
PASS_FILE="~/.ansible_password.txt"
SECURITY_OPTS="--vault-password-file $PASS_FILE --private-key=$PRIVATE_KEY"
CREATE_CMD="ansible-playbook create_kithe.yml”

Assertions:

  • The create_kithe playbook does not affect existing servers or change them. (That is the job of the update_kithe playbook.)

  • The Web and Jobs server’s IP addresses are not checked into Ansible. Therefore there’s no need to worry about any commits to that codebase.

1. Build

1. Jobs

$CREATE_CMD $SECURITY_OPTS --extra-vars "role=jobs tier=production"

2. Web

$CREATE_CMD $SECURITY_OPTS --extra-vars "role=web tier=production" --extra-vars "@group_vars/kithe_web_production_override"

Switch

  • Point the prod URL in your etc/hosts file locally to the web server.

  • Deploy master branch to all the new servers.

    bundle exec cap production deploy --trace

    (lightbulb)At this stage, if you point the prod URL in your etc/hosts file locally to the web server, you should see a functioning production website.

    (lightbulb)Visiting the workers page should show twice the usual number of workers (as of Aug. 2020, that’s 24 workers).

  • Point elastic IP to the new web server:

    • Select Elastic IPs

    • Select digicoll-production

    • Actions ➡ Associate address

    • Under the "Resource type", Choose "Instance".

    • Fill in the appropriate instance (the new prod web server)

    • Check Allow this Elastic IP address to be reassociated if already attached"

    • Click Associate.

  • Remove the etc/hosts entry and check that traffic is indeed routed to the new server.

  • Spin down the old web and jobs servers.

  • Announce the switch is complete.

3. Wrap up

  • Wait a couple days (question)

  • Turn off termination protection

  • Delete the old servers in AWS console

  • Delete the old disks left orphaned by the old servers

  • No labels