Rebuilding Web and Jobs servers (Obsolete)

Owned by Eddie Rubeiz
Last updated: Jul 30, 2021
1 min read

Our recipe for rebuilding the production scihist_digicoll web and jobs servers from scratch. We may not need to save this recipe.

0. Preliminaries

Some shell substitutions to help avoid typos and make the commands easier to read.

PRIVATE_KEY="~/.ssh/chf_prod.pem"
PASS_FILE="~/.ansible_password.txt"
SECURITY_OPTS="--vault-password-file $PASS_FILE --private-key=$PRIVATE_KEY"
CREATE_CMD="ansible-playbook create_kithe.yml”

Assertions:

  • The create_kithe playbook does not affect existing servers or change them. (That is the job of the update_kithe playbook.)

  • The Web and Jobs server’s IP addresses are not checked into Ansible. Therefore there’s no need to worry about any commits to that codebase.

1. Build

1. Jobs

$CREATE_CMD $SECURITY_OPTS --extra-vars "role=jobs tier=production"

2. Web

$CREATE_CMD $SECURITY_OPTS --extra-vars "role=web tier=production" --extra-vars "@group_vars/kithe_web_production_override"

Switch

  • Point the prod URL in your etc/hosts file locally to the web server.

  • Deploy master branch to all the new servers.

    bundle exec cap production deploy --trace

    At this stage, if you point the prod URL in your etc/hosts file locally to the web server, you should see a functioning production website.

    Visiting the workers page should show twice the usual number of workers (as of Aug. 2020, that’s 24 workers).

  • Point elastic IP to the new web server:

    • Select Elastic IPs

    • Select digicoll-production

    • Actions Associate address

    • Under the "Resource type", Choose "Instance".

    • Fill in the appropriate instance (the new prod web server)

    • Check Allow this Elastic IP address to be reassociated if already attached"

    • Click Associate.

  • Remove the etc/hosts entry and check that traffic is indeed routed to the new server.

  • Spin down the old web and jobs servers.

  • Announce the switch is complete.

3. Wrap up

  • Wait a couple days

  • Turn off termination protection

  • Delete the old servers in AWS console

  • Delete the old disks left orphaned by the old servers