Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Next »

Our recipe for rebuilding all the scihist_digicoll servers from scratch.

1. Build

There’s no need for downtime during this phase, as building new servers should in no way interfere with the existing ones.

(lightbulb) If there’s no rush, consider performing the build and switch phases on consecutive days.

DB
ansible-playbook create_kithe.yml --vault-password-file ~/.ansible_password.txt --private-key=~/.ssh/chf_prod.pem --extra-vars "role=database tier=production"

Add DB’s private IP to group_vars/kithe_production

redis_ip: new_private_ip_of_scihist_digicoll-database1-production
postgres_ip: new_private_ip_of_scihist_digicoll-database1-production

SOLR
ansible-playbook create_kithe.yml --vault-password-file password_file.txt --private-key=~/.ssh/chf_prod.pem --extra-vars "role=solr tier=production"

Add SOLR’s private IP to group_vars/kithe_production

solr_ip: new_private_ip_of_scihist_digicoll-solr1-production

Jobs
ansible-playbook create_kithe.yml --vault-password-file ~/.ansible_password.txt --private-key=~/.ssh/chf_prod.pem --extra-vars "role=jobs tier=production"

Web
ansible-playbook create_kithe.yml --vault-password-file ~/.ansible_password.txt --private-key=~/.ssh/chf_prod.pem --extra-vars "role=web tier=production" --extra-vars "@group_vars/group_vars/kithe_web_production_override"

Deploy

Deploy master branch to all servers.

(lightbulb) At this stage, if you point the prod URL in your etc/hosts file locally to the web server, you should see an empty production website.

2. Switch

  • Point the prod URL in your etc/hosts file locally to the web server.

  • Spin up the downtime server

  • Point the elastic IP for the prod web server to the downtime server

  • Spin down the existing prod servers

  • Copy the old database over to the new DB server.

  • Index
    bundle exec cap production invoke:rake TASK="scihist:solr:reindex"

  • Point elastic IP to the new web server:

    • Select "Elastic IPs"

    • Select "digicoll-production"

    • Actions -> Associate address

    • Under the "Resource type", Choose "Instance".

    • Fill in the appropriate instance (the prod web server)

    • Check "Allow this Elastic IP address to be reassociated if already attached"

    • Click "Associate".

  • Remove the etc/hosts entry and check that traffic is indeed routed to the new server

  • Announce the switch is complete

3. Wrap up

  • Wait a couple days (question)

  • Delete the old servers in AWS console

  • Delete the old disks left orphaned by the old servers

  • No labels