...

Our Ansible configuration is stored on Bitbucket (see below for the URL).Once you check out the above git repository from BitBucket, you'll find a more detailed description of the organization of our Ansible code at https://bitbucketgithub.orgcom/ChemicalHeritageFoundationsciencehistory/ansible-inventory/src/master/README.md .

Overview of the codebase

The information in the Bitbucket Ansible repository is organized into four main file types: the hosts file, a set of playbooks, roles called by each playbook, and encrypted variables files.

Hosts file (hosts)

This is a catalog of our Ansible-managed servers. Each server is associated with information about its purpose (tags) and instructions about how to contact it (ip addresses and hostnames). The file is organized by category; each category contains zero or more server IP's associated with it. These are:

• "Tier" categories:
  • prod: production servers.
  • stage: staging servers.
  • dev: development servers. (Currently empty; our dev servers are not currently managed by Ansible.)
• "Role" categories:
• fedora: a server on which Ansible knows to install Fedora.
• solr: a server on which Ansible knows to install SOLR.
• app: a server on which Ansible knows to install the digital collections application (chf-sufia.)
jobs:
a server whose purpose is to run background jobs.
• monitor: a server on which Ansible knows to install Netdata, our monitoring software (see https://github.com/firehol/netdata).
• aspace: an ArchivesSpace server (see http://archivesspace.org/).

Playbooks

These are the main tasks Ansible knows how to perform:

• build_ami.yml : Create an Amazon machine image, which serves as a template for the actual EC2 cloud servers. To list tasks:

  • ansible-playbook --list-tasks --extra-vars "hosts=stage" --extra-vars "role=solr tier=stage" build_ami.yml

• create_ec2.yml: Create a new server based on an AMI. To list tasks:

  • ansible-playbook --list-tasks --extra-vars "hosts=stage" update-box.yml

  • ansible-playbook --list-tasks --extra-vars "hosts=prod" update-box.yml

• resize.yml: Stop a server, resize its disks, and then restart it.
• sync.yml: Copy over data from production to staging. To list tasks:

  • ansible-playbook --list-tasks sync.yml

• update-box.yml: Runs updates on a server.

Roles

These are the subtasks of the playbooks (for instance, the recipe for installing Apache on a given server). They are listed in roles/*/tasks/main.yml files. The roles can also refer to:

• Subsidiary tasks, known as handlers: roles/*/handlers/main.yml
• Templates, typically for configuration files. These are roles/*/templates/* and are often stored as Jinja templates. (see http://jinja.pocoo.org/). For instance, the roles/apache/tasks/main.yml role has a choice of three Apache configuration files in roles/apache/templates/, which it uses depending on which server it's configuring.

Variables files

The values of the variables are used by the roles in building and modifying the servers. These are in group_vars/* and there is one for each category listed in the Hosts files. In certain cases, when a server belongs to two categories (e.g. it's both prod and solr), a special override file is used: e.g. group_vars/solr is overridden by group_vars/solr_prod_override .

...

Variables files are the only ones encrypted.

The password for ansible-vault is in

Shared/P/OthmerSupport/Computer LibraryServices/Digital Collections - Internal Access/ ./ansible-vault.txt

To view a file:

ansible-vault --vault-password-file FILE_CONTAINING_PASSWORD view ENCRYPTED_FILE

...

ls -1 ansible-inventory/group_vars | while read N ; do echo $N: ; \
ansible-vault --vault-password-file FILE_CONTAINING_PASSWORD \
view ansible-inventory/group_vars/$N | grep -i STRING_TO_LOOK_FOR ; done

...

See Editing Ansible for notes on current practices for editing.
Building a new machine on AWS with Ansible
Note: ansible-vault password and all current AWS keys are in shared network drive  Othmer Library\Digital Collections - Internal Access\Authentication - Confidential
If you do not have access, 
...
you will need to be added to the allowed group.
This assumes you have the main Ansible password inside a plain Ascii file at /your/ansible_password.txt .
Check ansible variables in the encrypted file
$ ansible-vault edit --vault-password-file /your/ansible_password.txt group_vars/all
 (will need password)
Look for # Use these temporarily for new instancesRIght now certain values such as fedora_ip, solr_ip, and the rest will need to be determined once the box has been built and a valid IP exists.
Generally speaking the best way to build boxes to minimize needing to go back and edit IPs is
Fedora
Solr
Sufia machines (riiif, app)
ensure your ssh key is listed under keys_to_add, this is needed for capistrano deploys and ssh access with your personal account.
run the ansible playbook
$ ansible-playbook create_ec2.yml --ask-vault-pass --private-key=/PATH/TO/KEY --extra-vars "role=ROLE tier=SERVICE_LEVEL" --extra-vars "@group_vars/ROLE_SERVICE_LEVEL_override"
Use chf_prod.pem for all production level machines
Use test.pem for all other machines
Select the role and service level of the machine you want to build.
OR, if you're re-running scripts on an existing machine: 
$ ansible-playbook -i hosts my_playbook.yml --ask-vault-pass [-e hosts=target]
target can be one of the groups in the hosts file: staging, production, dev, ec2hosts
Assign an elastic IP to the new box if if needs one
Consider naming the aws volumes for 'root' and 'data' – this isn't done in the scripts (but probably could be!)
Set up to use capistrano (below) or just deploy with capistrano (above)
Run configure_prod.yml if on production to set up e-mail password resets, ssl, and backup procedures.
Updating boxes with Ansible
New AWS Key
Generate a new ssh key on AWS (EC2 > Keypairs)
place it in ~/.ssh
chmod 0600.
useful command if you're having problems with the key: $ openssl rsa -in chf_prod.pem -check
Git repositories for ansible - structure and use (this section is out of date)
The code we use to administer Sufia via Ansible lives at https://github.com/curationexperts/ansible-hydra
A wrapper with local configuration lives at https://bitbucket.org/ChemicalHeritageFoundation/ansible-inventory. Wrapper contains:
our hosts file
our group_vars files
ansible-hydra as a git submodule
an ansible.config which points to ansible-hydra for roles_path.
A number of roles and plays for CHF specific customization
Aside: pull requests can be submitted via branches; there's really no need to fork this repo since we'll all be owners.
To use
$ git clone clone git@bitbucket.org:ChemicalHeritageFoundation/ansible-inventory.git
$ cd ansible-inventory
$ git submodule update --init
Subsequently, when you pull ansible-inventory and the submodule has been updated, just run
$ git submodule update
Playbook Notes
...
Creating a test box using Ansible
Scenario: You've changed the Ansible scripts that affect the build process for the jobs server. To make sure everything looks right, you want to spin up a new jobs2-stage on EC2 without disturbing the functioning of jobs1-stage.
Procedure:
Make sure the main Ansible password is available inside a plain text file at, e.g.,  /your/ansible_password.txt .
Change the "iteration" so your box has a different label from the existing jobs1-stage. (Use whichever editor you want, but some work better than others.)
EDITOR=emacs ansible-vault edit --vault-password-file /your/ansible_password.txt . group_vars/all
Find the line that mentions "iteration" and set it to an unused number.
iteration: 2
This will result in your new jobs box being labeled jobs2-stage.
List the tasks you'll be performing. This is helpful to make sure everything looks good before actually running the script.
ansible-playbook --vault-password-file /your/ansible_password.txt --list-tasks --extra-vars "role=jobs tier=stage" create_ec2.yml
Now actually create the box:
ansible-playbook --vault-password-file /your/ansible_password.txt --extra-vars "role=jobs tier=stage" create_ec2.yml
Note: this will modify your "hosts" file in the root directory of ansible-inventory by adding the new box's IP address to it.
In the EC2 dashboard, you'll see a new EC2 instance called jobs2-stage. You can log into it as you would any ec2 server.
Watch the progress of the ansible-playbook command in your terminal.
When you're done, you can get rid of the box in the EC2 console (but don't get rid of the wrong box!) by selecting it, then choosing Actions > Instance State > Terminate.
New AWS Key
Generate a new ssh key on AWS (EC2 > Keypairs)
place it in ~/.ssh
chmod 0600.
useful command if you're having problems with the key: $ openssl rsa -in chf_prod.pem -check