Process documentation

What version of fedora is running?

it's in the footer of the main page

for example http://localhost:8080/fedora/

Deploy the application with Capistrano

cd into the root dir of your chf-sufia repository

$ bundle exec cap [staging|production] deploy

ssh to the ec2 machine as 'ubuntu' user (because you will need sudo). you can create a new ssh config entry if you want; either use the key you downloaded in the first step or if you have added your own key you can use that.

$ sudo service tomcat7 restart
$ sudo service apache2 restart

Addt'l useful capistrano info

bundle exec cap staging deploy --dry-run --trace – shows you the whole deployment with capistrano task execution info.
hook syntax: http://capistranorb.com/documentation/getting-started/before-after/#

Deploy with downtime

bundle exec cap staging maintenance:enable REASON="a test of maintenance mode" UNTIL="12pm Eastern Time"

Deploy as usual / desired
Do anything else needed on the server that required the downtime

bundle exec cap staging maintenance:disable

Delete all the data

(Don't do this on prod!)

Shut down tomcat

rm -rf /opt/fedora-data/*

rm -rf /opt/solr/collection1/data/*

Delete database stuff (notifications, mostly)

psql -U chf_pg_hydra -d chf_hydra

delete from mailboxer_receipts where created_at < '2015-11-9';

delete from mailboxer_notifications where created_at < '2015-11-9';

delete from mailboxer_conversations where created_at < '2015-11-9';

delete from trophies where created_at < '2015-11-9';

Building a new machine on AWS with Ansible

(Note: ansible-vault password and all current AWS keys are in shared network drive)
Generate a new ssh key on AWS (EC2 > Keypairs)
1. place it in ~/.ssh
2. chmod 0600.
1. useful command if you're having problems with the key: $ openssl rsa -in chf_prod.pem -check
Check ansible variables
1. $ ansible-vault edit group_vars/all
2. Look for # Use these temporarily for new instances
3. ensure your ssh key is listed under keys_to_add
run the ansible playbook
1. $ ansible-playbook -i hosts ansible-hydra/create_ec2.yml --private-key=/path/to/your/.ssh/key.pem --ask-vault-pass
2. OR, if you're re-running scripts on an existing machine:
  1. $ ansible-playbook -i hosts ansible-hydra/my_playbook.yml --ask-vault-pass [-e hosts=ec2hosts]
3. note that if there's a failure during postgres setup handlers may not run – watch out for this. if this happens it's potentially best to start over completely.
Assign an elastic IP to the new box
Ask IT to give you a DNS entry for the elastic IP if desired
Consider naming the aws volumes for 'root' and 'data' – this isn't done in the scripts (but probably could be!)
Set up to use capistrano (below) or just deploy with capistrano (above)

Set up Capistrano (first-time use)

Create an entry for the deploy user in your .ssh/config:

Host staging
 Hostname NEW.IP
 User hydep
 #IdentityFile ~/.ssh/your_key
 ForwardAgent yes

This keeps us from publishing server names, etc, in the cap config files which live in our public repo.
don't change the Host designation without:
- Changing it in capistrano, e.g. deploy/staging.rb, to match
- Clearing it with everyone who might deploy (they'll have to change their ssh config as well.
this will use your personal ssh key – the one that matches your public key on github, which is added to the deploy user by ansible scripts.

Git repositories for ansible - structure and use

Code lives at https://github.com/curationexperts/ansible-hydra

Wrapper with local configuration lives at https://bitbucket.org/ChemicalHeritageFoundation/ansible-inventory. Wrapper contains:

our hosts file
our group_vars files
ansible-hydra as a git submodule
an ansible.config which points to ansible-hydra for roles_path.

Aside: pull requests can be submitted via branches; there's really no need to fork this repo since we'll all be owners.

To use

$ git clone clone git@bitbucket.org:ChemicalHeritageFoundation/ansible-inventory.git
$ cd ansible-inventory
$ git submodule update --init

Subsequently, when you pull ansible-inventory and the submodule has been updated, just run

$ git submodule update

AWS - add new user

IAM (identity & access management) > Users > Create New Users.
Enter user name, uncheck 'generate an access key', create
select newly-created username > 'permissions' tab > Attach policy (you can look at policies on other users if needed)
'security credentials' tab > 'assign an auto-generated password', check 'require user to create a new password at next sign-in'
download the security credentials.
(another option is to have the person stand right next to you and give themselves a temporary password)
Stick the credentials somewhere safe!
Get user to sign on and reset password.

General Notes

Notes from conversations with Alicia

AWS Web Console

Administration & security > IAM

issue and revoke access keys
ansible_access policy is copied from a full access policy.
- Need to disallow some things –
  - anything that allows termination of services (destroy. note: stopping is okay)
  - "describe" commands also not great
  - The user can't create users, manage policies, etc, so that's a good start.
delete snapshot, volume are useful for backups management (note: Alicia says there is no cost associated with snapshots)
These can also be made more granular, e.g. under 'Resource' -> that's the entire account but can be broken into individual machines or groups

Instance: keypair column. key pair -> ssh connection for the instance.

it's set once and cannot be changed.
you can add more keys on the machine itself.
You could also create an image from your instance, spin up another server w/ a new keypair, and use the image (e.g. if you lose the key)

Security group: the firewall configuration. any machine can have multiple security groups. the most permissive rule on each port is used.

Alicia tends to name this after the keypair.
if on CentOS (which we are not) you must remember to make iptables match these rules
do we still need to close port 8080 or has that been done in ec2 scripts? - done via console

User - can change access keys if these credentials are leaked

AWS Architecture

pricing - unless you make a specific decision it's pay-as-you-go

Architecture design: 1 dev machine, 1 higher end. Fedora data on separate volume. b/u every 24 hrs. Keep every 7 days + first of every month (keep 2)

Instances

Root device (disc at '/') typically ~8G, just for OS. EBS -> persistent storage

Block devices (e.g. /dev/sdb) mounted at '/opt' to hold solr and fedora

3rd device for fedora data

Use DNS or elastic IPs to swap machines

Volumes

See mounted discs. State shows whether they're attached

Misc

snapshots
elastic IPs - external IPs you can move between machines to avoid messing with DNS
AMI - Amazon Machine Image. like a snapshot of a machine. Can be used to script ways to do as-needed load balancing, e.g.

Security

Keep names of users secret, along with keys, other obvious things

The stack

Web server will be Apache, due to Alicia's greater experience with that server

Installing Ruby from source

Advantages: standard, stable, known version of ruby (no problems with apt updates coming at a bad time)
Disadvantage: security bugs – always stay a version behind

Postgres uses peer authentication

superuser is the postgres user, who can log in to db without a password (auto-authenticates when you are this user)
Alicia changes default auth to MD5, makes a new user, with restricted permissions, for the rails database
settings in database.yml

Solr

app/config and tomcat both need to know the name of the solr instance. Alicia has called it hydra.xml, but she's making it a variable. Not a security issue b/c we close the port (8080) to all but localhost. Not sure why i have a note that says $curl localhost:8080/my_context/

Use an ssh tunnel if you want to access your solr instance via the web without messing with the firewall!

$ ssh -L 2020:localhost:8080 target_machine
set up target_machine in sshconfig or remember you have to specify user, ip, cert, etc.
in browser, to go localhost:2020/solr_context
(things might break if there are redirects with ports in them)

Capistrano

shared files (linked files) are for two types of things

1. things that are stored in the repo without production configuration
2. things that for other reasons you don't want to change from one deployment to the next (this is mostly directories, like tmp/pids - you don't want old tmp/pids directories hanging around with processes in limbo)

there's room for various interpretations - for example, on sandbox deployments I like to keep the log directories per-deployment

that way if a deployment fails, I can easily isolate the relevant log messages, even after reverting to an earlier, functioning deployment so testing can continue

but for production environments, sharing the log directory and rotating the files makes more sense - especially if you test everything in a staging environment first so you only deploy successful code

Ansible

FYI, the ansible.cfg file included in the repo turns off host checking for ec2 instances. if you get rid of that, your playbook will fail after prompting you to add the RSA fingerprint to your known hosts list

there are two ec2-related bits

the launch-ec2 one does the heavy lifting - creates the instance and then creates and attaches a volume
then the ec2 role puts aws-specific tools on the box for backups - for obvious reasons, that had to come later in the process
Backup script pulls the same variable used to create the backups, so it can vary by machine (i.e. the production machine could use "CHF-prod" and the staging machine "CHF-stage") and each machine will delete its own backups over time.

vault

The credentials in the vaulted files are all new. For backups, I generated a new IAM user, new credentials, and a policy that only has access to snapshots. For creating instances, I generated new credentials on the existing IAM user and turned off the old credentials (because they will be on GitHub now if you know how to find them).

AWS, Ansible, & Capistrano