Skip to content

Commit d197e83

Browse files
authored
Merge pull request #5063 from IQSS/4990-ec2-ansible-scripting
4990 ec2 ansible scripting
2 parents 5390f28 + 746eec5 commit d197e83

9 files changed

Lines changed: 254 additions & 4 deletions

File tree

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,3 +43,4 @@ conf/docker-aio/dv/install/dvinstall.zip
4343
# or copy of test data
4444
conf/docker-aio/testdata/
4545
scripts/installer/default.config
46+
*.pem

doc/sphinx-guides/source/developers/coding-style.rst

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,20 @@ If you just downloaded Netbeans and are using the out-of-the-box settings, you s
9393

9494
If you know of a way to easily share Netbeans configuration across a team, please get in touch.
9595

96+
Bash
97+
----
98+
99+
Generally, Google's Shell Style Guide at https://google.github.io/styleguide/shell.xml seems to have good advice.
100+
101+
Formatting Code
102+
~~~~~~~~~~~~~~~
103+
104+
Tabs vs. Spaces
105+
^^^^^^^^^^^^^^^
106+
107+
Don't use tabs. Use 2 spaces.
108+
109+
shfmt from https://github.com/mvdan/sh seems like a decent way to enforce indentation of two spaces (i.e. ``shfmt -i 2 -w path/to/script.sh``) but be aware that it makes other changes.
96110

97111
Bike Shedding
98112
-------------
@@ -103,4 +117,4 @@ Come debate with us about coding style in this Google doc that has public commen
103117

104118
----
105119

106-
Previous: :doc:`debugging` | Next: :doc:`containers`
120+
Previous: :doc:`debugging` | Next: :doc:`deployment`

doc/sphinx-guides/source/developers/containers.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -410,4 +410,4 @@ Again, Dataverse Docker images on Docker Hub are highly experimental at this poi
410410

411411
----
412412

413-
Previous: :doc:`coding-style` | Next: :doc:`making-releases`
413+
Previous: :doc:`deployment` | Next: :doc:`making-releases`
Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
==========
2+
Deployment
3+
==========
4+
5+
Developers often only deploy Dataverse to their :doc:`dev-environment` but it can be useful to deploy Dataverse to cloud services such as Amazon Web Services (AWS).
6+
7+
.. contents:: |toctitle|
8+
:local:
9+
10+
Deploying Dataverse to Amazon Web Services (AWS)
11+
------------------------------------------------
12+
13+
We have written scripts to deploy Dataverse to Amazon Web Services (AWS) but they require some setup.
14+
15+
Install AWS CLI
16+
~~~~~~~~~~~~~~~
17+
18+
First, you need to have AWS Command Line Interface (AWS CLI) installed, which is called ``aws`` in your terminal. Launching your terminal and running the following command to print out the version of AWS CLI will tell you if it is installed or not:
19+
20+
``aws --version``
21+
22+
If you have not yet installed AWS CLI you should install it by following the instructions at https://docs.aws.amazon.com/cli/latest/userguide/installing.html
23+
24+
Afterwards, you should re-run the "version" command above to verify that AWS CLI has been properly installed. If "version" still doesn't work, read on for troubleshooting advice. If "version" works, you can skip down to the "Configure AWS CLI" step.
25+
26+
Troubleshooting "aws: command not found"
27+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
28+
29+
Please note that as of this writing the AWS docs are not especially clear about how to fix errors such as ``aws: command not found``. If the AWS CLI cannot be found after you followed the AWS installation docs, it is very likely that the ``aws`` program is not in your ``$PATH``. ``$PATH`` is an "environment variable" that tells your shell (usually Bash) where to look for programs.
30+
31+
To see what ``$PATH`` is set to, run the following command:
32+
33+
``echo $PATH``
34+
35+
On Mac, to update your ``$PATH`` to include the location where the current AWS docs install AWS CLI on the version of Python included with your Mac, run the following command:
36+
37+
``export PATH=$PATH:$HOME/Library/Python/2.7/bin``
38+
39+
After all this, you can try the "version" command again.
40+
41+
Note that it's possible to add an ``export`` line like the one above to your ``~/.bash_profile`` file so you don't have to run it yourself when you open a new terminal.
42+
43+
Configure AWS CLI
44+
~~~~~~~~~~~~~~~~~
45+
46+
Next you need to configure AWS CLI.
47+
48+
Create a ``.aws`` directory in your home directory (which is called ``~``) like this:
49+
50+
``mkdir ~/.aws``
51+
52+
We will be creating two plain text files in the ``.aws`` directory and it is important that these files do not end in ".txt" or any other extension. After creating the files, you can verify their names with the following command:
53+
54+
``ls ~/.aws``
55+
56+
Create a plain text file at ``~/.aws/config`` with the following content::
57+
58+
[default]
59+
region = us-east-1
60+
61+
Please note that at this time the region must be set to "us-east-1" but in the future we could improve our scripts to support other regions.
62+
63+
Create a plain text file at ``~/.aws/credentials`` with the following content::
64+
65+
[default]
66+
aws_access_key_id = XXXXXXXXXXXXXXXXXXXX
67+
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
68+
69+
Then update the file and replace the values for "aws_access_key_id" and "aws_secret_access_key" with your actual credentials by following the instructions at https://aws.amazon.com/blogs/security/wheres-my-secret-access-key/
70+
71+
If you are having trouble configuring the files manually as described above, see https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html which documents the ``aws configure`` command.
72+
73+
Download and Run the "Create Instance" Script
74+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
75+
76+
Once you have done the configuration above, you are ready to try running the "create instance" script to spin up Dataverse in AWS.
77+
78+
Download :download:`ec2-create-instance.sh <../../../../scripts/installer/ec2-create-instance.sh>` and put it somewhere reasonable. For the purpose of these instructions we'll assume it's in the "Downloads" directory in your home directory.
79+
80+
You need to decide which branch you'd like to deploy to AWS. Select a branch from https://github.com/IQSS/dataverse/branches/all such as "develop" and pass it to the script with ``-b`` as in the following example. (Branches such as "master" and "develop" are described in the :doc:`version-control` section.)
81+
82+
``bash ~/Downloads/ec2-create-instance.sh -b develop``
83+
84+
You must specify the branch with ``-b`` but you can also specify a non-IQSS git repo URL with ``-r`` as in the following example.
85+
86+
``bash ~/Downloads/ec2-create-instance.sh -b develop -r https://github.com/scholarsportal/dataverse.git``
87+
88+
Now you will need to wait around 15 minutes until the deployment is finished. Eventually, the output should tell you how to access the installation of Dataverse in a web browser or via ssh. It will also provide instructions on how to delete the instance when you are finished with it. Please be aware that AWS charges per minute for a running instance. You can also delete your instance from https://console.aws.amazon.com/console/home?region=us-east-1 .
89+
90+
Caveats
91+
~~~~~~~
92+
93+
Please note that while the script should work fine on newish branches, older branches that have different dependencies such as an older version of Solr are now expected to yield a working Dataverse installation. Your mileage may vary.
94+
95+
----
96+
97+
Previous: :doc:`coding-style` | Next: :doc:`containers`

doc/sphinx-guides/source/developers/index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ Developer Guide
2020
documentation
2121
debugging
2222
coding-style
23+
deployment
2324
containers
2425
making-releases
2526
tools

doc/sphinx-guides/source/installation/prep.rst

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,10 +38,12 @@ Installing Dataverse involves some system configuration followed by executing an
3838
Advanced Installation
3939
+++++++++++++++++++++
4040

41-
There are some community-lead projects to use configuration management tools such as Puppet and Ansible to automate Dataverse installation and configuration, but support for these solutions is limited to what the Dataverse community can offer as described in each project's webpage:
41+
There are some community-lead projects to use configuration management tools such as Ansible and Puppet to automate Dataverse installation and configuration, but support for these solutions is limited to what the Dataverse community can offer as described in each project's webpage:
4242

43-
- https://github.com/IQSS/dataverse-puppet
4443
- https://github.com/IQSS/dataverse-ansible
44+
- https://github.com/IQSS/dataverse-puppet
45+
46+
(Please note that the "dataverse-ansible" repo is used in a script that allows Dataverse to be installed on Amazon Web Services (AWS) from arbitrary GitHub branches as described in the :doc:`/developers/deployment` section of the Developer Guide.)
4547

4648
The Dataverse development team is happy to "bless" additional community efforts along these lines (i.e. Docker, Chef, Salt, etc.) by creating a repo under https://github.com/IQSS and managing team access.
4749

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
#!/bin/bash
2+
3+
# For docs, see the "Deployment" page in the Dev Guide.
4+
5+
SUGGESTED_REPO_URL='https://github.com/IQSS/dataverse.git'
6+
SUGGESTED_BRANCH='develop'
7+
8+
usage() {
9+
echo "Usage: $0 -r $REPO_URL -b $SUGGESTED_BRANCH" 1>&2
10+
exit 1
11+
}
12+
13+
REPO_URL=$SUGGESTED_REPO_URL
14+
15+
while getopts ":r:b:" o; do
16+
case "${o}" in
17+
r)
18+
REPO_URL=${OPTARG}
19+
;;
20+
b)
21+
BRANCH_NAME=${OPTARG}
22+
;;
23+
*)
24+
usage
25+
;;
26+
esac
27+
done
28+
29+
AWS_CLI_VERSION=$(aws --version)
30+
if [[ "$?" -ne 0 ]]; then
31+
echo 'The "aws" program could not be executed. Is it in your $PATH?'
32+
exit 1
33+
fi
34+
35+
if [ "$BRANCH_NAME" = "" ]; then
36+
echo "No branch name provided. You could try adding \"-b $SUGGESTED_BRANCH\" or other branches listed at $SUGGESTED_REPO_URL"
37+
usage
38+
exit 1
39+
fi
40+
41+
if [[ $(git ls-remote --heads $REPO_URL $BRANCH_NAME | wc -l) -eq 0 ]]; then
42+
echo "Branch \"$BRANCH_NAME\" does not exist at $REPO_URL"
43+
usage
44+
exit 1
45+
fi
46+
47+
SECURITY_GROUP='dataverse-sg'
48+
GROUP_CHECK=$(aws ec2 describe-security-groups --group-name $SECURITY_GROUP)
49+
if [[ "$?" -ne 0 ]]; then
50+
echo "Creating security group \"$SECURITY_GROUP\"."
51+
aws ec2 create-security-group --group-name $SECURITY_GROUP --description "security group for Dataverse"
52+
aws ec2 authorize-security-group-ingress --group-name $SECURITY_GROUP --protocol tcp --port 22 --cidr 0.0.0.0/0
53+
aws ec2 authorize-security-group-ingress --group-name $SECURITY_GROUP --protocol tcp --port 80 --cidr 0.0.0.0/0
54+
aws ec2 authorize-security-group-ingress --group-name $SECURITY_GROUP --protocol tcp --port 443 --cidr 0.0.0.0/0
55+
aws ec2 authorize-security-group-ingress --group-name $SECURITY_GROUP --protocol tcp --port 8080 --cidr 0.0.0.0/0
56+
fi
57+
58+
RANDOM_STRING="$(uuidgen | cut -c-8)"
59+
KEY_NAME="key-$USER-$RANDOM_STRING"
60+
61+
PRIVATE_KEY=$(aws ec2 create-key-pair --key-name $KEY_NAME --query 'KeyMaterial' --output text)
62+
if [[ $PRIVATE_KEY == '-----BEGIN RSA PRIVATE KEY-----'* ]]; then
63+
PEM_FILE="$KEY_NAME.pem"
64+
printf -- "$PRIVATE_KEY" >$PEM_FILE
65+
chmod 400 $PEM_FILE
66+
echo "Your newly created private key file is \"$PEM_FILE\". Keep it secret. Keep it safe."
67+
else
68+
echo "Could not create key pair. Exiting."
69+
exit 1
70+
fi
71+
72+
# The AMI ID may change in the future and the way to look it up is with the
73+
# following command, which takes a long time to run:
74+
#
75+
# aws ec2 describe-images --owners 'aws-marketplace' --filters 'Name=product-code,Values=aw0evgkw8e5c1q413zgy5pjce' --query 'sort_by(Images, &CreationDate)[-1].[ImageId]' --output 'text'
76+
#
77+
# To use this AMI, we subscribed to it from the AWS GUI.
78+
# AMI IDs are specific to the region.
79+
AMI_ID='ami-9887c6e7'
80+
# Smaller than medium lead to Maven and Solr problems.
81+
SIZE='t2.medium'
82+
echo "Creating EC2 instance"
83+
# TODO: Add some error checking for "ec2 run-instances".
84+
INSTANCE_ID=$(aws ec2 run-instances --image-id $AMI_ID --security-groups $SECURITY_GROUP --count 1 --instance-type $SIZE --key-name $KEY_NAME --query 'Instances[0].InstanceId' --block-device-mappings '[ { "DeviceName": "/dev/sda1", "Ebs": { "DeleteOnTermination": true } } ]' | tr -d \")
85+
echo "Instance ID: "$INSTANCE_ID
86+
echo "End creating EC2 instance"
87+
88+
PUBLIC_DNS=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID --query "Reservations[*].Instances[*].[PublicDnsName]" --output text)
89+
PUBLIC_IP=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID --query "Reservations[*].Instances[*].[PublicIpAddress]" --output text)
90+
91+
USER_AT_HOST="centos@${PUBLIC_DNS}"
92+
echo "New instance created with ID \"$INSTANCE_ID\". To ssh into it:"
93+
echo "ssh -i $PEM_FILE $USER_AT_HOST"
94+
95+
echo "Please wait at least 15 minutes while the branch \"$BRANCH_NAME\" from $REPO_URL is being deployed."
96+
97+
# epel-release is installed first to ensure the latest ansible is installed after
98+
# TODO: Add some error checking for this ssh command.
99+
ssh -T -i $PEM_FILE -o 'StrictHostKeyChecking no' -o 'UserKnownHostsFile=/dev/null' -o 'ConnectTimeout=300' $USER_AT_HOST <<EOF
100+
sudo yum -y install epel-release
101+
sudo yum -y install git nano ansible
102+
git clone https://github.com/IQSS/dataverse-ansible.git dataverse
103+
export ANSIBLE_ROLES_PATH=.
104+
ansible-playbook -i dataverse/inventory dataverse/dataverse.pb --connection=local --extra-vars "dataverse_branch=$BRANCH_NAME dataverse_repo=$REPO_URL"
105+
EOF
106+
107+
# Port 8080 has been added because Ansible puts a redirect in place
108+
# from HTTP to HTTPS and the cert is invalid (self-signed), forcing
109+
# the user to click through browser warnings.
110+
CLICKABLE_LINK="http://${PUBLIC_DNS}:8080"
111+
echo "To ssh into the new instance:"
112+
echo "ssh -i $PEM_FILE $USER_AT_HOST"
113+
echo "Branch \"$BRANCH_NAME\" from $REPO_URL has been deployed to $CLICKABLE_LINK"
114+
echo "When you are done, please terminate your instance with:"
115+
echo "aws ec2 terminate-instances --instance-ids $INSTANCE_ID"
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
#!/bin/bash
2+
3+
#This script gets all the instances from ec2 and sends terminate to them
4+
#Its pretty basic and probably shouldn't be trusted at this point. Namely:
5+
# - You can kill instances other people are using
6+
# - It will try to kill instances that are already dead, which makes output hard to read
7+
# - If it fails for some reason it's hard to tell the script didn't work right
8+
9+
INSTANCES=$(aws ec2 describe-instances --query 'Reservations[].Instances[].[InstanceId]' --output text)
10+
11+
aws ec2 terminate-instances --instance-ids $INSTANCES

scripts/installer/ec2-list-all.sh

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
#!/bin/bash
2+
# https://docs.aws.amazon.com/cli/latest/userguide/controlling-output.html
3+
INSTANCES=$(aws ec2 describe-instances --query 'Reservations[].Instances[].[InstanceId,KeyName,State.Name,PublicDnsName]' --output text)
4+
if [[ "$?" -ne 0 ]]; then
5+
echo "Error listing instances."
6+
exit 1
7+
else
8+
echo "$INSTANCES"
9+
fi

0 commit comments

Comments
 (0)