Technical Track Nosql

Ansible and AWS automation

7 min read

May 11, 2017

Overview

The goal was to create and manage a MySQL RDS instance and learn how to control AWS resources by using Ansible. With Ansible interacting with AWS it's best to determine how hard or easy it is to build and remove items in EC2, to determine the validity of Ansible for automation purposes in AWS. The big drive for a lot of companies is to implement some or all of their infrastructure into the cloud and with Amazon being one of the largest cloud providers it seemed the logical choice to start with. Ansibles main goal is to make automation of your infrastructure quick and simple, but how does it integrate with AWS? As a MySQL DBA, the ultimate goal is to develop more automated tasks around RDS and Aurora. This post will help develop a basic or initial understanding of using Ansible with AWS to be able to continue working on further automation opportunities.

AWS Authentication with Ansible

The first requirement is to authenticate to AWS and allow Ansible to execute commands into AWS to discover, build, and destroy objects. This was accomplished pretty quickly. In the interest of time, the export method of the credentials was used for the AWS User (IAM) Key and secret. The more preferred method is to use the Ansible Vault to securely store the credentials: https://docs.ansible.com/ansible/guide_aws.html https://docs.ansible.com/ansible/playbooks_vault.html Credentials

export AWS_ACCESS_KEY_ID=''
 export AWS_SECRET_ACCESS_KEY=''

Ansible Site

To help make the roles reusable and easily updated, the variables were placed in the main site.yml file for configuring all of the aspects from the network, bastion, and RDS. This helped centralize the configuration of the playbooks. One key thing to note about Ansible interacting with AWS is that the hosts parameter does not come into effect and it is configured for localhost. Site.yml

---
 # This playbook deploys the whole AWS Network, Bastion, Web, and RDS Instances. Web Configuration is currently manual
 # - Must have credentials exported for AWS IAM
 # - RDS may take up to 30 minutes to deploy the instance
 
 - name: Deploy RDS Infrastructure
  hosts: localhost
  connection: local
  gather_facts: false
  vars:
 
 # Global AWS Variables
  region: us-west-2
  aws_network_name: dev
 
 # VPC Variables
  vpc_cidr: 10.4.0.0/16
  public_subnet_cidr: 10.4.0.0/24
  public_subnet_az: us-west-2a
  private_subnet_1_cidr: 10.4.1.0/24
  private_subnet_1_az: us-west-2b
  private_subnet_2_cidr: 10.4.2.0/24
  private_subnet_2_az: us-west-2c
 
 # Bastion Variables
  ec2_id: "ami-4836a428"
  key_pair: "pythian-markwardt-west"
 
 # RDS Variables
  rds_user: root
  rds_pass: 
  rds_instance_type: db.m1.small
  rds_size_gb: 15
  rds_parameter_engine: mysql5.6
  rds_instance_engine: 5.6.34
  rds_parameters:
  - { param: 'binlog_format', value: 'ROW' }
  - { param: 'general_log', value: '1' }
 
  roles:
  - aws-network
  - aws-bastion
  - aws-rds

Building the AWS Network

Building the base network was also pretty easy for a first timer using Ansible AWS modules. There was a small learning curve, but ultimately it was pretty straight forward and the documentation (and examples) helped simplify a lot of the configuration. This required some pre-existing knowledge of setting up and configuring an AWS network. Below is a list of the primary components that were focused on in order to get an operating network with routing and internet access. - Build VPC - Build Subnets - 1 Public subnet to access the environment - 2 Private subnets for internal traffic. Two because the RDS Subnet group requires two for redundancy. - Internet Gateway for the VPC - Routing Table to allow the three subnets to talk to one another and then send any non subnet traffic to the Internet Gateway - Security groups to allow specific traffic into specific instances Building AWS Network Tasks

---
 - name: Build VPC
  ec2_vpc_net:
  name: "-vpc"
  state: present
  cidr_block: ""
  region: ""
 
 - name: Get VPC ID
  ec2_vpc_net_facts:
  region: ""
  filters:
  "tag:Name": "-vpc"
  register: vpc_facts
 
 - name: VPC ID
  debug: 
  var: vpc_facts['vpcs'][0]['id']
 
 - name: Create Private Subnet 1
  ec2_vpc_subnet:
  state: present
  vpc_id: ""
  cidr: ""
  region: ""
  az: ""
  resource_tags:
  Name: "-private1"
  register: private_subnet_1
 
 - name: subnet MySQL database servers ID 1
  debug: 
  var: private_subnet_1['subnet']['id']
 
 - name: Create Private Subnet 2
  ec2_vpc_subnet:
  state: present
  vpc_id: ""
  cidr: ""
  region: ""
  az: ""
  resource_tags:
  Name: "-private2"
  register: private_subnet_2
 
 - name: subnet MySQL database servers ID 2
  debug: 
  var: private_subnet_2['subnet']['id']
 
 - name: Create public subnet
  ec2_vpc_subnet:
  state: present
  vpc_id: ""
  cidr: ""
  region: ""
  az: ""
  resource_tags:
  Name: "-public"
  register: public_subnet
 
 - name: Subnet public ID
  debug: 
  var: public_subnet['subnet']['id']
 
 - name: Create internet gateway 
  ec2_vpc_igw:
  vpc_id: ""
  state: present
  region: ""
  register: igw
 
 - name: Internet gateway ID
  debug: 
  var: igw['gateway_id']
 
 - name: Set up public subnet route table
  ec2_vpc_route_table:
  vpc_id: ""
  region: ""
  state: present
  tags:
  Name: "-public"
  subnets:
  - ""
  - ""
  - ""
  routes:
  - dest: 0.0.0.0/0
  gateway_id: ""
  register: public_route_table
 
 - name: Bastion Security group
  ec2_group:
  name: "-bastion"
  state: present
  description: Security group for SSH Bastion to get into the servers
  vpc_id: ""
  region: ""
  rules:
  - proto: tcp
  from_port: 22
  to_port: 22
  cidr_ip: 0.0.0.0/0
  register: bastion_sg
 
 - name: Bastion Security group ID
  debug: 
  var: bastion_sg['group_id']
 
 - name: Web Security group
  ec2_group:
  name: "-web"
  state: present
  description: Security group for SSH Bastion to get into the servers
  vpc_id: ""
  region: ""
  rules:
  - proto: tcp
  from_port: 80
  to_port: 80
  cidr_ip: 0.0.0.0/0
  - proto: tcp
  from_port: 22
  to_port: 22
  group_id: ""
  register: web_sg
 
 - name: Web Security group ID
  debug: 
  var: web_sg['group_id']
 
 - name: MySQL Security group
  ec2_group:
  name: "-private"
  state: present
  description: Security group for private access
  vpc_id: ""
  region: ""
  rules:
  - proto: tcp
  from_port: 80
  to_port: 80
  cidr_ip: 0.0.0.0/0
  - proto: tcp
  from_port: 22
  to_port: 22
  group_id: ""
  - proto: tcp
  from_port: 3306
  to_port: 3306
  group_id: ""
  register: mysql_sg
 
 - name: MySQL Security group ID
  debug: 
  var: mysql_sg['group_id']

Building the Bastion

Now there's a working network in place, the next step is to set up a bastion server to SSH in order to access the RDS instance that will be built in the next stage. It's bad security practice to make the RDS or Database publicly accessible to the internet. When building the bastion, it will be placed into the the public security group and subnet in order to allow access into the network, because the database will only be accessible internally. The bastion is created using the default AWS EC2 AMI for the particular region that it will reside. A custom AMI can also be used that has pre-built settings. Finally an SSH key or key pair was created and the site.yml updated with both the AMI and key pair being used for the bastion. When the playbook is executed, the output for the bastion creation will provide the public IP that gets assigned, and the public security group that it is placed into allows for port 22 SSH access. Building AWS Bastion Tasks

---
 - name: "Get Public Subnet ID"
  ec2_vpc_subnet_facts:
  region: ""
  filters:
  "tag:Name": "-public"
  register: public_subnet
 
 - name: Subnet ID
  debug:
  var: public_subnet['subnets'][0]['id']
 
 - name: Get bastion SG ID
  ec2_group_facts:
  region: ""
  filters:
  group-name: "-bastion"
  register: bastion_sg
 
 - name: Bastion SG ID
  debug:
  var: bastion_sg['security_groups'][0]['group_id']
 
 - name: Create Bastion
  ec2:
  region: ""
  key_name: ""
  group_id: ""
  instance_type: t2.micro
  image: ""
  wait: yes
  wait_timeout: 500
  exact_count: 1
  instance_tags:
  Name: "-bastion"
  Environment: Dev
  count_tag: 
  Name: "-bastion"
  Environment: Dev
  vpc_subnet_id: ""
  assign_public_ip: yes
  register: bastion_facts
 
 - debug: "var=bastion_facts"
 
 - name: Capture Bastion public IP
  set_fact: 
  bastion_public_ip: ""
  when: bastion_facts['instances'] | length > 0
 
 - name: Capture Bastion public IP
  set_fact: 
  bastion_public_ip: ""
  when: bastion_facts['tagged_instances'] | length > 0
 
 - name: Display Public IP
  debug:
  var: bastion_public_ip

Building RDS

Now the network is in place and the bastion is built in order to get logged into the network. It's now time to create the RDS instance. Before creating the RDS instance a subnet group needs to be configured where the RDS instance will reside, and then a new parameters configuration is needed in order to set up the RDS instance. There are only a couple of simple configurations done in this playbook and this is sufficient to validate that the process is working. But if this was for a customer, it would have far more customizations to meet the customer's needs. Building RDS Tasks

---
 # tasks file for aws-rds
 - name: "Get Subnet ID for markwardt-private1"
  ec2_vpc_subnet_facts:
  region: ""
  filters:
  "tag:Name": "-private1"
  register: subnet_private1
  
 - name: "Get Subnet ID for -private2"
  ec2_vpc_subnet_facts:
  region: ""
  filters:
  "tag:Name": "-private2"
  register: subnet_private2
 
 - name: Get MySQL SG ID
  ec2_group_facts:
  region: ""
  filters:
  group-name: "-private"
  register: private_sg
 
 - name: Build RDS Subnet Group
  rds_subnet_group:
  region: ""
  state: present
  name: "-subnetgroup"
  description: Subnet Group for 
  subnets:
  - ""
  - ""
  register: subnet_group_results
 
 - name: Subnet group results
  debug:
  var: subnet_group_results
 
 - name: Build MySQL Parameters
  rds_param_group:
  state: present
  name: "-parameters"
  description: " Parameters"
  engine: ""
  immediate: no
  region: ""
  params: "{{item.param}}={{item.value}}"
  with_items: ""
 
 - name: Build RDS Instance
  rds:
  command: create
  instance_name: "-rds"
  db_engine: MySQL
  size: ""
  instance_type: ""
  username: ""
  password: ""
  region: ""
  subnet: "-subnetgroup"
  parameter_group: "-parameters"
  engine_version: ""
  vpc_security_groups: ""
  wait: yes
  wait_timeout: 1800
  multi_zone: yes
  tags:
  Environment: ""
  register: rds_results
 
 - name: RDS rds results
  debug:
  var: rds_results

Cleaning up

When working in the cloud, there are a lot of scenarios where cleaning up your infrastructure is a good idea. Below are some playbooks that almost completely removes all of the instances that were created (parameters, subnet groups), and all of the network components. The RDS snapshots were incapable of being cleaned up successfully. These would have to be automated using another method outside of Ansible modules as the current Ansible modules do not allow for discovery to remove the RDS snapshots. AWS RDS Cleanup Tasks

- hosts: localhost
  connection: local
  vars:
  region: "us-west-2"
  aws_network_name: dev
  tasks:
  - name: Remove RDS Instance
  rds:
  region: ""
  command: delete
  instance_name: "-rds"
  wait: yes
  wait_timeout: 1800
  - name: Remove RDS Subnet Group
  rds_subnet_group:
  region: ""
  state: absent
  name: "-subnetgroup"
  - name: Build MySQL Parameters
  rds_param_group:
  region: ""
  state: absent
  name: "-parameters"

AWS Bastion Cleanup Tasks

- hosts: localhost
  connection: local
  vars:
  region: "us-west-2"
  aws_network_name: dev
  tasks:
  - name: Get Bastion EC2 ID
  ec2_remote_facts:
  region: ""
  filters:
  "tag:Name": "-bastion"
  register: ec2_facts
  
  - name: Destroy Bastion
  ec2:
  instance_ids: "{{item.id}}"
  state: absent
  region: ""
  with_items: ""

AWS Network Cleanup Tasks

- hosts: localhost
  connection: local
  vars:
  region: "us-west-2"
  aws_network_name: dev
  vpc_cidr: 10.4.0.0/16
  public_subnet_cidr: 10.4.0.0/24
  private_subnet_1_cidr: 10.4.1.0/24
  private_subnet_2_cidr: 10.4.2.0/24
  gather_facts: False
  tasks:
  - name: Get VPC ID
  ec2_vpc_net_facts:
  region: ""
  filters:
  "tag:Name": "-vpc"
  register: vpc_facts
 
  - name: Destroy SG MySQL
  ec2_group:
  name: "-private"
  state: absent
  region: ""
 
  - name: Destroy SG Web
  ec2_group:
  name: "-web"
  state: absent
  region: ""
 
  - name: Destroy SG Bastion
  ec2_group:
  name: "-bastion"
  state: absent
  region: ""
 
  - name: Remove subnet route table
  ec2_vpc_route_table:
  region: ""
  state: absent
  tags:
  Name: "-public"
  vpc_id: "" 
  
  - name: Remove Public Subnet 
  ec2_vpc_subnet:
  state: absent
  cidr: ""
  region: ""
  vpc_id: ""
 
  - name: Remove Private Subnet 1
  ec2_vpc_subnet:
  state: absent
  cidr: ""
  region: ""
  vpc_id: ""
 
  - name: Remove Private Subnet 2
  ec2_vpc_subnet:
  state: absent
  cidr: ""
  region: ""
  vpc_id: ""
 
  - name: Create internet gateway 
  ec2_vpc_igw:
  vpc_id: ""
  state: absent
  region: ""
 
  - name: Remove VPC
  ec2_vpc_net:
  name: "-vpc"
  state: absent
  cidr_block: ""
  region: ""

Summary

Ansible was very enjoyable and easy to work with. The same goes for using Ansible to manage AWS. Even though there are other tools that can be used such as Terraform (https://www.terraform.io/), Ansible was pretty straightforward and intuitive for creating each of the components, discovering them, and then cleaning them up. It made it nice and simple to search for objects by tag, so it was able to tag the AWS pieces appropriately to allow for easy management of those objects dynamically. In coordination with using Ansible to integrate with AWS to build the network and instances, Ansible can then be used to manage and configure the EC2 operating systems as needed. The Ansible AWS modules make it almost a total package for building and managing an infrastructure in AWS.