Browsed by
Tag: ansible

Looping Through Ansible Hosts

Looping Through Ansible Hosts

For the past week I have been working on deploying Hashicorp’s Vault using Terraform and Ansible. As I was installing and configuring the Consul server, I came across an interesting problem with building the server configuration. I’ve been following instructions from DigitalOcean and while most of the configuration has been pretty straight forward, the config.json file proved to be a bit of a challenge.

According to the instructions, for a three node cluster, you only want to put the IP addresses for the other two server into the config.json file for the server, but it took me a little while to figure out how to get Ansible to do that. While it may seem straight forward to others, I had a hard time even finding information on the internet on how to do this, so I figured I would share it here.

My first iteration was to just get it set up to do put all the IP addresses in. This was accomplished by this:

However, this is not what I wanted. I wanted it to exclude the hosts’s IP address. After searching high an low, I finally found a Jinja2 tidbit that would get me what I wanted. I didn’t realize that you could an if right in with the for loop, so I just needed to add “if server != inventory_hostname” into the for loop so that it would exclude the host it was running on.

I can see this little tidbit coming in handy for all sorts of things.

Looking at Open Source Cloud Foundry

Looking at Open Source Cloud Foundry

We have been using Pivotal’s version of Cloud Foundry for the past year and while it has a lot of nice features around it, there are a couple of things about it that I have found rather frustrating. Unfortunately, it is the very things that Pivotal adds on top of Cloud Foundry that I find the most frustrating.

The first and biggest frustration is that we haven’t been able to figure out how to effectively automate the deployment of the cloud foundry environment. While they do provide a Cloud Formation template that will build out the correct AWS bits, we found it wasn’t very good overall and ended up rewriting most of it to add some pretty important bits such as allowing us to pick our own IP addresses, encrypting the databases, and building out the subnets in multiple availability zones.

Once the cloud formation template is run and the Ops Manager is running, they provide the ability to deploy the Ops Manager Director (Bosh) tile that will then allow you to deploy the Elastic Runtime tile (Cloud Foundry). To deploy these tiles, you must click through a number of web forms and fill in the values that you want to use. While this may work well for the novice, I want to be able to deploy a cloud foundry from scratch using automation, not clicking through a bunch of web forms.

The web forms also point to the second big complaint (which admittedly may be a feature for some) is how much Pivotal obfuscates the inner workings of Cloud Foundry. Initially we took advantage of this when deploying the app, but as we ran it and the need for troubleshooting came up it became much more annoying. Not knowing where to go to look at logs, how to log in (properly) and how to check on system health became more of a problem as we started to run more production workloads.

So we have decided to look at standing up an open source Cloud Foundry environment to see if having more direct control over the pieces will allow us to better automate and support our infrastructure. The tools that I have chosen to build out our proof of concept (POC) environment are: Terraform, Ansible, and Jenkins. I’ll be using a lot of the hacks and tricks that we have learned over the last year in our Pivotal environment.

Testing Ansible Galaxy Roles

Testing Ansible Galaxy Roles

With the push to move our roles to Ansible Galaxy as much as possible, we needed to come up with a good way to test the roles as we write them. Up until now, we would build and test them completely within Ansible against the specific system type that we planned to run on. While this works ok against the focused roles that we were writing, it doesn’t work very well for generalized roles that are expected to run on the many different Linux distributions that we run at Blackbaud.

To solve this, we have come up with a Vagrant configuration that allows us to test against multiple OSs both locally (via VirtualBox or VMware) or in the cloud (AWS). You can check out code here. To get started, simple clone the project to you your local machine.

git clone git@github.com:MarsDominion/vagrant-ansible-testing.git

The Vagrantfile in the master branch provides three test environments: aws-linux, centos7, and ubuntu. The aws-linux role will build an Amazon Linux host in AWS while the CentOS and Ubuntu nodes environments are vmware_desktop based nodes that are pulled from Atlas. This gives me a way to test our roles against both cloud and local instances. If you don’t have VMware Fusion or Workstation, you can change the provider from vmware_desktop to virtualbox and they should work as well.

Before launching the instances, you need to download your ansible roles to run. This is done with the ansible-galaxy command.

% ansible-galaxy install blackbaud.linux-hardening

And then update your playbook to include the roles:

- hosts: all
   become: true
   roles:
     - blackbaud.linux-hardening

Finally, set some variables to be able to connect to your Amazon Environment:

AWS_ACCESS_KEY_ID=KIAI3XQCPIPKSDJHSVQ
AWS_SECRET_ACCESS_KEY=onX5HfdsIpasdH6+E+JJCgNxIfzJWY1btZgU4LfQ
AWS_KEYPAIR_NAME=test_key
MY_PRIVATE_AWS_SSH_KEY_PATH=$HOME/.ssh/test_key.pem

Now we are ready to test the

vagrant up
# Brings up all three instances and tests

vagrant up <aws-linux|centos7|ubuntu>
# Brings up the specified instance and tests

It will launch each instance and run through the the Ansible on each node and show you the results. It will jump right into the next node when it completes the previous one, so keep an eye on the output to see the results. When you are done, you can simply destroy the nodes.

vagrant destroy -f 
Ansible Role Style Guide

Ansible Role Style Guide

Ansible Role Style Guide

In my last post, I discussed how to get started with creating an Ansible Galaxy role. This post will go into more detail on what comprises a roles and how we use it at Blackbaud for building Ansible roles that can be reused and shared throughout the company.

By default, running the ansible-galaxy init command will create the following directory structure:

ansible-role-linux-hardening/
|- defaults/
    |- main.yml
|- handlers/
    |- main.yml
|- meta/
    |- main.yml
|- tasks/
    |- main.yml
|- tests/
    |- inventory
    |- test.yml
|- vars/
    |- main.yml
.travis.yml
README.md

Depending on the role that is being created, it is possible that some of these directories will not be needed. It is recommended that you remove the directories that are not being used. Also, a templates directory isn’t created with ansible-galaxy init, but you can add it if it is necessary.

defaults

The default folder contains the default values of any of the variables that are used throughout the rest of the role (You can learn more about variable precedence here. This should be reserved for variables that mainly don’t often stray from the defined value but that may in certain specific instances.

handlers

The handlers directory contains the process handlers that can be notified when changes occur. For example, you can have a handler that restarts SSH when changes to the configuration file are made:

- name: restart ssh
   service:
     name: "{{ ssh_service}}"
     state: restarted

meta

The meta directory serves two purposes. The first is to define the ansible galaxy role, including the version needed, what platforms are supported, etc, and to define any dependencies that the role may have.

galaxy_info:
  author: Mark Honomichl
  description: Ansible for Hardening a Linux Host
  company: Blackbaud
  license: MIT
  min_ansible_version: 1.2
  platforms:
    - name: EL
      versions:
        - all
    - name: Amazon
      versions:
        - all
    - name: Ubuntu
      versions:
        - all
  galaxy_tags:
    - security
    - linux
dependencies: []

tasks

Tasks is where all the tasks of the role are defined. With larger roles, it is recommended that seperate tasks files are created for similar tasks and then called in the main.yml file via the includes directive.

- name: Update Ubuntu
   apt:
     upgrade: dist
  when: ansible_distribution == "Ubuntu"
  tags: update_os

- include: "hostname.yml"

test

The tests directory is included so that the role can be tested via Travis CI. No changes should need to be made to this directory.

vars

The vars directory is primarily used to define variables that can be different across platforms or environments. For example, in ubuntu the ssh service is called “ssh” while in EL it is called “sshd”. Rather than writing two handlers, we can write it as defined in the above handlers section and then define ssh_service as a varaible in file called Ubuntu.yml and CentOS.yml respectively, and then include the correct file in the tasks section with an includes call.

- name: Include OS Specific Variables
   include_vars: "{{ ansible_distribution }}.yml" 

.travis.yml

The .travis.yml file is generated by default to allow the role to run in Travis CI. By default, it spins up a container, installs ansible, and runs a syntax check against the code. If possible, it is recommended that you actually run the playbook by adding a second ansible-playbook command without the –syntax-check option.

README.md

Once the role is completed, update the README.md file with the information requested. Make sure this document clearly defines how to use the role. Update the Author information with the following:

Blackbaud
Created in 2016 by [Blackbaud](http://blackbaud.com/)
Getting Started with Ansible Galaxy

Getting Started with Ansible Galaxy

I started using Ansible in December when I joined Blackbaud, and while I do feel like the team is doing some really innovative stuff with our Ansible roles, most of the stuff we do is pretty straightforward. After spending the last six months building up a monolithic repository, we are starting to examine how we can break the roles out into Ansible Galaxy roles that can better be shared across the company. I figured since I was documenting how to create a Galaxy compatible role for the company, I would go ahead and share those instructions here as well.

Create a GitHub Repository

The first step is to create an empty git repository on GitHub. This can be done by logging into your account and clicking on the ‘New repository’ button. We prefix all of our roles at Blackbaud with ‘ansible-role-’ so that we can easily distinguish them when looking at the hundreds of repos that we have. For example, the linux-hardening role that we have is called ‘ansible-role-linux-hardening’.
We create the repo without any files (.gitignore, README.me, or LICENSE) so that we can manage all of that properly later. We generally start all of our roles as private so that we can have a chance to build it out and test it before we make it public.

Create the Local Repository

Once our empty repo has been created at GitHub, the next step is to create the Ansible Galaxy skeleton directory structure on a local machine. This is done with the ‘ansible-galaxy’ command.

ansible-galaxy init ansible-role-<ROLE-NAME>

Where <ROLE-NAME> is the name of the role (i.e. linux-hardening). Once the files are created you need to initialize a git repository, add the files, and push them up to GitHub.

cd ansible-role-assume-role
git init
git add .
git commit -m "Commit Skeleton Role"
git remote add origin https://github.com/blackbaud/ansible-role-assume-role.git
git push -u origin master

Now you can begin editing your role.

Sharing Ansible Roles

Sharing Ansible Roles

When I started back in December, Blackbaud did not have a single line of code written to help build, deploy or manage our environments. The Platform Engineering team has come a long way in these last six months. As of this morning the team is just shy of 50K lines of code, consisting primarily of Ansible and Python.
I’m really proud of the work that the team has been doing around Ansible. Over the next few weeks I am hoping to share some of those innovations, not just in this blog, but also via github. However, the one place that I feel that we have been lacking is in how we have build our Ansible directory out. We have built this monolithic repository that contains everything we use.
Some people (even here at Blackbaud) like the thought of a mono-repo, something where everything exists and dependencies are easy to satisfy. In this instance, however, having a mono-repo makes it very difficult for us to share our ansible with other teams within the organization. It is an all or nothing proposition. If you want to use our linux-hardening role, you have to download everything and try to either move it to a new directory or figure out how to add stuff to our repo.
This is antithetical to everything that we are hoping to accomplish. We want to be able to share individual roles across the organization in an organized way that allows our engineering teams (and operational teams) to use what they need without having to sift through tons of roles they will never need.
To make it easier to share roles, we will begin moving our roles into individual repositories and utilizing Ansible Galaxy to share individual roles between teams. By building self-contained units, not only will it help us better share roles between teams, but it will allow us to share some of our roles with the community.
Over the next few weeks, I’ll be sharing some details of how we are doing that.