Browsed by
Tag: Terraform

Problems with Terraform

Problems with Terraform

In my last post, I discussed using Terraform to build out the base components of an AWS environment. While running this code to build out the base environment has worked the way I intended, I have run into some pretty major issues with building out the next layer, which consist of a group of private subnets.

I ran into two key problems that I haven’t been able to solve. The first is around passing the counts from one environment to the next. In my base environment I set them as outputs and then import the state file as a data source, but when I try to use it, I get the error “value of count cannot be computed.”

The second issue is a little more complicated, but it comes down to setting variables in the module section of the file when the data doesn’t exist in base statefile. Essentially, if I don’t create a second nat gateway in the base setup, the no output shows up in the statefile. When I run the second set of Terraform scripts, I would like it to ignore or default if it doesn’t exist, rather than error.

At this point, I am pretty frustrated with it. I have decided that I am going to circle back and take another look at CloudFormation now that they have support for YAML and cross-stack variable and see if I can do everything that I want to do. I’ll post details later this week.

Terraform to Buildout AWS

Terraform to Buildout AWS

I started playing with Terraform a few months ago when I needed to spin up a prototype environment to do an evaluation of the open source version of Cloud Foundry. One of the results was some Terraform code that could bring up the essentials of an AWS VPC, which included the VPC itself, three public subnets, three NATs, three Elastic IPs (EIPs), and a Route53 hosted zone. While it might seem like overkill to use this many Availability Zones (AZs) for a prototype environment, once of the things we needed to test was how Cloud Foundry’s new multi-az support worked.

This was good for what I was working on at the time since I needed to test across muliple AZs, but it was problematic for most of the day-to day testing that I need to do as it would spin up (and charge me for) components that I didn’t really need. Most things I have to test don’t require three of everything. The challenge was, I do not want to maintain different code repositories for different use cases.

Luckily, I came across an article by Yevgeniy Brikman that had some interesting tips and tricks on how to do some loops and conditionals. The most interesting bit for the problem I am trying to solve was learning that in Terraform, a boolean true is converted to a 1 and a boolean false is converted to a 0. Yevgeniy then used an example that I proceed to incorporate into my code. Essentially, what I did was create three new variables in my environments file to define whether or not I wanted to create each of the three public subnets:

# public-subnets module
public-1a_create = true
public-1b_create = false
public-1c_create = false

Then for each resource, I added the count variable:

resource "aws_subnet" "public-1a" {
    count                   = "${var.public-1a_create}"
    vpc_id                  = "${var.vpc_id}"
    cidr_block              = "${var.public-1a_subnet_cidr}"
    map_public_ip_on_launch = true
    availability_zone       = "us-east-1a"
    tags {
        Name                  = "public-1a"
        Description           = "Public Subnet us-east-1a"
        Terraform             = "true"

Now I am able to spin up an AWS environment that only is only in one availability zone to do some testing, or bring it up in three for production. There are still a few other things that I am hoping to figure out, such as how to not set environmental variables for the second and third subnets even though you don’t need them and how to let Terraform deployments that build on the base to use the right NAT subnets if they use three AZs and the their are only two NATS. You can find the code in this repo.

Storing Terraform State Files in S3 When Using assume_role

Storing Terraform State Files in S3 When Using assume_role

We use a lot of different AWS accounts, so rather than managing credentials across all of them we have built a model where one is strictly used for managing users accounts (both locally and via ADFS).  From there, all of our interactive and automated logins use STS to assume roles in other accounts.  As we started to dive more into Terraform, I was excited to find that they supported this with the aws_assume_role resource.

However, as we started to implement this, we quickly ran into a problem: the terraform command itself doesn’t support assuming a role in another account, so we would either need to store the state files in one account (in this case, our auth account), or figure out how to allow the auth account to put the files an S3 bucket of the account we are working with.  since I don’t want to store ANY data in the auth account, I had to go figure out how to give users from my auth account access to the account I am working on.  In the end, it was relatively straight forward.  I just need to add a bucket policy in the target account and a policy in the auth account that I then attached to my team’s user group.

The first step is to create a bucket policy that allows my user to list the contents of the file and to also be able to get and put the state files.  I could probably lock the policy down more, and only restrict it to the terraform-state folder that I have in my bucket, but since I have full access outside of terraform anyways, I didn’t think it was as important.  This is the policy I used:

Once the bucket policy was in place, I added the below role to my auth account and attached it to my user group.  I figure as I put more accounts under Terraform control, I’ll just add additional resources.

Once the two policies were in place, Terraform was able to use the S3 bucket in the account we building out.

Looking at Open Source Cloud Foundry

Looking at Open Source Cloud Foundry

We have been using Pivotal’s version of Cloud Foundry for the past year and while it has a lot of nice features around it, there are a couple of things about it that I have found rather frustrating. Unfortunately, it is the very things that Pivotal adds on top of Cloud Foundry that I find the most frustrating.

The first and biggest frustration is that we haven’t been able to figure out how to effectively automate the deployment of the cloud foundry environment. While they do provide a Cloud Formation template that will build out the correct AWS bits, we found it wasn’t very good overall and ended up rewriting most of it to add some pretty important bits such as allowing us to pick our own IP addresses, encrypting the databases, and building out the subnets in multiple availability zones.

Once the cloud formation template is run and the Ops Manager is running, they provide the ability to deploy the Ops Manager Director (Bosh) tile that will then allow you to deploy the Elastic Runtime tile (Cloud Foundry). To deploy these tiles, you must click through a number of web forms and fill in the values that you want to use. While this may work well for the novice, I want to be able to deploy a cloud foundry from scratch using automation, not clicking through a bunch of web forms.

The web forms also point to the second big complaint (which admittedly may be a feature for some) is how much Pivotal obfuscates the inner workings of Cloud Foundry. Initially we took advantage of this when deploying the app, but as we ran it and the need for troubleshooting came up it became much more annoying. Not knowing where to go to look at logs, how to log in (properly) and how to check on system health became more of a problem as we started to run more production workloads.

So we have decided to look at standing up an open source Cloud Foundry environment to see if having more direct control over the pieces will allow us to better automate and support our infrastructure. The tools that I have chosen to build out our proof of concept (POC) environment are: Terraform, Ansible, and Jenkins. I’ll be using a lot of the hacks and tricks that we have learned over the last year in our Pivotal environment.