Browsed by
Tag: hashicorp

Terraform to Buildout AWS

Terraform to Buildout AWS

I started playing with Terraform a few months ago when I needed to spin up a prototype environment to do an evaluation of the open source version of Cloud Foundry. One of the results was some Terraform code that could bring up the essentials of an AWS VPC, which included the VPC itself, three public subnets, three NATs, three Elastic IPs (EIPs), and a Route53 hosted zone. While it might seem like overkill to use this many Availability Zones (AZs) for a prototype environment, once of the things we needed to test was how Cloud Foundry’s new multi-az support worked.

This was good for what I was working on at the time since I needed to test across muliple AZs, but it was problematic for most of the day-to day testing that I need to do as it would spin up (and charge me for) components that I didn’t really need. Most things I have to test don’t require three of everything. The challenge was, I do not want to maintain different code repositories for different use cases.

Luckily, I came across an article by Yevgeniy Brikman that had some interesting tips and tricks on how to do some loops and conditionals. The most interesting bit for the problem I am trying to solve was learning that in Terraform, a boolean true is converted to a 1 and a boolean false is converted to a 0. Yevgeniy then used an example that I proceed to incorporate into my code. Essentially, what I did was create three new variables in my environments file to define whether or not I wanted to create each of the three public subnets:

# public-subnets module
public-1a_create = true
public-1b_create = false
public-1c_create = false

Then for each resource, I added the count variable:

resource "aws_subnet" "public-1a" {
    count                   = "${var.public-1a_create}"
    vpc_id                  = "${var.vpc_id}"
    cidr_block              = "${var.public-1a_subnet_cidr}"
    map_public_ip_on_launch = true
    availability_zone       = "us-east-1a"
    tags {
        Name                  = "public-1a"
        Description           = "Public Subnet us-east-1a"
        Terraform             = "true"
    }
}

Now I am able to spin up an AWS environment that only is only in one availability zone to do some testing, or bring it up in three for production. There are still a few other things that I am hoping to figure out, such as how to not set environmental variables for the second and third subnets even though you don’t need them and how to let Terraform deployments that build on the base to use the right NAT subnets if they use three AZs and the their are only two NATS. You can find the code in this repo.

Using Vault to Manage AWS Accounts

Using Vault to Manage AWS Accounts

I’ve been putting off setting up the AWS backend in our Vault server for the last few months. I knew that I was going to need it eventually, but other priorities kept taking precedence so I have been pushing it off. This past week one of the application teams came to me with a requirement for needing to write a file to an S3 bucket.

Under normal circumstances, I would probably just go ahead and create an instance profile that could be applied to the system and the problem would be solved. The problem with this approach was that they wanted did not want other applications to be able to access it. Since we run containers, using instance profiles to control access would allow every container on that host access to the buckets.

Preparing the Environment

Setting up the AWS backend is pretty straight forward. To begin, you need to configure your environment to be able to interact with Vault.

export VAULT_ADDR=vault.example.com:8200
export VAULT_TOKEN=a38dc275-86d3-48bd-57ae-237a45d6663b

Once set, you can test your configuration using the curl command to go to the health endpoint.

% curl -k -X GET ${VAULT_ADDR}/v1/sys/health
{"initialized":true,"sealed":false,"standby":false,"server_time_utc":1477441389,"version":"0.6.2","cluster_name":"vault-cluster-2fbd0333","cluster_id":"d8056c7f-acbb-ae59-4ed4-3673f2d27d48"}

Initialize the AWS Backend

Once you have verified that the endpoint is working, you can create and configure the AWS Backend. Since we use multiple AWS accounts for each environment, I will mount different backends for each account.

curl -k -X POST -H "x-Vault-Token: ${VAULT_TOKEN}" -d '{"type": "aws",   "description": "AWS Backend", "config": {"default_lease_ttl": "360", "max_lease_ttl": "720"}}'  ${VAULT_ADDR}/v1/sys/mounts/aws-prototype

This command sets up the aws-prototype backend with a default lease time of 5 minutes and a max lease time of 10 minutes. Since the post doesn’t return anything, you can verify it with the mounts endpoint. If you don’t have jq, I highly recommend you download it, as it makes viewing JSON output much easier

curl -k -X GET -H "x-Vault-Token: ${VAULT_TOKEN}" ${VAULT_ADDR}/v1/sys/mounts|jq .

Configure AWS Backend

Once the mount is created, you will need to add AWS credentials to the backend. You will need to create an AWS user that has full IAM access so that it can create other users. We have automation that controls our IAM, but you can use a couple of IAM commands to set up the user you will need.

aws iam create-user --user-name HashiVault
aws iam attach-user-policy --user-name HashiVault --policy-arn arn:aws:iam::aws:policy/IAMFullAccess
aws iam create-access-key --user-name HashiVault

Once you create the access & secret keys, you can use them to configure the AWS backend.

curl -k -X POST -H "x-Vault-Token: ${VAULT_TOKEN}" -d '{"access_key": "XXXXXXXXXXXXXXXXXXXX", "secret_key": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", "region": "us-east-1"} ${VAULT_ADDR}/v1/aws-prototype/config/root

After the backend is configured, you can start adding roles to Vault. For the S3 access that we want, we need to create a role that creates a user that has the right policy attached.

curl -k -X POST -H "x-Vault-Token: ${VAULT_TOKEN}" -d '{"arn": "arn:aws:iam::aws:policy/AmazonS3FullAccess"}' ${VAULT_ADDR}/v1/aws-prototype/roles/S3-Access

You can verify that it was created properly by curling the endpoint and getting the credentials back.

curl -k -X GET -H "x-Vault-Token: ${VAULT_TOKEN}" ${VAULT_ADDR}/v1/aws-prototype/creds/S3-Access

Now that you are getting credentials, you can repeat the process for every account and/or role that you need to have setup.

Testing the S3 Access

I had a little problem testing my S3 access once I had everything configured. I wrote a quick little one-liner to get my creds and set them to the proper environmental variables.

CREDS=$(curl -k -X GET -H "x-Vault-Token: ${VAULT_TOKEN}" ${VAULT_ADDR}/v1/aws-prototype/creds/S3-Access);export AWS_ACCESS_KEY_ID=$(echo $CREDS |jq -r .data.access_key);export AWS_SECRET_ACCESS_KEY=$(echo $CREDS |jq -r .data.secret_key)

When I tried to download a file, I received and error.

download failed: s3://mybucket/testfile.txt to ./testfile.txt An error occurred (InvalidArgument) when calling the GetObject operation: Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4. You can enable AWS Signature Version 4 by running the command:

aws configure set s3.signature_version s3v4

I ran the command, but each typed I tried to run the aws s3 command, I received the same error. What I learned was that running the command updated my .aws/config file and added the following line to it:

[default]
s3 =
    signature_version = s3v4

Since I already have a ‘[profile default]’ set with nothing in it, I moved the S3 bits up underneath that and removed the ‘[default]’ block and everything started working as expected.

Using Vagrant to Test Galaxy Roles

Using Vagrant to Test Galaxy Roles

Last summer, I wrote a post about how we were using Vagrant to test Ansible roles across AWS and the Datacenter. This has worked well with a single AWS account, but it has proven to be a little trickier in our account layout, which uses a centralized account and STS roles. Initially, I had an assume_role script that we had written that would get and set the right bits for the vagrant file to work, but it wasn’t very elegant.

While working on some Ansible roles recently, I decided to take an afternoon and see what I could come up with to make everything a little easier. I’m pretty happy with the results. It’s much more streamlined and easier to run and maintain.

If you would like to try it out yourself, you can start by cloning (or forking first) the repo to your local system:

git clone git@github.com:MarsDominion/vagrant-ansible-testing.git

Once you have it cloned, you will want to change directories and then checkout the sts branch:

cd vagrant-ansible-testing
git checkout sts

From here, you will need to create an env.rb file in the top level of the directory and add the environmental variables you will want to use:

ENV['AWS_ACCESS_KEY_ID'] = 'XXXXXXXXXXXXXXXXXXXX'
ENV['AWS_SECRET_ACCESS_KEY'] = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
ENV['AWS_KEYPAIR_NAME'] = 'my-keypair'
ENV['MY_PRIVATE_AWS_SSH_KEY_PATH'] = '/Users/me/.ssh/my-keypair.pem'
ENV['AWS_SUBNET'] = 'subnet-xxxxxxxx'
ENV['AWS_SG'] = 'sg-xxxxxxxx' 

You can also optionally define the following variables (defaults are listed):

ENV['AWS_DEFAULT_REGION'] = 'us-east-1'
ENV['AWS_INSTANCE_TYPE'] = 't2.micro'
ENV['AWS_AMI'] = 'ami-9be6f38c' #(aws-linux)
ENV['AWS_EC2_USER'] = 'ec2-user'

After you have saved the env.rb file, you can update the requirements.yml and playbook.yml with your Ansible code. Vagrant will run the ansible-galaxy command with the -f (force) option on the “up” and “provision” vagrant sub commands. Once you have your Ansible files the way you want, all that is left is to run the vagrant command:

% vagrant up

You can iterate on your Ansible by running the vagrant command:

% vagrant provision

Once you have completed your Ansible testing, you can destroy the environment:

% vagrant destroy

That’s it. A quick and easy way to test your Ansible roles against an AWS server.

Authenticating Vault Using GitHub

Authenticating Vault Using GitHub

I have never been a big fan of creating and managing users on individual systems. I much prefer some sort of centralization of credentials, preferably that somebody else manages when people come and go. That is one of the key reasons I wanted to get the GitHub auth backend up and working in Vault.

Preparing the Environment

Setting up the GitHub authentication backend is pretty straight forward. The most difficult part was digging into how the policies work so that the teams that I add from GitHub have the right permissions. To begin, you need to enable the setup your environment.

export VAULT_ADDR=vault.example.com:8200
export VAULT_TOKEN=a38dc275-86d3-48bd-57ae-237a45d6663b

Once set, you can test your configuration by using the curl command to go to the health endpoint.

% curl -k -X GET ${VAULT_ADDR}/v1/sys/health
{"initialized":true,"sealed":false,"standby":false,"server_time_utc":1477441389,"version":"0.6.2","cluster_name":"vault-cluster-2fbd0333","cluster_id":"d8056c7f-acbb-ae59-4ed4-3673f2d27d48"}

Initialize the GitHub Auth backend

Once you have verified that the endpoint is working, you can create and configure the auth backend.

curl -k -X POST -H "X-Vault-Token: $VAULT_TOKEN" -d '{ "type": "github", "description": "Github OAuth Backend" }' $VAULT_ADDR/v1/sys/auth/github

You can verify that the backend was created successfully by doing a GET against sys/auth. If you don’t have jq, I highly recommend you download it, as it makes viewing JSON output much easier.

curl -k -X GET -H "X-Vault-Token: $VAULT_TOKEN" $VAULT_ADDR/v1/sys/auth|jq .

You should see the github backend in the output. Once you have verified that it has been created, the next step is to configure the backend by adding the GitHub organization that you will be authenticating against.

curl -k -X POST -H "X-Vault-Token: $VAULT_TOKEN" -d '{ "organization": "yourorghere" }' $VAULT_ADDR/v1/auth/github/config

Configure GitHub Team

Next you will need to create a policy that will allow you to actually do something (Deny is the default). This is my initial policy, and I’m sure it is not a great policy, but it is only a POC. Create a file called admin.hcl with the following code.

Once the file has been created, it needs to be uploaded to the server. That can be done through the sys/policy endpoint.

curl -k -X PUT -H "X-Vault-Token: $VAULT_TOKEN" -d @<(jq -n --arg a "$(<./admin.hcl)" '{ "rules": $a }')  $VAULT_ADDR/v1/sys/policy/admin

You can validate it by doing a GET against the same endpoint.

curl -k -X GET -H "X-Vault-Token: $VAULT_TOKEN" $VAULT_ADDR/v1/sys/policy/admin

Once the policy is uploaded, you can map it to a team in GitHub.

curl -X POST -H "X-Vault-Token: $VAULT_TOKEN" -d '{ "value": "admin" }' $VAULT_ADDR/v1/auth/github/map/teams/myteam

Verify Everything Works

Now you can test to ensure that everything works properly. Head over to Github and generate a Personal access token and then try to authenticate against Vault.

curl $VAULT_ADDR/v1/auth/github/login -d '{ "token": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" }'|jq .

This will return JSON that will give you a client_token you can use to access vault.

To make it easy, you could set your VAULT_TOKEN with the curl command.

export VAULT_TOKEN=$(curl ${VAULT_ADDR}/v1/auth/github/login -d '{ "token": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" }'|jq  -r .auth.client_token)

And then test that you are connecting properly to the system.

curl -k -X GET -H "x-Vault-Token: ${VAULT_TOKEN}" ${VAULT_ADDR}/v1/sys/mounts|jq .

Now you can set up other teams with more restricted access.

Using Vault as a Certificate Authority

Using Vault as a Certificate Authority

For the next few weeks we are doing a POC on Hashicorp’s Vault. While I am still learning about all of the functionality that Vault provides, there are a few key pieces I have already identified to check out in addition to just storing credentials. One of the big ones is the PKI backend. This would make it a lot easier for not just my team, but developers as well to generate SSL certificates. While I found some basic instructions on how to set it up from various sources (mentioned later), I decided to do my own write-up that would consolidate everything I learned.

Create the Root and Intermediate Certificates

Rather than writing up instructions on how to create a the root or intermediate CA’s, I will just post the instructions that I followed that were written up by Jamie Nguyen entitled OpenSSL Certificate Authority. For the purposes of this document, I followed the sections called Create the root pair and Create the intermediate pair.

It’s also possible to create a root certificate authority (and/or an intermediary certificate), but I prefer to do this outside of Vault so that my root certificate remains secure.

Initialize the PKI Backend

Once you have your root and intermediate certificates generated, the first think that you want to do is prepare them for upload to Vault. You can do that with by combining the intermediate certificate with your key.

cat intermediate/certs/ca-chain.cert.pem > /tmp/ca_bundle.pem
openssl rsa -in intermediate/private/intermediateCA.key.pem >> /tmp/ca_bundle.pem

These two commands will concatenate everything into one file called ca_bundle.pem. The next step is to initialize and configure the PKI backend. I was able to find some pretty good instructions on configuring it between Cuddletech’s website and a post by Joel Bastos. But since most of what I want to use Vault for will be driven from automation, I decide to focus on utilizing only the API (which made things just a little tougher).

The first step is to initialize some environmental variables that will make our commands easier to run. You’ll want to set the VAULT_ADDR to your URL of your vault server and VAULT_TOKEN to your login token.

export VAULT_ADDR=vault.example.com:8200
export VAULT_TOKEN=a38dc275-86d3-48bd-57ae-237a45d6663b

Once set, you can test your configuration using the curl command to go to the health endpoint.

% curl -k -X GET ${VAULT_ADDR}/v1/sys/health
{"initialized":true,"sealed":false,"standby":false,"server_time_utc":1477441389,"version":"0.6.2","cluster_name":"vault-cluster-2fbd0333","cluster_id":"d8056c7f-acbb-ae59-4ed4-3673f2d27d48"}

Configure the PKI Backend

After you have verified that the endpoint works, you can create and configure your PKI backend.

curl -k -X POST-H "x-Vault-Token: ${VAULT_TOKEN}" -d '{"type": "pki",   "description": "Test Root CA", "config": { "max_lease_ttl":     "87600h"}}'  ${VAULT_ADDR}/v1/sys/mounts/pki-test

This command creates a new PKI backend mount called “pki-test” and sets the max_lease_ttl to 10 years. You may want to adjust these settings to whatever is suitable for your environment.
Since the post doesn’t return anything, you can verify it with the mounts endpoint. If you don’t have jq, I highly recommend you download it, as it makes viewing JSON output much easier

curl -k -X GET -H "x-Vault-Token: ${VAULT_TOKEN}" ${VAULT_ADDR}/v1/sys/mounts|jq .

Once you have initialized the backend, you can upload the certificate bundle that you created by following the instructions noted above.

curl -k -X POST -H "x-Vault-Token: ${VAULT_TOKEN}" -d @<(jq -n --arg a "$(</tmp/ca_bundle.pem)" '{ pem_bundle: $a }') ${VAULT_ADDR}/v1/pki-test/config/ca

This command doesn’t return anything either. You can verify that it uploaded properly by trying to download the intermediate certificate.

curl -k -X GET -H "x-Vault-Token: ${VAULT_TOKEN}" ${VAULT_ADDR}/v1/pki-test/ca/pem

Create a Role

The final step is to configure a role to issue the certificates.

curl -k -X POST -H "x-Vault-Token: ${VAULT_TOKEN}" -d '{"allow_any_name": "true", "allow_ip_sans": "true", "max_ttl": "17520h"}' ${VAULT_ADDR}/v1/pki-test/roles/example-dot-com

You can verify that the role exists with a GET to the roles endpoint.

curl -k -X GET -H "x-Vault-Token: ${VAULT_TOKEN}" ${VAULT_ADDR}/v1/pki-test/roles/example-dot-com|jq .

Issue Certificates

Now we are all set to issue certificates from our Vault server. This can be done one of two ways. The first is to request a certificate and key from the Vault directly:

curl -H "X-Vault-Token: ${VAULT_TOKEN}"   -d '{ "common_name": "testhost.example.com" }' https://${VAULT_ADDR}/v1/pki-test/issue/example-dot-com | tee >(jq -r .data.certificate > test.example.com.cert) >(jq -r .data.private_key > test.example.com.pem) >(jq -r .data.ca_chain[] > test.example.com-chained.pem)

This will create three files in your directory, one that contains the key, one that contains the certificate, and one that contains the certificate chain. You can also send a CSR that you created to have a certificate generated.

curl -k -X POST -H "X-Vault-Token: ${VAULT_TOKEN}" -d @<(jq -n --arg a "test.example.com" --arg b "$(<../server.csr)" '{ common_name: $a, csr: $b }') ${VAULT_ADDR}/v1/pki-test/sign/example-dot-com| tee >(jq -r .data.certificate > test.example.com.cert) >(jq -r .data.ca_chain[] > test.example.com-chained.pem)

Since the key was generated separately, it won’t create a new key file, but it does generate the certificate file and the certificate chain.

That’s all it takes to get a functioning CA in Vault. I’m sure that I still have a whole lot to learn about configuring and securing the PKI backend, but for our POC I think this will work nicely.