One of the most advantages using cloud platforms is automation of resources managements, be their creation, changes and terminations. This not only helps make similar deployments and standard ones, avoiding very dissimilar environments that would be difficult to manage, but also allows self servicing by customers decreasing time to deploy labs, Q/A and other environments. It should also make quick to bring up an external customer environment in a short time going further and using a service catalog tool (which by the way, AWS provides with some additional integration).
For cloud architects and operators, CloudFormation (Infrastructure as a code) tool makes quicker set up and change of resources. Those are traceable via CloudTrail and can be controlled via policies stating who can use templates and which resources. In smaller environments CloudFormation templates can be managed by small team in a local repository such as file share, local disk or remotely on S3 (this can even use bucket versioning to help storing a small amount of versions if the environment changes rarely, but it is not best use of S3).
So once CloudFormation gets more use and the team increases to more than a few members, using a versioning tool such as Git or CodeCommit helps a lot in traceability, rollback and troubleshooting. With every committed change goes description and comments of what is the intent and targeted resources.
Going one step further CodePipeline can be used to monitor changes (commits) in CodeCommit repositories and start CloudFormation template updates, or even creation of a new stack. If a company already uses CodeCommit and CodePipeline to manage code, using them for CloudFormation management is a similar process. This demo intends to show a small setup with a single commit user creating and updating a small VPC.
CloudCommit set up
CloudCommit is the code versioning and management repository tool in AWS, Git based. It should be familiar to GitHub users for software development and collaboration projects. To start up it we will need a CodeCommit user with SSH RSA keys so access to the repository is private and secured via Unix command line. Note that AWS CodeCommit allows HTTPS connections using a different set up and both ways are supported in Windows as well. Reference document in AWS documentation here.
Creating a new IAM user:
- Go to IAM tool and create a new user with Programmatic access which allows CLI management. Click Next
- Click on Attach existing policies directly so we can add a standard AWS policy to the user. There are 3 policies allowing different permissions and for this demo we will use AWSCodeCommitFullAccess one. For production environments this should be more restrictive, probably PowerUser type where repositories cannot be deleted.
- Move next screen, review policy and create the new IAM user. Make sure you download the CSV with the Access key ID and Secret access key to set up git access.
Setting up git:
- Make sure you have git installed in your system. Installation in different platforms should be easy. Once done we should be able to invoke it as
iMac:~ aws$ git version git version 2.15.2 (Apple Git-101.1) iMac:~ aws$
- Let us run ssh-keygen to create a new key pair that will be used with the new IAM user. At the end of the process, the cat command will display the public key so we can add to the user profile.
iMac:~ aws$ iMac:~ aws$ ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/Users/aws/.ssh/id_rsa): /Users/aws/.ssh/codecommit_rsa Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /Users/aws/.ssh/codecommit_rsa. Your public key has been saved in /Users/aws/.ssh/codecommit_rsa.pub. The key fingerprint is: SHA256:Nxw13oUv9UBv8ci90iuflrTKspzZH21vptLfS4rXGS0 aws@iMac.local The key's randomart image is: +---[RSA 2048]----+ | o..o.| | o +o=+| | . . ++*| | . . o.+| | S + . +.| | . . Eo+| | oo=B| | ..B.*BB| | *+*=X=| +----[SHA256]-----+ iMac:~ aws$ iMac:~ aws$ iMac:~ aws$ cat /Users/aws/.ssh/codecommit_rsa.pub ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDE3oy5PHVHVVVITzJusWhSo2ihFNPtx7+mBypHX1TJThAIUmSK8oJtyYa1ncuDCV5P8rdxv3lgVIr6mTbxr6RY3v/FECxwt1uf4CiKaGwaJsMglPPRw8NneMTIRbhoV1Kae7gfokMmZc9g3MDtv+sipM2z6ZlVqnJqDISiNaYK6RY0eDW8X2KLYL0rgPoIEERUjjr+iAhcLMr0lkb6IHFMDGc+onbGPCKZ/W3/zP8TOeG83tG/xA3CiGTG/LC31TK6zRwKl5g7WhoUfaAE3KkrHz398a9Nm8q6NLIAgr2HguEFz9TaLbqmpphYi/p+lpByGYGzLd0nTS+HbKD3XjWd aws@iMac.local iMac:~ aws$ iMac:~ aws$ iMac: aws$
- Moving back to IAM user properties screen, we need to add the new generated public key to its profile. In the third tab Security Credentials , click on the Upload SSH public key and paste the contents of the key showed above.
- Now back to the local machine, we will setup the private key for SSH access. Go to .ssh directory to add a new configuration. The file will be named config and should contain the lines:
Host git-codecommit.*.amazonaws.com User ****USER-KEY-ID**** IdentityFile ~/.ssh/codecommit_rsa
Replace the user key ID field with one generated in you IAM user SSH credential. If you used a different name for the key file other than codecommit_rsa just replace it with yours. Also, protect the file access with chmod 600 config . Once done the .ssh directory should be like:
iMac:.ssh aws$ pwd /Users/aws/.ssh iMac:.ssh aws$ ls -la total 64 drwx------ 6 aws staff 204 Jul 25 08:24 . drwxr-xr-x+ 26 aws staff 884 Jul 18 14:37 .. -rw------- 1 aws staff 1679 Jul 25 07:47 codecommit_rsa -rw-r--r-- 1 aws staff 396 Jul 25 07:47 codecommit_rsa.pub -rw------- 1 aws staff 101 Jul 25 08:24 config -rw-r--r-- 1 aws staff 3844 Jun 18 14:22 known_hosts iMac:.ssh aws$
- Test the SSH connection using ssh git-codecommit.us-east-1.amazonaws.com
iMac:.ssh aws$ ssh git-codecommit.us-east-1.amazonaws.com The authenticity of host 'git-codecommit.us-east-1.amazonaws.com (18.104.22.168)' can't be established. RSA key fingerprint is SHA256:eLMY1j0DKA4uvDZcl/KgtIayZANwX6t8+8isPtotBoY. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'git-codecommit.us-east-1.amazonaws.com,22.214.171.124' (RSA) to the list of known hosts. You have successfully authenticated over SSH. You can use Git to interact with AWS CodeCommit. Interactive shells are not supported.Connection to git-codecommit.us-east-1.amazonaws.com closed by remote host. Connection to git-codecommit.us-east-1.amazonaws.com closed.
The -v parameter should help troubleshooting connection errors, permissions. Please, mind the key name file used matching its reference in the config file. Now return back to the folder where the repository will be created.
iMac:.ssh aws$ cd ~/Documents/Demos/CodeCommit-Pipeline/ iMac:CodeCommit-Pipeline aws$ ls -la total 0 drwxr-xr-x 4 aws staff 136 Jul 25 07:58 . drwxr-xr-x 7 aws staff 238 Jul 25 07:58 .. drwxr-xr-x@ 7 aws staff 238 Jul 25 06:42 CloudFormation automation.mindnode -rw-r--r--@ 1 aws staff 326 Jul 25 07:58 VPC-Demo.yml iMac:CodeCommit-Pipeline aws$
CodeCommit repository setup and git connection
A CodeCommit repository can now be created and linked with Git. To start, access the AWS CodeCommit console service screen under Developer Tools . Click on Get Started if this is the first repository and enter a name and description. After confirmation that the repository has been created we can select which types of notification. Leave as it is and select an existent test topic or create a new one.
Once saved it is time to connect the repository to Git, and alternatively one could work directly with the console adding and reviewing files, which is a great way to explore CodeCommit later. For now, let’s stick with regular Git tool. Select SSH as connection method and the OS. All 3 prerequisites have been completed by now so we can move on.
Select, copy the git command from the AWS console and run it on your command prompt. An example below shows the connection to the repository, new directory created and a message stating it is empty for now. I also moved a simple template for a VPC, file VPC-Demo.yml, in YAML format into the repository directory.
You can create a similar YAML file for the CloudFormation test by using this sample:
AWSTemplateFormatVersion: "2010-09-09" Description: VPC in North Virginia # # Resources: # # VPC # MyVPC: Type: AWS::EC2::VPC Properties: CidrBlock: "10.17.0.0/16" InstanceTenancy: default Tags: - Key: Name Value: MyVPC - Key: Environment Value: Testing # # END
Now, for connecting the repository:
iMac:CodeCommit-Pipeline aws$ pwd /Users/aws/Documents/Demos/CodeCommit-Pipeline iMac:CodeCommit-Pipeline aws$ git clone ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/CloudFormation Cloning into 'CloudFormation'... warning: You appear to have cloned an empty repository. iMac:CodeCommit-Pipeline aws$ ls -la total 0 drwxr-xr-x 5 aws staff 170 Jul 25 10:49 . drwxr-xr-x 7 aws staff 238 Jul 25 10:49 .. drwxr-xr-x 3 aws staff 102 Jul 25 10:49 CloudFormation drwxr-xr-x@ 7 aws staff 238 Jul 25 06:42 CloudFormation automation.mindnode -rw-r--r--@ 1 aws staff 326 Jul 25 07:58 VPC-Demo.yml iMac:CodeCommit-Pipeline aws$ mv VPC-Demo.yml CloudFormation iMac:CodeCommit-Pipeline aws$ cd CloudFormation iMac:CloudFormation aws$ ls -la total 0 drwxr-xr-x 4 aws staff 136 Jul 25 10:49 . drwxr-xr-x 4 aws staff 136 Jul 25 10:49 .. drwxr-xr-x 10 aws staff 340 Jul 25 10:49 .git -rw-r--r--@ 1 aws staff 326 Jul 25 07:58 VPC-Demo.yml iMac:CloudFormation aws$
Going back to the AWS console, we can confirm the repository is still empty, so let’s try few Git commands to update the repository.
First one is git status which should show a change on the local repository and new file VPC-Demo.yml
iMac:CloudFormation aws$ git status On branch master No commits yet Untracked files: (use "git add <file>..." to include in what will be committed) VPC-Demo.yml nothing added to commit but untracked files present (use "git add" to track)
Now we can add all files, and recheck status using git add -A and git status commands.
iMac:CloudFormation aws$ git add -A iMac:CloudFormation aws$ git status On branch master No commits yet Changes to be committed: (use "git rm --cached <file>..." to unstage) new file: VPC-Demo.yml iMac:CloudFormation aws$
Now that the VPC-Demo.yml file is added to the repo, it can be committed using git commit command. Typing it with -m parameter will run non-interactively, otherwise you will be presented with the default text editor.
Synch the new file to the repository with example below and refresh the AWS console screen to confirm the repository has it.
iMac:CloudFormation aws$ git commit -m "Adding the first version of CloudFormation template to the repo" [master (root-commit) 43df3a8] Adding the first version of CloudFormation template to the repo Committer: AWS <aws@iMac.local> Your name and email address were configured automatically based on your username and hostname. Please check that they are accurate. You can suppress this message by setting them explicitly. Run the following command and follow the instructions in your editor to edit your configuration file: git config --global --edit After doing this, you may fix the identity used for this commit with: git commit --amend --reset-author 1 file changed, 20 insertions(+) create mode 100644 VPC-Demo.yml iMac:CloudFormation aws$ iMac:CloudFormation aws$ git status On branch master Your branch is based on 'origin/master', but the upstream is gone. (use "git branch --unset-upstream" to fixup) nothing to commit, working tree clean iMac:CloudFormation aws$ iMac:CloudFormation aws$ git diff iMac:CloudFormation aws$ iMac:CloudFormation aws$ git remote origin iMac:CloudFormation aws$ git push Warning: Permanently added the RSA host key for IP address '126.96.36.199' to the list of known hosts. Counting objects: 3, done. Delta compression using up to 8 threads. Compressing objects: 100% (2/2), done. Writing objects: 100% (3/3), 459 bytes | 459.00 KiB/s, done. Total 3 (delta 0), reused 0 (delta 0) To ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/CloudFormation
[new branch] master -> master
iMac:CloudFormation aws$ git status On branch master Your branch is up to date with 'origin/master'. nothing to commit, working tree clean iMac:CloudFormation aws$
If you click on the demo file you should see its content and be able to edit it directly on AWS console. Going to the Commits submenu at the left you should be able to see the first commit information, the master branch, description and time completed.
Getting back to the terminal or prompt, let us try a file change, adding a subnet the VPC, checking the differences and committing and push the changes to the master branch of the repository.
Here is the updated file if you want to update the CloudFormation template. It only adds one subnet, one route table and associates both.
AWSTemplateFormatVersion: "2010-09-09" Description: VPC in North Virginia # # Resources: # # # VPC # # MyVPC: Type: AWS::EC2::VPC Properties: CidrBlock: "10.17.0.0/16" InstanceTenancy: default Tags: - Key: Name Value: MyVPC - Key: Environment Value: Testing # # # Subnets # # MySubnet1a: Type: AWS::EC2::Subnet Properties: AvailabilityZone: us-east-1a CidrBlock: "10.17.1.0/24" Tags: - Key: Name Value: MySubnet1a VpcId: !Ref MyVPC # # # Route Tables # # MyRouteTable: Type: AWS::EC2::RouteTable Properties: VpcId: !Ref MyVPC # # # Route Tables Associations # # MySubnet1aRouteTableAssociation: Type: AWS::EC2::SubnetRouteTableAssociation Properties: RouteTableId: !Ref MyRouteTable SubnetId: !Ref MySubnet1a # # # # END
Now that we have an edited file let’s check few Git commands with CodeCommit and finally commit and push the changes to master branch of the repository. This is just a bit of Git commands needed to check changed code, review it before updating the repository. Note the git diff command will show added lines with a ‘+’.
iMac:CloudFormation aws$ git status On branch master Your branch is up to date with 'origin/master'.
nothing to commit, working tree clean` iMac:CloudFormation aws$ ` iMac:CloudFormation aws$ ` iMac:CloudFormation aws$ # AFTER EDITING THE FILE` iMac:CloudFormation aws$ ` iMac:CloudFormation aws$ git status` On branch master` Your branch is up to date with 'origin/master'.`
Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git checkout -- <file>..." to discard changes in working directory)
no changes added to commit (use "git add" and/or "git commit -a") iMac:CloudFormation aws$ iMac:CloudFormation aws$ iMac:CloudFormation aws$ git diff diff --git a/VPC-Demo.yml b/VPC-Demo.yml index 0a9a970..9752611 100644 --- a/VPC-Demo.yml +++ b/VPC-Demo.yml @@ -4,7 +4,9 @@ Description: VPC in North Virginia # Resources: # +# # VPC +# # MyVPC: Type: AWS::EC2::VPC @@ -17,4 +19,39 @@ Resources: - Key: Environment Value: Testing # +# +# Subnets +# +# + MySubnet1a: + Type: AWS::EC2::Subnet + Properties: + AvailabilityZone: us-east-1a + CidrBlock: "10.17.1.0/24" + Tags: + - Key: Name + Value: MySubnet1a + VpcId: !Ref MyVPC +# +# +# Route Tables +# +# + MyRouteTable: + Type: AWS::EC2::RouteTable + Properties: + VpcId: !Ref MyVPC +# +# +# Route Tables Associations +# +# + MySubnet1aRouteTableAssociation: + Type: AWS::EC2::SubnetRouteTableAssociation + Properties: + RouteTableId: !Ref MyRouteTable + SubnetId: !Ref MySubnet1a +# +# +# # END iMac:CloudFormation aws$ iMac:CloudFormation aws$ git commit -a -m "Adding subnets to the VPC" [master e5e3d34] Adding subnets to the VPC Committer: AWS <aws@iMac.local> Your name and email address were configured automatically based on your username and hostname. Please check that they are accurate. You can suppress this message by setting them explicitly. Run the following command and follow the instructions in your editor to edit your configuration file:
git config --global --edit`
After doing this, you may fix the identity used for this commit with:
git commit --amend --reset-author`
1 file changed, 37 insertions(+) iMac:CloudFormation aws$ git push Counting objects: 3, done. Delta compression using up to 8 threads. Compressing objects: 100% (2/2), done. Writing objects: 100% (3/3), 595 bytes | 595.00 KiB/s, done. Total 3 (delta 0), reused 0 (delta 0) To ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/CloudFormation 43df3a8..e5e3d34 master -> master iMac:CloudFormation aws$
Back to the AWS console and refreshing the Commits page, you should see the new commit, time of its completion and brief commit ID by the end of the page. If you click on it, CodeCommit shows some of the compare tools.
When working with teams this can get multiple branches and pull requests, outside scope of this demo for now. Now, let’s start CodePipeline setup to automate CloudFormation execution of those changes.
AWS CodePipeline tool allows staging software development following several methods from modest to complex projects. In this demo we will initially set up a 2-stage process to monitor CodeCommit master branch of the CloudFormation repository and trigger CloudFormation create/update stack as needed. It should work with changes in the repository itself on CodeCommit editor, over a text editor in local repository copy as it is committed and pushed back to the repository.
Before creating a new pipeline, we need a new CloudFormation role with PowerUser policy. Go to IAM and select create new Role:
- In the AWS Service tab, select CloudFormation
- Click on Permissions button to move on, search and add for the policy named PowerUserAccess. Important note here is that this role can have advanced permissions boundaries, not applicable in the demo.
- Review the role name and click on Create role. As suggestion I used CustomCloudFormationPowerUser. Now we can go to create new pipeline.
To start up a new pipeline, go to AWS CodePipeline main window to create a new one. Note that one does not need all 6 steps in every project.
Step 1 – Name the pipeline CloudFormationPipeline
Step 2 – Choose CodeCommit as the source, select the demo repository CloudFormation and branch Master . Leave detection method as Amazon CloudWatch.
Step 3 – Build, select No Build
Step 4 – Deploy
- Choose AWS CloudFormation as the Deployment provider
- Action mode: Create or update a stack
- Stack name: CFNAuto
- Template file: VPC-Demo.yml as our previous template name
- Blank configuration file
- No changes in Capabilities
- Role name: Custom-CodePipelineAccessToCloudFormation that we previous set up in IAM before starting CodePipeline
Step 5 – A new role for CodePipeline should be created, named Custom-AWS-CodePipeline-Service. We are just adding custom wording to the role name as to make easier to identify not AWS standard ones when cleaning up.
Step 6 – Review the pipeline information and click on Create pipeline to complete it.
Immediately after the pipeline creation we should see the AWS CodeCommit running as stated as in progress blue line for a minute and the succeeded message in green if everything went alright. At this time the pipeline should move from the Source step to the Staging one. If the CloudFormation template is valid and the stack creation was successful we should get another green message.
Click on View Pipeline history and you should see a first build of our demo VPC pipeline.
We can check how the CloudFormation process went by entering on its console and selecting the Events tab. All doing well, it shows the VPC creation first followed by the route table and subnet, finishing with the associations. This demo is ran entirely on US-EAST-1 or North Virginia region, so mind this when describing resources in the CloudFormation template otherwise subnet Availability Zone will not match VPC and the whole stack will roll back.
As a last verification, going into the VPC service console we should see the new VPC and subnet created along with the new route table.
The new VPC (along with other previous test and default VPC for the region)
The new subnet. Note that the routing table does not (and should not) match the default VPC routing table if we successfully attached both. The main route table should allow all inside VPC communication but having a custom table can be useful when adding internet, NAT or VPN connections for a subnet.
With this simple demo of automation we can move on into adding changes to the CloudFormation template via Git updates.
Updating the CloudFormation via CodeCommit Git tools.
So in the event of change need in the infrastructure all we need to do is to edit the template and commit the change to our Git tool. Vantage here is versioning and history of changes can be kept, who done that and later, we can introduce approvals into the change flow. First things first, let us try adding a new subnet to the VPC, us-east-1b AZ with 10.17.2.0/24 CIDR.
Edit your template file adding the following lines in the subnet section:
MySubnet1b: Type: AWS::EC2::Subnet Properties: AvailabilityZone: us-east-1b CidrBlock: "10.17.2.0/24" Tags: - Key: Name Value: MySubnet1b VpcId: !Ref MyVPC
And under Route Tables Associations section:
MySubnet1bRouteTableAssociation: Type: AWS::EC2::SubnetRouteTableAssociation Properties: RouteTableId: !Ref MyRouteTable SubnetId: !Ref MySubnet1b
Now back to our terminal let us check Git status and commit the update.
iMac:CloudFormation aws$ iMac:CloudFormation aws$ git status On branch master Your branch is up to date with 'origin/master'.
Changes not staged for commit:` (use "git add <file>..." to update what will be committed)` (use "git checkout -- <file>..." to discard changes in working directory)`
no changes added to commit (use "git add" and/or "git commit -a")` iMac:CloudFormation aws$ ` iMac:CloudFormation aws$ ` iMac:CloudFormation aws$ git add -A` iMac:CloudFormation aws$ git commit -m "Adding a second subnet to the VPC"` [master c735027] Adding a second subnet to the VPC` Committer: AWS <aws@iMac.local>` Your name and email address were configured automatically based` on your username and hostname. Please check that they are accurate.` You can suppress this message by setting them explicitly. Run the` following command and follow the instructions in your editor to edit` your configuration file:`
git config --global --edit
After doing this, you may fix the identity used for this commit with:`
git commit --amend --reset-author
1 file changed, 16 insertions(+)` iMac:CloudFormation aws$ ` iMac:CloudFormation aws$ `
iMac:CloudFormation aws$ git push Counting objects: 3, done. Delta compression using up to 8 threads. Compressing objects: 100% (2/2), done. Writing objects: 100% (3/3), 336 bytes | 336.00 KiB/s, done. Total 3 (delta 1), reused 0 (delta 0) To ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/CloudFormation 7ae00a2..c735027 master -> master iMac:CloudFormation aws$
With all running well, we should see changes in the CloudFormation console, and so on the previous route table having a new subnet associated with.
VPC route table associations
This should give small demo of 3 AWS tools working together to manage infrastructure as a code. As one last step in this demo, we can add one more stage for authorization into the workflow.
Go back to the CodePipeline console, select the CloudFormationPipeline if not shown automatically and click on the Edit button. Just below the Source stage there is a ‘+’ Stage box where we can add another stage. Click on it and name Auth. On the Action Category, choose ‘Approval’ type, add an Action name such as ‘FirstApprover’ with type Manual approval. Choose the previous topic ‘Notify me’ for this region and under Comments enter something appropriate like “Please review and approve the changes to the VPC”. Save the pipeline change.
Once done, click on Release Change button even with no changes in the template file. We should get usual Source stage running for few minutes and the Auth stage holding for approval with a Review button. If the SNS topic was set up, an email with a link should be sent as well. For now click on the Review button and approve the change. Enter a comment before approval and the pipeline should move to the final stage Staging with no changes in the VPC or in the CloudFormation stack.
Cleaning up the demo
In order to not risk incurring costs, if you want to clean the lab up, it should be done in simple steps:
- Delete the CloudFormation stack
- Delete the CodePipeline pipeline
- Delete the CodeCommit repository
- Delete the CodeCommit IAM user
- Remove the local Git repository
- Check for an S3 bucket containing the template versions for CloudFormation and CodePipeline
- The new service roles named after “custom” prefix can also be deleted
This is far from most complete use of CloudFormation automation in AWS with its coding tools but should give a head start exploring them. As a more elaborate exercise, this lab can be expanded by using additional code commit users, simulating a larger admin group and mostly, working with different branches and environments, let’s say, being able to deploy a test or dev env into another VPC and/or account before committing to production.
As for learning more about Git and CodeCommit, AWS documentation page helps starting up a bit using step by step setup and tutorials and the excellent online / ebook Pro Git project by Scott Chacon and Ben Straub which dive deep into Git.
Edit: 2018–08–06 — I’ve made a not so quick video for commuters (47 min) showing similar setup working > https://youtu.be/9RENcc8PZTk — hope it is helpful to you!