This article was originally published as part of the 2014 AWS Advent series.
Introduction
This article assumes some familiarity with CloudFormation concepts such as stack parameters, resources, mappings and outputs. See the AWS Advent CloudFormation Primer for an introduction.
Although CloudFormation templates are billed as reusable, many users will attest that as these monolithic JSON documents grow larger, they become “all encompassing JSON file[s] of darkness,” and actually reusing code between templates becomes a frustrating copypasta exercise.
From another perspective these JSON documents are actually just hashes, and with a minimal DSL we can build these hashes programmatically. SparkleFormation provides a Ruby DSL for merging and compiling hashes into CFN templates, and helpers which invoke CloudFormation’s intrinsic functions (e.g. Ref, Attr, Join, Map).
SparkleFormation’s DSL implementation is intentionally loose, imposing little of its own opinion on how your template should be constructed. Provided you are already familiar with CloudFormation template concepts and some minimal ammount of Ruby, the rest is merging hashes.
Templates
Just as with CloudFormation, the template is the high-level object. In SparkleFormation we instantiate a new template like so:
1
|
|
But an empty template isn’t going to help us much, so let’s step into it and at least insert the required
AWSTemplateFormatVersion
specification:
1 2 3 |
|
In the above case we use the _set
helper method because we are setting a top-level key with a string value.
When we are working with hashes we can use a block syntax, as shown here adding a parameter to the top-level
Parameters
hash that CloudFormation expects:
1 2 3 4 5 6 7 8 9 |
|
Reusability
SparkleFormation provides primatives to help you build templates out of reusable code, namely:
- Components
- Dynamics
- Registries
Components
Here’s a component we’ll name environment
which defines our allowed environment parameter values:
1 2 3 4 5 6 7 8 |
|
Resources, parameters and other CloudFormation configuration written into a SparkleFormation component are statically
inserted into any templates using the load
method. Now all our stack templates can reuse the same component so
updating the list of environments across our entire infrastructure becomes a snap. Once a template has loaded a
component, it can then step into the configuration provided by the component to make modifications.
In this template example we load the environment
component (above) and override the allowed values for the environment
parameter the component provides:
1 2 3 4 5 |
|
Dynamics
Where as components are loaded once at the instantiation of a SparkleFormation template, dynamics are inserted one or more times throughout a template. They iteratively generate unique resources based on the name and optional configuration they are passed when inserted.
In this example we insert a launch_config
dynamic and pass it a config object containing a run list:
1 2 3 4 5 6 |
|
The launch_config
dynamic (not pictured) can then use intrisic functions like Fn::Join
to insert data passed in the config deep inside a launch
configuration, as in this case where we want our template to tell Chef what our run list should be.
Registries
Similar to dynamics, a registry entry can be inserted at any point in a SparkleFormation template or dynamic. e.g. a registry entry can be used to share the same metadata between both AWS::AutoScaling::LaunchConfiguration and AWS::EC2::Instance resources.
Translating a ghost of AWS Advent past
This JSON template from a previous AWS Advent article provisions a single EC2 instance into an existing VPC subnet and security group:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 |
|
Not terrible, but the JSON is a little hard on the eyes. Here’s the same thing in Ruby, using SparkleFormation:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 |
|
Without taking advantage of any of SparkleFormation’s special capabilities, this translation is already a few lines shorter and easier to read as well. That’s a good start, but we can do better.
The template format version specification and parameters required for this template are common to any stack where EC2 compute resources may be used, whether they be single EC2 instances or Auto Scaling Groups, so lets take advantage of some SparkleFormation features to make them reusable.
Here we have a base
component that inserts the common parameters into templates which load it:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
|
Now that the template version and common parameters have moved into the new base
component, we can
make use of them by loading that component as we instantiate our new template, specifying that the
template will override any pieces of the component where the two intersect.
Let’s update the SparkleFormation template to make use of the new base
component:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
|
Because the base
component includes the parameters we need, the template no longer explicitly
describes them.
Advanced tips and tricks
Since SparkleFormation is Ruby, we can get a little fancy. Let’s say we want to build 3 subnets into an existing VPC. If we know the VPC’s /16 subnet we can provide it as an environment variable (export VPC_SUBNET="10.1.0.0/16"
), and then call that variable in a template that generates additional subnets:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
|
Of course we could place the subnet and route table association resources into a dynamic, so that we could just call the dynamic with some config:
1 2 3 |
|
Okay, this all sounds great! But how do I operate it?
SparkleFormation by itself does not implement any means of sending its output to the CloudFormation
API. In this simple case, a SparkleFormation template named ec2_example.rb
is output to JSON
which you can use with CloudFormation as usual:
1 2 3 4 5 6 |
|
The knife-cloudformation plugin for Chef’s knife
command adds sub-commands for creating, updating,
inspecting and destroying CloudFormation stacks described by SparkleFormation code or plain JSON
templates. Using knife-cloudformation does not require Chef to be part of your toolchain, it simply
leverages knife as an execution platform.
Advent readers may recall a previous article on strategies for reusable CloudFormation templates which advocates a “layer cake” approach to deploying infrastructure using CloudFormation stacks:
The overall approach is that your templates should have sufficient parameters and outputs to be re-usable across environments like dev, stage, qa, or prod and that each layer’s template builds on the next.
Of course this is all well and good, until we find ourselves, once again, copying and pasting. This time its stack outputs instead of JSON, but again, we can do better.
The recent 0.2.0 release of knife-cloudformation adds a new --apply-stack
parameter
which makes operating “layer cake” infrastructure much easier.
When passed one or more instances of --apply-stack STACKNAME
, knife-cloudformation will cache the outputs of the named stack
and use the values of those outputs as the default values for parameters of the same name in the stack you are creating.
For example, a stack “coolapp-elb” which provisions an ELB and an associated security group has been configured with the following outputs:
1 2 3 4 5 6 7 8 9 10 11 12 13 |
|
The values from the ElbName and ElbSecurityGroup would be of use to us in attaching an app server auto scaling group to this ELB, and we could use those values automatically by setting parameter names in the app server template which match the ELB stack’s output names:
1 2 3 4 5 6 7 8 9 10 11 12 13 |
|
Once our coolapp_asg
template uses parameter names that match the output names from the coolapp-elb
stack, we can deploy the app server layer “on top” of the ELB layer using --apply-stack
:
1
|
|
Similarly, if we use a SparkleFormation template to build our VPC, we can set a number of VPC outputs that will be useful when building stacks inside the VPC:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
|
This ‘apply stack’ approach is just the latest way in which the SparkleFormation tool chain can help you keep your sanity when building infrastructure with CloudFormation.
Further reading
I hope this brief tour of SparkleFormation’s capabilities has piqued your interest. For some AWS users, the combination of SparkleFormation and knife-cloudformation helps to address a real pain point in the infrastructure-as-code tool chain, easing the development and operation of layered infrastructure.
Here’s some additional material to help you get started:
- SparkleFormation documentation - more detailed discussion of the concepts introduced here, and mmore!
- SparkleFormation starter kit - an example repository containing some basic templates for deploying a VPC and an EC2 instance inside that VPC.
- Sean Porter’s SparkleFormation ignite talk from DevOpsDays Vancouver 2014