Build a custom CLI to AWS Cloudformation with Boto3

Published in

Dronzebot

5 min readApr 5, 2017

I will be honest. This isn’t my best guide. Its also not my worst. Think that lands it squarely in the “medium of medium” camp.

What it is: A simple implementation of a Boto3 stack create and delete call using that includes all the code on github. So maybe that is helpful. I like helping. That is what https://dronze.com is all about. Helping.

Cloudformation is a way to provision or deprovision groups of AWS assets.

So this guide is how we made our bare bones implementation.

Creating Stacks With Boto3

Creating stacks with boto is easy. I would say the hardest thing is deciding how to transform a CLI based interface into something that an API can call. The goal is to make a set of commands that are pretty simple and will translate to the boto3 create_stack call spec:

response = client.create_stack(
    StackName='string',
    TemplateBody='string',
    TemplateURL='string',
    Parameters=[
        {
            'ParameterKey': 'string',
            'ParameterValue': 'string',
            'UsePreviousValue': True|False
        },
    ],
    DisableRollback=True|False,
    TimeoutInMinutes=123,
    NotificationARNs=[
        'string',
    ],
    Capabilities=[
        'CAPABILITY_IAM'|'CAPABILITY_NAMED_IAM',
    ],
    ResourceTypes=[
        'string',
    ],
    RoleARN='string',
    OnFailure='DO_NOTHING'|'ROLLBACK'|'DELETE',
    StackPolicyBody='string',
    StackPolicyURL='string',
    Tags=[
        {
            'Key': 'string',
            'Value': 'string'
        },
    ]
)

Wow. That’s a lot of stuff. I don’t think we need all that for a “simple” implementation. So this was the bare minimum we chose:

$ python create-stack.py -h
usage: create-stack.py [-h] --config CONFIG --name NAME 
--templateurl TEMPLATEURL --params PARAMS --topicarn TOPICARN
                       [--tags TAGS]optional arguments:
  -h, --help            show this help message and exit
  --config CONFIG       the config file used for the application.
  --name NAME           the name of the stack to create.
  --templateurl TEMPLATEURL
                        the url where the stack template can be fetched.
  --params PARAMS       the key value pairs for the parameters of the stack.
  --topicarn TOPICARN   the SNS topic arn for notifications to be sent to.
  --tags TAGS           the tags to attach to the stack.

And since the tags and params are pretty much the same:

{
    'ParameterKey': 'string',
    'ParameterValue': 'string',
    'UsePreviousValue': True|False
}

for the parameters and for the tags used for metadata related to the stack:

{
    'Key': 'string',
    'Value': 'string'
}

Why don’t we normalize the implementation that allows name value pairs to be parsed using a query string since they are so close:

# Parameters=[
#     {
#         'ParameterKey': 'string',
#         'ParameterValue': 'string',
#         'UsePreviousValue': False
#     },
# ],
def make_kv_from_args(params_as_querystring, name_prefix="", use_previous=None):nvs = parse_qs(params_as_querystring)#{'i': ['main'], 'enc': [' Hello '], 'mode': ['front'], 'sid': ['12ab']}
    kv_pairs = []
    for key in nvs:
        # print "key: %s , value: %s" % (key, nvs[key])
        kv = {
            "{0}Key".format(name_prefix):key,
            "{0}Value".format(name_prefix):nvs[key][0],
        }
        if use_previous != None:
            kv['UsePreviousValue'] = use_previouskv_pairs.append(kv)return kv_pairs

Believe it or not this is the hardest thing about the implementation because using boto3 is so easy.

We have a general way to create the boto3 client so we can choose to configure it, or use the fallback credentials approach that boto3 attempts natively:

def make_cloudformation_client(config=None):
    """
    this method will attempt to make a boto3 client
    it manages the choice for a custom config
    """
    #load the app config
    client = None
    if config != None:
        logging.info("using custom config.")
        config = load_config(args.config)
        client = boto3.client('cloudformation',
            config["AWS_REGION_NAME"],
            aws_access_key_id=config["AWS_ACCESS_KEY_ID"],
            aws_secret_access_key=config["AWS_SECRET_ACCESS_KEY"])
    else:
        # we dont have a configuration, lets use the
        # standard configuration fallback
        logging.info("using default config.")
        client = boto3.client('cloudformation')if not client:
        raise ValueError('Not able to initialize boto3 client with configuration.')
    else:
        return client

Then all you need to do is make the boto3 call:

# setup the model
template_object = get_json(args.templateurl)
params = make_kv_from_args(args.params, "Parameter", False)
tags = make_kv_from_args(args.tags)response = client.create_stack(
    StackName=args.name,
    TemplateBody=json.dumps(template_object),
    Parameters=params,
    DisableRollback=False,
    TimeoutInMinutes=2,
    NotificationARNs=[args.topicarn],
    Tags=tags
)

That is it. You will want to do some response handling (see code) but you are just about ready to try to create a stack. This can be called via our CLI app, create-stack:

python create-stack.py --name newstack01 --templateurl https://raw.githubusercontent.com/dronzebot/dronze-qlearn/master/cicd/cloudformation/ec2_instance_sg.json?token=AAY5LkQG0d0G4Uql5Y8T-74L2BuGjKfNks5Y7kTAwA%3D%3D --params "KeyName=dronze-oregon-dev&InstanceType=t2.small"   --tags "name=newstack01&roo=mar" --topicarn arn:aws:sns:us-west-2:705212546939:dronze-qlearn-cfINFO       2017-04-05 08:17:07,009 make_cloudformation_client           50  : using default config.
INFO       2017-04-05 08:17:07,041 load                                 628 : Found credentials in shared credentials file: ~/.aws/credentials
INFO       2017-04-05 08:17:07,675 _new_conn                            735 : Starting new HTTPS connection (1): cloudformation.us-west-2.amazonaws.com
INFO       2017-04-05 08:17:08,345 main                                 111 : succeed. response:{"StackId": "arn:aws:cloudformation:us-west-2:605211536939:stack/newstack01/fd7ccbf0-1a11-11e7-a878-503ac9316861", "ResponseMetadata": {"RetryAttempts": 0, "HTTPStatusCode": 200, "RequestId": "eeef8f29-1a12-11e7-8b60-5b681d5e1677", "HTTPHeaders": {"x-amzn-requestid": "eeef8f29-1a12-11e7-8b60-5b681d5e1677", "date": "Wed, 05 Apr 2017 15:17:08 GMT", "content-length": "380", "content-type": "text/xml"}}}

When you watch what happens in EC2 you see something kindof like this:

And there you have it. A simple custom client for cloudformation and boto3.

Deleting Stacks with Boto3

Deleting stacks is even easier. This implementation includes a simple parser to allow retained resources to be excluded as comma separated string.

The help:


$ python delete-stack.py -h
usage: delete-stack.py [-h] — name NAME [ — retain RETAIN] [ — log LOG] [ — config CONFIG]arguments:
 -h, — help show this help message and exit
 — name NAME the name of the stack to create.
 — retain RETAIN the names (comma separated) of the resources to retain.
 — log LOG which log level. DEBUG, INFO, WARNING, CRITICAL
 — config CONFIG the config file used for the application

There really isn’t much to the the implementation because we are using the same function to make the client:

#load the client using app config or default
client = make_cloudformation_client(args.config)retained_resources = []if args.retain and len(args.retain)>0:
            retained_respources = args.retain.split(",")response = client.delete_stack(
            StackName=args.name,
            RetainResources=retained_resources
)

Running delete stack is super easy:

$ python delete-stack.py --name newstack01INFO 2017–04–04 18:03:17,576 make_cloudformation_client 50 : using default config.
INFO 2017–04–04 18:03:17,677 load 628 : Found credentials in shared credentials file: ~/.aws/credentials
INFO 2017–04–04 18:03:18,187 _new_conn 735 : Starting new HTTPS connection (1): cloudformation.us-west-2.amazonaws.com
CRITICAL 2017–04–04 18:03:18,961 main 58 : succeed. response: {“ResponseMetadata”: {“RetryAttempts”: 0, “HTTPStatusCode”: 200, “RequestId”: “a7e04f04–199b-11e7–8a78–11614ee92102”, “HTTPHeaders”: {“x-amzn-requestid”: “a7e04f04–199b-11e7–8a78–11614ee92102”, “date”: “Wed, 05 Apr 2017 01:03:18 GMT”, “content-length”: “212”, “content-type”: “text/xml”}}}

Using a Configuration File

I don’t pass configurations on the CLI, to me the args on the CLI are about runtime not config. We have a config file that has the static configs in it:

AWS_ACCESS_KEY_ID=[my_access_key]
AWS_SECRET_ACCESS_KEY=[my_secret_access]
AWS_REGION_NAME="us-west-2"
LOG_LEVEL="INFO"

Boto3 is capable of auto configuration, and it will behave like aws CLI and attempt to find configs from ~/.aws/credentials but if you want explicit configs that is available using the config option in the CLI. If you do this the debug level will default to INFO.