Unforgettable deploy: keep resources coupled with Cloudformation Nested Stacks

Requirement

This website is served by an AWS Cloudfront distribution. The distribution has a cache behavior with a lambda@edge function attached to it to complete with “/index.html” the urls ending with a slash character.
Before this post, the Cloudformation Stack with the lambda and the one with the cloudfront distribution were separated. The only link between the two was the output value exported by the former and read by the latter.
Here’s what the AWS web UI lists:

The Cloudfront distribution can’t live without the lambda, so the deployment of the lambda should be done within the distribution one. The risk of having the two stacks completely separated is that an updated version of the lambda is not immediately referenced in the Cloudfront distribution (which is exactly what happened in the previous deploys of the website)

Taken approach and what’s needed

AWS Cloudformation Nested Stacks can be useful. One child stack is referenced in a parent stack and, when the parent is deployed, the resources of the child stack are deployed first.

Some resources should be prepared because they can’t be referenced in the stack itself, or it will end in a loophole

• The s3 bucket that will contain the artifact (a zipfile in this case) with the lambda code
• The s3 bucket that will contain the deployed cloudformation template, that is, with the path to the nested stack yaml file resolved

Folder structure

So the folder structure will be:

/
|rootstack.yaml 
|----/lambda-edge
|----|index.mjs
|----|lambdastack.yaml

Preparing the lambda

So, starting from being in the root folder of the project, first thing is preparing the lambda. Compress the source javascript, create the s3 bucket and then upload the compressed source

cd lambda-edge
# (beg my pardon, this is for Windows)
Compress-Archive .\index.mjs .\lambda-edge.zip
# if the bucket is still not ready
aws s3 mb s3://lambda-artifacts-bucket
aws s3 cp lambda-edge.zip s3://lambda-artifacts-bucket

Now the lambda Cloudformation stack can be deployed as a nested stack the lambda is ready in the referenced path

Stack configurations

You can view the full child and parent stacks in the Gist section here

Child stack template output section

The child stack contains the outputs that will be referenced in the parent stack

Outputs:
  LambdaFunctionVersion:
    Description: LambdaFunctionVersion Arn
    Value:
      Ref: LambdaFunctionVersion
    Export:
      Name: LambdaFunctionVersion-Arn

Reference the child stack in the parent stack

Declaring the nested stack

The child stack contained in the parent stack is simply a resource of the type AWS::Cloudformation::Stack itself:

"LambdaEdgeCloudFrontStack":{
    "Type" : "AWS::CloudFormation::Stack",
    "Properties": {
        "TemplateURL":"lambda-edge/lambda-edge.yaml"
    }
}

Reference the output values

Here’s how the child stack values are referenced in the parent stack. This happens in the cache behaviors of the Cloudfront distribution. The ARN of the lambda to be referenced in the distribution points to the output of the child stack:
(Here’s a reference to an example: Cloudformation Nested Stack example)

"LambdaFunctionAssociations": [
    {
        "EventType": "viewer-request",
        "LambdaFunctionARN": {
            "Fn::GetAtt": [
              "LambdaEdgeCloudFrontStack",
              "Outputs.LambdaFunctionVersion"
            ]
        }
    }
]

Package the stack

Once that all the resources are ready, the packaging of stack can be issued with:

aws cloudformation package\
 --template-file parent-stack.json\
 --s3-bucket bucket-for-templates\
 --output-template-file target\packaged-template.json

And here’s the nested stack reference that points to the actual s3 location that can be found in packaged-template.json:

LambdaEdgeCloudFrontStack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: https://s3.amazonaws.com/cf-templates-***-us-east-1/***.template

Updating the stack

Once that the stack is packaged, a deploy can be issued to update the stack. This will remove the reading of the exported values and inserting the creation of the Lambda@edge function via the nested stack (paramters are not needed, aws will reuse the same values present in the existing stack).

aws cloudformation deploy\
 --template-file target\packaged-template.json
 --stack-name parent-stack-name\
 --capabilities CAPABILITY_NAMED_IAM

All done? Not yet, this command will lead to errors, as the subsequent section will tell.

The unexpected stuff

Having the configuration as it is, two clashes will happen:

Exported output values can’t be duplicated

The lambda standalone stack and the new lambda nested stack have the same output export name, which will produce this error

That is:

Embedded stack B was not successfully created:
Export with name XYZ is already exported by stack A 

Remediation

The output in the child stack can be left without the exports, since the reference is only between parent and child stacks.
New output section for nested stack is:

Outputs:
  LambdaFunctionVersion:
    Description: LambdaFunctionVersion Arn
    Value:
      Ref: LambdaFunctionVersion

Lambda role name can’t be duplicated

Using nested stacks will lead to having to identical lambda functions: one for dev pipeline and one for the prod stacks. Both lambda functions will have their instance of the role, and those can’t have the same role name: before correcting it, this error would prevent the stack update:

That is:

Embedded stack B was not successfully created:
The following resource(s) failed to create:
[LambdaRoleForCF]. 

Remediation

This one is a little trickier. It involved adding a parameter for the parent stack in order to distinguish between dev and prod stacks, and passing it to their respective child stack, in order to differentiate the role name with a suffix containing “dev” or “prod”.
This is the stage paramter in the parameters section in the parent stack:

"Stage": {
  "Description": "Distribution stage",
  "Type": "String",
  "Default" : "dev",
  "AllowedValues" : ["dev", "prod"]
}

And this is the stage parameter in parameter section in the child stack:

  Stage:
    Type: String
    AllowedValues:
        - dev
        - prod

And this is the usage of the Stage parameter in the child stack, in order to change the role name according to the stage:

RoleName: !Join [ "-", [ "LambdaRoleForCF", !Ref Stage ] ]

Updating the stacks

Now that all the issues have been fixed, the final deploy can we issued with the same command as before. The only difference is on using parameter-override, so that dev and prod parent stacks can be differentiated

aws cloudformation deploy --template-file target\packaged-template.json\
  --stack-name website-prod --capabilities CAPABILITY_NAMED_IAM\
  --parameter-overrides Stage=prod

Here we have the AWS web ui showing the updated dev stack

the updated prod stack

and their respective nested stacks

Cleaning

Now the standalone lambda containing the edge function can be removed (only the prod one is left), since its output is not referenced anymore by the parent stack. The deletion can be done via the ui, eliminating the stack and its resources.

Gist references

Child stack

Child stack containing the lambda

Parent stack

Parent stack containing the distribution

Next actions

The activities discussed in this post have fixed the drifts of the stacks that were present due to the lambda being updated in different times. Also the point “automate on cloudformation the deploy of the lambda “ can be marked as done. One step that can be added is having a cloudformation template that contains only the precondition buckets (for cloudformation templates and lambda artifacts) and a codebuild pipeline that automates the steps described above (e.g. packaging and deploying)