Cloudformation templates for Cloudfront automatic cache invalidation using Lambda within CodePipeline

In this post I’m going to show how I triggered an automatic cache invalidation for the Cloudfront distribution that is serving this website. As in the previous posts, all the resources will be provisioned via CloudFormation.
At the end of the post the CLI commands to create and / or update the resources will be shown.

The manual procedure

Once that the markdown file for a post is written and a local compilation / rendering has been made, the markdown source can be pushed on the git repo. That triggers the AWS Codepipeline that will download the source, render the markdown into html, and push the result to the S3 bucket served by Cloudfront.
Since Cloudfront is serving the S3 bucket, caching is in place. Newly pushed content won’t be visible until the cache expires, which is not feasible. So, after a successful compilation and pushing to S3, I manually get to Cloudfront distribution invalidations and fire a new invalidation. This way I’m sure that subsequent requests to the website will get the newly updated content.
In the images below the steps for manual invalidation are shown:

Go to CloudFront / Distributions, and search for “Invalidations” tab

Then selecting the last successful invalidation (shown below on the very left) and “copy to new” (upper right)

And then confirming the copy of the invalidation with the last path (the path /* is fine since AWS charges per invalidation, regardless of how much deep it is)

The invalidation takes a few minutes to be completed, and then the website is good to go. This is a mundane and forgetful-prone task, so I’m better automating it.

Automation setup

There is not an “invalidate cache” action that can be directly call from CodePipeline. A Lambda that actually creates the invalidation is needed and must be called as an action in the CodePipeline structure.
Let’s see in details the two resources:

The Lambda function

The Lambda function will leverage boto3 Python libraries to create the invalidation and notify the pipeline about the outcome (credits to the website in the reference section).
Let’s see some highlights. At the end of this section the link to the gist with the full source is provided.

• it reads an environment variable that has the ID of the cloudfront distribution whose cache has to be invalidated.

  Read cloudformation ID from environment

  DistributionId=os.environ.get("CLOUDFRONT_DISTRIBUTION_ID"),

• it contains the code to notify the caller (CodePipeline) if its execution was successful or not

  Caller notifications

  code_pipeline.put_job_failure_result(
      jobId=job_id,
      failureDetails={
          "type": "JobFailed",
          "message": str(e),
      },
  )
  /* .... */
  code_pipeline.put_job_success_result(
      jobId=job_id,
  )

• The number of invalidations (1) and the path are hard-coded since this function is really for creating one and just one invalidation.

  Invalidation specifications

  "Paths": {
      "Quantity": 1,
      "Items": ["/*"],
  },

Click to view the Gist with the Lambda Python code

Lambda Cloudformation stack and how to reference it

The Lambda cloudformation stack is similar to the one presented for the 301 redirects and URL rewriting in edge locations ( here’s the post); there are a few differences, though (at the end of the paragraph there is the gist with the full code):

• In the lambda’s resources, an environment variable is declared and its value is read from the cloudformation stack that contains the cloudfront distribution (referenced in the python code as CLOUDFRONT_DISTRIBUTION_ID).
  To be able to read that from here, the cloudformation stack that contains the cloudfront distribution has to list the variable as an output and flag it to be exported:

Cloudfront distribution environment variable reference

Parameters:
  CloudformationExportVar:
    Type: String
...
Environment:
        Variables:
          CLOUDFRONT_DISTRIBUTION_ID:
            Fn::ImportValue:
              !Ref CloudformationExportVar

And here’s the export in the stack hosting the cloudfront distribution

Exported value from cloudfront distribution resource

"Outputs": {
        "CloudFrontDistributionId": {
            "Description": "ID of the CloudFront distribution",
            "Value": { "Fn::GetAtt": ["CloudFrontDistribution", "Id"] },
            "Export": {
                "Name": { "Fn::Join": ["-", [{ "Fn::Sub": "${AWS::StackName}-CloudFrontDistributionId" }, { "Ref": "Stage" }]]}
            }
        }
    }

• There is no need for the edge permission (only lambda.amazonaws.com is needed)

services for the AssumeRole action

AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action: "sts:AssumeRole"

• Other than the BasicExecutionRole, three other permissions must be granted for the creation of the invalidation and the notification back to the CodePipeline
  • cloudfront:CreateInvalidation
  • codepipeline:PutJobFailureResult
  • codepipeline:PutJobSuccessResult

Additional permissions for basic lambda role

Policies:
  - PolicyName: InvalidateCloudfrontDistributionPolicy
    PolicyDocument: 
      Version: "2012-10-17"
      Statement:
        - Effect: Allow
          Action:
            - cloudfront:CreateInvalidation
            - codepipeline:PutJobFailureResult
            - codepipeline:PutJobSuccessResult
          Resource: "*"

That should put the lambda in place for the purpose.
Here below you can view the full gists of the cloudformation stack of

• the lambda
  Click to view the Gist
• the cloudfront distribution (which is the parent stack as shown in the older articles)
  Click to view the Gist

CodePipeline stage

The action can be added in CodePipeline as a new stage. From there the lambda can be referenced. Here’s how the new stage will look like after cloudformation template has been deployed (You can see that the action is referring to the lambda and still no runs shown):

Codepipeline Cloudformation stack

Let’s start from the addition of the new stage in CloudFormation. We can see

• the parameter that will reference the exported value from the lambda stack. The parameter value contains the name of the exported variable, and will reference the lambda function name
• the action
• the updated permissions in order to allow the calling of the lambda function from the pipeline:

New stage in Codepipeline

"Parameters": {
  ...
  "InvalidationLambdaExported": {
    "Description": "Lambda function performing the cache invalidation",
    "Type": "String"
  }
}
... stages ...
{
  "Actions": [
    {
      "ActionTypeId": {
        "Category": "Invoke",
        "Owner": "AWS",
        "Provider": "Lambda",
        "Version": "1"
      },
      "Configuration": {
        "FunctionName": {"Fn::ImportValue":{"Ref":"InvalidationLambdaExported"}}
      },
      "Name": "InvalidateCloudFrontCacheAction"
    }
  ],
  "Name": "InvalidateCloudFrontCacheStage"
}
...
"Policies": [
  {
    "PolicyDocument": {
      "Statement": [
        {
          "Action": [
            ...
            "lambda:invokeFunction"
          ],
          "Effect": "Allow",
            "Resource": "*"
        }
      ],
      "Version": "2012-10-17"
    },
    "PolicyName": "MyCodePipelineRolePolicy"
  }
]

The full reference to the cloudformation template can be found in the gist below:

Click to view the Gist cloudformation for the Cloudfront distribution

Below is how the new CodePipeline stage should turn out if everything was successful:

And that should do for adding a new stage with a new action calling the lambda and invalidating the cache

Cloudformation CLI commands

Here’s the AWS CLI commands (legit ones!) that have been fired in order to create and / or update the cloudformation stacks (and the lambda, of course):

CloudFront stack update

aws cloudformation package --template-file marcoaguzzi.json --s3-bucket cf-templates-e5ht2sji9no7-us-east-1 --output-template-file target\packaged-template.yaml
aws cloudformation deploy --template-file target\packaged-template.yaml --stack-name marcoaguzzi-website-prod --capabilities CAPABILITY_NAMED_IAM --parameter-overrides Stage=prod

It should appear the exported output variable:

New lambda creation, passing the export from above

cd lambda-cloudfront
compress-Archive .\index.py .\lambda-cloudfront-invalidate-prod-20240101.zip
aws s3 cp lambda-cloudfront-invalidate-prod-20240101.zip s3://lambda-artifacts-bucket-maguzzi/
aws cloudformation create-stack --stack-name lambda-invalidate-cloudfront-prod --template-body file://lambda-invalidate.yaml --capabilities CAPABILITY_NAMED_IAM --parameters ParameterKey=ZipDate,ParameterValue=20240101 ParameterKey=Stage,ParameterValue=prod ParameterKey=CloudformationExportVar,ParameterValue=marcoaguzzi-website-prod-CloudFrontDistributionId-prod

The new stack is created and the lambda name is exported in outputs:

CodePipeline update referencing the lambda exported variable in parameters

aws cloudformation update-stack --stack-name marcoaguzzi-stack-codepipeline-prod --template-body file://marcoaguzzi-codepipeline.json --parameters file://parameters-codepipeline-prod.json --capabilities 
CAPABILITY_IAM

And now the Codepipeline should have the last stage as shown in the pictures above, ready to invalidate the cache after the website deploy to S3 :-)

References

• AWS: Creating a CloudFront Invalidation in CodePipeline using Lambda Actions
• AWS Cloudformation user guide