As we know with great power comes great responsibility, on-demand backups are the real saviour if your Dynamo DB got corrupted abruptly. Also, it helps you to meet the compliance requirements for the ages if your client keeps bothering you like your Ex. But I don't have the tenacity to initiate the Backup every day. Hence, I devise a Serverless solution for all the lazy people like me out there to automate the DynamoDB backup. Where, we can explicitly mention the backup intervals, table names and backup retention period. I have achieved this using a conjunction of AWS services like Cloudformation, Lambda and Cloudwatch.
Fact-check: Why do we even doing this, contrary by using DynamoDB point-in-time recovery (PITR) we can restore to any point in time within typically 5 minutes before the current time?
- After you enable point-in-time recovery, you can restore to any point in time within typically 5 minutes before the current time. But the major drawback is that the point-in-time recovery process always restores to a new table and for that reason, the source code and workflow need to be changed and requires re-work.
- In contrast, we can even enable on-demand backup in every 5 mins interval. At the time of restoration of a table, we will first delete the table else will encounter the following error "Table already exists". Post that, proceeds with the restoration keeping the table name same and thus there is no overhead to change the table name in the source code and workflow.
- But the same cannot be achieved in point-in-time recovery as we cannot delete the table ahead of the restoration to keep the table name same.
- Point-in-time Recovery maintains continuous backups for the last 35 days only. To meet the compliance for more than 35 days you need to use On-Demand backups.
- The following solution is less expensive as we can provide custom backup interval and retention period.
just do it (Solution Briefing)
The Cloudformation template has been used to keep the solution implementation minimalistic, refer the diagram above to know which services are being used to execute this. Here, we have used cloud watch event to trigger the lambda function at a specific interval.
Lambda function stores the python script which interact with other services to automate the use-case, also the following python script has been embedded in the Cloudformation template as an inline function code to keep the integration minimalistic. (
To check the python script,click hereDisclaimer: Make sure your Cloudformation stack and DynamoDB table are in the same region.
1. Navigate to cloudformation (if you are doing it for the first time click on get started and proceed) & click create stack, keep everything default and put the following URL in Amazon S3 URL & click Next.
https://automationking.s3.amazonaws.com/DynamoDBBackup.json2. In the next window, you specify the stack name for e.g DynamoDBBackup and parameters. Parameters are defined in your template and allow you to input custom values when you create or update a stack. (Click Next afterwards)
- DDBTableName: Enter DynamoDB Table name separated using comma(,) . The table name is case-sensitive. For e.g. Table1 , Table2 , Table3
- BackupRetention: Here you mention the number of days to keep the backup.
- Backup Interval: Here you mention the trigger interval in minutes. Following expression, rate(1440 minutes) will auto-populate. Replace only the numeric value (1440) with required minutes to set the backup interval or else keep the DEFAULT i.e. 1440 mins ~ 24 Hours. For e.g. to take 12 hours as input convert 12*60 = 720 minutes and replace it with 1440 i.e. rate(720 minutes).
3. In step 3, keep everything default and it is advisable to add a tag for easy identification.
4. In step 4, you will review the configuration, make sure you adhere with all the instructions. Finally acknowledge, that AWS CloudFormation might create IAM resources and hit create stack , Voila!’ you just break your boredom.
Important notes
IAM resources: By default, you need to create IAM resources so that the lambda function gets the following permissions to execute all the operations, you can review it under Execution role in Lambda Function.
- Create , Delete and List DynamodB Backups.
- CreateLogGroup , CreateLogStream and PutLogEvents in Cloud Watch which gives permissions to lambda function only to its Log group.
Cost: (US East (N. Virginia))
- Continuous backups (PITR) $0.20 per GB-month
- On-demand backup $0.10 per GB-month
- Restoring a table $0.15 per GB for both Continuous backups (PITR) and On-demand backup.
URLs:
- To check the Cloudformation template click here.
- To check the python script click here.
- You can always reach out to me on Linkedin.