Connect EC2 Remote Nodes from Ansible Control Node by retrieving secrets from SSM Parameter Store

I always wondered, what is the most secure way to connect to remote machines from an Ansible Control Node. Though, Ansible Provides features like Ansibles Vault, Filters & etc. to encrypt Sensitive data like Private Key or Password which primarily use to SSH/WinRM to the remote machines.

Thus, thought to share a new way to connect to the remote nodes without storing the sensitive data physically throughout the lifecycle of the Ansible Controller Node instead only during the Ansible playbook execution & retrieving the Secure String from a Secured Cloud Vault like AWS Systems Manager Parameter Store.

It uses AWS Key Management Service (KMS) to encrypt & decrypt secure strings. KMS is FIPS 140-2 validated hardware security modules (HSM) and supports FIPS 140-2 validated endpoints.

Hence the goal here is to do a basic Infra Setup in the AWS with a Pre-configured Ansible Controller Node and a Remote Node. The Ansible Controller will fetch the remote instance, Private Key, from SSM Store Parameter and use it to connect to the Target Node and execute the Ansible Playbook. Don't worry, in the Demo, all required steps automated through Cloudformation Template.  Thus, Non-AWS users won't find it difficult & easy to grasp. Let's get started.

Prerequisites:

You need to have an AWS Free Tier Account to run a CloudFormation Template which will do the basic setup for the Upcoming Demo.
All the resources provisioned through CloudFormation Template comes under Free Tier Eligibility.

How to do it:

Here I have used Amazon Linux 2 EC2 Instances to achieve the objective. Though, a similar solution would work on other Cloud Native Services hosted on Azure, GCP, Windows VMs, Containers Technology, etc. as well as On-Prem Infrastructure. So, feel free to explore them.

Let’s assume we have an Amazon EC2 cluster where we need to promote one instance, as an Ansible Control node either in a Private subnet or Public Subnet. Thus, if the Ansible control node wants to connect with the remote EC2 instances, the Inventory file needs to point to the Private Key of the EC2 key-pairs which is stored physically somewhere within the running instance. As per our objective, we are going to integrate the Ansible Control node with a highly available AWS native service which can store Sensitive information by using De facto standards to encrypt & decrypt data securely, e.g., AWS SSM Parameter Store or Secret Manager.

Thus, by providing correct permission to the Ansible Control Machine, it can fetch the Remote Host, login credentials safely from a Cloud Vault during a full Ansible run and store it temporarily until the execution completed.

How Does it work:

Ansible controller machine that runs on an EC2 instance should use an IAM role, the role supplies temporary permissions that Ansible controller machine can use when they make calls to AWS SSM Parameter store to retrieve the Secure String without using Password based authentication mechanism. When we launch an EC2 instance, we specify an IAM role to associate with the instance. Applications that run on the instance can then use the role-supplied temporary credentials to sign API requests. For better understanding read the following official AWS Document.

1. So let's see something happen now.

1.1 AWS Ansible Infra Step Up:

Cloudformation template used to keep the solution implementation minimalistic, refer the diagram above to know which services used to build the Demo Environment.

Click-here to check the step-by-step screenshots to build the Demo Environment by using Cloudformation.

In this solution, there are two nodes inside a Public subnet. Though, always go for a Secure Environment hosted in a private subnet. All interaction with that Environment should go through a Bastion Host but for this Demo, let's stick with Public Subnet for simplicity. Here, one node is an Ansible Controller Node & another node is a Target Node.
By attaching an IAM role for Amazon EC2 to the Ansible Controller Node, it can send API request to the SSM Parameter Store and retrieve the required Secure Strings. Though, the Policy attached to the IAM role should have limited access to the SSM Parameter Store where the Ansible Controller Node can only retrieve the secrets of its remote nodes.
By using User data script, the Ansible Controller Node preconfigured with the Ansible Service for Ansible related Administration & required Demo Playbook cloned from the GitHub Repository for further execution.

1.2 Store the Private Key (EC2 Key-Pairs) in SSM ParameterStore:

Navigate to Login to AWS Console > AWS Systems Manager > Parameter Store. Alternatively, click here to land up to the required page (N. Virginia Region). Refer the below Screenshot to fill up the details to create the Secure String, put the Private Key (.pem format) value as shown below, the private key must match with the Key-Pair used in the Previous section (1.1) . Note: Make sure the Cloudformation Region and SSM Parameter Store Region are same, here we have used N. Virginia Region.

1.3 How To Run The Ansible Playbook

After successfully creating the Ansible Environment, note down the Public IP of Ansible Controller Node and Private IP of Target Node as instructed here, Step 5. Login to the Ansible Controller Node by using an SSH Client, you need the Public IP and Key-Pair to get into the Ansible Controller Node, refer the Link if you get stuck.

The Ansible Project used here is kept in the following Github Repository, kindly go through it to understand its functionality. Now let's try to understand how to use the Playbook.

1. First, change the current directory to Ansible Playbook Directory, ansible-project and check if all the required files are present from the Github Repo? After that update the AnsibleTagetNode Private IP in the TargetNode.yml file. Thus, run the below Commands to do the same, make sure the 10.10.10.28 (Dummy IP) replaced with AnsibleTagetNode's Private IP.

2. After the initial setup, you can run the Playbook by running the below command.

#Change the Directory to ansible-project 
cd /home/ec2-user/ansible-project/    

#Append the AnsibleTargetNode Private IP
sudo sh -c "echo 'ansible_host: 10.10.10.28' >> /home/ec2-user/ansible-project/host_vars/TargetNode.yml"

#Run the Playbook
sudo ansible-playbook playbook.yaml -i inventory

Clean-Up:

Congratulation for coming so far. Here, let's do the most important part by cleaning up the resources provisioned through Cloudformation. Click here to go to Cloudformation Stack Page (N. Virginia), if you have hosted the Stack in another region, kindly switch the Region in the Top Right Corner. Look for the Stack you have created, select and delete the Stack. It will remove all the resources provisioned earlier hassle-free.

Caveat:

In the Following Role, (SecureStringParameterStore) (Click Here, to goto mains.yaml), you need to Update the Right Region where you have stored the Cloudformation & SSM Parameter Store, to retrieve the Secure String from the SSM Parameter Store. Here we have used us-east-1 which denotes N.Virginia Region. (Click here to check the AWS Region Code).

We've come a long way, kindly Share & Comment only if you find it useful and help me on my mission to educate and familiarize people in the world of digitization 💪 This is a Free tutorial and all my upcoming tutorials will be free and accessible from Public forums.Appreciate if you drop me a note on LinkedIn & share your opinion. Don't worry, I don't bite 👻 so don't shy away 🏃🏻‍♀️ 🏃🏻. Your feedback will help me to come up with more awesome content on the internet.