Schedule check for forgotten running EC2 with AWS Lambdas
Have you ever got surprised by an AWS invoice charging you for resources you forgot running in strange regions around the world, while doing some experiments with your account?
Well, I did, more than once.
After spend lot of time looking for a tutorial on how to write a serverless function to monitor forgotten resources running consuming my credits in AWS with no success, I decided to my research and create my own solution.
I came up with a free tier AWS set of resources solution consisting of a Lambda Function running every day on a particular AWS region, checking for running EC2 instances around all AWS regions for more than a pre configured time, and notifying me via email.
In this post, I will go through step by step on how to create your own global EC2 resource monitoring, spamming across a variety of AWS technologies:
- Lambda functions
- Simple Notification Service SNS
- CloudWatch Events and Logs
- IAM Roles and Policies
- NodeJS and AWS SDK
To complete this exercise, you require an AWS account with access to all these resources. If you don’t have one, I recommend creating your own account, as AWS offers a free tier which allows you to practice all different services they provide, without spending any money.
This exercise can be executed and left running forever under the AWS Free Tier.
Ready? Let’s start!
The monitoring code
AWS Lambda supports various language. I decided to use NodeJs as I feel particularly comfortable with the async/await and Promises way to write concurrent code.
Before starting configuring AWS resources, let’s look at the monitoring code and understand what it does
On the first block (lines 1–5) we are initialising AWS SDK, alongside with SNS and EC2 clients. As I’m based in Sydney, all the resources are going to be created on ap-southeast-2 region, but you can use your own region to run the EC2 instance check. We are also setting the Topic SNS ARN which you will have to update accordingly after creating your own Topic, and configuring how many minutes an EC2 has to be running to trigger the Email.
On line 9, we are querying a list of existing AWS regions and just after, lines 11–13 we are creating a list of Promises, ie, code that will be executed asynchronously, to get a list of EC2 instances in each region.
On line 15, we wait for all Promises to complete, and them, using some functional programming, we build a list of all EC2 instances that are running (State.Code=16) for more than maxMinutes.
Lastly, the block from line 29 to 38 we are publishing an SNS event for every EC2 instance running, that will result in notification Emails.
Simple Notification Service SNS
First we will create the service responsible for sending emails. We will do that using SNS, which works on a Topic/Subscriber fashion. Every message published to a topic will be distributed to every subscriber. Subscriber’s can be either HTTP or Email endpoints. In our case, we will use Email.
Done! At this point, we setup a SNS topic that will forward every message to the subscribed email. Take note of your generated Topic ARN, as it is the universal resource identifier that will be used in the Lambda code later.
Now let’s create the core of the system, the serverless Lambda Function that will run on a scheduled basis, looking for running EC2 instances in every AWS region across the globe.
In the new Lambda function dialog, type in:
Runtime: Node.js 8.10
Role: Create new role from template
Role name: check-running-ec2-instances
Policy Templates: SNS publish policy
IAM Roles and Policies
Every Lambda function requires an IAM Role to run under. When we created the Lambda function in the previous example, a new IAM Role was created to grant pre-defined permissions to the function.
In order to inspect EC2 instances, and publish events to an SNS topic, this role requires extra policies to do so.
We can see in the pre created Role, there is already two attached Policies.
- AWSLambdaBasicExecutionRole: is added by default, and basically grants write access to ClodWatch Logs, so the function can record it’s execution logs.
- AWSLambdaSNSPublishPolicy: we selected on the Lambda creation dialog, and grants permissions to publish on the SNS Topic.
In order to monitor our EC2 instances, we need to add a new EC2ReadOnly Policy to this role
Lambda Functions accepts a variety of triggers to run. One of them is CloudWatch Events, which can be configured to trigger a event on a scheduled basis.
We want our function to run at least once per day, so we are going to create a new CloudWatch Event scheduled rule, and plug it as a trigger in our Lambda Function
Our new CloudWatch Event rule should have a schedule as Event Source, with a Fixed rate of 1 day. On the Targets, we select Lambda Function, and our check-running-ec2-instance function. To correctly trigger the function execution, CloudWatch needs a input body. As our function doesn’t expect anything, let’s just put a random Constant JSON text.
CloudWatch is also the service where AWS resources direct their logs. By default, every new Lambda function created will also have a Log Group created in CloudWatch, whose messages never expires by default.
As our function will execute every day, let’s change the log retention period so we don’t store unnecessary data.
Testing the whole thing
Now with the whole infrastructure in place, let’s create an EC2 instance and test our Lambda Function.
Now that we have a running EC2 instance, we can go back to our Lambda Function and manually test it.
Congratulations! You finished the tutorial, and now you have a free serverless infrastructure checking for forgotten EC2 instances around the globe in your AWS account.
With this exercise, you learned how to
- create Topic SNS and configure emails subscriptions
- create new Lambda functions and how to test them
- Attach IAM Policies in IAM Roles
- Create scheduled CloudWatch Events and configure CloudWatch logs expiration
No more surprises on AWS invoices at the end of the month.