How to Ship Cloudflare logs to Sumo Logic with Lambda

With the advent of Cloudflare’s new Enterprise Log Share (ELS) REST API, it is now easy to download and inspect a zone’s access logs by a single RayID (Cloudflare’s fancy term for a request ID), a defined time period, or from the last RayID downloaded. In our case, we ship all of our other logs to Sumo Logic so everyone can analyze and parse them in any way imaginable!

Unfortunately, there seems to be no cut-and-dry way to integrate the two. Thankfully, AWS Lambda functions are now able to be run on a scheduled interval using CloudWatch Scheduled Events — sort of like you would a cron job on a server.

There are a couple of clear benefits of using a Lambda function instead of a job on a physical/virtual server you run yourself:

Low cost: Execution of a Lambda function costs next to nothing, and it may actually cost nothing if you remain within the free tier AWS hands out every month
Ease of development: In my case, I used the Serverless framework to allow me to quickly get up and running, invoke a function locally (which is key to developing and iterating quickly), and it makes deployments simple. I also recommend checking out apex, too.
Speed: Lambda functions are fast. This is important for things which require speed, such as parsing and ingesting logs for a high-traffic site like ours.
No servers to build/maintain: Write the code, test and deploy — that’s it! No Chef, no Docker, no nothing… just write your function, ship it and it is live.

How It Works

First, you’re going to want to create a new HTTP source to ingest the Cloudflare logs. Set the source category, host, etc. at SumoLogic to make the configuration of the Lambda function easier.

Next, create a new Serverless function:

$ mkdir -p ~/cloudflareToSumoLogic

$ cd ~/cloudflareToSumoLogic

$ serverless create \--name cloudflareToSumoLogic \--template aws-nodejs \--path ~/cloudflareToSumoLogic

Let’s walk through the serverless.yml configuration file:

functions:

# This is the name of the Lambda function cloudflareToSumoLogic:

**_\# This is the name of the handler; this is the "main"_    \# _function called by Lambda_**  
handler: index.handler

**_\# This function was written to be run every one minute  
# Any adjustments will require code changes..._**  
events:  
  - schedule: rate(1 minute)

**_\# Required environment variables:_**  
environment:  
  **\# Grab this URL from the Sumo Logic source you just created**  
  SUMO\_ENDPOINT: ...  
  **\# Get these values from the Cloudflare admin UI**  
  CLOUDFLARE\_ZONE\_ID: ...  
  CLOUDFLARE\_AUTH\_EMAIL: ...  
  CLOUDFLARE\_AUTH\_KEY: ...

Finally, you’re going to need the function:

Now, try it out!

$ serverless invoke local -f cloudflareToSumoLogic

startTime: 2017-02-11T15:07:00.000ZendTime: 2017-02-11T15:08:00.000Z

cloudflareOpts: { method: 'GET',hostname: 'api.cloudflare.com',path: '/client/v4/zones/.../logs/requests?start=1486825620&end=1486825680',headers:{ 'X-Auth-Email': '...','X-Auth-Key': '...' } }

res.statusCode: 200

res.headers: { date: 'Sat, 11 Feb 2017 15:38:40 GMT','content-type': 'application/json','transfer-encoding': 'chunked',connection: 'close','set-cookie': [ '__cfduid=...; expires=Sun, 11-Feb-18 15:38:39 GMT; path=/; domain=.cloudflare.com; HttpOnly' ],vary: 'Accept-Encoding','x-cf-request-id': '00000000-0000-0000-0000-000000000000','strict-transport-security': 'max-age=31536000','served-in-seconds': '0.355',server: 'cloudflare-nginx','cf-ray': '...-ORD' }

Log events: 659

Everything look good? It’s time to deploy:

$ serverless deploy

Serverless: Packaging service...Serverless: Uploading CloudFormation file to S3...Serverless: Uploading service .zip file to S3 (9.45 KB)...Serverless: Updating Stack...Serverless: Checking Stack update progress........................Serverless: Stack update finished...Serverless: Removing old service versions...

Service Informationservice: cloudflareToSumoLogicregion: us-east-1api keys:Noneendpoints:Nonefunctions:cloudflareToSumoLogic

Analyzing Ingested Logs in Sumo Logic

Now that your logs are being ingested every minute, you can run the following query and you should be able to start seeing results like the screenshot above:

Replace _sourceCategory with the HTTP source category that you created earlier for your Cloudflare logs. This snippet parses the JSON lines automatically, translates the log timestamp timestamp to a time object, replaces the _messageTime with the log timestamp, then remove the raw JSON object and its ingest timestamp.