How to develop a serverless chatbot (for Hangouts Chat)

How to develop a serverless chatbot (for Hangouts Chat): find reminders & notify users

Introduction

This post is part of a series where you can learn how to develop, monitor (and later debug and profile) a serverless chatbot (for Hangouts Chat). Please refer to the other posts linked below for context.

Intro & architecture
Process incoming chat messages
Send messages to users
Find reminders & notify users (this post)
Monitoring

Notify users about their reminders (Steps 7 to 10)

Lastly, users need to be notified of their reminders. This is where Cloud Scheduler comes in. It publishes a message to a Pub/Sub topic every 60 seconds, that triggers a Cloud Function which looks for reminders to send. If it finds some, it follows the same pattern as described earlier where a message is published to the reminder-bot-messages-out Pub/Sub topic, from where another Cloud Function picks it up and sends it to the Hangouts Chat API (we already developed that 👍).

Step 7 — Publish an event every 60 seconds (PR #17)

We treat the scheduler like any other service, even though it does not actually contain any source code. This will help us later when we look at infrastructure as code (IaC).

npx lerna create reminder-bot-scheduler creates the new directory and boilerplate code. For now, we don’t need the __tests__ and lib directories — delete them.
Add the following deploy script to packages/reminder-bot-scheduler/packages.json:

// packages/reminder-bot-scheduler/package.json

{ ... "scripts": { "deploy": "gcloud beta scheduler jobs create pubsub reminder-checker --schedule "* * * * *" --topic reminder-bot-checker --message-body "{}"", ... } }

Lastly, as is the case for the other two services, let’s add a convenience deploy script to the root package.json: "deploy:reminder-bot-scheduler": "lerna run deploy --stream --scope=\"reminder-bot-scheduler\"".

Before this can be deployed, open the Google Cloud Console and enable App Engine by doing the following:

In the navigation, click APIs & Services > Library.
Search for App Engine Admin API and click on it.
Click ENABLE. (you may have to wait for a few minutes for the changes to propagate)

Now, run npm run deploy:reminder-bot-scheduler from the root directory.

Validation: Open the Google Cloud Console, navigate to Cloud Scheduler and make sure your reminder-checker is listed. Keep in mind the job fails until we have the reminder-bot-checker Pub/Sub topic created. Let’s do that next.

Step 8 — Create the reminder-bot-checker Pub/Sub topic and the reminder-bot-checker Cloud Function (PR #18)

This should be a familiar task, so please refer to previous instructions if more details are needed or review the pull request changes:

npx lerna create reminder-bot-checker (git commit)
Add a deploy script to the new service’s package.json and one for convenience to the root package.json. (git commit)
Set up the function’s boilerplate code and log the input values. (git commit)
Deploy the function from the root with npm run deploy:reminder-bot-checker.

Validation: The Cloud Scheduler job no longer fails and in the logs for the reminder-bot-checker, you will see that it gets called once per minute.

Step 9 — Look for reminders that are due (PR #19)

This requires the @google-cloud/firestore NPM package to find reminders with certain criteria. To install, run npx lerna add @google-cloud/firestore --scope=reminder-bot-checker from the project root.

The business logic is in its own module:

// packages/reminder-bot-checker/lib/db.js

const Firestore = require('@google-cloud/firestore');

const db = new Firestore({
  projectId: process.env.GCP_PROJECT,
  timestampsInSnapshots: true
})

const findReminders = async () => {
  try {
    const remindersRef = db.collection('reminders');
    const queryRef = remindersRef
      .where('remind_at', '<=', new Date())
      .where('status', '==', 'new');
    const querySnapshot = await queryRef.get();
    return querySnapshot.docs;
  } catch (error) {
    console.error(new Error(`Reminders could not be found due to: ${error}`))
  }
}

module.exports = {
  findReminders
}

Before this works, the query requires an index. If you forget to do that, the logs will display a URL you can click to create the index. Alternatively, open the Google Cloud Console and follow these steps:

In the navigation, click Firestore > Indexes.
Click Create Index.
Set the fields so it matches the following screenshot. It will take a few minutes to complete.

A composite index to find reminders

Validation: To make sure we find the reminders, we can print them (or at least the number of reminders found) to the console. To do that, modify the packages/reminder-bot-checker/lib/reminder-bot-checker.js as follows:

// packages/reminder-bot-checker/lib/reminder-bot-checker.js

const { findReminders } = require('./db')

exports.reminderBotChecker = async (event, context) => {
  ...

const reminders = await findReminders()
  console.log(`Found ${reminders.length} reminders to be sent.`)
}

Deploy the reminder-bot-checker function, wait for a minute or two and verify the logs show a number of reminders that were found. Something along these lines:

Reminders found in Firestore

Step 10 — Send a message to the user for each reminder (PR #20)

We already built a helper module to send messages to the reminder-bot-messages-out Pub/Sub topic for the reminder-bot-receiver. Let’s re-use that by copying it to the reminder-bot-checker function: cp packages/reminder-bot-receiver/lib/pubsub.js packages/reminder-bot-checker/lib/.

Also, let’s add the @google-cloud/pubsub NPM package: npx lerna add @google-cloud/pubsub --scope=reminder-bot-checker.

With that in place, all that’s left is processing each reminder found in the database and publish an event to the reminder-bot-messages-out Pub/Sub topic. Remember? Any events published to that topic will be processed by the reminder-bot-sender Cloud Function and sent to the Hangouts Chat API as a chat message.

// packages/reminder-bot-checker/lib/reminder-bot-checker.js

const { sendMessage } = require('./pubsub')

exports.reminderBotChecker = async (event, context) => {
  ...

const reminders = await findReminders()
  console.log(`Found ${reminders.length} reminders to be sent.`)

let reminderIndex = 1
  for (const reminderDoc of reminders) {
    const reminder = reminderDoc.data()
    const reminderId = reminderDoc.id
    console.log(`Processing reminder with ID ${reminderId} (${reminderIndex} of ${reminders.length})`)

    await sendMessage({
      spaceName: reminder.parent,
      threadName: reminder.thread,
      message: `Your reminder: ${reminder.subject}`
    })
    reminderIndex++
  }
}

A note on the for-loop: It currently sends messages in parallel. An improvement here would be to use Promise.all() to clean this up a bit and likely make it a bit more efficient.In addition, if for some reason the function runs for more than 60 seconds, a new instance will get started automatically. This may lead to two functions processing the same reminders. In my production reminder bot, processing times are far from 60 seconds, but it’s likely one of the next tasks I’ll work on to improve.

Validation: Once the reminder-bot-checker function is deployed, this is it — almost. Every minute, you will now receive a chat message for each reminder in the database. That’s great, but once a reminder is sent, we need to flag is as such so the reminder-bot-checker doesn’t continuously send messages for the same reminders.

Step 10 — Flag reminders as sent (PR #21)

Lastly, two minor changes will ensure reminders are flagged as sent once they’re published to the Pub/Sub topic.

// packages/reminder-bot-checker/lib/db.js

const setReminderStatusSent = async (reminderId) => {
  await db.collection('reminders').doc(reminderId).update({ status: 'sent' })
}

module.exports = {
  ...
  setReminderStatusSent
}

And:

// packages/reminder-bot-checker/lib/reminder-bot-checker.js

...
    await sendMessage({
      spaceName: reminder.parent,
      threadName: reminder.thread,
      message: `Your reminder: ${reminder.subject}`
    })
    setReminderStatusSent(reminderId)
    reminderIndex++
...

Validation: Once the reminder-bot-checker Cloud Function is deployed, you will get one more chat message for all reminders in the database. One minute later, no reminder is sent to Hangouts Chat.To wrap this up, add the following two reminders by messaging the Reminder Bot directly:

“Take a break from reading blog posts in two minutes”
“Go for a walk in five minutes”

Pay close attention to the reminders the chatbot sends you over the next five minutes.

Summary

That’s a wrap. You built your own Reminder Bot and it’s time to take a break, get some fresh air. Don’t just stop here though, the architecture and foundation you learned help you build any type of chatbot. Maybe try to hook up a bot to another messaging platform or look at ChatOps and how you can write a simple bot to help at your work.

👏 ❤️

Next Steps

A copy of the Reminder Bot we built is deployed in a production environment where I get to maintain and support it. As time goes by and I learn more about that process, I will continue to write posts and share how that goes.

The first and most important next step is to talk about monitoring. Head over here to see what you can put in place to ensure the reminder bot properly runs and reminds users of their reminders.

How to develop a serverless chatbot (for Hangouts Chat) — Find reminders & notify users