How to Erase Expired Docs Automatically with MongoDB (TTL index)

Recently I needed to delete some documents that I saved in MongoDB after some time. I can think of a few examples of why we would want to delete data after some time:

logs / events
api keys / access tokens
non-active users
etc

We could do it by running a cronjob that deletes the data, delete the data every time we insert new data or any other solution.

Luckily for me, my wife told me that MongoDB already has that mechanism built-in.

TTL index

TTL (Time-To-Live) indexes are special single-field indexes that MongoDB can use to automatically remove documents from a collection after a certain amount of time.

A background thread in MongoDB reads the values in the index and removes expired documents from the collection (usually every minute).

For example, to create a TTL index on the

lastModifiedDate

field of the

eventlog

collection, use the following operation in the mongo shell:

db.eventlog.createIndex( { "lastModifiedDate": 1 }, { expireAfterSeconds: 3600 } )

The indexed field must be BSON date type or an array of BSON dates
If the indexed field in a document is not a date or an array that holds a date value(s), the document will not expire.
If a document does not contain the indexed field, the document will not expire.

Conditional delete

As of MongoDB 3.2, a collection can be partially indexed using a specified filter expression,

partialFilterExpression

. TTL index can also be used with partial indexes.

For example:

Delete documents that created 1 hour ago if

state

equals to

TMP

db.eventlog.createIndex(
  { created_at: 1 },
  { expireAfterSeconds: 3600, partialFilterExpression: { state: 'TMP' } }
);

Delete documents that created 1 day ago if

count

is lower than

db.eventlog.createIndex(
  { created_at: 1 },
  { expireAfterSeconds: 86400, partialFilterExpression: { count: { $lt: 5 } } }
);

Additional info on partial indexes

Real-world example

Recently I finished developing BundleMon, which is a free tool that helps you monitor your apps bundle size.

One of the components in BundleMon is a service that saves historic reports in order to compare bundle size between branches. So when you open a PR, BundleMon saves a record with the current bundle size report.

No need to save the report for more than 30 days, so I just added a TTL index:

db.reports.createIndex(
  { creationDate: 1 },
  { expireAfterSeconds: 2592000, partialFilterExpression: { prNumber: { $exists: true } } }
);

Also published at https://dev.to/lironer/delete-expired-documents-automatically-with-mongodb-ttl-index-l44