Applications for Tarantool 1.7. Part 1: Stored procedures

Original article available at https://habrahabr.ru/company/mailru/blog/334266/

Hi there! In this article, I’d like to share my experience creating applications for Tarantool 1.7. It’s the first in a series of tutorials that might be useful both to those who’ve already decided to give Tarantool a try and to those who are still looking for a solution to streamline their projects.

The whole series will cover an existing Tarantool application, and this tutorial will touch upon such topics as installing Tarantool, storing and accessing data, and writing efficient stored procedures.

Tarantool is a NoSQL database that stores data in RAM or on disk (depending on the storage engine) and ensures persistence via a well-thought-out mechanism called a write-ahead log (WAL). Tarantool boasts a built-in LuaJIT (just-in-time) compiler that allows executing Lua code. You can also create stored procedures in C.

Why write your own Tarantool applications

There are two main reasons:

1. This speeds up your service. Storage-side data processing reduces data traffic, and bundling several requests into a stored procedure helps minimize network latency.
2. Ready applications can be reused. Tarantool ecosystem is actively evolving, with people creating more and more open-source Tarantool applications. With time, some of them become part of Tarantool itself. Such packages shorten development time for new services.

Naturally enough, this approach has its downside. Tarantool can’t take full advantage of a multi-core CPU, so if you strive to make your service scalable, you’ll need to shard your database and design an appropriate project architecture. On the bright side, as the number of requests grows, the workload becomes easy to scale.

Now I’m going to walk you through the creation of a Tarantool application that implements an API for registering and authenticating users. It offers the following capabilities:

• registration and authentication via email in two steps: creating an account and confirming the registration and setting the password;
• registration with social network credentials (FB, VK, Google+);
• password recovery.

For an example of a stored procedure for Tarantool, we’ll take a look at the first step of email registration — getting a confirmation code. To make it more interactive, you can check this GitHub page and follow along.

Let’s go!

Installing Tarantool

You can find detailed installation instructions for different operating systems on this site. For example, to install Tarantool on Ubuntu, you’ll need to run the following commands in the console:

curl http://download.tarantool.org/tarantool/1.7/gpgkey | sudo apt-key add -release=`lsb_release -c -s`

sudo apt-get -y install apt-transport-https

sudo rm -f /etc/apt/sources.list.d/*tarantool*.listsudo tee /etc/apt/sources.list.d/tarantool_1_7.list <<- EOFdeb http://download.tarantool.org/tarantool/1.7/ubuntu/ $release maindeb-src http://download.tarantool.org/tarantool/1.7/ubuntu/ $release mainEOF

sudo apt-get updatesudo apt-get -y install tarantool

Let’s check the installation has been successful by typing tarantool and entering the interactive administrator console.

$ tarantoolversion 1.7.3–202-gfe0a67ctype ‘help’ for interactive helptarantool>

This is where you can try your hand at Lua programming. If you’re not too familiar with Lua, here’s a short tutorial to get you started.

Registration via email

It’s time to take it one step further and write our first script that creates a space that will be holding all the users. A space is analogous to a data storage table. Data itself is stored in tuples (arrays holding records). Each space must have one primary index and can have several secondary ones. Indexes can be defined on a single or multiple fields. Below is a space scheme for our authentication service:

As you can see from the image, we’re using indexes of two types: HASH and TREE. A HASH index allows finding tuples by a fully matching primary key and must be unique. A TREE index supports non-unique keys, enables searches by the first part of a composite index, and lets us streamline key sorting, since key values within an index are ordered.

The session space holds a special key (session_secret) used for signing session cookies. Storing session keys allows logging users out on the service side, if necessary. A session also has an optional link to the social space. It’s necessary for validating the sessions of those users who log in with social network credentials (we’re checking the validity of the stored OAuth2 token).

Before we start writing the application itself, it’s worth taking a look at the structure of the project:

tarantool-authman├── authman│ ├── model│ │ ├── password.lua│ │ ├── password_token.lua│ │ ├── session.lua│ │ ├── social.lua│ │ └── user.lua│ ├── utils│ │ ├── http.lua│ │ └── utils.lua│ ├── db.lua│ ├── error.lua│ ├── init.lua│ ├── response.lua│ └── validator.lua└── test├── case│ ├── auth.lua│ └── registration.lua├── authman.test.lua└── config.lua

Paths specified in the package.path variable are used for importing Lua packages. In our case, packages are imported relatively to the current directory, that is tarantool-authman. However, if necessary, import paths can easily be extended as follows:

-- Prepending a new path with the highest prioritypackage.path = “/some/other/path/?.lua;” .. package.path

Before creating the first space, let’s put all the needed constants into separate models. We need to give each space and each index a name. It’s also necessary to specify the order of fields in a tuple. For example, here’s what the authman/model/user.lua model looks like:

-- Our package is a Lua tablelocal user = {}

-- The package has the only function — model — that returns a table-- with the model’s fields and methods-- The function receives configurations in the form of a Lua tablefunction user.model(config)local model = {}

-- Space and index namesmodel.SPACE_NAME = ‘auth_user’model.PRIMARY_INDEX = ‘primary’model.EMAIL_INDEX = ‘email_index’

-- Assigning numbers to tuple fields-- Lua uses one-based (!) indexingmodel.ID = 1model.EMAIL = 2model.TYPE = 3model.IS_ACTIVE = 4

-- User types: registered via email or with social network-- credentialsmodel.COMMON_TYPE = 1model.SOCIAL_TYPE = 2

return modelend

-- Returning the packagereturn user

When handling users, we’ll need two indexes: unique by id and non-unique by email address — when two different users register with social network credentials, they may be assigned the same email address, or even no email address at all. As for the users who registered the regular way, our application will make sure their email addresses are unique.

The authman/db.lua package contains a method for creating spaces:

local db = {}

-- Importing the package and calling the model function-- The config parameter is assigned a nil (empty) valuelocal user = require(‘authman.model.user’).model()

-- The db package’s method for creating spaces and indexesfunction db.create_database()

local user_space = box.schema.space.create(user.SPACE_NAME, {if_not_exists = true})user_space:create_index(user.PRIMARY_INDEX, {type = ‘hash’,parts = {user.ID, ‘string’},if_not_exists = true})user_space:create_index(user.EMAIL_INDEX, {type = ‘tree’,unique = false,parts = {user.EMAIL, ‘string’, user.TYPE, ‘unsigned’},if_not_exists = true})end

return db

UUID will serve as a user id, and we’ll be using a HASH index for full-match searches. The index for searches by email will be two-part: (user.EMAIL, ‘string’) — user’s email address, (user.TYPE, ‘unsigned’) — user type. As a reminder, the types have been defined in the model a bit earlier. A composite index enables searches not only by all the fields, but also by the first part of the index; therefore, we can search by email address only (without the user type).

Let’s enter the interactive administrator console and try to use the authman/db.lua package.

$ tarantoolversion 1.7.3–202-gfe0a67ctype ‘help’ for interactive helptarantool> db = require(‘authman.db’)tarantool> box.cfg({listen=3331})tarantool> db.create_database()

Great, we’ve just created the first space! One thing to keep in mind here: before calling box.schema.space.create, you need to configure and run the server via the box.cfg method. Now we can perform some simple actions within the space we created:

-- Creating userstarantool> box.space.auth_user:insert({‘user_id_1’, ‘exaple_1@mail.ru’, 1})— -- [‘user_id_1’, ‘exaple_1@mail.ru’, 1]…tarantool> box.space.auth_user:insert({‘user_id_2’, ‘exaple_2@mail.ru’, 1})— -- [‘user_id_2’, ‘exaple_2@mail.ru’, 1]…-- Getting a Lua table (array) with all the userstarantool> box.space.auth_user:select()— -- — [‘user_id_2’, ‘exaple_2@mail.ru’, 1]— [‘user_id_1’, ‘exaple_1@mail.ru’, 1]…

-- Getting a user by the primary keytarantool> box.space.auth_user:get({‘user_id_1’})— -- [‘user_id_1’, ‘exaple_1@mail.ru’, 1]…

-- Getting a user by the composite keytarantool> box.space.auth_user.index.email_index:select({‘exaple_2@mail.ru’, 1})— -- — [‘user_id_2’, ‘exaple_2@mail.ru’, 1]…

-- Changing the data in the second fieldtarantool> box.space.auth_user:update(‘user_id_1’, {{‘=’, 2, ‘new_email@mail.ru’}, })— -- [‘user_id_1’, ‘new_email@mail.ru’, 1]…

Unique indexes restrict the insertion of non-unique values. If you need to create some records that may already be in a space, use the upsert (update/insert) operation. You can find the full list of available methods in the official documentation.

Let’s extend the user model with a capability to register users:

function model.get_space()return box.space[model.SPACE_NAME]end

function model.get_by_email(email, type)if validator.not_empty_string(email) thenreturn model.get_space().index[model.EMAIL_INDEX]:select({email, type})[1]endend

-- Creating a user-- Fields that are not part of the unique index are not mandatoryfunction model.create(user_tuple)local user_id = uuid.str()local email = validator.string(user_tuple[model.EMAIL]) anduser_tuple[model.EMAIL] or ‘’return model.get_space():insert{user_id,email,user_tuple[model.TYPE],user_tuple[model.IS_ACTIVE],user_tuple[model.PROFILE]}end

-- Generating a confirmation code sent via email and used for-- account activation-- Usually, this code is embedded into a link as a GET parameter-- activation_secret — one of the configurable parameters when-- initializing the applicationfunction model.generate_activation_code(user_id)return digest.md5_hex(string.format(‘%s.%s’,config.activation_secret, user_id))end

The code snippet below uses two standard Tarantool packages — uuid and digest — and one user-created package — validator. Before you can use them, they need to be imported:

-- Standard Tarantool packageslocal digest = require(‘digest’)local uuid = require(‘uuid’)-- Our application’s package (handles data validation)local validator = require(‘authman.validator’)

When declaring variables, we’re using the local operator that limits their scope to the current block. Otherwise, these variables will be global, which is what we need to avoid due to potential name collisions.

Now let’s create the main package — authman/init.lua — that will hold all the API methods:

local auth = {}

local response = require(‘authman.response’)local error = require(‘authman.error’)local validator = require(‘authman.validator’)local db = require(‘authman.db’)local utils = require(‘authman.utils.utils’)

-- The package returns the only function — api — that configures and-- returns the applicationfunction auth.api(config)local api = {}-- The validator package contains checks for various value types-- This package sets the default values as wellconfig = validator.config(config)

-- Importing the models for working with datalocal user = require(‘authman.model.user’).model(config)

-- Creating a spacedb.create_database()

-- The api method creates a non-active user with a specified email-- addressfunction api.registration(email)-- Preprocessing the email address — making it all lowercaseemail = utils.lower(email)

if not validator.email(email) then  
  return response.error(error.INVALID\_PARAMS)  
end

_-- Checking if there already exists a user with a given email  
-- address_  
local user\_tuple = user.get\_by\_email(email, user.COMMON\_TYPE)  
if user\_tuple ~= nil then  
  if user\_tuple\[user.IS\_ACTIVE\] then  
    return response.error(error.USER\_ALREADY\_EXISTS)  
  else  
    local code = user.generate\_activation\_code(user\_tuple\[user.ID\])  
    return response.ok(code)  
  end  
end

_-- Writing data to the space_  
user\_tuple = user.create({  
  \[user.EMAIL\] = email,  
  \[user.TYPE\] = user.COMMON\_TYPE,  
  \[user.IS\_ACTIVE\] = false,  
})

local code = user.generate\_activation\_code(user\_tuple\[user.ID\])  
return response.ok(code)  

end

return apiend

return auth

Great! Now users can create accounts.

tarantool> auth = require(‘authman’).api(config)-- Using api to get a registration confirmation codetarantool> ok, code = auth.registration(‘example@mail.ru’)-- This code needs to be sent to a user’s email address so that they-- can activate their accounttarantool> code022c1ff1f0b171e51cb6c6e32aefd6ab

That’s it for now. The next article will be about using ready Tarantool packages, networking, and implementing OAuth2 in tarantool-authman.

Thanks for reading and stay tuned!