What is ETL 💭?
ETL is short for extract, transform, load, three database functions that are combined into one tool to pull data out of one database and place it into another database.
- Extract is the process of reading data from a database. In this stage, the data is collected, often from multiple and different types of sources.
- Transform is the process of converting the extracted data from its previous form into the form it needs to be in so that it can be placed into another database. Transformation occurs by using rules or lookup tables or by combining the data with other data.
- Load is the process of writing the data into the target database
In the sample apps given by hasura, serverless-etl app is one among them. I will be showing how to implement it using Next.js .
This is the link to the example given by hasura.
The main features of this app will be :-
- Add Book to the database
- From the database data is saved in an algolia index
- Search for the book in the algolia index
How will we implement it ?
We will be using graphql mutations to insert book details into the database and when something is inserted into the database an event is triggered which will update the algolia index with the inserted data from the database and this data can be retrieved from the algolia index when searched. That’s it! it is that simple.
Pre-requisites:
- Algolia account
- Hasura GraphQL Engine (HGE) installation
Now that you are ready, Lets begin!
Setting up basic Next.js project and Graphql Engine
First step is to create a next.js project using yarn create next-app etl-example . This command will create a basic boilerplate for a next.js project on which we will be working on.
Add the packages and scripts which are not present in your package.json file from the gist below and install it using yarn install.
Next setup a Hasura GraphQL engine using the the one-click deploy option and note down the deployed url and create a config.js file in the root folder with the following code
replace the herokuapp-url with the url of the deployed hasura engine. Add authentication headers in the config object if you have any.
Creating Book Table
We will be having a single table name book which will have the details of the book like title and author. For that we have to go to the Hasura GraphQL engine console and create the following table
book table
Setting up Algolia Index
Next step is to setup an Algolia Index, For that follow the instructions given below.
- Sign-up for Algolia account
- Go to the Dashboard,
- Click on
Indicestab on the left sidebar - Click on
New Indexbutton - Name it as
demo_serverless_etl_app - Goto
API Keystab on the sidebar - Copy the
Application ID(we'll call this**ALGOLIA_APP_ID**) - Click on
All API Keystab and then clickNew API Keybutton - Give description as
server key - Choose the
demo_serverless_etl_appindex - Select
Add recordsunder the ACL section - Click
Generateand copy this API key from the list (we'll call it**ALGOLIA_ADMIN_API_KEY**)
Setting up webhooks
For adding the data into the algolia index when some data is inserted into the database, we will be needing a webhook which will be triggered on data insertion and this webhook will add the inserted data into the algolia index. For having a single api endpoint there is no point of having a seperate server. What we can do is setup a node server and add the api endpoint for the webhook and route all other url’s to the next.js handler.
For the node server we have to create a file named server.js in the root directory and copy the content from below. The route /webhook will be used to trigger an event when data is inserted into the table to add it into the algolia index. Add your algolia APP_ID and the ADMIN_API_KEY in the empty space present in the code.
Building the front-end
Next we will setup our page where we will take in the author name and book title which will be added to the database and also a search bar where you can search for any book. Create a file named index.js inside post folder and add the following code and also replace the algolia api variables with your values.
Search Component
The InstantSearch component provided by react-instantsearch-dom package will do the search magic automatically. You do not have to worry about coding all those searching stuff 😉. The SearchBox component will render the searchbox and Hits component will render the results of your search
Deploy
Now that wehave the server ready, deploy it anywhere for example glitch and you will get a url for the deployed server note that down, let it be <SERVER_URL>.
Creating an Event Trigger
Our next step is to create an event trigger on adding a new book to the database, for this go to Events tab in Hasura Graphql Engine Console and add a new trigger. Replace <TRIGGER_URL> with <SERVER_URL>/webhook.
Trigger
Now you can open <SERVER_URL> in your browser and see the magic! 😉
And that’s it, we have the ETL-demo ready🙌 . I haven’t added any stylings in the code. If you want to make it look more awesome, just add the required css and make it better 😎.
Link to the gists :- index.js , server.js , package.json
About Me
My name is Aswin VB. I am an undergraduate student at the Indian Institute of Information Technology, Allahabad, pursuing my B.Tech degree in Information Technology.
I love creating and learning new things which i always do when i have free time. I love to code in JavaScript and Python.