Web scraping is one of the most useful tools for any data practitioner to have.
As you know, real life is not a Kaggle dataset.
Most of the data does not exist neatly, just for you, structured in a file or database, waiting until you use it.
If you ask me though, I’d say that collecting data from the web is incredibly fun. So Kaggle or not, you should be ok after reading through this tutorial.
Since day one, I’ve been amazed to see things done automatically. Huge amounts of data -- which would take months of cumbersome work in order for it to be collected manually --- can now be gathered in a matter of seconds.
Usually, these processes are done with the help of some very powerful programming languages, such as Python (my personal favorite), Ruby, or even C++.
Although very effective, the above scenario makes web scraping out of reach for people without a programming background.
A few years ago -- before I learned to code -- I was trying to collect data about football matches by copy and pasting it manually in an Excel sheet. Well, when I realized how much time it would take me, I gave up.
In this article, we’ll see how a no-code scraping tool can be the solution -- not only for non-coders -- but also anyone that could use some data being collected within a few clicks -- or even virtually with no clicks at all.
Getting Started
The first big advantage of a no-code tool for web scraping is, obviously, the friendly interface and the lack of a coding requirement.
Also though, this approach makes it possible to take advantage of features that no programming language could provide.
First of all, if your need is to collect data from the world’s most famous websites, then everything you need to do is… well, nothing.
Yes, the entire scraper is already built-in for you.
Let’s say I want to gather information about a particular product on Amazon. I mean, it’s almost the holiday season, right?
All you have to do is to select the Amazon template and then tell them the zip code and what you’re looking for:
Here’s the data we collected with basically just a couple of clicks and that can be easily exported to an Excel sheet, a CVS file, or even a SQL database.
No time wasted writing any code or configuring any environment! It does not get any easier than this.
Going Deeper
But of course, if we’re talking here about not being dependent on a programming language, we’re not going to be dependent on built-in templates, no matter how easy they make our lives at some points.
Such a tool obviously needs to be able to scrape any website you want and not only pre-settled pages. The beauty here is that you can scrape any page you like with virtually no clicks.
As an example, let’s use quotes.toscrape.com -- a website built for scraping-learning purposes, so it’s a good choice for this exercise.
If you enter this URL (or any URL you want) a built-in browser will be opened and there will be a button to auto-detect the webpage data.
This is a machine-learning algorithm, which is trained to identify patterns on the website. It shows the user how the data is structured, as well as the best way to collect it.
By choosing this approach on the website that we’re using as an example, all the information about each quote is already identified and you can even see a preview of the data.
Now, the Tips pop-up keeps making your life easier by suggesting new steps to make your scraper more powerful. In this case, you can easily create pagination to grab the quotes from all the pages on the website.
Selecting the Data Manually
OK, machine learning algorithms are great, but sometimes they cannot do everything for you and that’s why it’s important to have the option of manually choosing the data that you want collected.
We’re now collecting data about cryptocurrencies here.
Looking at the website, we can see a table with the top ten cryptocurrencies (according to this website) where information such as price, market cap, volume, etc. are available. That’s what we’re looking for.
If you use the auto-detect button on this particular page it will not select the data on this table. It will select the news headlines at the bottom.
I mean, it's great that the algorithm automatically creates a way to click on the “Show more” button and scroll down the page for more news. Sadly, that's not the data we came for.
So we have the option of selecting, with a few clicks, the entire table for extraction. Just like this:
And just like that, it’s possible to select and extract basically anything from any page you want.
More Advantages
Besides all of this, no-code web scraping also comes with all of the advantages of software like Octoparse.
Advantages such as a dashboard where you can monitor all your scraping tasks at the same time, the option of running tasks locally or in their cloud where a complete infrastructure of IP addresses and a backup of the data are already provided, task scheduling, and easy connection to SQL databases.
A recently added and very cool feature is the possibility to export the scraped data you have stored on the cloud to several types of applications such as Dropbox, Google Sheets, MongoDB or even to upload a new file directly to Google Drive.
All these can be done by connecting the user’s account in each of these applications to the Octoparse account through integration with Zapier, which allows a trigger to be set up so your data can be automatically stored whatever you want as soon as it is collected from the web. All of this with no code needed at all, of course.
Wrapping Up
As we have seen thus far, web scraping has broken the barriers of programming and can now be done in a much simpler and easier manner all while enjoying a friendly interface. Most importantly, not a single line of code is needed!