In this story i would go through how to begin a working on deep learning without the need to have a powerful computer with the best gpu , and without the need of having to rent a virtual machine , I would go through how to have a free processing on a GPU , and connect it to a free storage , how to directly add files to your online storage without the need to download then upload , and how to unzip file for free online .
I am going through how i am beginning my deep learning project using google colab that allows you to start working directly on a free Tesla K80 GPU using Keras, Tensorflow and PyTorch, and how i connect it to google drive for my data hosting , I would also share some techniques i have used to automatically download data to google drive without needing to first download them , and then uploading them to google drive , then I would go through how you can unzip your files online , without the need to unzip them in python code .
So to sum all things up i would be talking
- Google Colab
- Connecting to Google Drive
- Downloading data directly to google drive without need to download then upload (from an external link , another google drive)
- unzip files online
so lets start
Google Colab
google colab is a free to use Jupyter notebook , that allows you to use free Tesla K80 GPU it also gives you a total of 12 GB of ram , and you can use it up to 12 hours in row .
You can either
- begin a new google colab
- or upload a github ipyn notebook
so lets begin with how you can create a google colab
1- Begin a new google colab
1-a) go to https://colab.research.google.com
1-b) select Google Drive Tab (to save your new google colab to google drive)
2-c) select New Python 3 Notebook (you can also select python 2 notebook)
a blank notebook would be created to your google drive , it would look like this
You can change the runtime of your notebook from selecting the runtime button in the top menu , to
- change which python version you are using
- choose a hardware accelerator from ( GPU , TPU )
2- upload a github Jupyter notebook to google colab
you can automatically upload an already made Jupyter notebook from github automatically to colab , this truly helps in testing code on the fly , so lets try it out
2-a) this is a Jupyter notebook that i have created for text summurization
2-b) lets also go to https://colab.research.google.com , but this time we would select github tab
3-c) then we just paste the Jupyter notebook link , and click upload
You can even read more in this tutorial
As you have seen , its rather easy to start a colab project , but for any real world data project , it would require a truly huge datasets , there are many ways to do this , but to truly achieve the goal of having truly huge datasets , you would need the power of google drive .
So lets connect our google drive to our google colab notebook.
Connect to Google Drive
in the newly created notebook , add a new code cell
then paste this code in it
!apt-get install -y -qq software-properties-common python-software-properties module-init-tools!wget https://launchpad.net/~alessandro-strada/+archive/ubuntu/google-drive-ocamlfuse-beta/+build/15331130/+files/google-drive-ocamlfuse_0.7.0-0ubuntu1_amd64.deb!dpkg -i google-drive-ocamlfuse_0.7.0-0ubuntu1_amd64.deb!apt-get install -f!apt-get -y install -qq fusefrom google.colab import authauth.authenticate_user()from oauth2client.client import GoogleCredentialscreds = GoogleCredentials.get_application_default()import getpass!google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret} < /dev/null 2>&1 | grep URLvcode = getpass.getpass()!echo {vcode} | google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret}
!mkdir -p drive!google-drive-ocamlfuse drive
this would connect to your drive , and create a folder that your notebook can access your google drive from
It would ask you for access to your drive , just click on the link , and copy the access token , it would ask this twice
after writing this code , you run the code by clicking on the cell (shift enter) or by clicking the play button on the top of your code cell
then you can simply access any file by its path in form of
path = "drive/test.txt"
by this you can load truly huge file , as your google drive can hold up to 115GB and google colab offers a good ram of 12 GB so this would open the way for truly great data apps
But a new problem would arise , as most of the data that one would use in his data projects would be in the gigabytes , so to work with them in your google colab you would need to download them , then to upload them , this could really take much time , as most people have really slow upload speeds , which would make working on a multi gigabyte datasets quite an issue , so that what the next section discovers .
I have managed to use some services that allow you to remote download links directly to your google drive without need to download them , and then uploading again to your drive .
Also there is another service that allows you to have a copy of a file (that is hosted on another google drive ) directly to your google drive ,
so Lets begin
Downloading data directly to google drive without need to download then upload
primarily you have 2 scenarios , either the file that you need is
- hosted on a website , that you can download the dataset through a link
- the needed file is hosted on another google drive
lets discover the first scenario
1-a) Download from external link
Here we would use multcloud , multcloud is a service that contains a free plan that allows you to Manage your cloud services , one of its services that we would use is its ability to download files directly to your google drive ,
You just paste your link , and wait for the link to be downloaded directly to your google drive , using he free plan you would be able to transfer 50GB/month
After signing up
Fist Add Google Drive to multcloud
then authorize your account
after sign in , and after registering your google drive , click on your google drive
then click on upload
then click on upload url
then paste your link
then you can see your download progress by click on the top button , then url tasks
then a menu would appear , where you can track your progress , or even add urls from here
1-b) Download from another google drive
sometimes there is a case where your files actually are on another google drive , and you just try to have a copy of them to your google drive
I have tried using multcloud in this case , using the direct google link to download the files , but with no success , so i came across Copy, URL to Google Drive , which enables you to easily copy files between different google drives
paste your link , name it , then save to google drive
after authenticating , you just click save to google drive
But most of the datasets are found in a zip format , they would need to be unzipped to be able to deal with them in your project .
Some people unzip their files inside google colab using python code , but i find that there are much more productive ways , a way that don’t require writing code and maintaining the code for different zip formats , one of them is using cloudconvert
Unzip Files online for free
cloudconvert is an app that is available to use with google drive , that allows you to unizp your files for free , it can unzip a huge range of zip formats , so lets begin
Fist lets connect more apps in your google drive
search for cloudconvert , then hit enter , then add the app
then lets open the zip through cloudconvert
then you select the format to be extract , then you hit start converting (the checkbox of save to drive must be checked)
then your zip would be saved to your drive , this is a much convent way than writing code to unzip
through the above sections , we walked through how you can build a data project without need of a strong device , without need of renting one , or without having to have a huge expensive online storage , or without the need to download and then upload , and we presented a way of unziping files online
I truly hope this was beneficial to you , I am looking forward to hear your comments , and please tell me if it was beneficial to you .