Python built-in zip library is commonly used to create archive. However, there is a concern when creating zip files using built-in library. Consider the case which we are zipping files larger than our available memory, we would easily run out of memory.
I was building a feature that requires zipping of files and upload to our Django backend storage. After digging around the internet, I summarize the logic I used to support this feature.
Use NamedTemporaryFile Instead Of Memory
A NamedTemporaryFile resides in secondary memory instead of main memory, thus using it does not consume extra memory.
Pseudocode
1. Create a NamedTempFile2. Create a ZipFile with NamedTempFile as file output3. Write files into ZipFile4. Move the cursor of the NamedTempFile back to the beginning5. Wrap it with Django File6. Inject a file name7. Upload it to storage
Sample Code
Assume we have a model as in models.py, the logic of using secondary memory to create zip files lies in zipping.py.
<a href="https://medium.com/media/253403bafa88ed3a06ab5e2b8bbb93b2/href">https://medium.com/media/253403bafa88ed3a06ab5e2b8bbb93b2/href</a>
I hope this helps. Please leave me a comment if you have better idea on reducing memory load when zipping files.
Your clap will definitely drive me further. Give me a clap if you like this post.