Скачать или смотреть How to Stream Unzip into Split Gzip Files Efficiently

Learn how to efficiently split a zip file into multiple gzip files using streaming in memory without saving the entire file to disk.
---
This video is based on the question https://stackoverflow.com/q/68979753/ asked by the user 'Nir' ( https://stackoverflow.com/u/719001/ ) and on the answer https://stackoverflow.com/a/68979845/ provided by the user 'Nir' ( https://stackoverflow.com/u/719001/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: stream unzip into split

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Stream Unzip into Split Gzip Files Efficiently

If you've ever worked with large zip files, you might have encountered the need to split the contents into smaller files. In particular, you may want to split them into gzip files without incurring the overhead of writing the entire file to disk first. This can save time and system resources, especially with larger archives. If this describes your situation, you're in the right place!

The Problem: Splitting a Zip File

You have a zip file called a.zip, and you want to split it into multiple gzip files. The goal is to keep everything in memory for improved performance. Many people initially toy with commands like:

[[See Video to Reveal this Text or Code Snippet]]

or

[[See Video to Reveal this Text or Code Snippet]]

However, you might run into issues. Using unzip can lead to unwanted output or errors, such as:

Unwanted Output: The unzip process might output filenames or messages, which can disrupt the flow for gzip.

File Not Found Error: The second command could fail, complaining about 'a_' for reading: No such file or directory.

The Solution: Adding the Right Options

The key to solving this problem lies in the right command options. Specifically, you need to use the -c and -q flags with the unzip command. This combination allows you to extract only the contents quietly, stripping out unwanted messages while sending the output to be gzip'ed and split.

Command Breakdown

Here’s the final command you should use:

[[See Video to Reveal this Text or Code Snippet]]

Explanation of Key Options:

unzip -c -q a.zip:

-c: This flag extracts files to stdout (standard output), which is crucial for streaming the output directly into the next part of the command.

-q: This flag suppresses the normal output (the inflation progress and filenames), providing a clean output stream for processing.

split -C 2M -d -a 3 --filter 'gzip > $FILE.gz':

-C 2M: This option sets the maximum size of each split file to 2 megabytes.

-d: This tells the split command to use numeric suffixes for the output files.

-a 3: This specifies the number of digits in the numeric suffixes to be 3, allowing for a larger number of split files.

--filter 'gzip > $FILE.gz': This uses a filter to gzip each piece created by split.

Conclusion

By leveraging the correct command-line flags, you can effectively split a zip file into multiple gzipped files without ever needing to touch the disk for temporary storage. This is not only efficient, but also enhances the speed of operations, especially with larger datasets.

Now you're equipped with the knowledge to handle zip files more effectively using simple yet powerful command-line tricks! If you have any further questions or need assistance with other commands, feel free to ask.