Скачать или смотреть How to Parse Large JSON Files with jq in Streaming Mode

Learn how to efficiently parse large JSON files using the `jq` command-line tool in streaming mode. This guide provides step-by-step instructions and examples for transforming your JSON data without overwhelming system memory.
---
This video is based on the question https://stackoverflow.com/q/69047630/ asked by the user 'CYC' ( https://stackoverflow.com/u/8621405/ ) and on the answer https://stackoverflow.com/a/69048741/ provided by the user 'peak' ( https://stackoverflow.com/u/997358/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: jq parse json with stream flag into different json file

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Efficiently Parsing Large JSON Files with jq in Streaming Mode

Working with large JSON files can be challenging, especially when they exceed several gigabytes in size. Loading massive files into memory can lead to performance issues or even crashes. Fortunately, the jq command-line tool offers a powerful solution for parsing JSON data in a memory-efficient manner by utilizing the streaming mode. If you’re dealing with a 20GB JSON file and want to extract specific data efficiently, you've come to the right place.

The Challenge

Let's say you have a JSON file named data.json, structured like this:

[[See Video to Reveal this Text or Code Snippet]]

From this file, you aim to extract the information under the tooldatareports key. Your end goal is to achieve a specific JSON structure, where you obtain a list of dictionaries, each containing a data key with the associated time-series data.

Understanding the Data Transformation

Desired Output Structure

The desired output should look like this:

[[See Video to Reveal this Text or Code Snippet]]

Initial Attempt and Limitations

Your first attempt to extract this data using the command:

[[See Video to Reveal this Text or Code Snippet]]

This approach, while functional, presents several issues:

The output is not formatted as a list of dictionaries as required.

The time and value pairs are contained within separate lists, complicating the desired structure.

The Solution: Using Streaming Mode with jq

To effectively parse the JSON file without loading the entire content into memory, a more refined approach is required. Here’s how to side-step the limitations encountered in your initial command using jq in streaming mode:

The Command

[[See Video to Reveal this Text or Code Snippet]]

Explanation

jq -n --stream: This starts jq in streaming mode.

[fromstream(inputs ... ]: This retrieves the input JSON data in a streaming fashion.

index("data"): Identify the index of the key “data” in the input.

select($ix): Filter to keep only relevant entries based on the index found.

.[0] |= .[$ix:]: Adjust the structure to keep only the required data.

This command effectively restructures your JSON output to match your desired format while maintaining low memory usage.

Conclusion

In this guide, we explored how to utilize jq for parsing large JSON files in streaming mode. By understanding both the initial problem and implementing the right command, you can efficiently process even the largest of data sets without running into memory constraints. This technique will not only enhance your workflow but also enable you to handle data-driven tasks more effectively. Happy coding!