Скачать или смотреть How to Use jq to Extract and Transform JSON Data with External Commands

Learn how to invoke external commands in conjunction with `jq` to extract JavaScript from JSON structures effectively.
---
This video is based on the question https://stackoverflow.com/q/75661161/ asked by the user 'alturkovic' ( https://stackoverflow.com/u/5291611/ ) and on the answer https://stackoverflow.com/a/75661821/ provided by the user 'knittl' ( https://stackoverflow.com/u/112968/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: How can I invoke external command to transform json using jq?

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
How to Use jq to Extract and Transform JSON Data with External Commands

When working with JSON data, one of the common tasks is to process information in a structured manner. Many developers leverage tools like jq for this purpose, but what happens when you need to use external commands to transform data? This guide will explore how to extract JavaScript <script> tags from an HTML string contained in a JSON structure using jq and a tool called pup.

The Problem at Hand

Imagine you have a JSON file that contains a URL and an HTML string as follows:

[[See Video to Reveal this Text or Code Snippet]]

You want to extract all the JavaScript <script> tags from the html field and store them into a new JSON structure. Here’s the enhanced output you desire:

[[See Video to Reveal this Text or Code Snippet]]

While jq excels at manipulating JSON data, it cannot directly call external programs like pup from within its operations. So, how can we achieve this?

The Solution Explained

Using POSIX Shells

The most straightforward way to solve this requires using POSIX shells that allow piping multiple commands. You can execute the transformation as follows:

[[See Video to Reveal this Text or Code Snippet]]

Breakdown of the Command:

Extract URL:

jq '{url}' input.json: Extracts the url from your JSON input.

Extract Scripts using pup:

jq -r '.html' input.json | pup 'script[type="text/javascript"] text{}': This command takes the HTML content and uses pup to extract all <script> tags of type text/javascript.

Format Scripts for JSON:

jq -Rs '{scripts: .}': This takes the raw output (just text, not JSON) from the previous command and converts it into a JSON array under the key scripts.

Combine Results:

jq -s 'add': Merges the JSON objects from previous commands into one final object.

Alternative Method: Using --arg

Another, albeit less efficient, method is invoking jq with the --arg option. This necessitates reading the input twice:

[[See Video to Reveal this Text or Code Snippet]]

Key Points:

The $(...) allows you to execute the embedded command and insert the output as a variable in the main jq command.

Like the previous method, this will also yield a similar output, but reads the input twice, which could be less efficient for larger datasets.

Sample Output

Both methods will produce output similar to the following:

[[See Video to Reveal this Text or Code Snippet]]

Conclusion

Transforming JSON data with external commands requires a good grasp of how jq interacts with shell commands. Although jq has its limitations, by cleverly orchestrating your commands within a shell, you can achieve complex data extraction and manipulation tasks. Now you can effectively transform and structure your JSON data combined with pup to extract JavaScript content seamlessly.

Exploring such workflows can enhance your data processing capabilities, especially when working with web scraping and information extraction tasks!