Скачать или смотреть Solving JSON Transformation in Nifi with JOLT: Flattening Arrays and Shifting Keys

Transforming complex JSON structures in Nifi using `JOLT` can be daunting. Discover how to flatten your JSON effectively while moving specific keys.
---
This video is based on the question https://stackoverflow.com/q/63927494/ asked by the user 'Tokolosk' ( https://stackoverflow.com/u/13133754/ ) and on the answer https://stackoverflow.com/a/63943145/ provided by the user 'Jagadesh' ( https://stackoverflow.com/u/6112907/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: JOLT transformation for Nifi, flattening array and moving keys

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Flattening JSON Arrays and Moving Keys in Nifi using JOLT

Handling nested JSON structures can be a challenge, especially when you need to transform them for analysis or integration purposes. In this guide, we address a common problem: how to flatten a nested JSON output while moving part of it to a different level using JOLT in Apache Nifi.

The Problem

Imagine you have a JSON structure from an energy meter's readings that looks somewhat like this:

[[See Video to Reveal this Text or Code Snippet]]

Your goal is to create a flattened version of the JSON output like this:

[[See Video to Reveal this Text or Code Snippet]]

In this output format, each entry has the timestamp moved into the same level as the values. Let's explore how to achieve this transformation with JOLT specifications.

The Solution

To perform the required transformation, we will use a set of JOLT specifications. Here’s how to break it down into steps:

Step 1: Create the Initial Shift Operation

You will start with a shift operation to extract the required parts of the nested structure. Below is the JOLT specification you'll need:

[[See Video to Reveal this Text or Code Snippet]]

Explanation

readings: Accesses the main array containing the reading objects.

*: Acts as a wildcard to encompass all meters and their respective records.

records: Navigates into the records of each meter to find timestamps and value fields.

@ : This captures all values and their keys and moves them up in the hierarchy while preserving them in an array format.

Step 2: Shift Timestamp into the Same Level

Now, you need to introduce an additional transformation to take the timestamp out of the array and place it at the same level as the respective values. Here's the second shift operation:

[[See Video to Reveal this Text or Code Snippet]]

Explanation

This operation will couple each timestamp with the corresponding values effectively, enabling each output object to contain both timestamp and data fields like DateTime and ExportWh.

Conclusion

Now you have a clear understanding of how to flatten JSON structures and move desired keys using JOLT transformations in Apache Nifi. By following these steps and leveraging the shift operation creatively, you can transform complex JSON data into a simpler, flatter structure suitable for various purposes like reporting or analysis.

With these tools, you’re well on your way to mastering JSON transformation in Nifi!

Feel free to comment or ask questions if you need further assistance with Nifi and JOLT transformations.