Скачать или смотреть How to Create and Read Delta Tables with map Types without Spark in Python

Learn how to create and read Delta Lake tables with map column types in Python without using Spark, leveraging PyArrow and deltalake.
---
This video is based on the question https://stackoverflow.com/q/79329985/ asked by the user 'Frank' ( https://stackoverflow.com/u/9388056/ ) and on the answer https://stackoverflow.com/a/79349088/ provided by the user 'Frank' ( https://stackoverflow.com/u/9388056/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: DeltaTable map type

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to drop me a comment under this video.
---
Introduction

Delta Lake supports complex column types like MAP<STRING, TIMESTAMP>, which can be easily created with Spark. However, creating such tables without Spark, especially using Python libraries like pyarrow and deltalake, can be challenging due to type compatibility and schema enforcement.

This guide explains how to create a Delta table with a map type using Python only, and how to query it efficiently.



Problem

Attempting to create Delta tables with map columns via PyArrow's Arrow Table API often results in errors like:

pyarrow.lib.ArrowTypeError: Could not convert 'a' with type str: was expecting tuple of (key, value) pair

NotImplementedError: ArrowSchemaConversionMode.passthrough is not implemented to work with DeltaSchema

These errors occur because the expected input data types and schema handling for maps aren't straightforward in the current Python ecosystem.



Solution

You can construct a map type Delta table in Python by:

Using PyArrow to define an explicit schema that includes a map_ type field.

Creating your data as a list of dictionaries where the map column is a Python dictionary.

Building a PyArrow Table directly from this data and schema.

Writing the table to Delta format with the write_deltalake method from the deltalake Python package.

Step-by-step Code Example

[[See Video to Reveal this Text or Code Snippet]]

Notes:

Use pa.map_(key_type, item_type) to define map columns properly.

The data for the map should be a Python dict.

Passing an explicit PyArrow schema helps avoid conversion errors.

Use the Rust engine in write_deltalake for better compatibility and performance.



Reading Delta Tables with Map Types

You can read the Delta table that contains map types efficiently using duckdb:

[[See Video to Reveal this Text or Code Snippet]]

This leverages DuckDB’s ability to query Delta Lake tables directly, preserving map types.



Summary

Define map columns using PyArrow’s pa.map_ types explicitly.

Convert your data to a PyArrow Table with the specified schema.

Use write_deltalake from the deltalake Python package (preferably with the Rust engine) to write tables.

Use DuckDB for convenient and performant reading of Delta tables with complex types.

This approach avoids Spark altogether and works well with modern Python data processing stacks.