Скачать или смотреть Pushing data into Snowflake via Snowpipe using Apache NiFi

Quick demo showing how to use the recently added processors and controller services (added in Apache NiFi 1.19) to push data into Snowflake via Snowpipe.

Dataset used for the demo:
https://www.kaggle.com/datasets/seatt...

Key Pair Authentication:
https://docs.snowflake.com/en/user-gu...

SQL Statement to create the table:
create or replace TABLE MYDATABASE.PUBLIC.INVENTORY (
BIBNUMBER NUMBER(38,0),
TITLE VARCHAR(5000),
AUTHOR VARCHAR(500),
ISBN VARCHAR(5000),
PUBLICATIONYEAR VARCHAR(500),
PUBLISHER VARCHAR(5000),
SUBJECTS VARCHAR(5000),
ITEMTYPE VARCHAR(10),
ITEMCOLLECTION VARCHAR(10),
FLOATINGITEM VARCHAR(10),
ITEMLOCATION VARCHAR(10),
REPORTDATE VARCHAR(35),
ITEMCOUNT NUMBER(38,0)
);

SQL Statement to create the pipe:
create or replace pipe MYDATABASE.PUBLIC.MYPIPE auto_ingest=false as COPY INTO INVENTORY FROM @%INVENTORY FILE_FORMAT = (TYPE = 'CSV', FIELD_DELIMITER = ',', FIELD_OPTIONALLY_ENCLOSED_BY = '"');