Deploy Talend Job into Snowflake Using Airflow

Описание к видео Deploy Talend Job into Snowflake Using Airflow

create or replace TABLE AIRFLOW_SNF.AIRFLOW_SCHEMA.AIRFLOW_TABLE (
ID NUMBER(38,0),
NAME VARCHAR(16777216),
COMPANY VARCHAR(16777216)
);

truncate table airflow_schema.airflow_table;

create or replace file format my_csv_format
type = csv field_delimiter = ',' skip_header = 1
field_optionally_enclosed_by = '"'
null_if = ('NULL', 'null')
empty_field_as_null = true;

CREATE OR REPLACE STAGE AIRFLOW_SNF.AIRFLOW_SCHEMA.AIRFLOW_TABLE_stage
URL = 's3://s3stag/landing_directory/'
CREDENTIALS = (
AWS_KEY_ID = 'AKIATNIWVCQAYAQTXCPS',
AWS_SECRET_KEY = 'INY4AN472hKwlPFnnmi1rdp0oJlAH\’
)
FILE_FORMAT = my_csv_format;

show stages;
list @AIRFLOW_SNF.AIRFLOW_SCHEMA.AIRFLOW_TABLE_stage;

select * from AIRFLOW_SNF.AIRFLOW_SCHEMA.AIRFLOW_TABLE;

Execute the below code to start the server:
------------------------------------------------
source venv/bin/activate
airflow db init
airflow webserver

In a new terminal
source venv/bin/activate
airflow db init
airflow scheduler

Git Hub link: https://github.com/8867156429/Deploym...
Talend open studio download and install:    • Talend Download, SQL, SnowFlake  
Snowflake Download and install:    • SnowFlake ( Extract using PANDAS)  

CICD using git hub:    • Snowflake_GitHub_Schemachange_CICD  

Комментарии

Информация по комментариям в разработке