9. How to do full load from On Premise SQL Server till ADLS using Azure Synapse Pipelines

This video tells how to do full load from On Premise SQL Server till ADLS in a Parquet format using Azure Synapse Pipelines.

We used Azure SQL DB to store the metadata details. We loaded data from On premise SQL server to ADLS layer.

To prepare the metadata we have used this query :

select a.TABLE_SCHEMA as Table_Schema, a.TABLE_NAME as Table_Name ,a.TABLE_CATALOG,
'select '+ string_agg(b.column_name,',') + ' from ' + a.TABLE_SCHEMA+'.'+a.TABLE_NAME
as Query, 'Y' as IncrementalFlag, 1 as Isactive
from INFORMATION_SCHEMA.tables a
inner join INFORMATION_SCHEMA.COLUMNS b
on a.TABLE_SCHEMA=b.TABLE_SCHEMA and a.TABLE_NAME=b.TABLE_NAME
where TABLE_TYPE='BASE TABLE'
group by a.TABLE_SCHEMA,a.TABLE_NAME,a.TABLE_CATALOG

#azuredatafactory
#azuresynapseanalytics
#datafactory
#azure