Скачать или смотреть Export to Cloudsql (PostgreSQL) from Cloud DataProc | PySpark Job - Google Cloud Platform

Export to Cloudsql(Postgresql) from Cloud DataProc | PySpark Job - Google Cloud Platform

github url: https://github.com/vigneshSs-07/Googl...

#Install libraries
pip3 install pyspark
pip3 install sqlalchemy

Start a cluster using the gcloud CLI
gcloud dataproc clusters start ${CLUSTER_NAME}
--region=${REGION}

Command to create single node cluster for sql instance disk size configuration
PROJECT_ID=learngcp-ace-guide-342819 REGION=us-east1 INSTANCE_NAME=mypostgresql CLUSTER_NAME=dataproc-demo1 gcloud dataproc clusters create ${CLUSTER_NAME}
--region ${REGION}
--scopes sql-admin
--single-node
--master-machine-type n1-standard-2
--master-boot-disk-size 500
--image-version 2.0-debian10
--initialization-actions gs://goog-dataproc-initialization-actions-${REGION}/cloud-sql-proxy/cloud-sql-proxy.sh
--metadata "enable-cloud-sql-hive-metastore=false"
--metadata "additional-cloud-sql-instances=${PROJECT_ID}:${REGION}:${INSTANCE_NAME}"

Command to submit pyspark job
gcloud dataproc jobs submit pyspark datatoPostgresql.py --cluster=dataproc-demo1 --region=us-east1 --jars=gs://demo_bucket_97/jars/postgresql-42.2.20.jar

Stop a cluster using the gcloud CLI
gcloud dataproc clusters stop ${CLUSTER_NAME}
--region=${REGION}

Resources:
https://github.com/GoogleCloudDatapro...

Apache Beam - Playlist
   • Apache Beam - Python SDK  

Subscribe to Channel: https://lnkd.in/ehFZbVH5

Follow us in LinkedIn: https://lnkd.in/gDT3ESdm
Follow us in Instagram: https://lnkd.in/gZ278ShA
Follow us in Facebook: https://lnkd.in/gQGF_3Eb
Follow us in Twitter: https://lnkd.in/gh7dZACW
Join in Telegram channel: https://lnkd.in/guFt2sAg
Join in WhatsApp group: https://lnkd.in/gAqkuDPA

Connect with me here:
Instagram:   / thearrow0494  
LinkedIn:   / vignesh-sekar-sujatha-02aa9b125  

🙏🙏🙏🙏🙏🙏🙏🙏
YOU NEED TO DO BELOW THINGS to support my channel
1. LIKE
2. SHARE
&
3. SUBSCRIBE
TO MY YOUTUBE CHANNEL

#gcpcloud #dataproc #pyspark