Cricket Statistics Data Pipeline in Google Cloud using Airflow | Data Engineering Project

Looking to get in touch?
Drop me a line at [email protected], or schedule a meeting using the provided link https://topmate.io/vishal_bulbule Cricket Statistics Data Pipeline in Google Cloud using Airflow,Dataflow,Cloud Function and Looker Studio

Data Retrieval: We fetch data from the Cricbuzz API using Python.

Storing Data in GCS: After fetching the data, we store it in a CSV file in Google Cloud Storage (GCS).

Cloud Function Trigger: Create a Cloud Function that triggers upon file upload to the GCS bucket. The function will execute when a new CSV file is detected and trigger dataflow job.

Cloud Function Execution: Inside the Cloud Function, we will have code that triggers a Dataflow job. Ensure you handle the trigger correctly and pass the required parameters to initiate the Dataflow job.

Dataflow Job: The Dataflow job is triggered by the Cloud Function and loads the data from the CSV file in the GCS bucket into BigQuery. Ensure you have set up the necessary configurations.

Looker Dashboard: BigQuery serves as the data source for your Looker Studio dashboard. Configure Looker to connect to BigQuery and create the dashboard based on the data loaded.

Github Repo for all code used in this project
https://github.com/vishal-bulbule/cri...
============================================

Associate Cloud Engineer -Complete Free Course
   • Associate Cloud Engineer -Complete Fr...  

Google Cloud Data Engineer Certification Course
   • Google Cloud Data Engineer Certificat...  

Google Cloud Platform(GCP) Tutorials
   • Google Cloud Platform(GCP) Tutorials  

Generative AI
   • Generative AI  

Getting Started with Duet AI
   • Getting started with Duet AI | Google...  

Google Cloud Projects
   • Google Cloud Projects  

Python For GCP
   • Python for GCP  

Terraform Tutorials
   • Terraform  Associate Certification(00...  

Linkedin
  / vishal-bulbule  

Medium Blog
  / vishalbulbule  

Github Repository for Source Code
https://github.com/vishal-bulbule

Email - [email protected]

#dataengineeringessentials #dataengineers #dataengineeringproject #airflow #dataflow #cloudcomposer #bigquery #looker #googlecloud #datapipeline