Data Engineering Use cases

 How to become a data engineer?

• expertise in SQL

• programming skills (python, scala, java)

• processing frameworks (Apache Spark, Apache Beam, Hadoop, Apache Flink)

• data warehousing tools like Hive, Google BigQuery, Amazon Redshift 

• streaming  tools such as Apache Kafka, Google Dataflow, Google pubsub, Amazon SNS, Amazon SQS, Apache storm etc.

• workflow management tools such as Apache Airflow, Google cloud composer, Amazon Data Pipeline, AWS Glue, Managed Workflows etc.

I have created some use cases which are useful in data engineering and I will keep on adding different use cases, technical concepts, scenarios going forward. They can  be helpful while designing and implementing data pipelines in your project. 

Use cases:

1. Joins and group by

2. Window functions with joins

3. Running Aggregates

4. Run Spark on Google Cloud Platform

5. Schedule spark jobs on GCP - Composer

6. Write dataflow custom template