Show HN: Data Engineering Project for Beginners

Re:Show HN: Data Engineering Project for Beginners by jkm2155 on Sunday June 07, 2020 @09:47AM (#23447294)

Simple project to help beginners get started with data engineering. This is my fist post on HN, any feedback would be greatly appreciated.

[ Parent | on HN ]

Re:Show HN: Data Engineering Project for Beginners by gifflar on Sunday June 07, 2020 @12:15PM (#23448391)

Thanks for the illuminating post. I like how Apache Airflow is used to move the pyspark script to a S3 location so that it can be read by the EMR step. I remember working on a project where we wanted to automate a data pipeline using Airflow and had this problem of how to get our pipeline scripts to the right locations.

[ Parent | on HN ]

Re:Show HN: Data Engineering Project for Beginners by jkm2155 on Thursday June 11, 2020 @12:49PM (#23490745)

@gifflar Glad you found it illuminating :). Yea moving spark script to S3 using a Airflow task is usually the easiest.

[ Parent | on HN ]