How to Become a Data Engineer in 2021

Re:How to Become a Data Engineer in 2021 by StreamBright on Monday January 11, 2021 @09:23AM (#25729137)

2021? More like 2010. Hadoop is getting deprecated rapidly and more companies split their write and read workloads. Separated storage and compute is also popular. Scala is not used that much, I think it is not worth the time investment. More and more companies go for Kotlin instead of Java when these want to tap into the Java ecosystem.

[ Parent | on HN ]

Re:How to Become a Data Engineer in 2021 by adilkhash on Monday January 11, 2021 @09:37AM (#25729315)

Hadoop is still widely used in enterprises (especially in banks), if you have experience working with Hadoop ecosystems it is a big plus anyway.

[ Parent | on HN ]

Re:How to Become a Data Engineer in 2021 by StreamBright on Monday January 11, 2021 @10:45AM (#25730431)

Yes, that is the status quo.

There is also some trends:

https://trends.google.com/trends/explore?date=today%205-y&ge...

[ Parent | on HN ]

Re:How to Become a Data Engineer in 2021 by pjmlp on Tuesday January 12, 2021 @01:58AM (#25743339)

Kotlin's future is tied with Android, on the JVM it will be another Scala in 5 years time.

If JetBrains gets lucky, they might manage to create a cross-platform Kotlin eco-system as they are trying hard to push, as means to sell InteliJ licenses.

Lets see if it doesn't end like like Typesafe.

[ Parent | on HN ]

Re:How to Become a Data Engineer in 2021 by smattiso on Monday January 11, 2021 @10:50AM (#25730509)

Are you working in this field? I am looking for a consultant to setup a modern data processing pipeline for a data driven hardware product I am building.

[ Parent | on HN ]

Re:How to Become a Data Engineer in 2021 by StreamBright on Monday January 11, 2021 @01:11PM (#25733133)

Yes, I am moving companies to their next data pipeline, that is my specialty. I have added my email to my profile, you can reach out to me.

[ Parent | on HN ]

Re:How to Become a Data Engineer in 2021 by kentm on Tuesday January 12, 2021 @11:04AM (#25748662)

What does your typical stack look like?

[ Parent | on HN ]

Re:How to Become a Data Engineer in 2021 by StreamBright on Saturday January 16, 2021 @05:32PM (#25806157)

Depends. Just some random mixture of stacks: PrestoDB, S3, Airflow, Luigi, Dremio, Athena, Hive LLAP, EMC Isilon, Kafka.

My favorite so far is S3 + PrestoDB with either ORC or Parquet files. It is a solid DWH solution for most enterprises on the cloud. (Cloud or not is a different discussion). It works for small scale (50TB) to really high scale (50PB). There are some (very few) gotchas and moving parts as opposed to Hadoop + co. You can combine it with Kafka for streaming data and you got yourself a pretty solid data solution.

[ Parent | on HN ]