Bump!

I thought the _Learning Spark_ book was a pretty good introduction. Databricks offers it for free here [0]

[0] https://pages.databricks.com/rs/094-YMS-629/images/LearningS...

Just get a job since you are already senior. You can learn it on the job. Find a few tutorials if you must, but people should be able to pick it up in a few weeks for basic work.

I'm a Data Engineer which uses Spark daily. I guess the only important cert would come from Databricks, but I think it will be more worth your while to read the book mentioned here and try to do a little project ingesting/transforming data

If you have experience in any data frame library (like Pandas), and SQL, you can pick up PySpark pretty easily... With the one caveat that writing good data pipelines in any language gets much harder when you start looking at ways to actually processes big data (~20+TB). Modern SQL engines are so good though.

Having used spark for the past 8 years or so, it's definitely a solid basic for data engineering. I use it for generating reports the most, but sometimes we have large projects to get data into different staging databases. I use it a lot with ElasticSearch or a parquet. Basically it helps you write large joins and flatten the result to a database that can more quickly perform aggregations on that flattened result (like Elasticsearch) or a columnar database.

honestly I love this post and it's great to read but I'm also here to share ideas of What I've experience so far, although this may not concern the book, but it's very important for you to know, 2 days ago I saw a guy and he's was named Timothy, he posted a review here, about jbee spy team, are first I didn't believe, I thought it was a scam and I even reported this account, well, I was having issues with my bitcoin account, because it was hacked by scammers, and there was a lot of money in that account, so I needed a way out I tried going to the Bitcoin office but there was nothing to do about it, so I contacted them, and within 10 minutes my account was restored, and thay help me track down the scammer too, honestly I was so happy but I was angry at the same time because i reported someone who is trying to help us, please Timothy forgive me I can't bring your account back but I can find a way to reach out to others and let them know what happened

you can contact them on telegram +44 7456 058620 or via email [email protected]

if you need any assist, I can bet you 100% sure