News for hackers. Stuff that matters.

A comment on Show HN: Prophecy.io – Cloud Native Data Engineering. Parent comment

Re:Show HN: Prophecy.io – Cloud Native Data Engineering by ibains on Friday August 16, 2019 @09:27PM (#20721324)
The variation of complexity of transforms across Enterprises and teams within Enterprises is quite large.

Spark and Hive (that I was product manager for, and used by hundreds of Enterprises) are used quite heavily in data engineering. Many transforms fit well into relational/set based model.

In your particular case, perhaps due to the nature of your transforms, it might not have been the best fit.

Also, if your data is small, it's not always a great fit. However, in many large companies, if they have 500 transforms and 30 are small - there is simplicity in using a single technology.

[ Parent | on HN ]
Re:Show HN: Prophecy.io – Cloud Native Data Engineering by dvt on Saturday August 17, 2019 @03:31PM (#20725826)
Just to be clear, I'm not discounting the work you're doing with Prophecy. We were using Databricks, and I can definitely see how a platform like Prophecy is a step up. I have a personal rule that I will always prop up any budding startup or entrepreneur (the golden rule and all that).

So huge congrats on your release!

[ Parent | on HN ]