Re:Show HN: Prophecy.io – Cloud Native Data Engineering
by jeremy_k on Friday August 16, 2019 @11:38PM
(#20721846)
> In reality, most companies will have datasets that could be ETL'd on my iPhone and a Spark cluster is just overkill.
I like that you said this and I've seen a similar sentiment before. I was first exposed to it watching an ElixirConf keynote from José Valim[1] about the Flow framework they built in Elixir where he summarized it as "For between 40-80% of the jobs submitted to MapReduce systems, you'd be better off running them on a single machine." which referenced the paper Musketeer: all for one, one for all in data processing systems [2].
While I'm no Data Engineer myself, I do often wonder if distributing the workload is always better? The anecdote above indicates that a powerful single multicore machine may be right solution for many.
Now that isn't to discount what Prophecy is trying to do, my company just went through a huge re-platforming moving from on on premises to the cloud and it is not easy; any company trying to tackle that space is on the right track. But I just wonder if its overkill for most use cases?
I like that you said this and I've seen a similar sentiment before. I was first exposed to it watching an ElixirConf keynote from José Valim[1] about the Flow framework they built in Elixir where he summarized it as "For between 40-80% of the jobs submitted to MapReduce systems, you'd be better off running them on a single machine." which referenced the paper Musketeer: all for one, one for all in data processing systems [2].
While I'm no Data Engineer myself, I do often wonder if distributing the workload is always better? The anecdote above indicates that a powerful single multicore machine may be right solution for many.
Now that isn't to discount what Prophecy is trying to do, my company just went through a huge re-platforming moving from on on premises to the cloud and it is not easy; any company trying to tackle that space is on the right track. But I just wonder if its overkill for most use cases?
[1] - https://www.youtube.com/watch?v=srtMWzyqdp8 [2] - http://www.cs.utexas.edu/users/ncrooks/2015-eurosys-musketee...