News for hackers. Stuff that matters.

A comment on Command-line Tools can be 235x Faster than your Hadoop Cluster (2014). Parent comment

Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by bpodgursky on Tuesday March 08, 2022 @12:31AM (#30596756)
You can write some SQL in Spark, but

1) Why would you want to maintain your own Spark infrastructure? Spark on Kube is a huge improvement over YARN but you still have to deal with OOMEs, filled disks, Kube upgrades, pushing custom images to container registries, etc etc etc.

2) Snowflake is probably 10-50x as performant as Spark for data manipulation. I don't know what kind of unholy demonic incantations Snowflake is doing on the backend to support their SQL performance, but it's really freaking fast. There's just no other way to cut it.

I've spent 5-10 years eking every ounce of performance I can get out of a Hadoop/Spark cluster. I'm not trying to be unreasonable about this. I would love for OSS to be competitive; it's great for the world, and it would be great for my skill set and earning potential.

But it's not a contest, and if you think standalone Spark is going to be a viable competitor in a couple years, you are deluding yourself. Make informed choices about your career and investment.

[ Parent | on HN ]
Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by Foobar8568 on Tuesday March 08, 2022 @02:10AM (#30597232)
A lot of articles I read about snowflake involves data vault which is a massive turn off. And when their tech lead (Kent Graziano) is a prominent figure in the DV bullshit...
[ Parent | on HN ]
Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by fritkot on Tuesday March 08, 2022 @02:56AM (#30597554)
Snowflake and DV have no interdependency whatsoever. Snowflake is just a database. Whether you use DV to model the data inside of it or dimensional modelling or "big wide tables" is completely up to you, there's nothing about it that requires or benefits DV in particular.
[ Parent | on HN ]
Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by secondcoming on Tuesday March 08, 2022 @04:07AM (#30598016)
What is DV?
[ Parent | on HN ]
Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by belter on Tuesday March 08, 2022 @06:45AM (#30598874)
Count not knowing what it is, as a blessing. Run if you can.

https://en.wikipedia.org/wiki/Data_vault_modeling

Edit: As the Wikipedia article has no Criticism section I will add some references:

http://kejser.org/the-data-vault-vs-kimball-round-2/

https://timi.eu/blog/data-vaulting-from-a-bad-idea-to-ineffi...

[ Parent | on HN ]
Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by fritkot on Tuesday March 08, 2022 @04:24AM (#30598101)
It's a data modelling method for data warehouses, it can be used in Snowflake or on any other data management platform.
[ Parent | on HN ]
Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by mohanmcgeek on Thursday March 10, 2022 @08:57PM (#30635241)
> Snowflake is probably 10-50x as performant as Spark for data manipulation

Wow is this for a fact? I haven't used either in a while but I saw the blog post from databricks and Spark was more performant than snowflake.

I assumed that's what I'll also get when i run spark on kube

[ Parent | on HN ]
Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by rxin on Tuesday March 08, 2022 @12:43AM (#30596822)
You should try Databricks, especially the new Photon engine powering Spark. In general more performant than Snowflake in SQL and a lot more flexible. (There are some cases in which Databricks would be slower but the perf is improving rapidly.)
[ Parent | on HN ]
Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by belter on Tuesday March 08, 2022 @03:47AM (#30597906)
Probably an oversight on your part, but I would argue would be elegant to disclose you are one of the co-founders.
[ Parent | on HN ]
Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by Fiahil on Tuesday March 08, 2022 @01:01AM (#30596897)
Databricks has an extremely bad API. So, sure, your Spark jobs might be a little bit faster some times, but why would you use it if you can't even read logs of running jobs?
[ Parent | on HN ]
Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by Lucasoato on Tuesday March 08, 2022 @06:03AM (#30598614)
Databricks is amazing, the Delta Live Table technology is incredible. It's very hard to approach problems like Data Lineage and Data Quality, but that platform does it in the right way.

My only concern is that they offer just a managed cloud product. That's cool for startups, but large enterprises sometimes need more governance and ownership than that.

[ Parent | on HN ]
Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by soulbadguy on Tuesday March 08, 2022 @01:04AM (#30596917)
Very surprised by this. Do you have a reference ?
[ Parent | on HN ]
Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by fnord123 on Tuesday March 08, 2022 @02:30AM (#30597360)
Fyi, rxin is co-founder of databricks.
[ Parent | on HN ]
Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by soulbadguy on Tuesday March 08, 2022 @11:37AM (#30601901)
That explains it
[ Parent | on HN ]
Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by StreamBright on Tuesday March 08, 2022 @02:56AM (#30597563)
The biggest selling point of Snowflake for most of the customers is that they do not need to maintain the infrastructure.
[ Parent | on HN ]
Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by fritkot on Tuesday March 08, 2022 @02:56AM (#30597567)
Of course you would say that it's more performant and flexible ...TCP-DS was just a PR ploy
[ Parent | on HN ]