News for hackers. Stuff that matters.
Re:SQL should be your default choice for data engineering pi... by afpx on Monday January 30, 2023 @11:10AM (#34581658)
Spark is useful in that it’s pretty cheap and flexible. I can grind through 10 TB of data for like $50 and a few hours. If my experiments work, I can use it as a component in other projects. Plus, I can use (slightly non-standard) SQL right in pyspark, if I want to. And, it often performs faster because of the query optimizer.

If your db is scalable enough (like Snowflake) and you have the money, you can use SQL directly in the same way. But, I've seen data analysts writing bad SQL against Snowflake costing thousands of dollars per query.

It’s really tough to future proof anything. At least with pyspark you will have the code. And, there will always be experts out there. Worse case, if it’s important enough, someone will wrap it in an API and use it until something better comes along (see: 50 year old mainframes still in use).

Re:SQL should be your default choice for data engineering pi... by dangwhy on Monday January 30, 2023 @12:35PM (#34583332)
> I can grind through 10 TB of data for like $50

Is this hosted databricks ?

Re:SQL should be your default choice for data engineering pi... by afpx on Monday January 30, 2023 @01:06PM (#34583830)
EMR
Re:SQL should be your default choice for data engineering pi... by berkle4455 on Monday January 30, 2023 @01:17PM (#34584006)
$50 to analyze 10TB over the course of a few hours is an insanely expensive and inefficient setup.
Re:SQL should be your default choice for data engineering pi... by afpx on Monday January 30, 2023 @01:19PM (#34584040)
Not analyze. Transform. I'm not talking about a query.

But, tell me more. I always like a better way.

Is it more efficient to read/write from SSD? Right now, I need everything in memory.

Re:SQL should be your default choice for data engineering pi... by berkle4455 on Monday January 30, 2023 @01:34PM (#34584288)
a single node of clickhouse would likely be sufficient for a fraction of the cost and time. use attached nvme disks (ebs on aws) or a physical local disk.
Re:SQL should be your default choice for data engineering pi... by eskaytwo on Monday January 30, 2023 @03:39PM (#34586036)
Polars would be worth looking into too. $50 and hours seems orders of magnitude too high vig 10tb
Re:SQL should be your default choice for data engineering pi... by rnk on Monday January 30, 2023 @03:51PM (#34586193)
I don't want to pile on, but I do see startup after startup that is focused on improving the cost/perf payoff of pyspark. Pyspark seems to be basically implemented by having the databricks query planner embedded in it, then it generates multiple queries to evaluate it. I think there's nothing inherent in that process that should make it faster than raw sql queries. There's theoretical advantages of maybe looking at the overall giant pyspark query and decomposing it into efficient pieces. Compare that to someone who wrote a series of separate sql queries to accomplish the same thing, and the qp can't take advantage of any reused computation.
Re:SQL should be your default choice for data engineering pi... by threeseed on Monday January 30, 2023 @04:37PM (#34586786)
> I think there's nothing inherent in that process that should make it faster than raw sql queries

You can choose when to cache the data in memory.

And it's the fact that you can write the code in Python, Scala etc instead of SQL.