News for hackers. Stuff that matters.
Re:SQL should be your default choice for data engineering pi... by simonw on Monday January 30, 2023 @09:34AM (#34580024)
I worry that if I write significant amounts of code in pyspark it will have fallen out of fashion in ten year's time and someone will end up either rewriting it or running an ancient, unmaintained version of pyspark forever just to keep my stuff working.

I'm much less worried about that happening to SQL scripts.

Re:SQL should be your default choice for data engineering pi... by mritchie712 on Monday January 30, 2023 @10:32AM (#34580964)
In addition to being good at what it does, SQL has Lindy on it's side like no other language used today.
Re:SQL should be your default choice for data engineering pi... by rnk on Monday January 30, 2023 @10:50AM (#34581279)
Lindy?
Re:SQL should be your default choice for data engineering pi... by mritchie712 on Monday January 30, 2023 @10:52AM (#34581310)
Re:SQL should be your default choice for data engineering pi... by rnk on Monday January 30, 2023 @03:45PM (#34586115)
Thanks, that's an idea that makes a certain sense. The counter point is that technical things get replaced quickly at times when something vastly superior arrives. We don't use cpm computers or apple IIs any more. I think saying sql has the lindy affect is a clever slam against it. We don't use system 360 assembler any more. X86 seemed to be certain to own the world, maybe arm or risc-v will usurp it.
Re:SQL should be your default choice for data engineering pi... by rnk on Monday January 30, 2023 @10:50AM (#34581273)
PySpark is such a performance killer. How many startups will there be that just have a goal to improve performance and reduce cost. It locks you into a small number of platforms (bc many dbs don't have an impl and details vary), every database vendor eventually struggles to make their own implementation.
Re:SQL should be your default choice for data engineering pi... by afpx on Monday January 30, 2023 @11:10AM (#34581658)
Spark is useful in that it’s pretty cheap and flexible. I can grind through 10 TB of data for like $50 and a few hours. If my experiments work, I can use it as a component in other projects. Plus, I can use (slightly non-standard) SQL right in pyspark, if I want to. And, it often performs faster because of the query optimizer.

If your db is scalable enough (like Snowflake) and you have the money, you can use SQL directly in the same way. But, I've seen data analysts writing bad SQL against Snowflake costing thousands of dollars per query.

It’s really tough to future proof anything. At least with pyspark you will have the code. And, there will always be experts out there. Worse case, if it’s important enough, someone will wrap it in an API and use it until something better comes along (see: 50 year old mainframes still in use).

Re:SQL should be your default choice for data engineering pi... by dangwhy on Monday January 30, 2023 @12:35PM (#34583332)
> I can grind through 10 TB of data for like $50

Is this hosted databricks ?

Re:SQL should be your default choice for data engineering pi... by afpx on Monday January 30, 2023 @01:06PM (#34583830)
EMR
Re:SQL should be your default choice for data engineering pi... by berkle4455 on Monday January 30, 2023 @01:17PM (#34584006)
$50 to analyze 10TB over the course of a few hours is an insanely expensive and inefficient setup.
Re:SQL should be your default choice for data engineering pi... by afpx on Monday January 30, 2023 @01:19PM (#34584040)
Not analyze. Transform. I'm not talking about a query.

But, tell me more. I always like a better way.

Is it more efficient to read/write from SSD? Right now, I need everything in memory.

Re:SQL should be your default choice for data engineering pi... by berkle4455 on Monday January 30, 2023 @01:34PM (#34584288)
a single node of clickhouse would likely be sufficient for a fraction of the cost and time. use attached nvme disks (ebs on aws) or a physical local disk.
Re:SQL should be your default choice for data engineering pi... by eskaytwo on Monday January 30, 2023 @03:39PM (#34586036)
Polars would be worth looking into too. $50 and hours seems orders of magnitude too high vig 10tb
Re:SQL should be your default choice for data engineering pi... by rnk on Monday January 30, 2023 @03:51PM (#34586193)
I don't want to pile on, but I do see startup after startup that is focused on improving the cost/perf payoff of pyspark. Pyspark seems to be basically implemented by having the databricks query planner embedded in it, then it generates multiple queries to evaluate it. I think there's nothing inherent in that process that should make it faster than raw sql queries. There's theoretical advantages of maybe looking at the overall giant pyspark query and decomposing it into efficient pieces. Compare that to someone who wrote a series of separate sql queries to accomplish the same thing, and the qp can't take advantage of any reused computation.
Re:SQL should be your default choice for data engineering pi... by threeseed on Monday January 30, 2023 @04:37PM (#34586786)
> I think there's nothing inherent in that process that should make it faster than raw sql queries

You can choose when to cache the data in memory.

And it's the fact that you can write the code in Python, Scala etc instead of SQL.

Re:SQL should be your default choice for data engineering pi... by btilly on Monday January 30, 2023 @11:37AM (#34582235)
About 20 years ago I wrote a bunch of complex reports using SQL. The engineer who came after me tried briefly to understand it, decided it all needed to be rewritten in "a real language", broke all the reports, and they never really got them back.

So merely writing in a language that we believe will be around forever does not prevent your code from being rewritten.

Re:SQL should be your default choice for data engineering pi... by remus on Monday January 30, 2023 @12:10PM (#34582865)
I think that's always a risk (the old "which idiot wrote this? clearly needs a full rewrite!" followed by the rewriter rediscovering all the bugs the original had fixed), but with a well established language like SQL hopefully the risk is lower.
Re:SQL should be your default choice for data engineering pi... by btilly on Monday January 30, 2023 @02:36PM (#34585209)
The fact is that, word for word, SQL is incredibly more efficient for reporting than most programming languages. But since few programmers treat it as a real language, it tends to be written without formatting. Which makes it hard to read.

Also unfamiliarity will make SQL feel inefficient for non-SQL programmers. The experience of rewriting SQL teaches some this lesson. But others find that the familiarity of the language of the rewrite makes it seem better to them, even though it is objectively worse. An objectively worse that only becomes visible when someone needs to optimize it for performance reasons. And it is seldom the programmer who did the inefficient write who has the skills to optimize it back to what SQL did in the first place!

(Yeah, I've had to do those optimization fixes.)

Re:SQL should be your default choice for data engineering pi... by xupybd on Monday January 30, 2023 @03:06PM (#34585620)
Many developers, good developers haven't worked with complex SQL queries.

So on seeing one they can sometimes think it's quicker to redo this in a language I know rather than upskill enough to understand this.

Re:SQL should be your default choice for data engineering pi... by rvanlaar on Monday January 30, 2023 @04:25PM (#34586598)
While I agree with the sentiment, in practice I haven't seen it go over well. I find the tooling around SQL to be severely lacking.