News for hackers. Stuff that matters.
Re:What SQL Analysts Need to Know About Python (2016) by goatinaboat on Monday November 04, 2019 @01:59PM (#21445080)
I’ve seen devs just run select * from table then filter it and sort it in their own code. Then they complain “the database is slow” when it’s spending all its time shipping gigabytes of data they don’t need to them!
Re:What SQL Analysts Need to Know About Python (2016) by SJetKaran on Monday November 04, 2019 @04:30PM (#21446705)
From what I encountered, this is generally the case when someone is in the "analysis/reports" mode. Rather than get summary statistics on each column, find number of nulls, etc by writing a sql query, they instead get the data into the Python/R instance, and use general purpose functions, utilities, etc. "Programmers are expensive" statement probably applies here as well. I'm not trying to be defensive here, just saying that this might be one reason.
Re:What SQL Analysts Need to Know About Python (2016) by mrbungie on Monday November 04, 2019 @07:58PM (#21448549)
If you believe "Programmers are expensive", then you should do as much as you can do with a declarative data manipulation language (usually SQL, you can also consider sequences of text manipulations tools using pipes) and leave that last 15-5% of high-value work to a more powerful but also verbose imperative lenguage (usually Python, but any).

Asking for what you want is considerably faster than saying how you want it done.

Re:What SQL Analysts Need to Know About Python (2016) by goatinaboat on Tuesday November 05, 2019 @01:46AM (#21450248)
From what I encountered, this is generally the case when someone is in the "analysis/reports" mode

I understand this use case, but this is in actual application code!

Re:What SQL Analysts Need to Know About Python (2016) by starpilot on Monday November 04, 2019 @02:31PM (#21445453)
Is anyone working on a translator for pandas dataframe syntax to SQL?
Re:What SQL Analysts Need to Know About Python (2016) by alexhutcheson on Monday November 04, 2019 @02:48PM (#21445671)
In the R world, dbplyr[1] does this amazingly well.

[1] https://dbplyr.tidyverse.org/

Re:What SQL Analysts Need to Know About Python (2016) by airstrike on Monday November 04, 2019 @08:26PM (#21448726)
tidyverse is just an mind-boggingly amazing ecosystem of packages

Forget Da Vinci, the first man to be cloned should be Hadley Wickham

Re:What SQL Analysts Need to Know About Python (2016) by faizshah on Monday November 04, 2019 @04:45PM (#21446852)
Re:What SQL Analysts Need to Know About Python (2016) by starpilot on Monday November 04, 2019 @08:56PM (#21448898)
Dang, Ibis doesn't support Redshift or SQL Server. I'm also having trouble understanding what it really is - it's an entire framework for big data it seems and not just a translator. What I'd really like is just that, something that turns pandas dataframe operation into ANSI SQL. So input pandas2sql('tablename["col"]') -> "select col from tablename". Something really simple to use.
Re:What SQL Analysts Need to Know About Python (2016) by goatinaboat on Tuesday November 05, 2019 @01:48AM (#21450257)
Pandas has from_sql and to_sql methods that are compatible with SQLAlchemy if you insist on using an ORM, that gets you most of the way there...
Re:What SQL Analysts Need to Know About Python (2016) by philipov on Wednesday November 06, 2019 @03:25AM (#21460651)
SQLAlchemy is more than just an ORM. It also has sql expression language, for writing queries using python without using any ORM features.
Re:What SQL Analysts Need to Know About Python (2016) by medecau on Monday November 04, 2019 @04:25PM (#21446666)
Re:What SQL Analysts Need to Know About Python (2016) by shrikant on Monday November 04, 2019 @05:48PM (#21447496)
Re:What SQL Analysts Need to Know About Python (2016) by pojzon on Monday November 04, 2019 @03:06PM (#21445871)
Im surprised ppl dont use ORM libs for this instead..
Re:What SQL Analysts Need to Know About Python (2016) by bshipp on Monday November 04, 2019 @03:36PM (#21446214)
ORMs are just as capable as any developer of generating bad SQL queries that crush databases and plug network connections.
Re:What SQL Analysts Need to Know About Python (2016) by pojzon on Tuesday November 05, 2019 @02:59AM (#21450505)
Yes, but in the example used, all orm libs I know would fetch only a single column or two (often for a single row). While "select *" would fetch everything.