News for hackers. Stuff that matters.
Re:Polars by dkga on Monday January 08, 2024 @07:16PM (#38920516)
Actually pandas is not the original open source data frame library, perhaps only in Python. There is a very rich tradition in R on data.frames, which includes the unjustly neglected data.table.
Re:Polars by p4ul on Monday January 08, 2024 @07:38PM (#38920664)
Yep! Unless I'm mistaken, R (and its predecessor S) seems to have been the first to introduce the concept of a dataframe.

One could also argue that dataframes are basically in-memory database tables. And in that case, S and SQL probably tie in terms of the creation timeline.

Re:Polars by ayhanfuat on Monday January 08, 2024 @07:42PM (#38920699)
The difference is dataframes can also be seen like matrices. You can do row operations, row + column operations, multiply rows and columns, multiply different matrices, transpose them etc. These kind of things don't really make sense in DB tables (and they are generally not supported and you jump through hoops to do similar things in DBs).
Re:Polars by p4ul on Monday January 08, 2024 @07:46PM (#38920736)
Yes, that's totally fair; dataframes are more flexible in that sense.
Re:Polars by p4ul on Monday January 08, 2024 @09:57PM (#38921690)
Oh, and another important difference is memory layout. The dataframe implementations mostly (or all) use column-major format. Whereas most conventional SQL implementations use row-major format, I believe.
Re:Polars by theLiminator on Tuesday January 09, 2024 @02:16AM (#38923328)
I think most OLTP databases are row oriented whilst most OLAP are column.
Re:Polars by kkoncevicius on Tuesday January 09, 2024 @03:19AM (#38923655)
> The difference is dataframes can also be seen like matrices. You can do row operations, row + column operations, multiply rows and columns, multiply different matrices, transpose them etc.

I think this is overblowing the similarities to matrices. Matrices have elements all of the same type, while data.frames mix numbers, characters, factors, etc. You certainly cannot transpose a data.frame and still have a data.frame that makes sense. Multiplying rows would not make sense either, since within one row you will have different types of data. Unless you have a data.frame that is all numeric, but in that case one should probably be using a matrix in the first place.

Re:Polars by ayhanfuat on Tuesday January 09, 2024 @04:48AM (#38924196)
> Unless you have a data.frame that is all numeric, but in that case one should probably be using a matrix in the first place.

They still have their advantages with row/column labels, NaN handling etc. These are not operations I am speculating about by the way. I am most familiar with pandas and the dataframe there has transpose, dot product operations and almost all column operations have their correspondence in rows (i.e. you either sum(axis=0) or sum(axis=1)).

Re:Polars by kkoncevicius on Tuesday January 09, 2024 @06:03AM (#38924693)
Oh, based on the comment you replied to I thought this was about R. In R matrices can handle NaNs and NAs, have column and row labels, have dot products and much more.
Re:Polars by xwowsersx on Monday January 08, 2024 @09:13PM (#38921359)
I feel like the predecessor of R should be Q!
Re:Polars by p4ul on Monday January 08, 2024 @09:49PM (#38921650)
The way that I've heard the story, S was short for "statistics", and R was chosen because the authors were _R_obert [Gentleman] and _R_oss [Ihaka].

Statisticians are funny!

Re:Polars by 7thaccount on Monday January 08, 2024 @07:24PM (#38920592)
Yeah. I think Wes McKinney liked the data frames in R, but preferred the programming language of Python. I've heard somewhere that he also got a lot of inspiration from APL.
Re:Polars by Cacti on Monday January 08, 2024 @11:40PM (#38922407)
R is literally designed to do statistics and has first class support and language feature support for many specialized tasks in statistics and closely related fields.

Python is literally designed to be easy to program with in general.

Well, it turns out when you’re dealing with terabytes of data and TFLOPS, the programming becomes more important than the math. Not all R devs are happy about this and they are very loud about it.

But it shouldn’t really surprise anyone. That is literally how those languages are designed.

Most of the R devs I know like this are just butthurt they are paid less and refuse to switch because they’re obstinate, or they’re a little scared they’re being left behind. first group is all over the place, but the second group tends to skew older of course

Re:Polars by disgruntledphd2 on Tuesday January 09, 2024 @05:54AM (#38924638)
> Most of the R devs I know like this are just butthurt they are paid less and refuse to switch because they’re obstinate, or they’re a little scared they’re being left behind. first group is all over the place, but the second group tends to skew older of course

Look, I started with R and use mostly Python these days, but this is not really a fair take.

R is (still) much, much, much better for analytics and graphing (the only decent plotting library in python is a ggplot clone). The big change (and why Python ended up winning) is that integrating R with other tools (like web stuff, for example) is harder than just using Python.

pandas (for instance) is like an unholy clone of the worst features from both R and Python. Polars is pretty rocking, though (mostly because it clones from Spark/dplyr/linc).

It's another example of Python being the second best language for everything winning out in the marketplace.

That being said, if I was starting a data focused company and needed to pick a language, I'd almost certainly build all the DS focused stuff in R as it would be many many times quicker, as long as I didn't need to hire too many people.

Re:Polars by mjhay on Tuesday January 09, 2024 @08:34AM (#38925973)
R is heavily influenced by Scheme. Not only is it heavily functional, but it has metaprogramming capabilities allowing a high level of flexibility and expressiveness. The tidyverse libraries use this heavily to produce very nice composable APIs that aren't really practically possible in Python.

R is fine. The issue is more in the ecosystem (with the aforementioned exception of the tidyverse).

Re:Polars by RamblingCTO on Wednesday January 10, 2024 @11:49AM (#38940561)
Imho forecasting is still way better in R than in Python
Re:Polars by Cacti on Monday January 08, 2024 @11:01PM (#38922144)
My friend. You cannot make people like R. We all know about and study data.table, so it’s not neglected, we just don’t use that implementation.

Mainly because R sucks for anything that isn’t statistics.

Re:Polars by bonadrag on Tuesday January 09, 2024 @08:06AM (#38925709)
> which includes the unjustly neglected data.table

So so true.

I was working on an adhoc project that needed a quick result by the end of the day. I had to pull this series of parquet files and do some quick and dirty analysis. My first reflex was to use python with pandas, quick and easy. Python could not handle the datasets, too large. I decided to give R and data.table a go and it went smoothly. I am usually a python user but from time to time I feel compelled to jump back to R and data.table. Phenomenal tool.

Re:Polars by jgalt212 on Wednesday January 10, 2024 @02:01PM (#38941879)
Indeed.

Python is great for getting things into a data.frame. But once in a data.frame, R is so much easier to work with for all things stats and ml-related.