As a working data scientist I'd say it's completely the opposite: a good tabular data manipulation package is the single most valuable tool in my tool box. And R's packages (either data.table or dplyr) are definitely way better than pandas. There's no comparison.

I would be hard-pressed to find a working data scientist whose definition of data science is "that thing you do with sklearn, Deep Learning and Numpy".

Tabular data is great for many usecases, but saying that image, audio, and video analysis is not data science seems like a weird variant of gatekeeping to me.

> Tabular data is great for many usecases, but saying that image, audio, and video analysis is not data science seems like a weird variant of gatekeeping to me.

Most problems are mostly tabular, IME.

I completely agree that text, images and video are much, much better handled by Python (that's why I use and know both).

Of course, my post doesn't imply such a silly statement.

> "Data science is that thing where you do sklearn, Deep Learning and Numpy" is not a working data scientist's perspective.

It could be. It's such a broad job title and it looks so different across different companies and teams that the main tool for one data scientist might be something that another data scientist never has to touch. Different data science jobs prioritise different tools, that's all.

Right, so defining data science as 90% sklearn+DL+numpy is just as silly as saying that it's 90% table manipulation. That's exactly my point.

Still, if anyone here has managed to find a data science job in which tabular data management is not a sizable piece of what you do, I'd like to know some details!

I imagine there are data scientists who operate primarily on unstructured rather than tabular data. Part of my current job involves stuff like text classification, and it's not that difficult to imagine someone for whom that's a more sizable proportion of their day-to-day.

Still, my suspicion -- at least from my corner of data science -- is that such individuals are rare, and that most data scientists do make use of tabular data more often than not.

I totally get what you mean - I would suspect that when you work with unstructured data, tabular data manipulation is maybe 20-40% of what you do, and when you work with structured data, it's more like 60-80%.

I worked as a datascientist for a couple of years and tabular data was a very small part of my job. I spent far more time with image-analysis and JSON, both of which I found R sucks at.

maybe we are casualties of the vague definition of "data science," but in my experience numpy is too low-level for most of what I consider DS, and pandas/matplotlib are _much_ more central than sklearn or pytorch. Even if your definition only encompasses deep learning research, surely plotting is still indispensable?

I'll also add my vote for the superiority of data.table and ggplot2 to any Python alternatives. the bloat and verbosity of pandas is a daily struggle

Just curious. In which way is data.table superior to pandas? Really interested about it! From my personal experience pandas is just sometimes a bit slow.

I'm more a dplyr man myself, but data.table is much faster than pandas, most noticeably IMO when reading large files. It's also extremely succinct if you're into that sort of thing (though I find it a bit obfuscated). pandas is a lot of things, but "fast" and "concise" are not two of them.

Got it. Regarding fast you have something like Vaex on python side (but not sure how fast it realy is). For me I had with pandas the most issues using it's multiindex.

> For me I had with pandas the most issues using it's multiindex.

Yessss. I loathe indices, and have never been in a situation where I was better off with them than without them.

> Regarding fast you have something like Vaex on python sid

I've never used Vaex, but I've used datatable (https://github.com/h2oai/datatable) and polars (https://github.com/pola-rs/polars). Polars is my favorite API, but datatable was faster at reading data (Polars was faster in execution). I'll have to give Vaex a try at some point.

Pandas is the PHP of data science. Pretty badly designed, but immensely popular because it got there first and had no real competition (in Python) for years.

I just love how much more terse and fast it is, someone else linked a benchmark below. There's definitely a learning curve though.

If you already think pandas is slow I think you'll be surprised how much more strongly you feel after using data.table!

Data.table is Faster to write and faster to perform

https://h2oai.github.io/db-benchmark/

> If you're doing data science aren't sklearn, DL, and numpy getting you 90% of the way there anyway?

Not really, tbh. Most of my jobs (even when the primary output was models) require spending a _lot_ of time data wrangling and plotting. R is much, much better for this kind of exploratory work.

But if I need to integrate with bigger systems (as I normally do), there's a stronger push for Python to reduce complexity and make it easier for SE's to understand and maintain (some of) the code.