> All of these languages have libraries that produce the same results, the difficulty is mangling the data into the correct input format. Python's list comprehensions are much, much easier to use than MATLAB matrices, R's data frames, Java's ArrayLists, etc.

At the end of the day, for machine learning applications, your data is in a tabular format. (in Python, a pandas data frame) Yes, Python has a few tricks like list comprehensions for speeding up data processing into that analyzable form. R has a few tricks for processing tabular data as well. (e.g. dplyr).

There are tradeoffs and the skill is finding which works best. Using a single programming language is a bad philosophy even for non-statistical developers.

Only learning a single language is a bad idea, but using a single one, it it fits the problem is a good one.

The even bigger advantage of python is everything else: web scraping, interfacing with weird APIs, hitting the database, consuming and emitting obscure formats, parsing text, calling operating system services, and a million other things. That and much much better abstractions for building larger systems out of reusable components and much better tooling for serious software engineering.

R or Matlab can be fine at exploratory data analysis, making charts, running miscellaneous bits of non-programmer-grad-student code found on the internet, etc., but as soon as you want to do anything other than data analysis, they quickly become annoying.

If you haven't looked at the R ecosystem in awhile, there is a package for each of those use cases. (scraping/API: rvest; database: dplyr; parsing text: stringr, etc).

Yes, Hadley Wickham is primarily responsible for the popularity of R.

We’re talking about an order of magnitude difference in number of packages (82096 on PyPI vs. 8551 on CRAN) and their maturity, and such a naïve metric probably undersells the difference in variety of use cases.

If you picked 100 random production python projects out of a hat, no more than a small handful of them would be remotely appropriate to build using R.

And that's entirely fine. R is great at being a quick and dirty statistical analysis language for people doing data exploration.

I suspect that the utility of more packages increases only logarithmically. Having 10x more packages doesn't mean it's 10 times more useful. If your obscure need isn't in the first 8,000 packages, it probably won't be in the next 80,000. That's just how power laws work. And any common task you can think of will probably be in the top 8,000.

I totally buy that argument, but there's probably not a huge overlap in the set of "common tasks" for R and python.

Pandas for Python is almost re-imaging of R (Base 0 instead of base 1 UGH) very closely and the "common task" match closely, but I still use R over Python.

Well then all the more reason to use R, if those are the set of tasks you want to do.

how about quick and friendly?

Wickham has done a ton but but R had great popularity even before those projects, especially with projects like bioconductor.

Dplyr is not for databases at all. Its a piping operator that simplifies complex syntax a lot.

Yes, you can use dplyr to connect and query data from a database. https://cran.r-project.org/web/packages/dplyr/vignettes/data...

Yes and no - the dplyr example code heavily used pipes but that's really a feature of Magrittr package. You can used dplyr without pipes.

Plus dplyr essentially allows you to treat tables in a database as if they are data.frames.

I think R's data frames is by far the most intuitive abstraction. There's a reason why it was duplicated in Python, Spark or Julia. And R has a lot of utilities for converting input strings, dates, JSON etc.

data.table and dplyr take data frames up a notch!

https://github.com/Rdatatable/data.table/wiki/Getting-starte...

https://cran.rstudio.com/web/packages/dplyr/vignettes/introd...

xts is quite useful as well, when you need to do modeling/analysis in an especially time-oriented manner:

http://joshuaulrich.github.io/xts/

I'm glad I read this comment. After checking some of the docs I think I will have a go at Python for data wrangling. List comprehensions look... friendly.

R still rules for plotting and running canned statistical procedures but sometimes I feel like if I stop programming R for a week I forget how to use it effectively... E.g. Forgetting to add stringsAsFactor=FALSE to everything, forgetting rbind() can overwrite column names, forgetting I have to define my own string concatenation operator in every script.

If Python can save me some of the frustration involved in manipulating data frames that will be nice.

I'd recommend reading http://r4ds.had.co.nz for an R workflow that eliminates a lot of those pain points.

(Except for infix string concatenation - I've never really understood why people prefer that to paste(). Maybe if you're not thinking in vectors?)

A reply from the man himself! Thanks for the link. I'll have a go.

I do like the look of the dplyr library a lot. Combining functions like select and group_by with the pipe operators creates code that is reminiscent of SQL- very nice for readability.

You're going to love Spark if you haven't tried it yet.

Another powerful tool I will keep in mind!

I think this thread illustrates a kind of tension between those coming from an IT/big-data/web oriented background and the more traditional statistics/science/engineering side.

The IT side bring a lot of very powerful and scalable tools to the table. However there are aspects of traditional work which I suspect are lost on some big-data people.

For example, in my line of work (physical asset mgmt) we deal with a lot of very small datasets, very poor quality datasets (e.g. some guy's favourite spreadsheet) and also cultural issues (some engineers are inherently averse to changing systems, and spending decisions are inherently political). In this situation, there is a limit to the benefit of more powerful/scalable tools, and it is advantageous to use tools which are considered high quality and vetted by the community.

R is in a good position here as it has the pedigree of being accepted by the academic stats community, as well as actually being a great tool.

dplyr or data.frame is THE DATA MANIPULATION library just like ggplot2 is THE PLOTTING program for all other languages.

I'm primarily a python user and agree with most of your post but I do find myself going back to R for many of the more esoteric statistical methods. Ie if i want a specific sort of penalized regression it may not yet be implemented in python.

List comprehension is nice, but Hadley's packages for R blow Python out of the water when it comes to data wrangling.

Personally, I think anyone who works with data extensively should be familiar with both Python and R.

I use both Python and R for analytics projects. Python falls quite short on statistics and time series, especially if one is doing exotic/advanced stuff.

By the end of the day I just want to get my job done, and I select the best tool for the job. I have gained a lot by working with both languages.

On a sidenote, one cannot talk about the success of R without mentioning RStudio (amazing IDE for working with data)

R has the best plotting library (ggplot) by far.

Python the second best option for just about everything.

R has a wonderful ecosystem that does amazing data manipulation (dplyr for example)

Python is a good choice but it isn't "clearly better."

I know few about other languages. But as the data size growing, I wonder to know if Python has a disadvantage on efficiency over others because it's interpreted?

Can you give some concrete examples of the benefits of wrangling in python over R and a general sense of how much time this would save a user?

I really would be interested because I started in Python and R and now use R exclusively due to manipulation. R (dplyr) really is the best manipulation system have ever used. Perhaps people try to do loops in R and other non-R like ways and give up. R is fundamentally a Functional Language and people try to band a square into a round hole and give up?

Julia is great too for this kind of work

Whats the status of matpootlib nowadays? Graphs are a major major attractir/detractor

If I use python I have to write half of the algorithms I use myself. Worse, I have to write a gazillion helper functions nyself. It just won't do

I think you got downvoted because people didn't know what you meant. You're right though. Although R is laughably inferior to python as a programming language, it is vastly more work to try to do statistical data analysis in python than in R. I recommend using both languages and using csv or whatever format to exchange data sets.

> using csv or whatever format to exchange data sets

Have you used Pandas in the past year or two? I'm curious why you would exchange data sets using csv or another format between Python and R when you could easily call the R function of interest from within Pandas (using rpy2) and not even worry about data interchange.

It's definitely not vastly more work to do statistical analysis using Python Pandas than in R anymore, perhaps it was several years ago.

EDIT: And carlmcqueen mentioned feathers in response, which is a collaboration between the developers of Pandas and R to create interoperable on-disk data frames for both languages. Point being, between rpy2 and feathers (and probably other projects), you definitely don't need to use intermediary csv files anymore to move data back and forth between R and Python.

> It's definitely not vastly more work to do statistical analysis using Python Pandas than in R anymore, perhaps it was several years ago.

Haha, what? What statistics can you do in pandas? You can do some statistics in python by cobbling together stuff from scipy and statsmodels (maybe I'm out of date, is there more?). I see a few modules for regression and stuff in pandas but they are marked as deprecated. I think perhaps you and I mean different things by "statistics". R provides a vast ecosystem covering, for example

- Gold standard implementations of simulation, PDFs, quantiles of any probability distribution you can mention (in python you can find some of this in scipy; not pandas. But scipy is a real mess compared to R and not as comprehensive.)

- Gold standard implementations of any classical hypothesis test you can mention

- Gold standard implementations of computational methods for fitting generalized linear models, mixed models, frameworks for MCMC samplers, graphical models, HMMs, and a vast amount of other stuff I'm not clever enough to name right now let alone understand.

Really any statistical procedure -- whether "classical" or "modern"/"computational statistics" -- in R you will find it, and furthermore it will be basically the reference implementation / gold standard.

That's not mentioning the plotting tools and the numerical computing and clean linear algebra syntax. But that's it, no more: the people who go further and suggest using R for building a web server or web scraping or something mostly haven't used real programming languages.

You're missing the point. The python ecosystem can't compete with R on the statistics front -- it would be crazy to try. That's certainly not the aim of pandas.

> you definitely don't need to use intermediary csv files anymore to move data back and forth between R and Python.

Perhaps not, but doesn't it please you to have a well-defined interface (a serialization format) between the two languages? I haven't tried Rpy2 for years. I don't like to have two different languages get their tentacles into each other like that if I can avoid it, but I'm sure it's a good project which has its use cases.

EDIT: thanks, I hadn't seen feather. That looks like the thing to use.

feather is a great way to move data between the two as it is blazingly fast and native in python and R.

With pandas, this is no longer true.

How pandas solves statistical problems or the thousand utility functions that are missing? Please, explain.

Have you even looked at pandas? I get the impression you haven't. Pandas has most of the statistical utility functions that R does, and for those few that it lacks, Python/Pandas also has available an easy-to-use FFI interface to R via rpy2.

Take a look: http://pandas.pydata.org/pandas-docs/stable/

EDIT: Specified name of interface (rpy2).

Have you even looked at statistics? I get the impression you haven't. Pandas has aggregation and that's it. I don't really like to start those language wars, but what the heck. Your precious pandas is actually worse than the R alternatives in every way. It is slower than data.table while having comparable syntax. Dplyr is comparable in speed while having WAY better syntax.

> Pandas has most of the statistical utility functions that R does

Pandas is not shipping with most of the stuff you can get on CRAN.

I have heard of people that could not get the same results in Python as they get in R.

Have you looked at R's source? Some common functions contain dozens of unsourced magic numbers (and the comments indicate they have been modified over time).