packrat has helped a lot with version control of packages, but it still doesn't quite feel like the right solution.
I've been really impressed in the last 5 years how far R has come in these areas though, so like you I keep coming back. By the time I start getting over the learning curve other places, R seems to have developed better tooling for what I want to accomplish anyway and I can come back and right cleaner, clearer, better software faster in R.
I recently had to loop through 1.3Gb of data (5000 files) and merge just one column from each file into a new dataset. It did so in ~2 hours. Yet the loop was just ~5 lines of code.
I wonder if you tried doing things like:
* preallocate a list, then do.call(cbind, your_data) * Same as above, but with some of the faster alternatives to cbind like dplyr::bind_cols or data.table::cbind * Use data.table, which has far faster joins than base R (so does dplyr) if you were doing a true merge/join
If it was truly just adding a column rom each file together into a file, these kinds of tasks are much better using UNIX tools, in my experience.
Another example is the immutable structure that causes R to be a memory hog. Creating copies of data everywhere. But, again if you plan well and execute the 'best' solutions you can avoid the giant pitfalls but will rarely ever beat a equally well written python equivalent.
The bigger issue is that while R is liked by statisticians it lacks many of the features for the software development. We run across difficulties with logging, version control of packages, speed, size of docker image, build time etc. But, with these drawbacks I keep coming back because I develop faster and better in R.