But... and here's the big but...I almost never actually meet anyone capable of putting all these steps together in SAS these days that actually understands the SAS computation model end to end.
And SAS's strength, a computation model not being limited by memory by default, becomes a performance weakness when everyone reads/writes every step out to disk and programs without understanding all those little intricacies. SAS hasn't helped any of this by trying to move its eco system away from "programmer" to "application users", so now "programmers" can pick up an interpreted language like R with in-memory default vectorised operations and beat SAS.
Course, I'd still recommend places move to python/R these days because of the broader ecosystems, university talent pool, and avoiding the extensive lock in of proprietary software, but I still feel I have to reflexively respond to "R faster than SAS" claims :p
And yes technically SAS is faster than R but part of the equation is how many people can make SAS code faster than R/python. I had maybe, 1-2 people that could write efficient SAS code.
One version we had was a bunch of macro producing hash merge plus the whole how can I do something without having to get out of the data step. Just horrible. Number of characters in a line of code? You forgot your quote somewhere and now you have to run the magic line.
I hope I'm not too emotional when I say I hope SAS disappears from my industry and we embrace less adversarial licensing.
I'm being emotional when I say I have a soft spot for it because of some nostalgia and occasionally dropping in to do some "rock star" programming moments with it. But that's the opposite of what I'd want if/when I was running my own ship.
I too almost always try to steer myself and others away from it now because of the licensing/customer hostility. It's absolutely ridiculous...
I have been trying to help with exactly this (and your breadcrumbs help) but it is tricky for me since I am used to open source/*nix environment where you can use much different tools and also information and tutorials are distributed much more widely.
That combined with a general computer science background and you can start to put the whole thing together.
I'd be lying if I said I hadn't considered writing one, but at my age I'd honestly ask why write one for an old proprietary system and make business for someone else when, if I'll ever go back long term, they can pay me an exorbitant amount as a consultant. Might as well start writing 'the dark arts of COBOL' :p
It uses dplyr and data.table syntax to manipulate data on disk
Why do you believe it will be the future, and what do you see as the barriers to roll-out? I ask as someone who is curious about when/whether to start investing in Julia competence
https://julialang.org/blog/2012/02/why-we-created-julia
I've been playing around with it. As a Python/MATLAB guy, the syntax is very friendly. I can see it displacing Python in production code where you need speed and might avoid some of the heavy Python DS libraries. Overall it seems like a thoughtful combo of a lot of good numerical programming features.
But so far we have seen great development. Flux is a truly beautiful ml library. Being a compiled language remove a lot of headache when building production images. The syntax, the full utf support in variable name. Package management is great. Having that abstraction layer between CPU, GPU so you don't have to rewrite code. Dispatch based on signature, type management. I don't see it going away soon. It took me 13 years to make them transition out of SAS, good thing cloud computing come around and someone realised the clusterfuck of having to manage SAS licence in the cloud.
Right now the only way to do that without significant performance costs is to drop down into C or avoid the problem completely by using Julia.
Having worked with both R and Python on large datasets, I think both languages are really easy until they aren’t. Eventually you hit a performance wall.
I get that R is not for everyone but used correctly it is a beast.
Now this is anecdotal, but we have in the insurance industry what we call on level premium calculators. It is basically a program that will rerate all policies with the current set of rates.
Our current R program can rate 41000 policies a second fully vectorized on a user laptop that has a an i5 from 2015.
In contrast, the previous SAS program could do 231 policies a minute on xeon 64 core processor from 2017.
For our workload and type of work, R has been a godsend.
Bonus, we can put what our data scientist develop in R directly in production. (after peer review, testing, etc, not different than any other production code)
Back when I started in 2005, we modeled in some proprietary software like Emblem, used Excel to build a first draft premium calculator, rebuilt the computation in SAS for the onlevel program and sent specs to IT to rebuilt the program again for production. All three had to produce the same results.
I've tried Python, Go, Rust, Julia. I'd say Python could be a good alternative but speed of data.table, RStudio IDE and ease of package management in R makes R an obvious choice for us. I believe Julia to be the future but so far the adoption rate in house has been low.