Re:R Passes SAS, but Python Leaves Them Both Behind
by rm999 on Sunday March 05, 2017 @03:50PM
(#13798524)
This is a great point. Python has been a solid data engineer/pipeline language for years now. It makes it easy for data scientists and data engineers (or people who wear both hats) to work in the same environment and codebase.
Re:R Passes SAS, but Python Leaves Them Both Behind
by est on Monday March 06, 2017 @12:11AM
(#13800832)
The only area of un-touched by Python is web frontends and mobile apps.
Hope Web assembly would change that.
Re:R Passes SAS, but Python Leaves Them Both Behind
by thearn4 on Sunday March 05, 2017 @05:22PM
(#13799161)
Off-topic, but what is a "data engineer"?
Re:R Passes SAS, but Python Leaves Them Both Behind
by nl on Sunday March 05, 2017 @11:47PM
(#13800763)
In my team, "data engineer" refers to someone who deals with the ETL process, but also some feature engineering (ie, designing and extracting features. There's a lot of cross over here with the work Data Scientists do, but often the tools are different: for example, in the Python world the data engineer will use Pycharms while the data scientist will use Jupyter.
Re:R Passes SAS, but Python Leaves Them Both Behind
by nerdponx on Sunday March 05, 2017 @05:34PM
(#13799237)
I'm sure someone can come along with a better description, but it's kinda like devops/sysadmin but specifically for data storage and access.
You could also look it up...
Re:R Passes SAS, but Python Leaves Them Both Behind
by sdenton4 on Sunday March 05, 2017 @11:26PM
(#13800696)
At some point, people realized that 90% of data science work was building pipes and keeping them clean. And then data engineering was invented so that the data scientists wouldn't have to get their hands as dirty. :)
Re:R Passes SAS, but Python Leaves Them Both Behind
by fnord123 on Monday March 06, 2017 @04:55AM
(#13801695)
A snappier version which gives the flavour: Data scientists use {tensor flow, scikit-learn, parquet, hadoop, spark, scrapers, etc}. Data engineers write {tensor flow, scikit-learn, parquet, hadoop, spark, scrapers, etc}.
Re:R Passes SAS, but Python Leaves Them Both Behind
by xapata on Sunday March 05, 2017 @09:24PM
(#13800274)
I'd include database engineer in that bin. We used to call people DBAs but that seems a bit limiting now.
Re:R Passes SAS, but Python Leaves Them Both Behind
by mehaveaccount on Sunday March 05, 2017 @06:49PM
(#13799646)
That's a good description for a data engineer from my end.
We've had a lot of success weaning people of R and MATLAB in finance (so replace "data scientist" with "quant"). Of course if you work in a field that doesn't have the libraries and can't build them in house then your mileage will certainly vary.