News for hackers. Stuff that matters.
Re:Why your F# evangelism isn't working by throwaway2037 on Wednesday May 31, 2023 @04:18AM (#36135826)
Thank you to reply. Your definition of data engineer makes sense. From my experience, I would not call it a new role. People were doing similar things 25 years ago when building the first generation of "data warehouses". (Remember that term from the late 1990s!?)

I am surprised that you are using Python for data transformation. Isn't it too slow for huge data sets? (If you are using C/C++ libraries like Pandas/NumPy, then ignore this question.) When I have huge amounts of data, I always want to use something like C/C++/Rust/C#/Java do the heavy lifting because it is so much faster than Python.

Re:Why your F# evangelism isn't working by Hasnep on Thursday June 01, 2023 @02:46AM (#36148225)
Yes, it's definitely a new word for an old concept, same as the term data scientist for data analyst or statistician.

I find Python is fast enough for small to medium datasets. I've normally worked with data that needs to be loaded each morning or sometimes hourly, so whether the transformation takes 1 minute or 10 minutes it doesn't matter. The better way is of course to dump the data into a data warehouse as soon as possible and then use SQL for everything, so I only use Python for things that SQL isn't suited for, like making HTTP requests.