Then the BI people (who had never written a line of code except maybe some SQL) got jealous, started calling themselves "data engineers", broke everything ("what do you mean 'we need tests???'"), and ruined the title.
Back then his bank started building a data science/engineering dept, and he jumped ship, and essentially did the same work for 2x the salary. Let me tell you, understanding Black-Scholes and what the data actually means and how to source it, as well as what finance people actually are looking for is a skill that's 1000x as difficult to get as learning R/Python/Pandas etc.
I think the specialisms are really helpful for hiring - data engineering and data science involve a lot of different things so it's a lot easier to hire someone if you don't require skill in both.
That said, I think we have a culture problem in defining our roles narrowly, I've seen data scientists look down at data-pipeline parts of their job, and ignoring model latency and deployment; data engineers be deeply uncurious around what happens to the data after they put it in a warehouse; etc.
The best people I've worked with, within data and elsewhere, are the ones who are curious about every aspect of the stack, even if they mostly focus on one side.
Me too. There were no data engineers when I was a baby data scientist (15 years ago now).
I think that this was a terrible idea (the splitting out of roles). I remember once being asked in an interview if I was able to pull my own SQL or would I need an analyst. That kind of split always struck me as absurd, given that the only way to build a decent model is to know the data very well first..
> That said, I think we have a culture problem in defining our roles narrowly, I've seen data scientists look down at data-pipeline parts of their job, and ignoring model latency and deployment; data engineers be deeply uncurious around what happens to the data after they put it in a warehouse; etc.
I honestly feel like this is driven by the titles. I will note that most of the unsuccessful/bad data scientists I've met were the ones who only cared about the models. As I always said to people, I can run sklearn in a loop over model families if that's all you bring.
> The best people I've worked with, within data and elsewhere, are the ones who are curious about every aspect of the stack, even if they mostly focus on one side.
This is probably just a good signal in tech in general, some curiosity is really required to be successful in this industry.
15, 20 years ago, me and my T-shaped skills could get by just fine with a database server, a NAS, and some frank conversations about priorities.
Nowadays it’s not enough to to know how to work with databases and files. There’s also all these platforms: databricks, snowflake, new relic, sentry, datadog, GitHub actions, prefect, dbt, kubernetes, cassandra, elastic, and on and on and on. Each brings a new set of idiosyncrasies, its own terms for things, its own enterprise integration headaches. It’s just a hell of a lot of knowledge to as one person to cram into their head on top of whatever skills and domain expertise form the actual core of their job.
That makes sense for an enterprise. Enterprises have a lot of bodies to throw at the problem, so they can have an expert for each tool who’s available to support everyone else. And they like to have support contracts and SLAs, which you can get from à B2B vendor but not from the KISS principle. And they don’t mind great heaping piles of unnecessary and mutually conflicting abstractions because that sort of thing is just SOP for a large bureaucracy.
But if you want - need - a system that one person with T-shaped skills can reasonably fit into their head, this is not the way.
Every era views itself as an intricate snowflake, but they rarely are.
And I do think it’s more about how you build things than it is about the calendar. The calendar is just a proxy for what’s fashionable. For example, the startup I currently work at is in the process of incrementally simplifying our stack. Because we agreed that it was getting difficult to understand and expensive to operate. We don’t view this as turning the clock back to 2006; the discussions have framed it as a judicious application of the KISS principle. And yes, it has helped us break down knowledge silos and enable team members - especially data scientists - to be more self-sufficient.
30 min livebug fix and review for me on a job I specced out? Ticket to another team to review and schedule for next sprint and thus 2 weeks of metadata likely lost or malformed.
These setups look great (but full of invisible friction) when things are going well but shit the bed as soon as you come across the simplest of human errors that needs a solution.
And yet if a data person were to try the same on your specialization, surely this would also be well and good.
An analyst then loads this up with whatever visualization software or 3rd party BI tools and interprets the findings and I've found this varies from giving a simple presentation after AI tools do everything for them to doing full on R Studio analytics such as these models https://www.stat.cmu.edu/~cshalizi/ADAfaEPoV/
If the analyst then selects a feature set from the data and performs machine learning I've found they are called 'Data Scientists' otherwise they are analysts but these titles mean something different everywhere.
Stakeholders prefer a single point of dev contact so you can never get rid of the role, and you can only repeat that for so many domains or solutions before you run out of hours in the day. Trying to spread this ownership across a team results into either horizontal or vertical specialization which are the two dysfunctions. I'm not sure why this is like this in data.
I get that some specialisation is useful, but I would like to see more T-shaped data folks.