Dollars to donuts, this stuff always seems to be in that enterprise buzzword laden minefield of trendy make-work. There seems to be a tremendous amount of money and energy behind it for some reason, though.

I've seen some customers of ours being quoted six and seven figure prices by their internal data science teams to essentially slurp and transform a handful of SQL tables. It's the kind of thing that shouldn't cost six or seven hours to write the code for.

> I've seen some customers of ours being quoted six and seven figure prices by their internal data science teams to essentially slurp and transform a handful of SQL tables.

They don't wanna do it so they give a ridiculous estimate

I'm a firm believer that data science should be pulled out of IT and put into the business. However, storage is so cheap that you should never NOT collect data if you can. It's better to dump it into a data warehouse and wait for somebody who can use it to come along than it is to never collect it.

Then, and only then, should you look at your service levels and determine if there is a need for some sort of Hadoop or Spark infrastructure.

>I'm a firm believer that data science should be pulled out of IT and put into the business.

I say this from personal experience. The major risk here is that Data Scientists will constantly get pulled in as resources to support business fires and high visibility projects that should be handled by other people.

Otherwise what happens is this: "I know you are working on that Data Science project that is scoped for 4 weeks, but can you do this Business Analyst thing the insert exec name here asked for by this Friday?"

Data Scientists need to be shielded from that day to day analyst stuff, otherwise they'll never be able to do their jobs. One of the ways to do this is to put them in IT in a business facing consulting role. IT is typically (not always) is more focused on a proper solution, not day to day triage. So they can filter out projects that aren't appropriate.

This may end up repeating the mistakes of a decade or two ago: after a business leader gets two requests rejected, they are very unlikely to go back with a third request - they will simply outsource every future request. Soon the IT departments were being downsized because they couldn't get business leaders to use them instead of going outside.

I like data too but that approach has several hazards. The most obvious is liability: if you store it, you have to be responsible for protecting against misuse.

The second, however, is more subtle but often more damaging: people often assume that the data they have is the right data or complete so the drunkard's lampost problem is easy to fall into and people might not realize it as quickly because, hey, those numbers were based on so much data it took a day to run the query! Web analytics is notorious for that — people would make statements about performance, browser support, etc. from a bunch of log files and miss that this was skewed by bot traffic, confuse their server's response time as being a reasonable proxy for the user's perceived load-time, etc.

> However, storage is so cheap that you should never NOT collect data if you can.

Unless it involves customers and clients, in which case it's actually a sneaky liability... At least in terms of PR, if not legality.

Data should be moved out of IT and put into Data. Data headed up by a Chief Data Officer should be a first class citizen of every company. Not f’in IT, not Buisness. Data.

The businesses usually think they have a lot of data because they look it in terms of number of years of data but in practice their datasets are only a couple hundred megabyte large. Yet they setup Spark, Hadoop, and all of that stuff to "extract insights" from it. Usually all they need is someone who knows python to write a script to parse their data and put it in Postgres

I think this is a key point - too often, data "scientists" are brought in, given lots of tools, paid lots of money, and then told to just count things. And build dashboards that count things.

The data scientists that I have seen that are successful and bring the most value and seem to have the most satisfaction are those that spend time actually doing analysis and provide deep insights into the problems they are looking in to. The answers are the easy part - asking the right question is the difficult part.