My job as a consultant is to tell you "your data fits in RAM" and charge you $20k. While it may seem expensive, the ROI is about 14 days, because now you don't need to hire that data scientist.

In my experience its a tough sell. We deliver results, but people don‘t like the idea that what they are doing is (gasp!) pedestrian.

You nailed it. People don't get promoted for leading pedestrian projects, they get promoted for leading challenging, innovative, groundbreaking projects.

This is an accurate statement.

Sometimes tech reminds me rich, bored, stay at home SO's that are constantly redecorating their house. Not because they need it, but because they are bored and the next trendy design looks cool anyway.

It comes from both top and bottom. At the top managers want to justify their salaries to their managers so they always redesign / rebuild / reorganize, even if things work pretty well as is. At the bottom new programmers fresh out of college want to assert themselves. The best way to do is to propose that everything existing is old and shit and needs to be rewritten. So they volunteer of course.

How does someone get into a position like that? Are you an independent consultant or do you work for a company? Is this exclusively what you consult on?

His comment while accurate was probably a bit dramatized for effect.

My guess is he is a consultant hired to do a typical project (i.e., help me move X into the cloud, help me re-architect our data model for big data, help me implement AI for our dog walking app) and at that point he just shows them they aren't ready for it or flat out don't need it.

It's just my guess, but it's what consultants do. The bad ones are happy to take on your project and charge you $400 bucks an hour. The good ones unfortunately, deal with the dilemma of turning down lucrative work in the spirit of doing what's right.

...and the disillusioned ones realize that there's a line of bad consultants just waiting for them to turn the project down...

My guess is that at some point you get tired of it and take solace in the fact that you're at least helping them to do it right.

After all, if they think they are going to hit big data scale and want the tools to handle it, you aren't completely a bad guy if you help them do it right, especially if you've already advised them not to do it.

My apologies. I'm not a consultant. That was a joke.

I regret to inform you that my original comment was sarcasm. I'm not a consultant, but would be perfectly willing to do the job as I described. The hardest part would be keeping a straight face while demanding the $20k.

It's amazing how many startups don't re-evaluate their infrastructure requirements every couple of years and get stuck with numbers in their heads reflective of hardware prices five years ago.

No need for clever optimizations - Moore's Law will bail you out a lot of times.

So pretty much anything over a TB and we should be doing "big data" stuff?

Im curious. What do you mean by "actual ML"? Isn't regression always a form of curve fitting? So is actual ML = curve fitting with fancier models (like multi layer perceprtrons)?

SVM, other kernel methods, Bayesian networks, genetic algorithms, clustering etc.

> Isn't regression always a form of curve fitting?

Sure and that's been done before for many years. I was just saying that today everyone who was doing that, isn't doing curve fitting or regression analysis anymore but "AI" and "Deep Learning" (doesn't matter if there are not neurons involved).

I think it's a true scotsman fallacy. We don't have a good definition for ML. So we reject methods and problems as not "real" ML". Ive done some logistic regression in my work and hesistated to call it ML. But Ive read a survey about tools used by people who do ML and logistic regression was the top item on that list.

Neurons are just an inspiration from biology. You can call them layers of neurons or you can call them matrices and do matrix multiplication. Nothing special about neurons.

I understand your argument that people like to use buzzwords and you don't like this. But it's a genral problem which applies to everything, not just ML.

Cloud technologies were also same before. People still keep burning cash on AWS even if they don't get a single customer. Many never understand actual use case of AWS and it's various services.