News for hackers. Stuff that matters.
Re:Big Data's Big Problem: Little Talent by tgflynn on Saturday April 28, 2012 @08:57AM (#3902834)
This technology will change the world more than the Internet or any other technology in human history.

You're right that adoption is very slow. I'm convinced that businesses could save trillions of dollars by applying existing weak AI to their problems. Why aren't they doing it ? For one thing there's a huge gulf between the average business person's understanding of what is possible and what actually is. On the other hand the people who understand the technology don't have domain experience in various businesses. You can't develop solutions if you don't know what the problems are and it's very hard to guess at what economically relevant problems exist in fields you've never worked in.

There are probably other barriers as well. Domain experts are unlikely to champion technologies that may, well, replace them. Bayesian networks were developed in academia 20 years ago that outperformed doctors at medical diagnosis. Why aren't they being applied ? There are probably many reasons but I suspect conscious or unconscious resistance on the part of the medical community plays a significant role.

As for a talent shortage, I don't buy it. I'm exactly the sort of person this article talks about, with a strong mathematical background, excellent implementation skills and real world experience in developing and applying machine learning algorithms that have made millions of dollars for my former employers. I have had a website and a LinkedIn profile for over a year that make this fairly clear. How many consulting inquiries have I had ? Exactly zero.

Re:Big Data's Big Problem: Little Talent by dbecker on Saturday April 28, 2012 @11:04AM (#3903112)
Whether or not you have these skills: potential employers need to SEE them. There are lots of pretenders out there, and employers are appropriately wary.

Are you showing employers results on your web page that a worse ML practitioner can't match.

Putting results from a kaggle competition on my LinkedIn page landed me my current job (and I am still contacted by potential employers every couple weeks).

The employers of the world aren't stupid, but they aren't omniscient either. So you need to make it easy for them to see that you have the skills you claim.

Re:Big Data's Big Problem: Little Talent by tgflynn on Saturday April 28, 2012 @11:36AM (#3903189)
Did you win the competition ? If not what was your rank, if you don't mind sharing ? I'd be really interested to know how much impact this could have.
Re:Big Data's Big Problem: Little Talent by dbecker on Sunday April 29, 2012 @11:18AM (#3906167)
I was #4 at the time my current employer contacted me. I've continued in the competition with a small team of their employees, and we are currently #2.

I don't know how sensitive the number of job contacts is to your ranking. My hunch is that a lower ranking would still establish credibility.

Re:Big Data's Big Problem: Little Talent by tgflynn on Saturday April 28, 2012 @12:02PM (#3903259)
I couldn't reply to Estragon's comment below, so I'm replying here.

I'm convinced of it primarily because I'm convinced that the majority of activities that human workers perform do not require general intelligence (ie. strong AI). This includes most manual tasks : cleaning, cooking, customer service in restaurants, etc, but also many tasks performed by office or even "knowledge" workers.

A product my previous company developed replaced over 20K workers over a period of years. Few people even know it exists.

Once you are aware of this and follow the news you see it happening in virtually all domains: e-discovery systems replacing attorney hours, automated news story generation, etc. One area that has been lagging is robotics but this will start to develop very quickly, especially with the new DARPA Robotics Challenge.

Now to be clear many of the "Big Data" applications people are talking about may not fall into this human labor replacement category, but the underlying technologies are essentially the same.

Re:Big Data's Big Problem: Little Talent by csomar on Saturday April 28, 2012 @10:13AM (#3902989)
I have had a website and a LinkedIn profile for over a year that make this fairly clear

It doesn't work like that. You LinkedIn profile might easily land you any job in Software development, but not consulting.

In my opinion, if you want to do consulting for big corp. you should figure out what it takes to it. An attractive website and presentation, few buzzwords, client testimonials, business cards, and the other blablabla. Yes, it's irrelevant (and shitty) to what you are actually doing, but that's actually the world of consulting.

Re:Big Data's Big Problem: Little Talent by BrianLy on Saturday April 28, 2012 @10:38AM (#3903047)
I think you can go further than that. To get people asking for your time as a consultant you have to demonstrate experience and get close to vendors who already support clients you are interested in. For example, targeting a niche "big data" problem with a particular tool, and then developing a relationship with the community supporting that tool. That gives you access to the people who are looking for consulting.
Re:Big Data's Big Problem: Little Talent by waterlesscloud on Sunday April 29, 2012 @03:24AM (#3905392)
Big Corp - "We can't find people who can do X!!!!"

Person who can do X - "I've been saying over here I can do X."

Big Corp - "Oh. We don't look over there, it's not how it's done."

I think I'm seeing part of the issue here.

Re:Big Data's Big Problem: Little Talent by tgflynn on Saturday April 28, 2012 @10:19AM (#3903007)
That's definitely not my world and one reason I left big-corp in the first place.

EDIT ADDED: It seems like a really broken market if buyer decisions are completely orthogonal to the product being purchased.

Re:Big Data's Big Problem: Little Talent by tomjen3 on Saturday April 28, 2012 @11:42AM (#3903204)
I hope you are right. If you are, you have identified a potentially very, very lucrative option for a start-up. Broken markets can provide you a lot of money when you fix them.
Re:Big Data's Big Problem: Little Talent by lgieron on Saturday April 28, 2012 @12:22PM (#3903312)
This market is already (at least partially) covered by small consulting shops which provide sales front for competent freelancers who don't feel like doing the whole corporate networking&sales ritual.
Re:Big Data's Big Problem: Little Talent by ced on Saturday April 28, 2012 @03:40PM (#3903856)
Where do I find these small consulting shops?
Re:Big Data's Big Problem: Little Talent by lgieron on Sunday April 29, 2012 @07:14AM (#3905709)
Well, the companies I know tend to do their recruiting based on word-of-mouth recomendations - so they have to know you from some previous job or you need to be recommended by someone etc. It's a tiny sample of a couple companies though, and that probably doesn't generalize to all "small consulting shops".
Re:Big Data's Big Problem: Little Talent by Estragon on Saturday April 28, 2012 @11:44AM (#3903213)

  I'm convinced that businesses could save trillions of
  dollars by applying existing weak AI to their problems.
Why are you convinced of this? Do you have any case studies with obvious broad applications?

  I have had a website and a LinkedIn profile for over a year that make
  this fairly clear. How many consulting inquiries have I had ? Exactly
  zero.
Neither LinkedIn nor your ISP are responsible for marketing you. Have you been presenting your software at restaurant conferences? Do you have any case studies where it saved someone x% of their food costs?
Re:Big Data's Big Problem: Little Talent by zheng on Saturday April 28, 2012 @09:15AM (#3902869)
Slightly off-topic, but your website breaks after visiting the RDMS page, as all the other links seem to be relative, so they attempt to go to pages such as /products/people.html
Re:Big Data's Big Problem: Little Talent by tgflynn on Saturday April 28, 2012 @09:47AM (#3902927)
Sorry about that, thanks for letting me know. It should be fixed now.