News for hackers. Stuff that matters.
Re:Why becoming a data scientist is not easier than you think by michaelochurch on Tuesday October 16, 2012 @08:34AM (#4659482)
I was a math major in college, with a focus on pure math. I did a year or grad school (math PhD program) and left to work on Wall Street (and worked for a couple startups, and Google, in that mix). In all, I spent 6 years as a mix of quant, trader, software engineer, startup entrepreneur, data scientist.

The software engineering career is in somewhat of a mess right now. It comes down to the "bozo bit" problem. Being a software engineer (even with 10+ years of experience, because there are a lot of engineers who only do low-end work and don't learn much) is not enough to clear the bozo bit, and you won't be able to prove that you're good unless you have a major success, and it's hard to have that kind of success without people already trusting you with the autonomy to do something genuinely excellent.

It's not enough to write code, because LoC is a cost and source code is rarely actually read at the large scale. At least for backend developers, the only work-related (as opposed to political) way to establish that you're worth anything as a engineer is to have an architectural success, but it's very hard to have architectural successes unless you've established yourself as an "architect" to begin with. So there's a permission paradox: you can't do it until you've proven you can, and you can't prove you can do it until you've done it. Hence, the vicious politics that characterize software "architecture" in most companies.

Functional programming is one way to put yourself head-and-shoulders above the FactoryFactory hoipolloi. The problem is that most business people don't understand it. They just think Haskell's a weird language "that no one uses". Elite programmers get that you're elite if you know these languages, but most companies are run by people of mediocre engineering ability (and that's often just fine, from a business standpoint).

It is true that functional programming is superior to FactoryFactory business bullshit, but not well-enough known. Good luck making that case to someone who's been managing Java projects for 10 years. What is better known is that mathematics is hard. It's a barrier to entry. I doubt more than 5% of professional programmers could derive linear regression.

So I see the data science path (and yes, it takes a long time to learn all the components, including statistics, machine learning, and distributed systems) as a mechanism through which a genuinely competent software engineer can say, "I'm good at math, and I can code; therefore, I deserve the most interesting work your company can afford to fund." It's a way to keep getting the best work and avoid falling into that FactoryFactory hoipolloi who stop advancing at 25 and are unemployed by 40.

Re:Why becoming a data scientist is not easier than you think by ericmoritz on Tuesday October 16, 2012 @11:13AM (#4660153)
I don't know how you can say that knowing functional programming inherently makes you a better programmer than another. A bad programmer is a bad programmer regardless of domain.

I have seen some ninja FP artists that have written some really terse solutions in a FP style which are completely unreadable to the uninitiated.

Also, I do not think that there is anything intrinsically difficult about FP. Knowing it proves that you are ambitious and that you have mental plasticity but not necessarily that you are a good developer.

You're just as likely to become that weird dude that writes code no one understands as you are to become Mr Wizard.

Re:Why becoming a data scientist is not easier than you think by Evbn on Tuesday October 16, 2012 @12:02PM (#4660411)
Mr Wizard was accessible to children!
Re:Why becoming a data scientist is not easier than you think by coliveira on Tuesday October 16, 2012 @10:26AM (#4659952)
A big problem that science graduates have is that the market doesn't believe in academic training. A big company will take a smart engineer and train him/her on a particular technique (data mining, for example) rather than hiring someone with that particular skill (usually from grad school). Even if they eventually hire such a person, the new hire will usually be working for the in-house "expert" until they can prove themselves, which is pretty much a political process.

It is rare the case when a company will hire an academic person to directly lead a company effort, and when this happens it is because the person is already famous and has a lot of consulting/real world experience.

The path you mentioned is great if you can find a company where you can make a major contribution using your skills, but for the reasons I mentioned it has still the same problem (bozo bit) that you cited before.

Re:Why becoming a data scientist is not easier than you think by tomrod on Tuesday October 16, 2012 @11:59AM (#4660390)
> It is rare the case when a company will hire an academic person to directly lead a company effort

I actually hear of this happening a lot for economists in Silicon Valley, even with unfamous people.

Re:Why becoming a data scientist is not easier than you think by jeremyjh on Tuesday October 16, 2012 @10:31AM (#4659968)
I would expect it to be a lot harder to break into writing trading algorithms than into enterprise architecture at most companies. The latter I found so easy it was both embarrassing and frightening. ("really? this is IT? this how we decide how to spend this 12 million dollars?! with this powerpoint?")

I guess at least on the math side you can prove your algorithms against historical data and so merit assessments could be fact based.

Re:Why becoming a data scientist is not easier than you think by PaulHoule on Tuesday October 16, 2012 @02:19PM (#4661119)
it's easier to break into algorithmic trading because you can trade on your own account

i got interested in financial semantics lately and was amazed to find how many people are doing algorithmic trading with accounts anywhere from $10k to a few mil. in fact i know of many startups developing products and services aimed at such people.

if you read the MSM you'd think it's all about HFT and big players, but I think algorithmic trading today is a lot like day trading was in the late 90's.

Re:Why becoming a data scientist is not easier than you think by hknews on Tuesday October 23, 2012 @08:49AM (#4687795)
Thanks for the insight about algorithmic trading. Any tips on finding out more? eg: good aggregators or blogs?
Re:Why becoming a data scientist is not easier than you think by PaulHoule on Wednesday October 24, 2012 @11:04AM (#4693223)
try the "Algorithmic Trading" group on LinkedIn; when I joined it I was amazed to find I already had hundreds of people who were contacts in that Group.
Re:Why becoming a data scientist is not easier than you think by tomrod on Tuesday October 16, 2012 @11:56AM (#4660373)
One would see open-source projects and volunteer coding project (such as the one I will post at the end) are a good way to get a leg up.

Being able to derive a linear regression is less important than being able to see the end from the beginning and being able to read math. I know a coder-cum-CEO (freshly minted) who has a hard time understanding how to program a numerical derivative, yet has produced amazing works based on what he does know. I said it before--we all need each other, especially as the focus has been on specialization (so that no one person has the entire skillset requred).

[1]http://www.volunteermatch.org/search/index.jsp?v=true&af...

Re:Why becoming a data scientist is not easier than you think by PaulHoule on Tuesday October 16, 2012 @09:24AM (#4659717)
Right on.

You can make Java a lot more fun if you use generics, functional patterns and frameworks like Google's Guava, even if you still need to create a FactoryFactoryFactory once in a while.

Re:Why becoming a data scientist is not easier than you think by Evbn on Tuesday October 16, 2012 @12:04PM (#4660421)
But "fun" isn't the same as efficient and maintainable.
Re:Why becoming a data scientist is not easier than you think by PaulHoule on Tuesday October 16, 2012 @02:15PM (#4661102)
So far as maintainable I'd say the internal DSLs I've made in Java as as maintainable as internal DSLs made in other languages -- that is, its very maintainable until you run into some complexity cliff.

When I program Scala or C# I tend to curse type erasure in Java generics, but when I actually program Java the generics system makes me smile. I can very efficiently turn what's in my head into a design and code.

In a language like Haskell whatever you gain from the language you tend to pay back in missing or buggy standard libraries. If I was writing a webapp in any off-brand language, for instance, I wouldn't trust that the urlencode function works correctly until I'd written some test cases.

I'll admit functional Java is a little verbose (can't write a closure in just one line) but the Hotspot compiler is very good at inlining functional code and collecting the garbage from it. I can definitely say that I get much better performance w/ functional Java than I usually see with the Interpreter pattern with two orders of magnitude less code size.