News for hackers. Stuff that matters.
Re:So Many Research Scientists, So Few Openings as Professors by p4wnc6 on Thursday July 14, 2016 @02:31PM (#12096212)
Saying "MATLAB is for other people" isn't the sign of a poor candidate. Saying "you will need to use MATLAB for this job" on the other hand is overwhelmingly a sign of a bad job, in the same way that any usage of Excel VBA whatsoever is a sign to run, run, run for the hills -- that firm is absolutely doing it wrong.

I've also heard the opposite of your first example used as a criticism of candidates too. After completing my undergrad degree in math and then grad degrees in statistics, I was astounded how in industry, either describing yourself more as an engineer or scientist who does not work on the abstract math stuff or describing yourself as someone who is very interested in abstract math will both cause you to get rejected.

The only way to win is to happen to have done a lot of difficult abstract math in the past and remember it all well enough to pass tricky interviews, but then to be overwhelmingly happy and satisfied with a job that will not ever ask you to use it and will instead burn you out on dumb shit stuff like fitting a regression to KPI data and using t-stats to directly do (fallacious) model comparison.

The biggest career risk I've seen from having done very computer science-heavy statistics is underemployment. Your math chops will be pure credential, sometimes used by your manager to try to win arguments from authority about e.g. that dumb shit KPI regression. But you will 100% never be given open-ended modeling work that could actually have a positive impact on your business's bottom line.

Essentially, you are hired to be some more senior person's sycophant political darling. You function internally much the same way that a big consulting company functions externally -- people already know what they want to hear, they just want you to tell them what they want to hear, slap together some plausible-sounding rationalizations for it, dress it up with buzzwords about big data and "insights", and be a show pony for talking about it.

They emphatically do not want you for doing anything that would be called "real" work.

There are occasional exceptions, especially in certain teams at certain established tech companies. But then, landing a position inside one of those teams is nothing but the same lottery as winning a professorship all over again.

Re:So Many Research Scientists, So Few Openings as Professors by nradov on Thursday July 14, 2016 @03:36PM (#12096619)
Welcome to the real world of being an employee. It's common in every field for new employees to be academically overqualified and think their bosses are incompetent and irrational. If you don't like it then try starting your own company. I'm not being flippant; that's a serious suggestion and would give you a more realistic perspective.
Re:So Many Research Scientists, So Few Openings as Professors by p4wnc6 on Thursday July 14, 2016 @05:43PM (#12097393)
Same tired, cliche replies as usual ... you're probably also assuming that my opinions aren't based on job experience, yet they are based on several years of experience in many positions, where I did the 40-hour grind, delivered business solutions for months-long and years-long projects, etc.

I'm not talking about academic overqualification. I'm talking about someone hiring you, talking at length about how you are being hired specifically to do X because it matters to the company's bottom line, and then after you're hired they switch it and say actually you're going to do Y but you're going to be a political mouthpiece for X.

One common set of values is

Y = statistically invalid model fitting that actively causes the business to lose money but which is easier to reduce to pliable metrics for political jockeying

X = (deep) machine learning and/or Bayesian stats

It's not at all about academic overqualification. The actual business need, for reals, can benefit from the pragmatic and cost-effective use of the tools, and the person is actually skilled in using to do exactly that.

Yet, they are prevented politically.

As for starting a company, I think I would guess that's one of the least plausible ways of doing important or useful work. You'll only get funding if it is a trite variation on consumer bullshit -- even though consumers themselves don't want that and would rather that your labor is allocated to solving more fundamentally important social problems that there just isn't money for solving. Plus, you'll be so burnt out over all the auxiliary stuff like HR, marketing, sales, that you won't actually do any of the underlying quant work that was the whole reason for starting the company in the first place.

It kills me how people here seem to think that "start your own company" is some kind of "put up or shut up" gauntlet to throw down to challenge someone who is lamenting the shitty state that things are in.

"Start your own company" is not some venerated challenge-call for those brave few who want to change the world. Starting your own company is just a different format of the same bullshit phenomenon.

Fixing what's broken inside of companies and organizations that already have huge leverage and capital to positively impact fundamentally important problems -- that is perhaps worthwhile, if you can manage to deal with the political fighting without getting too burned out.

Re:So Many Research Scientists, So Few Openings as Professors by S4M on Friday July 15, 2016 @03:18AM (#12099387)
You seem to be missing the parent's point. He's saying that it's very common to be told that you are hired to do X and made to do Y (for the record, it happened to me once, for X being stochastic calculus for derivatives pricing and Y being excel jockeying), and if the reason for that are the politics caused by a big structure, then move to a smaller structure. You can create a company that does consulting in a quantitative field, so you don't have to seek funding to hire a team to build product.
Re:So Many Research Scientists, So Few Openings as Professors by p4wnc6 on Friday July 15, 2016 @10:33AM (#12101207)
I disagree. I think the parent is trying to say that businesses hired overqualified people and then give them less engaging work for valid business reasons.

What I'm trying to say is that you're told you'll be hired for X, and, crucially, that it's easily verifiable that X actually would help solve the business problem better than Y. Failing to do X actively hurts the business.

Yet you're still forced to do Y. It's not because in the real world you only needed simple, trustworthy Y to get the job done. No, you're failing to get the job done, need X to get it done, are told you're the one to bring X to the table, and then you're made to do Y for destructive political reasons.

The point I'm trying to make is that there's no defensible "real world" pragmatism to support the focus on Y nor the bait-and-switch to hire someone who knows X. Whatever the reasons for that, they are not about improving the firm nor making money for the firm. They are about optimizing a bonus or promotion or whatever for a single individual or some small faction, even at the expense of the organization's overall progress.

Re:So Many Research Scientists, So Few Openings as Professors by mattkrause on Thursday July 14, 2016 @03:46PM (#12096689)
What's with the Matlab hate?

Sure, it's closed source, expensive, and can be clunky, but it is not a horrible tool for many jobs that involve a mix of signal processing and other data analysis.

Scipy is often a viable altenrative, but if the company already has a bunch of Matlab code, rewriting things in Python probably wouldn't be worth it...

Re:So Many Research Scientists, So Few Openings as Professors by eli_gottlieb on Thursday July 14, 2016 @05:03PM (#12097152)
>Sure, it's closed source, expensive, and can be clunky

And slow. Don't forget slow. Closed-source, expensive, clunky, difficult to read, and slow.

(The first time I took machine-learning in grad-school, our assignments were in Matlab, and testing them required me to stay up until 04:00 in the morning at least once a month. A validation run just would not take anything less than five to six hours. The second time I took the class, I took it in the computer-science faculty, and the professor gave the assignments in scipy/numpy. Validating and debugging that was easy.)

(Just this year, I actually went and rewrote a probabilistic program in Haskell from a probprog package based on Python. Interpreted languages are fucking slow for numerical jobs.)

Re:So Many Research Scientists, So Few Openings as Professors by krapht on Thursday July 14, 2016 @05:41PM (#12097386)
I'm a little skeptical. Yeah, interpreted languages are slow, but under the hood numpy / Matlab call out to BLAS/LAPACK. There's a constant overhead per function call, but if you "vectorize" your computations to work on arrays instead of single values, the difference is not that great.

Seriously, Matlab = MATrix LABoratory. It's a DSL for matrix math, and it's really pleasant to use for that.

Also, my Matlab code is a joy to read. It's not Matlab, it's that most people who write Matlab code aren't professional programmers / don't care about making it pretty.

Re:So Many Research Scientists, So Few Openings as Professors by mattkrause on Thursday July 14, 2016 @06:22PM (#12097643)
Yeah, I'm not sure I believe that either--it's calling the same libraries python calls. It is possible to write matlab code that runs incredibly slowly--using eval to make anonymous functions calls to (badly-strided) individual elements of an array--but decent matlab and numpy code should be pretty comparable.

> It's not Matlab, it's that most people who write Matlab code aren't professional programmers

This, 1000x, this. There's also the time-honored tradition of a quick one-off script "to see if it works" that somehow becomes permanent.

Re:So Many Research Scientists, So Few Openings as Professors by eli_gottlieb on Thursday July 14, 2016 @06:38PM (#12097730)
Eh. It's more that Matlab code can either call out to fast linear-algebra libraries or go through an interpreter, and you have to memorize which language constructs do which. I'm definitely a professional programmer, but I still can't write fast Matlab code for that reason.
Re:So Many Research Scientists, So Few Openings as Professors by p4wnc6 on Thursday July 14, 2016 @05:34PM (#12097341)
The slowness and the pan-everything global namespace for the primary libraries are the two biggest problems. The interface for performance-targeted optimizations (mex files) is awful and doesn't hold a candle to Cython. These are downsides that would prevent me from using an open source and free tool ... so when you add that it's closed source and expensive, it just simply does not ever make sense.

If a place finds themselves having a lot of MATLAB code, it means that earlier they didn't refactor and retool when the choice was cost effective to do so. That's strong evidence that for whatever they are working on right now they also will not refactor it to at least try to avoid the future costs of current poor designs. That's a huge red flag.

I guess if they paid an insanely high salary or gave you some other type of assurance that you personally valued, you could trade it off against the red flag evidence. But generally these places also tend to just hire maintenance engineers, since they know they can't offer interesting work. It's best just to avoid and work for places that bend over backwards to refactor and evolve their tooling over time specifically to prevent this problem.

Re:So Many Research Scientists, So Few Openings as Professors by mattkrause on Thursday July 14, 2016 @06:37PM (#12097724)
> If a place finds themselves having a lot of MATLAB code....That's a huge red flag.

Serious question: do you feel that way about other older languages too? Would you balk at a C++ shop (which isn't doing high performance or low level stuff?)

Re:So Many Research Scientists, So Few Openings as Professors by p4wnc6 on Thursday July 14, 2016 @08:47PM (#12098290)
I would only balk at a couple of things, MATLAB, VBA, and extensive server-side Javascript being the main red flags of poor tech culture.

I have some additional skepticism about (a) shops that very quickly adopted Go and then display a cult-like dogma about how super perfect and awesome at all things networking that Go is and everything else isn't; and (b) shops that use Scala or Clojure purely as "better Java" and often have thin, entirely unfunctional layers wrapping bad legacy Java code -- in such places, the Scala and Clojure often exists specifically because they had a hard time recruiting people to be legacy Java maintainers, and having them do it through one layer of indirection, such as Scala, let them tap into a more vibrant labor market with a lot of people who just naively believed that e.g. usage of Scala == adherence to good quality standards.

The age of the language has nothing to do with it, and I actually quite enjoy C programming. I'm not a big fan of C++ code bases in which some of the more esoteric language features are greatly abused, but generally feel like C++ is a great tool and would in fact enjoy the chance to dig more into in a future job.

I also quite like code maintenance and legacy code extension, because when the company takes it seriously and values that kind of work, it involves a lot of cool abstraction, working with interfaces, design tests a la the Feathers book on testing in legacy code. I think that stuff is quite fun, and a healthy culture of refactoring makes it interesting to work on legacy systems.

While it's not an iron-clad rule, certain kinds of legacy choices, however, are not really compatible with a spirit of quality, legitimately valuing maintenance or refactoring, etc. MATLAB and Excel VBA are by far the strongest indicators that it's not a good situation.

The items above are just language choices, too. There are also many other things to worry about. For example, when a place uses MATLAB, it often means some of the work they do is scientific / quantitative prototyping. Do they enforce good code standards even at the prototype stage so as to minimize the distance between research prototypes and production? Failing to do this is a seriously gigantic red flag. It can mean that the management chain values the domain science more than the quality of the implementation, and often this means that domain scientists are allowed some of the following

- to manage their own working environments (leading to lots of awful "but it works on my machine" errors)

- to write every thing as giant, messy, linear scripts that start off with 200 lines of boilerplate data loading and model setup code that should be factored into a library but is instead copy/pasted and finish off with 200 more lines of custom plotting code that also should be factored into its own internal library for standardizing reports and charts

- to turn things into short-term fire drills for production programmers solely by virtue of them being "someone who uses MATLAB, not <some real language> that we use in production", and get management support for this.

- never bother to learn object oriented or functional programming principles -- google a design pattern and then just paste some ill-conceived bastardization of it wherever they feel, and then argue with production engineers that you can't refactor it "because it's a design pattern"

- ... I could go on.

When I see places that heavily use MATLAB or R for research systems, I break out in a cold sweat, since those languages are entirely unsuitable for professional software design. They are good for ad hoc linear algebra, optimization, model fitting, plotting, and statistics. But the software design underlying how those things are carried out in MATLAB and R does not translate at all to a system where the scientific code is maybe 1% and the business reporting code, customer-facing services code, etc., are the 99%. And putting the scientific code at 1% is generous even for a company that is solely about scientific computing or quantitative services.

Anyway, there is just a large difference between organizations that work from a systems engineering perspective first, and build that way, and mercilessly require non-programmer PhDs to get up to speed on actual, principled software development even for doing ad hoc domain specific work.

Re:So Many Research Scientists, So Few Openings as Professors by mattkrause on Friday July 15, 2016 @02:44PM (#12103001)
> management chain values the domain science more than the quality of the implementation

That is the crux of the issue. Many Matlab and R users don't see themselves as producing software per se. They are producing "recommendations" or models or papers; the code is just a means to that end, and so who cares if it is terrible? No one will do this exact thing again anyway... Obviously, that is pretty myopic. If you have good "infrastructure" code, it makes writing the one-off parts easier, faster, and less buggy--which it makes it easier to turn the one-off code into infrastructure.

Out of curiosity, how could I convince you that joining our group (which does have a ton of matlab code) wouldn't be awful? Or what would convince you to hire someone whose last job was predominately matlab?

Re:So Many Research Scientists, So Few Openings as Professors by cf on Thursday July 14, 2016 @05:34PM (#12097340)
Oh really. Is this Haskell program anywhere online?
Re:So Many Research Scientists, So Few Openings as Professors by eli_gottlieb on Thursday July 14, 2016 @06:11PM (#12097580)
If the resultant paper gets published, the repo on github will go public and the link will be in the paper.
Re:So Many Research Scientists, So Few Openings as Professors by cf on Thursday July 14, 2016 @06:32PM (#12097702)
I ask since I work on probabilistic programming in Haskell and am always looking for more use cases.

Good luck and I hope the work is published.

Re:So Many Research Scientists, So Few Openings as Professors by eli_gottlieb on Friday July 15, 2016 @08:17AM (#12100422)
Hey, can you tell me what probprog system for Haskell you work on? I might have been meaning to write a sampler for you.