Re:Technical Interview Performance by Editor, OS, and Language
by tedsanders on Tuesday September 19, 2017 @02:34PM
(#15287497)
Be careful equating 'evaluate' with 'interview', when TripleByte has incentive to obfuscate. I suspect hundreds of engineers may apply through Triplebyte each week, but they may interview only a small fraction of those. If true, this makes your point about small sample size even more important. I was disappointed their charts didn't have confidence intervals or count numbers. Makes it harder to guess at how the numbers generalize.
Re:Technical Interview Performance by Editor, OS, and Language
by jayant123 on Tuesday September 19, 2017 @10:24PM
(#15290292)
Also, my guess is that the success rate for interviews would be fairly low, making any comparison of the success rates for different languages/editors even more statistically shaky. They should definitely publish the total sample size and success percentage for these results to be meaningful.
Example: Say they hired 5 out of a 100 applicants who used Python, and 2 out of 20 applicants who used Ruby. That's a 5% success rate vs a 10% success rate, a 100% increase. I'd hardly call that a significant result though.
Disclaimer: I'm no statistician either
> At Triplebyte we evaluate hundreds of engineers every week.
> I pulled the editor / language data for every Triplebyte interview over the last year.
Based on that, if we suppose "hundreds" means "low hundreds," this suggests a sample size of 5K-15K.
That's not a bad sample size for the total population.
But if 35% use Python and only 5% use Ruby, then you're comparing a sample of 1,750 (on the low end) to a sample of 250. It wouldn't surprise me if the Ruby data is more shaky than the Python data, and thus it's not a fair comparison.
Same goes for a lot of the other metrics where the sample sizes vary.
~ Disclaimer: I'm no statistician (but I play one on Correlated.org).