News for hackers. Stuff that matters.
Re:Stanford's 2017 acceptance rate hits record low: 5.7% by mjn on Saturday March 30, 2013 @01:05AM (#5464483)
I'd be interested in reading something about what they're doing to keep the process working in that case.

I'm involved in various academic conferences, and acceptance rate is a big sticking point, which gets more problematic as the number gets lower. It's very easy to put on a symposium with a 60-70% acceptance rate: you just filter out the stuff that is clearly not good, and is not going to contribute. It's not much harder to go to 40-50%: you raise your bar for how well thought out something needs to be, how well it needs to ground itself in the existing research, and how good the prose itself needs to be.

But once you start going south of that, things quickly get problematic. Now you start rejecting decent stuff, for reasons that depend on random assignment of reviewers: some reviewers are harsher than others, and if you don't correct for it, that impacts things. You have not easily comparable factors: is borderline contribution in area A better or worse than borderline contribution in area B? How precisely do you weight every possible factor? Is interestingness, rigor, or clearness of communication more important? You also, as you get towards 10-20% acceptance rates, easily end up slipping into a mode where papers get accepted more because of "can't find a reason to reject it" rather than "would actually want to accept it". Reviewers become very critical (since most papers need to be rejected), and look for any reason they can find to reject a paper. So what survives is not the best papers, but the papers nobody could find a good enough reason to reject. What that usually means is either 1) airtight incremental work; or 2) blockbuster media-friendly work.

More generally, I think it gets exponentially harder to make any kind of review process meaningful as you accept a smaller and smaller percentage of applications. I believe I can honestly set up a screening system for accepting the top 30% of an applicant pool with at least acceptable error. But the top 5%? It starts looking like rolling dice. If Stanford does it better, I'd love to know how!

Re:Stanford's 2017 acceptance rate hits record low: 5.7% by dvt on Saturday March 30, 2013 @03:49AM (#5464796)
100% agreed. As a matter of fact, just about all schools on the top USNWR rankings are a crap shoot (http://colleges.usnews.rankingsandreviews.com/best-colleges/...)

You can get in one and not get in 20, get in 20 and not get in the one you really wanted to get into. I go to one of those schools and I personally know people that got into Berkeley and wanted to go to UCLA or got into Caltech when they wanted to go to Stanford, or didn't get into the state schools and got into Stanford. A friend of mine wanted to go to MIT for undergrad, but instead got into Cornell (and later went to MIT for grad school.) The undergraduate admission process at elite schools really makes zero sense.

All (most?) elite schools use a "holistic" method of evaluating applications. Which is basically a magical black box that spits out yes's and no's.

Re:Stanford's 2017 acceptance rate hits record low: 5.7% by aneth4 on Saturday March 30, 2013 @01:45AM (#5464569)
Two things:

- They are just as happy with a random selection of 7% from the top 15-20% of applications

- There is zero "error". They select who they select, and are not just looking for exactly the top applicants - in fact that has little meaning. What does top mean? Is someone with incredible musical talent but slightly weaker than another applicant in math better or worse? They want a solid diverse group of amazing students. This is not angel investing, where they are afraid of missing out on the one big hit.

Re:Stanford's 2017 acceptance rate hits record low: 5.7% by just2n on Saturday March 30, 2013 @02:28AM (#5464664)
Unless there is a 100% pass rate, there is definitely "error." Picking someone who you couldn't find a problem with but who is incapable of passing your courses counts as picking the wrong person.
Re:Stanford's 2017 acceptance rate hits record low: 5.7% by mtpearce on Saturday March 30, 2013 @02:56AM (#5464707)
The six year completion rate of the 2006 cohort was 95% (http://ucomm.stanford.edu/cds/2012). This rate has been consistent, both over time and at other top schools. Of the 5% that don't complete, it's rare that the student is incapable of completing (aaron swartz, tiger woods, etc). The admissions department will be the first to tell you that they pass on _a lot_ of qualified people. Even though most people believe that they were an 'admissions mistake', I still don't think that 'error' is the right term.
Re:Stanford's 2017 acceptance rate hits record low: 5.7% by barry-cotter on Saturday March 30, 2013 @02:49AM (#5464692)
The only elite college in the US with a failure rate higher than 2% is Cal Tech. Everyone else provides ample opportunities to switch to less rigorous majors with lower standards. Given that they're admitting 18 year olds with no previous experience of living away from home they're not realisitically going to approach elite professional school levels of graduation unless they make it really easy to pass.
Re:Stanford's 2017 acceptance rate hits record low: 5.7% by duaneb on Saturday March 30, 2013 @03:36AM (#5464771)
Lower standards or a better fit? I would find some subjects, like studio art, prohibitively difficult.
Re:Stanford's 2017 acceptance rate hits record low: 5.7% by ohazi on Saturday March 30, 2013 @04:01AM (#5464813)
I went to Caltech for undergrad. I know people who got "burnt out" and switched to non-science majors, but the core curriculum would still require everyone (even an English major) to take quantum mechanics, differential equations, etc. They've made some changes to the core curriculum requirements in the last few years, but I think this is still the case.
Re:Stanford's 2017 acceptance rate hits record low: 5.7% by aneth4 on Saturday March 30, 2013 @04:05AM (#5464817)
Even if the admission process had no "error" as you call it, there would still be kids who fail. Future behavior and circumstances are not predictable or controllable to that extent, no matter how good the admissions process is.

Regardless, the goal of admissions is not to pick the exact top students, but to create a solid class. That may involve taking some chances on kids who provide diversity or experience in areas other than pure academics. If those kids fail, it is not error, but intentional risk. Even the "best" (whatever that means) kid carries some risk, and may fail due to personal or incidental circumstances that could not have been known during the admissions process.

Re:Stanford's 2017 acceptance rate hits record low: 5.7% by seanmcdirmid on Saturday March 30, 2013 @04:58AM (#5464881)
Lets not conflate paper acceptance with college acceptance, the former is based on one piece of work that can be subjectively measured, at least, the latter is more of a crapshoot.

I'm a big fan of low acceptance rates in conferences. There are way too many mediocre papers that get into conferences I care about even when the acceptance rate is low. Also, the surge in papers submitted from China has pushed everyone's acceptance rates down, and quality isn't necessarily getting better. Disclaimer: I'm submitting from china these days.

In computer science, if you look at low accept conferences like OSDI/SOSP (systems), you'll see some safe (uncontroversial) papers getting accepted but most of them are just good and interesting. Now, compare that to CHI where the accept rate is still competitive but the quantity is huge. The quality there seems much less consistent.

When I review for conferences these days, I'll get a stack of papers where one or two are obvious good, one is controversial that we argue over, and 17 get binned after 5 minutes (the rest if the time you spend on the paper is just to give the authors useful feedback).

Re:Stanford's 2017 acceptance rate hits record low: 5.7% by mjn on Saturday March 30, 2013 @12:06PM (#5465777)
Yeah, I agree there's a big difference. What reminded me of it is that making finer distinctions get more error-prone as you try to take "the best of the best of the best" and add on more "of the bests". CHI is one of the conferences where I think the low acceptance rate tends towards: 1) a lot of randomness in the reviews; and 2) a tendency towards safe "I did a user study and here are p-values" type of work.

I agree there are some times when it's easy to accept only 10% or so: when there is a lot of obvious crap submitted. I'm currently organizing a conference with about a 30% acceptance rate, and it's somewhat painful, though. It's not one of the main catch-all conferences like OSDI, CHI, AAAI, SIGGRAPH, etc. which everyone submits to. Instead mostly people in the area submit to it, and a large proportion of the submissions are legitimately good. I would say 50% solid submissions from regular members of the community, 25% not-great submissions from people legitimately in the area, and 25% either crap or clearly off-topic.

So it'd be easy to choose which 50% to accept, but when you're accepting 30%, now some good work is being rejected, just because you have to hit the arbitrary acceptance-rate target to count as "selective" in certain U.S. universities' bibliometric systems. If we went down to a 20% acceptance rate (as some people are pushing, not b/c of quality but solely b/c it makes the conference look better bibliometrically), we'd have to fairly arbitrarily reject some more good papers.

We half-joke that the way to fix it next year is to spam the CfP to clearly unsuitable places, in order to elicit a flood of crap submissions that we can then easily reject, allowing us to hit a 30% (or 20%, or lower) acceptance rate without having to reject any good papers.

Re:Stanford's 2017 acceptance rate hits record low: 5.7% by seanmcdirmid on Saturday March 30, 2013 @07:35PM (#5467056)
My point with CHI is that they probably admit too many papers, or at least are very unfocused. OSDI and SIGGRAPH have high bars (not sure about AAAI), they don't so much as have a paper limit but just very high standards.

> Instead mostly people in the area submit to it, and a large proportion of the submissions are legitimately good.

Working groups or invite only workshops might work better in this regard. It depends on your area though.

> So it'd be easy to choose which 50% to accept, but when you're accepting 30%, now some good work is being rejected, just because you have to hit the arbitrary acceptance-rate target to count as "selective" in certain U.S. universities' bibliometric systems.

Its more about how tenure cases work in universities. Your judges don't read your papers, they just look at your bib.

> If we went down to a 20% acceptance rate, we'd have to fairly arbitrarily reject some more good papers.

In my reviewing experience, the PC never had to reject papers based on too many good papers; if anything we had to accept papers of questionable quality to fill the conference up. The rejection notices that PC chairs send out about there being too many good papers are often lies. But maybe our fields have different problems :) I'm a firm believer in accepting all the good papers and rejecting all the bad ones.

> We half-joke that the way to fix it next year is to spam the CfP to clearly unsuitable places, in order to elicit a flood of crap submissions that we can then easily reject, allowing us to hit a 30% (or 20%, or lower) acceptance rate without having to reject any good papers.

Just use Weibo to advertise your conference. There is a lot of talent in China and India who want to make it big, not all of it is great yet, but it will definitely push down your accept rates.

Re:Stanford's 2017 acceptance rate hits record low: 5.7% by daniel-cussen on Saturday March 30, 2013 @01:24AM (#5464517)
I remember noticing this in high school as I prepared for Stanford's review process. It's not so much the pursuit of greatness as the pursuit of non-failure.
Re:Stanford's 2017 acceptance rate hits record low: 5.7% by asafira on Sunday March 31, 2013 @03:10AM (#5467984)
I've talked with professors that do physics graduate school admissions decisions. All in all, they told me that it is not too difficult to get 700-800 applications down to 100-150. Unfortunately, they need to get it down to more like 60-70, and they all agree it's probably not very much better than random from there =/.