Interesting, your experience matches mine very closely! I once quit a doctoral program in disgust but once I had enough money to fund myself, I could go back to become, perhaps similar to you, a "not regular PhD student" and create a more suitable situation for myself. Not only did I become free in my choice of research topic and collaborators/supervisors. More importantly, I was able to reject proposed, "faster" topics that would have led more immediately to bite-sized, publishable results. As a result, I was able to find and work on something I believe could be fundamental in my field.
On a sidenote, I just started re-reading Kuhn's Structure of Scientific Revolutions. I am growing more and more convinced that the right way to do something important in today's academic environment (that's the link to OP) is the sort of half-half position that seems to be our shared experience. If you are too much of an insider, you might end up doing "normal science". If you are too much of an outsider, the risk is high that you are/become just another crank. Half-half seems like a good position. Maybe a few decades ago, it was still possible to hold an academic position and maintain one's independence. Today, a full-fledged, conventional academic career seems hard to square with a free spirit, what with all the dross of admin and the drag of teaching duties.
Note that Kuhn doesn't think that normal science is bad in general. Normal science - incremental, systematic research within a scientific community with clear, well-defined rules and solution methods - is necessary and worthwhile, IF you have a good paradigm. When Scott Aaronson publishes a new proof about some obscure or exotic quantum computational complexity class, that is valuable normal science. When the Intel researchers in the 80s or 90s came up with the umpteenth optimization trick that enabled next year's scheduled Moore's Law microprocessor speed doubling, that is valuable normal science.
The problem comes when researchers try to do normal science without a good paradigm, either because they never had one to begin with, or because their old paradigm broke down. For example, it looks like the statistics-based paradigm that guided research in psychology is breaking down, because of the replication crisis. Kuhn's advice to people in this situation would be to re-examine the philosophical basis of the field, NOT to continue on doing business as usual.
Yeah, I did not write that well. I was thinking too much about myself. From a population point of view, I agree with you (or Kuhn) that "normal science" is a necessary and valuable part of the enterprise. It would be inefficient to have everybody constantly questioning everything. You need the "normals" to milk a given paradigm for all its worth and the rebels to keep the paradigm, if you will, "on its toes" and replace it when the time has come.
It seems to me commerce is not dissimilar. Not everybody can or should be a wide-eyed entrepreneur. You need people who compete within a business model (in the process wringing all the efficiencies to be had out of it) as well as entrepreneurs who create and thus compete between business models.
I wonder if there is an optimal proportion between entrepreneurial/rebellious "predators" and "normals".
I am not familiar with Kuhn but the statements you echo resonate deeply with me. I hope in some places academics do shift their paradigm (when necessary) but from what I've seen at uni the outlook is bleak. Researchers do x for years, built up their reputation by doing so and at some point their ego is so big that they don't dare asking themselves whether they've wasted years doing something very stupid. Even worse, these people favor PhD candidates that are willing to follow their footsteps.
This is really the way to go. You have to love your field so much that you're willing to save up enough to support yourself to do independent research in your field. I've been thinking of doing that. Saving up enough to support myself to work on a research-esque project (i.e. not a startup with necessarily any revenue-generating potential). Your love for your field (in my case, computer science) has to be so great that you're willing to forego the currently ridiculously high developer salaries for a number of years to try to advance the field as a whole (which ideally benefits society on the whole, but doesn't benefit you in a financial sense anywhere as much).
I also have the sense that academics live in a sort of bubble, prioritizing concerns that are secondary, such as ego contests, impact factors, building CVs etc. (to me at least, this bubble is reminiscent of another microcosmic bubble sector - the army). Of course there are among them true scientists, with intact childlike curiosity and passion, but they are increasingly hard to spot in the overcrowded academia.
One needs to be a part-time scientist so that he has enough time to concentrate on a subject.
prioritizing concerns ... impact factors
To a very good first approximation, academics are assessed by other academics.
When did it vote
academics are assessed by
other academics
There are additional requirements that didn't exist in the past, like gender, ethnic and national quota.
There is Goodhart's law, and scientists develop ways of hacking this system, but there is no free lunch, and the effort that goes into successful bids for funding is major now.
I close with a link to a successful request for research funding from 1921:
https://dirnagl.files.wordpress.com/2014/01/warburg-dfg.jpg
https://dirnagl.com/2014/01/14/otto-warburgs-research-grant/
This is a fundamental misunderstanding of the problem being discussed.
The problem is the pressure to publish frequently, which leads to low quality science. Many of the nobel laureates had at most two, often zero, papers in graduate school. Now, they might require a graduate student to have seven or eight papers to get out. Some of this is a shift to sharing more data, but a lot of it is just science of minimal value being packaged up and pushed out.
And the drive for this does come from generic metrics, that rely not on scientific understanding, but on the ability to count. The government sometimes prioritizes funding by papers/$ funded for grant, papers/year, etc.
I think government officials don't have a direct interest in supporting the current inefficient publishing model, but it's a model that scientists themselves have accepted and reinforced since the 60s. If scientists put forward an alternative model that works better, i dont think governments or the wider public would object.
Social science has shown that a group of people can follow some rule even if all members disagree privately. This happens because there is social status, peer pressure and incomplete information, and so people tend to think that the others agree with the status quo.
I think because it seems so counter-intuitive you're ignoring this empirical observation. Scientists simply, believe it or not, don't have "complete control" over the situation.
I talk with several of my peers who are unhappy with the choices academia offers, and would be open to the thought of pursuing research without being a career academic in the conventional sense. The biggest hurdle that comes up is the feeling of being alone -- not knowing other people who are trying to hack the system and do things differently.
Since there seems to be some traction along the lines of independent research pursuits here, I'd like to pass out a quick survey I just created, for people currently trying to pursue academic research on their own path (or interested in doing so): https://goo.gl/forms/wLvkGPstpCCBJY513
It's very preliminary, but I thought it might be a good start to get some discusion going. I invite responses/comments, either here, or on the survey form.
The whole situation really makes me want to ultimately do my research outside of the establishment, but I recognize that collaborations and publications are a key part of credible science. In order to do that I figure that I will need a platform for recruiting participants and conducting experiments for the kind of research that I would want to do. Ideally the platform would be compelling enough to bring in researchers in academia (or elsewhere). So I'm working on that platform. That's the programming project that brought me to HN.
I filled out your questionnaire. Not sure if my project is related to your frustrations, but it seems somewhat close. Either way I share your pain.
As far as publishing goes. I've got papers that I'm working on now, and I've got several collaborations going that extend beyond my current program to faculty in other universities. So I'm building a network of people who are interested in the kind of research that I am doing that I could potentially facilitate.
The bottom line is that if I'm sitting on data or access to data, I'm going to have no shortage of academics who would like to get involved. They don't need to be superstars, just researchers who publish, and I'm doing pretty well in that department already. My immediate well-funded faculty supervisor's eyes light up when I talk to him about the kind of datasets I could potentially get for him. So there's potential funding on the academic side as well, at least on small to moderate scales.
Bottom line is that the online experimentation tools for large swaths of psychological research (questionnaires mainly but also cognitive tasks) aren't well handled by the big current players in that market. We're just not a big enough value proposition. The tools aren't designed for us. We really need for participants be able to just login to the experiment and be taken through the whole process. We need tools to track their participation, send reminder emails, etc that are built in to the experiment rather than cobbled together around it. And what most of us would like is for it to spit out a cohesive dataset at the end, ideally with at least some of the data-cleaning done (and documented) for us. A lot of research is cobbled together from a bunch of different tools that spit out separate chunks of data that need to be painstakingly re-constituted at the end. This may not be true for all academic research, but it is for the research that I'm targeting. So there's a value proposition there with a market of tens of thousands of grant-funded researchers.
In addition, I have access to several local sources of potential data collection (outside of academia) that are extremely appealing to an important subset of the researchers who might use my platform. I'm currently working with three such groups, but if things worked well there, there are hundreds if not thousands of such places in North America and Europe (and elsewhere). So I'd potentially be able to offer researchers not only a fairly comprehensive suite of tools for collecting data, but I'd have a growing network of recruitment sites so researchers could really just design it and click 'go' (and pay me).
This isn't all of it, just the research side. I'm pretty sure that the same set of tools can be used to go after a couple other non-academic markets as well. I've got at least 2 other non-academic revenue opportunities that I think are pretty solid and can be explored in tandem.
Then there are the long-term plans that change everything... (cartoonish evil laugh)
Deep learning is a big thing, yes? Hinton, Bengio, LeCun, etc. All that work originated in universities under completely conventional structures. Companies snatched it up once it was clear it would be the next big thing.
Self driving cars might have some impact? Thrun, Fox & Burgard were key players in probabilistic state estimation, wrote the Book, and trained the car people at $(BigCo). At universities.
Just to balance out some of the 'academia has no idea what it's doing' statements in this thread. It's not perfect but it's not ineffective.
Although this is a bit depressing and one of the reasons I left academia, is this not partly a reaction to the changing nature of science? With many fields well established with strong foundations there are arguably fewer revolutionary ideas required. It looks like science is going to be more evolutionary and less revolutionary, with emphasis shifting from the individual researcher to large teams.
Take for example the LHC where managing the huge teams and technology is (I imagine) significantly more of a challenge than interpreting the data or deciding what experiments to run. A bit of an extreme example perhaps but large projects are probably going to become more prevalent as most of the lower-hanging fruit is claimed.
This seems largely a byproduct of the funding model, which seems unlikely to change in academia.
> cardboard boxes for a living? Flipping burgers?
Or working as a patent clerk, maybe?
Yes, pretty much anything outside the academic process was what I meant, so that your livelihood does not depend on the outcome of your research, because then you have to tailor your research directions towards the safe and accepted. And so your straw men actually would work, though suboptimally because the income is probably too low to allow you free time.
Also, misquoting me by leaving out the "really" makes your argument disingenuous, because the "really" made clear that the statement is not a 100% but a trend (if a somewhat overwhelming one), and 4 exceptions (names?) don't exactly disprove a generalization.
It would also be interesting if you see your four examples as holdouts from a different era, or general exceptions. I'd wager they're more of the former rather than the latter, and that their number is dwindling.
Does it still work though? In the days of Einstein, I imagine you could slack off easier in such a position than nowadays.
The problem these days is that we're inundated with busywork not only in academia, but in most occupations.
Even in general at very large scale, a world of limited and shrinking economic opportunity but unlimited and expanding education (well, until the future student loan crash arrives), means you can try to pile on busywork but the people shamming and pencil whipping it are better at shamming and pencil whipping and everything else, than any other cohort in the history of humanity...
As a cultural touchstone look at the movie "Office Space" still relevant today. He stares at the walls for an hour or two every day trying to look busy. Sure you might have to pass three interviews with ten people while proving P=NP and writing a javascript compiler to GET the job, but to DO the job you have to update TPS report headers every couple weeks and that's about it, other than loading letter size paper into the laser printer and staring at walls looking busy. Not to spoil the plot of the movie but the lead actor is not a theoretical physicist although if he were, it would be a dream job for him, work about an hour a week then look really busy the other 39+ hours while working on physics.
Even if contemplation is enough, most "manual" jobs require you to use your brain. Not necessarily in an interseting way, but enough to distract you from contemplation. It can't all be moving heavy objects from one pile to another.
Now that I've gone back to do my PhD, the only reason I can do something I consider meaningful is because I am not a regular PhD student. Interestingly, that's also the feedback I get, though as "helpful" advice that while what I am doing may be both good and important, it is unlikely to lead to success in academia. With the implication that I should stop doing it and concentrate on something more reasonable. Fortunately, I am not particularly interested in success in modern academia, so I get to do something I consider both good and important.
A related issue is that there really is no such thing as a senior researcher. Instead, professors are turned into research managers, responsible for helping their charges' careers, who then also turn into research managers. Actual research appears to be mostly a still not entirely avoidable side-effect. (And this seems similar to the way the only real way to advancement in industry is to switch to management, all dual-track equivalence rhetoric aside).