> The final approach I've encountered is where research problems come from an external body.

For a good helping of the government research organizations, they hire academics and practitioners as Program Managers (PMs) who gather the experts in the field to come up with research programs, agendas, and seedling programs. Of course, the PMs have to sell areas to their higher ups too, but it does have some degree of dynamism and alignment with academia. The amount of resources that could be brought to bear on a problem is impressive.

I do wonder about this whole notion of “success” of research. Hundreds of years ago, wealth patrons sponsor the “researchers” of their times. Creative people found problems that interested them which sometimes led to unexpected results and new areas. I don’t think serendipity should be discounted.

>Creative people found problems that interested them which sometimes led to unexpected results and new areas. I don’t think serendipity should be discounted.

From my own experience as a researcher, when conferences and meetings are online (as they have been for most of the pandemic), you lose a lot of opportunities to make unexpected connections with new people and ideas. Serendipity helps jostle you out of a local optimum.

Definitely. Before Covid, I routinely returned from the major conferences with a notebook full of potential ideas, some of which I could explore, some of which I handed out like candy in the local environment to colleagues and students, and some of which become very relevant years later.

The virtual conferences during the last two years were useless in retrospect. I saw the presentations of the papers I was interested in and that's it, I don't even remember anything specific from any of them, somehow the format was so routine. I'm glad that this is ending and we're having proper conferences again. I understand that researchers in the 'global south' had a different perspective with remote conferences being more accessible, but I've come to a conclusion that for me virtual conferences are a waste of time - I'd rather just read the published papers asynchronously at my convenience instead of framing it as an event; if you don't want a conference for some reason, make it a journal instead.

> You can probably imagine my astonishment when I realised that in some communities there is a widely accepted list of "the next problems to work upon".

It would be nice if there was some curated list of problems to work on, where the items on the list are obtained and sorted through some consensus mechanism. I know HN is secretly obsessed with To-do lists, so perhaps this "Super To-do list" is something someone finds interesting to build.

Related question: where do startup ideas come from?

How is that even a problem? Ideas bubble into my head much faster than I have the time or ability to work on them. Granted, most of them are crap, but that's because most of them are outside of my field of expertise.

As for usefulness, I seem to remember that it was the father of category theory who on his deathbed said, "I can die happy, knowing that my research will never have any practical applications". Now it's the basis for most type systems.

One of the hardest parts of doing good research is figuring out what a good question is. “Good” here encompasses a lot of things, such as

* is this an interesting problem to me?

* is this an interesting problem for my students?

* is this an interesting problem for the community?

* codes my team have the skill set to solve this problem?

* is this problem even within the reach of existing techniques?

* if not, what are the intermediate steps required to get there?

> Granted, most of them are crap

Well lol there you go. He probably wants good ideas not crap ones.

Regarding "scouting mode", one time proven trick is assembling a large stable of grad students to do the heavy lifting and tackle the boring parts of research, while one spends most time scouting. The downside is senior researchers tend to devolve into managers. One inspiring thing about the likes of Feynman is it seems they were able to keep tackling problems and doing a lot of the heavy lifting themselves, even late in their careers.

After devolving into managers, the most senior/successful people reach a point where they can contract-out their management functions and return to doing actual research themselves. Or they are just so great from day one that they tell their bosses they don't want to administer grad students/junior researchers.

Even the particularly successful/well known ones must take care to not fall victim of the Carl Sagan effect. See https://www.jneurosci.org/content/jneuro/36/7/2077.full.pdf

TLDR: Some current Einsteins may be hiding, in case their peers think they are slacking by wasting time making their discoveries well know by laypeople, and becoming famous, instead of doing their job.

> one time proven trick is assembling a large stable of grad students

Not likely to be able to achieve that in the UK!

I’ve worked in industry supporting UK govt research by helping to define research programmes. This might involve doing horizon scanning to identify tech that could address a customer requirement and then defining risk reduction tasks that would raise the technology readiness level (TRL). Industry could then bid to execute these risk reduction tasks, which would then involve a mix of engineers and academics as required.

That’s all a bunch of corporate speak that I, as a researcher, am unable to understand, and cannot see why that would be attractive to me in terms of defining my research direction.

So are you are a corporate spook for QinetiQ then?

My rule: is your research idea likely to bring money to the table in 10 years or less? Don't research it! There is enough of this kind of research going on already.

When a mommy DARPA Grant and daddy Defense Contractor love each other very much...