I've been funded by DARPA in one way or another and DARPA is not big science. DARPA is an applied R&D agency, meaning they want to see a path from what you are talking about to some broad application, preferably in the military space. If you can't describe that path to them, the odds of getting funded by them are low (this is drawn from experience of winning more than 10mil in funding from DARPA on many different programs).

The DARPA model is also not big science because the researchers on the outside do not pick the problems, the PMs at DARPA do. The model works when a visionary becomes a PM and is given a "big" (20-80 mil) bucket of money to disperse to researchers to enact their big ideas. If the PMs are truly visionary, then this works. If they are not, you wind up with a sort of soup of crap.

Additionally, since it's applied, DARPA will do frequent (once a quarter and once a year) check-ins, measurements, and progress reports. If you don't measure up during those, your funding gets cut. So as a researcher, it is a challenge if what you want to do is a little bit off the path of what the PM wants to do.

I have much more limited exposure to IARPA but it seems to be the same way there.

I have more experience with NSF, which is the total opposite: researchers propose their own projects and there are infrequent touchpoints, and no cut points really. However, NSF will give you one to two orders of magnitude less money than DARPA.

I work on several DARPA programs and agree with most of this. But some programs are by design far more basic or applied than others. Also worth noting - DARPA generally doesn't pay contractors anything like what the private sector pays employees.

The programs (and their vision, and their success) depend quite a bit on the PM, and on their ideas and engagement. To a lesser extent success depends upon the SETAs as well.

> DARPA generally doesn't pay contractors anything like what the private sector pays employees.

Eh. You can get like a $200/hr rate. If you're a small company with low overhead, you can get paid quite competitively with the private sector in base comp. Difficult to match big G stock grants, though, no one is becoming a millionaire off of this work.

Unless of course what you do on the grant can be turned into a billion dollar company, because (generally) contractors walk away from research programs with liberal rights to the IP they create, with the government retaining some rights to use, but very rarely (IME) retaining direct ownership. Of course, it's probably not that likely that what you do can turn into a billion dollar company but hey, one can dream...

Thankfully one doesn't (usually) need to create a billion dollar company to become a millionaire.

Gosh, this is an interesting thread...

Request: can you give an AMA on how one actually proposes/gets funding from darpa or at least an ELI5 on the topic...

What sort of idea should one have where you say to yourself "gee, I should hit up darpa with this!"

Honestly there is a truth and a super-truth, to use theater terms.

The truth, is that you should look at the Broad Agency Announcements (BAAs) that DARPA, specifically the Information Innovation Office (I2O) makes. There will be big PDFs that have lots of boiler plate and look really boring, but these are basically documents written by a program manager (PM) that describes what their envisioned research program is, how they see it being divided up, and what kind of researchers they see doing work in each area of their research program. A research program is split up into one of many (between 3 and 8) technical areas (TA) and there will be language in the BAA describing how many TAs one performer might propose to.

Overall a TA can be described at a high level as one of a few roles: a blue team, a red team, and a white team. There will usually be more than one blue team, at most one red team, and precisely one white team. A blue team does the R&D on the program. Each blue team can have their own unique approach to addressing the research challenges laid out by the PM. Usually the blue teams have competing or complimentary approaches, it is seen as a waste of money to pay two blue teams to do exactly the same thing. A red team evaluates the work done by the blue team. Exactly what this means depends on what the blue team builds. The white team "holds the room together" by fitting the PMs vision together with what the blue team does. The white team usually has a much closer relationship with the PM and usually the white team is pre-determined by the time the BAA is made public.

Blue teams are usually teams of 2-4 companies / universities. These teams usually form before a BAA comes out due to people "in the know" getting a rumor about an upcoming program, or just after it comes out, based on prior relationships. It's kind of rare to have a blue team that is only one entity, but depending on the entity and the size of the program, it can happen.

The path to a successful proposal is demonstrating that your team has ideas that are relevant to the PM and that you can successfully execute them. You show this by having a stack of prior work, good ideas, and a well formed and coherent proposal. Think of the proposal as an audition - if you as a team can't get it together for a month and write a 30 page document describing what you want to do, you probably can't keep it together for 4 years working on something. Your team reads the BAA, does some analysis, figures out how your ideas and past work can apply to the research program proposed by the BAA, and then you write the story up. If you do a good job, and your story is more compelling than other people, you get the money.

The super-truth is that it's an old boys club with a lot of luck and nepotism.

The way people get "in" is when their peers go to DARPA to be a PM. Then you, the researcher, can think "Well Dr. Smith knows about X and likes X+Y, I do some Y, I have an idea that could use some funding, what would Dr. Smith like to read about." Then you write it up and e-mail it to them.

Luck is a big factor here because PMs get e-mails like this all the time. It's almost essential to already be in the PMs rolodex/friend group to get some of their cycles. They like to say that they want to have people outside their circle approach them with new ideas but that's kind of a white lie, there are lots of people in the world and a finite amount of the PMs time.

Seems like most of the best opportunities in general are exactly like this. It's not about what you know, but who you know...

Oh it's worse than that, it's what you know and who you know.

It's not who you know, it's who knows you.

If you see humanity as a brain and a human like a neuron, the network produces outcomes unimaginable from looking at the individual. Which is easy if you look at an individual stone-age human, and even they are actually more advanced thanks to culture and groups (tribes) than an individual human would be (which would be more akin to a "Mowgli", grown up without learning from and support of the group).

So it's no surprise who you know is important - the brain is mostly about connectivity, not about the individual neuron.

Just a model for thought, obviously not a complete description ("every model is wrong"). I think the value of this model, if you can warm up to it, is that you stop worrying that "it's all about the connections" - because it really is and it's useful that way because that's a major point of how a network works. So do work on your connectivity! And also just like in the brain, a few high-quality connections are worth much more than a thousand low-quality ones.

Fantastic insight. Thank you.

I think the biggest difference is the funding went from unrestricted to being very specific. A lot of questions are asked before money is given. But the answers for an interesting project may involve concepts and vocabulary that do not yet exist, ideas that are not currently trendy. While I can understand the desire to avoid misallocation, this model assumes the funder has a better idea of what needs resources that the funded. From the second point in the answer:

> 2. Fund people not projects — the scientists find the problems not the funders. So, for many reasons, you have to have the best researchers.

So yes, the total amount may be there but is now diverted.

> While I can understand the desire to avoid misallocation, this model assumes the funder has a better idea of what needs resources that the funded.

Reminds me of a (probably the) reason we're doing standardized testing in schools. A decent teacher is able to teach and test kids much better than standardized tests, but the society needs consistency more than quality, and we don't trust that every teacher will try to be good at their job. That (arguably, justified) lack of trust leads us as a society to choose worse but more consistent and people-independent process.

I'm starting to see this trend everywhere, and I'm not sure I like it. Consistency is an important thing (it lets us abstract things away more easily, helping turn systems into black boxes which can be composed better), but we're losing a lot of efficiency that comes from just trusting the other guy.

I'd argue we don't have many PARC and MIT around anymore because the R&D process matured. The funding structures are now established, and they prefer consistency (and safety) over effectiveness. But while throwing money at random people will indeed lead to lots of waste, I'd argue that in some areas, we need to take that risk and start throwing extra money at some smart people, also isolating them from demands of policy and markets. It's probably easier for companies to do that (especially before processes mature), because they're smaller - that's why we had PARC, Skunk Works, experimental projects at Google, and an occasional billionaire trying to save the world.

TL;DR: we're being slowed down by processes, exchanging peak efficiency for consistency of output.

> A decent teacher is able to teach and test kids much better than standardized tests, but the society needs consistency more than quality, and we don't trust that every teacher will try to be good at their job.

It's more about it being way to hard to fire bad teachers and standardized tests, in theory at least, are an attempt at providing objective proof of bad teaching that won't be subject to union pushback and/or lawsuits.

I think a lot of the desire for standardized testing would dissolve if principals could just make personnel decisions like normal managers. And, of course, it's up to their superiors to hold them accountable for being good at that job.

> ...we're losing a lot of efficiency that comes from just trusting the other guy.

I think people have been trusting public schools a lot. That generally worked out great for people in affluent suburban districts. And it worked out horribly for people in poor, remote, and urban districts.

Well, that's my point expressed in different examples. Standardized testing makes the system easier to manage from the top, and makes the result consistent. Firing "bad teachers" is a complex problem - some teachers can get fired because they suck at educating, but others just because the principal doesn't like them, etc. With standardized rules and procedures, you try to sidestep the whole issue, at the cost of the rules becoming what you optimize for, instead of actual education (which is hard to measure).

> I think a lot of the desire for standardized testing would dissolve if principals could just make personnel decisions like normal managers.

That could maybe affect the desire coming from the bottom, but not the one from the top - up there, the school's output is an input to further processes. Funding decisions are made easier thanks to standardized tests. University recruitment is made easier thanks to standardized tests. Etc.

> I think people have been trusting public schools a lot. That generally worked out great for people in affluent suburban districts. And it worked out horribly for people in poor, remote, and urban districts.

That's the think I'm talking about. I believe it's like this (with Q standing for Quality):

  Q(affluent_free) + Q(poor_free) > Q(affluent_standardized) + Q(poor_standardized)
  Q(affluent_standardized) < Q(affluent_free)
  Q(poor_standardized) > Q(poor_free)

E.g. education without standardized tests may be better for society in total in terms of quality, but then it totally sucks to be poor. Standardized educaion gives mediocre results for everyone.

The usual "business school" approach to preventing metrics from causing unintended effects is to assemble a balanced scorecard of a dozen or metrics. If you do it right the negative factors of each cancel each other out. So for example to evaluate public school teachers instead of just looking at student improvements in standardized test scores relative to their peers you could also factor in subjective (but quantified) ratings from principals / students / peers, number of continuing education credit hours, student participation levels in extracurricular programs, counts of "suggestion box" ideas, etc.

http://www.balancedscorecard.org/Resources/About-the-Balance...

Public education is a bad comparison because it drags in a lot of contentious political debates about how we deal with poverty, racism, difficult home environments, taxation, etc. Based on a fair amount of reading on the subject I think the primary thing driving the standardized testing push is the dream that with enough data we'll find some miracle teaching technique which will avoid the need to tackle the actual hard problems.

Agreed. Optimization for process over outcomes seems to be a recurring problem in large organizations. Unfortunately it's not hard to understand why decision maker's prefer this approach given the pressures of shareholders/voters. No one gets fired for following the process.

A similar issue was just discussed yesterday in a thread on Jeff Bezos' letter to shareholders:

"A common example is process as proxy. Good process serves you so you can serve customers. But if you’re not watchful, the process can become the thing. This can happen very easily in large organizations. The process becomes the proxy for the result you want. You stop looking at outcomes and just make sure you’re doing the process right. Gulp."

https://news.ycombinator.com/item?id=14107766

Standardized testing has also restricted teachers to standardized teaching. They must complete X number of units from the binder every Y number of days. This leaves little time for the teacher to come up with their own lesson plans. I think there needs to be a balance between a teacher's autonomy in the classroom and standardization. I've heard many teachers complain while I was in school that they felt like they don't have the time to focus on the things they want to do in class. In my elementry school, the class would go to a different teacher for either STEM or liberal arts. I think this is a good start since because teachers can solely focus on material that is related, they can tie in lesson plans and make sure that the standardized subjects are taught efficiently to leave time for the other topics they want to focus on.

I remember in middle school, I had a couple teachers who would say "ok, this is the standardized stuff I have to say" and then go into a very boring lecture of what we just had been learning except the way we first learned it was interesting and engaging. They were simply dotting their i's to make sure that they did their job in conveying the material the state wanted us to know.

But is it really a trade-off , for every teacher or scientist ?

Sure , some teachers could use the freedom to teach better(and create new teaching methods) , but statistically , won't most teachers perform better when driven by above average methods ?

And as for research , sure we need those big breakthroughs , but we also need many people to work on the many small evolutionary steps on each technology . Surely we cannot skip that ?

The issue here is IMO that the trustless processes deliver consistency and reliability at such huge costs to quality of outcome, that the outcome becomes below average. In other words, an average teacher could do a better job free than directed by standardized testing, but the results would be so varied in quality and direction[0] as to be useless within the current social framework. And it's not that the society couldn't handle it, it's that when building systems, we want to make things more consistent, so they can be more easily built upon.

Basically, think of why you build layers of abstraction in code, even though it almost always costs performance.

--

[0] - e.g. one teacher focusing on quality math skills, another on quality outdoors skills, etc.

Layers of abstraction in technology not always cost in performance, when you look historically and system wide. For example the fact that digital technologies we're modularized(physics+lithography/transistors/gates/../cpu's/software developed without tight coupling/optimization) allowed faster improvements across the whole stack , according to some researchers, who also think, that in general, modular technologies advance faster. And modularity always requires some standardization.

And as for education, i don't think standardization always leads for performance loss. For example ,the area of reading has seen a lot of research, and one of the results is "direct instruction", one of the best methods to teach reading, and is a highly standardized method.

But maybe what's your saying is true for standardized tests in current implementation.

You always trade something off when you're abstracting. Some things become easier, but at the expense of other things becoming more difficult.

To use your example - more modularized technologies are more flexible and can be developed faster, but they no longer use resources efficiently. A modular CPU design makes design process easier, but a particular modular CPU will not outperform a hypothetical "mudball" CPU designed so that each transistor has close-to-optimal utilization (with "optimal" defined by requirements).

Or compare standard coding practices vs. the code demoscene people write for constrained machines, in which a single variable can have 20 meanings, each code line does 10 things in parallel, and sometimes compiled code is itself its own data source.

--

The way I see it, building abstractions on top of something is shifting around difficulty distributions in the space of things you could do with that thing.

Sure, in the small, a few developers working - fully optimizing stuff usually works better.

But on bigger design spaces, when you let hundreds of thousands of people collaborate, create bigger markets faster, and grab more revenue - you enable a much more detailed exploration of the design space.

And often, you discover hidden gold. But also - if you've discovered you've made a terrible mistake in your abstractions - you can often fix that, maybe in the next generation.

This is one of the most important points

> It's probably because of the mythos of private entrepreneurship and the uncomfortable narrative of all this so-called "free market capitalism" being supported by billions in govt funds...

I think this is a really bad straw man. There's broad bipartisan support for funding research into good ideas, assuming it's done the right way. The problem, though, is that the billions in government research spending we already allocate are subject to incredible amounts of lobbying and earmarking.

We'll have a NASA allocation, but it has to take place in the state of a particular Senator. Or the research will be funneled through the department of defense and received by defense contractors.

And, besides, for every DARPAnet, we have many "Where does it hurt most to be stung by a bee?" (1).

It's hard to evaluate whether we would still end up in a net positive situation. I don't have hard feelings against people who are both enthusiastic and skeptical about government funding grants.

(1) https://www.flake.senate.gov/public/_cache/files/ef6fcd58-c5...

I really dislike research hit jobs like the link you posted. At best they are disingenuous, at worst they are outright lies. The funding attached to the projects is for the total grant amount provided, which may fund multiple projects, staff, or overhead.

From the link:

METHODOLOGY Specific dollar amounts expended to support each study were not available for the projects profiled in this report. Most were conducted as parts of more extensive research funded with government grants or financial support. The costs provided, therefore, represent the total amount of the grant or grants from which the study was supported and not the precise amount spent on the individual studies. This is not intended to imply or suggest other research supported by these grants was wasteful, unnecessary or without merit.

Agreed. I poked around for more primary sources but the office of a senator seemed less controversial than partisan news organizations. Actual links to studies would be less useful since the "interesting bit" would be a note about funding coming from the NSF or something.

It's an incomplete and distorted picture of how bureaucracy funds research. I brought it up to add counterpoints and context to the implication that government is actually good at this sort of thing if we'd just get over free market capitalism (or something).

You think the office of an avowedly partisan senator would be less contentious than a putatively 'neutral' news organization?

Than the sources I found? Yes.

> And, besides, for every DARPAnet, we have many "Where does it hurt most to be stung by a bee?" (1).

Did they invest the same amount in these two?

They invest more than DARPAnet in programs that get earmarked. Entire jets and battleships are researched, developed, and manufactured for political reasons.

I was saying that just pointing out that programs like DARPAnet might not make up for all the other nonsense.

And we can flip that around as well, maybe PARC was spun off from some successful grants, but we're not taking into account these sorts of failures when attaching the price tag to those initial grants.

Siri was an inferior product comparing to others not funded by government (even after Apple took over). So that is not a glorious example of Government funding. Government is extremely ineffective, and most funds are waste. But sure some stuff will appear from all that loot.

Inefficient != ineffective. If you're accessing a resource via the Internet, you're testament that public funding of R&D can work.

That was not what the parent was saying. US Government spends large amount of money every year. They are bound to make some things work. Private sector makes some things work also. So, the question is : 'which should be preferred' ? If you agree that government spending is less efficient in making things work, you should prefer private spending. Unless you believe there are things private spending cannot do which is a reasonable case to demand public funding for research. For example, some argue that the quarter-to-quarter nature of how private companies operate precludes them from making investment in long running research. But companies like Google do run multi-year research projects.

DARPA doesn't spend that much money, a few billions a year , and has a pretty good track record , probably better than most corporations .

Yes, i mean, its not like a company locked the telephone away because they feared a damage to their gramophone disc sales. That would be shortsighted madness, company's are incapable off, if you read the history books released by their PR-Departments.