For example, lots of organisations will have diversity officers and training. Very few do things such as taking names, languages spoken (obviously you would leave any that are relevant to the job) and other clues to sex and race from job applications before shortlisting people for interview.

A signal of an act. Not the act itself. Post l-modernisms ugly head rears itself once again.

In the security realm we have security theater. In the social realm, we have compassion theater. "We are spending money on problem X" shows that the supporters of the spending care about people with problem X. Any proposed cut to that spending shows that the opponents to the spending hate people with with problem X and don't want to help them. Meanwhile, the money actually supports those in charge of the spending and various suppliers of goods and services.

I've seen this mentioned in the context of leadership. Someone said the best leaders get people to do things they wanted to do anyway. They just act as a kind of meeting point (or 'lightning rod', take your pick) that senses the general mood, and figures out a way forward that most of the group can agree with. Repeated (sometimes continuous) back-and-forth is important here.

In politics it's often the other way around: many people are unhappy, politician comes up with a plan to do something, and tries to push that through. Such action often fails as it -mostly- ignores feedback from the people the changes are imposed upon.

Acknowledging a problem is important. But more important is understanding the problem through the eyes of those affected by it, understanding how an action would change things from that point of view, and adjusting control knobs as you go. Few leaders are good at ALL those things.

But as software people we all know that turning a measure into a target yields problems.

Measuring success is good, but we also need to exercise the ability to make small updates and continue measuring, instead of just using that data to argue on the side of "throw it all out!" or "we can't touch it, it's essential!"

Not having a metric set out in advance may, in fact, be more rigging resistant! unfortunately.

But the real deal is that none of it matters because the program itself is the goat sacrifice. It's not there to work, it's there to signal the proponents values and ingratiate themselves with the goat suppliers. It's actually better for theses purposes if it DOESNT work, because then you can perpetually campaign on solving the same problem.

For example, if you implement a free lunch program for all public school students in a state, you actually get to subtract out the amount of money that each school district spends dealing with reduced and free lunch program qualifications for a subset of students, all of the paperwork involved in maintaining that list, and the system that manages their account balances at the cafeteria register.

This example provides an easily understood illustration of why so many social programs fail. Simply put, the people proposing and creating the programs have zero understanding of the people whose lives they are trying to influence and control. Like generals ensconced in their offices located not even on the same continent as the front line they study maps and tables and hear the advice of experts with even less direct experience than themselves and then dispatch orders that arrive too late and attempt to command a reality that does not exist. The reports they receive back show nothing has changed or the situation has got worse and yet they never adapt or see their own fault. Why should they? They've suffered no consequence.

https://www.health.gov.au/sites/default/files/study-of-the-i... (page 12)

They also tried quitting more often (p=0.013)

https://pmc.ncbi.nlm.nih.gov/articles/PMC3710988/

“That’s why the worst possible reaction to a failed intervention is resignation. We may not know enough to improve our lot right now, but perhaps we could figure it out, if we tried.”

From reading the whole post, I gather that he’s truly not advocating some libertarian angle, he really is writing on the observation that policy is hard, and it’s nearly impossible to actually predict all the effects of societal intervention.

From my reading, I’m not sure even sure that he’s advocating caution or reluctance to social policy. Just goat sacrifices!

It is saying, acts must be done to reaffirm the commitment to the goal, the goat is a litmus test as to whether the goal is worth it. Not to abandon the goal, some things are worth the sacrifice, but to be aware of it.

There was a post here about Mississippi miracle https://en.wikipedia.org/wiki/Mississippi_miracle recently which was due to one of the programs mentioned in the original post claiming it produced no improvements.

As in the original post is flat put lying to push an agenda. You can google for like 2mins on many of the points and see counter evidence.

It's a bit discouraging that there's no clear answer to that question even on this relatively well-studied intervention.

To be clear, I don't want to play the game of paper vs. paper. I simply think there is more depth of detail here than can be nicely summarised in a comment.

You can still try and revert back if it fails.

It is not a surprise that when people are given virtually no power to do anything, they have a hard time accomplishing very much.

Unintended consequences are inevitable. The danger isn't that they'll exist - it's that they'll be ignored entirely or ignored until they cannot be, that the desire for panacea produces a willful blindness to their existence or myopic fixation on their effects to the point where the value of the intended consequences are ignored.

Another relevant example here is vaping: an amazing win for harm reduction in existing smokers, but an attractive on-ramp to the addiction for some who would have avoided it otherwise. No panacea.

Yes, usually there are unintended consequences. The most important unintended consequence here was the anti-vaping sentiment engendered by the anti-smoking campaign choosing to conflate nicotine with the harms from inhaling combustion byproducts and tacitly embracing a cateforical, puritanical morality regarding substance use.

This is documented, but it is not clear how much this is due to packaging vs. Australia having the highest tobacco tax in the world.

> and to smoke more

{{Citation needed}}

This claim is directly contradicted by this Cochrane review. https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD...

"Findings were mixed amongst the four studies assessing consumption, with some studies finding no difference and some studies finding evidence of a decrease."

This also ignores that the intervention was intended to reduce the number of people taking up smoking and to encourage current smokers to quit, which appears to have happened.

> In response to the policy, smokers switched from more expensive to cheaper cigarettes and reduced their overall tobacco expenditure and expenditure intensity. However, as smoking became less costly, smokers consumed more cigarettes.

And yes, even if that's true, it's separate from the question of how effective it was as stopping nonsmokers from starting.

I mean I think that's definitely true in some cases but it's a wide wide world out there, and the universe of things that count as "social programs" is impossibly vast, and some of them are quite well researched and simply work when rolled out. Congestion pricing in NYC for instance was the product of policy nerds and seems like it simply worked.

I completely agree there's unintended consequences and regulation can make things worse but not to the point of a universal anti government ethos.

E.g. if the tobacco industry is forced to choose between either hard bans on smoking, or a highly visible but largely symbolic measure like changing the packaging, they will probably choose the latter.

Want to cut a social program because you don't want to pay for it, but don't want to look overtly evil? First, shoot out the kneecaps in the name of efficiency. Second, notice that it can't walk anymore and cut it entirely. Optional third step: give a high-minded speech about how attempts to solve problems are doomed so we shouldn't even try, or shouldn't try without an infinite amount of meetings first.

This happens a lot.

WASHINGTON (AP) Recent reports indicate the federal goat sacrifice program has failed to produce a meaningful improvement in the success rate of public policy over the past five years.

The program, which costs taxpayers over $1 billion annually at a minimum of $1 million per goat, has also had little impact on out of control spending in unrelated government programs, a finding that reportedly surprised the bill's original cosponsors.

Advocacy groups have condemned the practice as inhumane. Proponents, meanwhile, blame rising costs on the $300 million spent each year on therapy for all individuals involved. Critics are now questioning how much of that funding actually goes toward mental health care, and whether more attention should be paid to the care of the goat prior to slaughter.

> That would be a beautiful study, but it wasn’t the study that was actually conducted. The “lottery tickets” in question were in fact football cards, each one with a picture of a player, their team, etc. Some participants were given the opportunity to choose their football card and some had a card handed to them. Later, just before the drawing, all of the participants were asked how much it would take to buy their card back. On average, the pickers asked for $8.67, while the not-pickers said $1.86.4

> I’m sure you can see the obvious and less exciting explanation for these results: when people were given the chance, they picked their favorite players. The people who didn’t get to pick ended up with a card they liked less. This “illusion of control” effect is merely a “reality of people having preferences and acting on them” effect.5

Alternatively, people who are knowledgeable of a subject will have selfishly biased opinions about it, which they value more than unknowledgeable people will value an arbitrary stance on the subject. From experience, this seems to affect things beyond situations of chance like lotteries, and infests collaborations like businesses in ways such as Not Invented Here Syndrome or generally poor management choices that fail to consider situational factors (e.g. overarching decrees that are ineffective in certain situations/locations/applications).

And the ignorant are always the loudest.

The fresh minted safety guy with his shiny new clipboard and vest is a fool running around the plant spewing rabid idiocy straight out of the internet comment section.

His boss who's been doing things for a decade is more measured, knows you can't tell a guy how to use a screwdriver if you also want to tell him how to use a ladder

Silliness of the highest order. I still have those numbers written down. They've got the same chance as any other set - but this set are _mine_. I might just be the only person alive who was there, now I reflect upon it. Huh

The idea that such ceremony induces humility may be true in a limited individual sense, in that a person may come to believe that they are nothing but their deity is everything. But at the same time it elevates the status of whoever is the current puppet master of their deity. The end result is net uptick in hubris, on behalf of their lord, whose relayed word is sufficient without any need to resort to such crudities as science or debate.

> For example:

> In 2009, Congress spent $3 billion on “School Improvement Grants” for the worst-performing schools in the country. As far as we can tell, those grants did nothing.

> Universities that hired a Chief Diversity Officer between 2001 and 2019 did not end up with more diverse students, faculties, or administrators.

> The American government spent $280 million to help Afghan women get jobs, and now they can’t really say whether any women got any jobs.1

> Remember 20 years ago, when giving laptops to kids in developing countries seemed like the most important thing to do? Well, a 10-year RCT of 531 rural schools in Peru found zero effects of “One Laptop Per Child” on academic performance.2

> Researchers developed four different messages that were supposed to convince parents to vaccinate their kids. None of the messages worked, and one of them accidentally made parents more likely to believe that vaccines cause autism.

Something that needs addressing on all but the laptop one: there are parties who had a vested interest in seeing these example programs fail. In order:

* people who think public schools exist only as instruments of mass indoctrination for secular values and who would rather not have their money go to that, and/or would like to be the investors in "market-based" solutions to America's supposed education problems

* people who think that anyone and everyone who isn't white, male, middle-class-or-above, and Judeo-Christian gets into college programs because the government forced them in there, and find other ways to get those people to leave, because they see them as a threat to their freedom to do and say whatever they want in the spaces of higher education, and a threat to their sense of intelligence and superiority

* the Taliban. I hope I don't need to explain that one further.

* medical grifters who can't push you things like bleach enemas and colloidal silver if you aren't sick because you got vaccinated, among others

I can't speak to what happened in Peru, but these are all examples where you can easily point to people in American or Afghan society participating in the efforts in bad faith, and thus sabotaging them.

Solution:

Make graffiti classes mandatory in school. Ensure they are taught by gym teachers with no discernible artistic interest or talent. Subsidize a Graffiti text book industry. Create schools of approved graffiti. Launch a "America's Got Graffiti" television show with celebrity randos.

Graffiti will be dead within ten years. :-)

If corporate HR departments and schoolmarms do it, it's not cool.

I had a similar idea about vaccination back during COVID. Have Hillary Clinton be quoted saying it's good that Republicans aren't getting vaccinated so that way they'll die of COVID. Then make Trump be quoted saying they should discourage Democrats from getting vaccinated so they die of COVID. Result: everyone gets vaccinated.

Looking at policy-making it seems we routinely mistake having a good reason to want something to happen for having good evidence that our proposed intervention will actually make it happen.

Perhaps the crucial missing ingredient is a requirement that policies contain their own mechanisms for being proved wrong?

You have to make sure your end goal is realistic and attainable. The Soviet example is extreme, but sometimes even changes that look deceptively simple aren’t realistic due to deeply embedded social issues or plain human nature.

Generally, though, I agree that our social priorities seem to be misplaced.

If I'm investigating a bug and find that the program is still misbehaving after my fix, my usual reaction is to dig deeper, find out what I missed or how my fix changed the program's execution. My reaction would not be "eh, better not fix anything at all".

> Most of those are left-coded failures, so here are some right-coded failures:

"I'm neither left nor right, I'm just an unpolitical neoliberal..."

These interventions target very complex systems. It's not unexpected that we need to try a bunch of things that don't work before we find something that does, to the extent that most of the things we try are ineffective, right?

His proposed change at the end is that we must always guess and check (emphasis on check) for now, and we may eventually get into a "Psychohistory" level of understanding in the future, but we are far from it now.

Only if the value of whatever smidgen of knowledge you gain (assuming you do actually gain any) outweighs the much-more-than-smidgen cost of the intervention to all the people who weren't helped, and in a significant number of cases were harmed, by it.

As you can see, I don't think whatever benefits are gained actually do outweigh the costs. Which means that no, continuing to try such interventions is not a win.

The lesson we should be learning from all this is to stop trying to do so many things top down. That whenever you see some kind of large scale statistic that you think is not what it should be, the last thing you should do is concoct a large scale intervention to try to change it. Still less should you sacrifice a poor goat who did nothing to deserve it, just to underscore the point that your intervention won't work.

The goal of science can only be to learn whether it helps people, who it helps, who it hurts, etc. Then it can be implemented, or not, depending on the results.

Longitudinal studies tracking populations over time does not imply that sociological intervention experiments, especially political ones, will be reliably recreated over time.

If my hunch were to price right, anything we'd learn there would have a short half live and may not hold out past, say, a generation or two.

I can say anecdotally that my grandparents would respond differently to many situations and interventions than I would. They grew up during the Great Depression and he flew in the war, those two events alone very much impacted how they viewed things for the rest of their life.

Looking at those only a generation behind me and even they seem to respond very differently than I'd expect those in my generation would.

The question I was raising was whether said results would be useful beyond a decade or two into the future, not across different cities today.

No, paying $3B for information of dubious value and which was pretty obvious ahead of time is not a win.

The reason the article resonated with me is because policies and interventions are often sold as "this thing is self-evidently great, and if you oppose it you're an idiot", rather than "this seems promising, so let's give it a try and scale it up if it works well". When I look at proposed measures at my local government level (for example) almost all of the supporting literature focuses on why the problem is important, rather than why the proposed interventions are likely to succeed. Being "effective" as a legislator means enacting policy, so if you want to be effective you try to keep the conversation on the first part.

It's quite interesting to appreciate that these issues were observed, and written about, some 2400 years ago. Times when ostensibly everything would have been practically alien. Yet it emphasizes that the thing that's really changing is not humanity, so much as it is technology. We're still trying, and failing, to deal with the exact same issues as we were thousands of years ago.

And mind you that's just the first time it was written down. I wouldn't doubt that these observations and issues long predate even Plato. Because I think one thing that's become clear is that he wasn't necessarily unimaginably brilliant so much as he was simply capable of objectively seeing the patterns in society and recording them. Though perhaps that is some form of brilliance in its own right.

[1] - https://en.wikipedia.org/wiki/Ship_of_State

There aren't too many people comfortable with intentionally letting others suffer for the sake of the scientific method, and those who do have a hard time getting past ethics review boards.

Instead we have this reality where we throw money at problems and claim credit whenever something good happens, and claim we did good work when nothign changes.

I'm not sure what to even discuss about this article due to the low quality. I guess there overall point is that policies and interventions don't necessarily scale, or perform the same across cultures/cohorts, or even accomplish their intended goals. I would largely agree with those points, but not due to the way the article got there.

it's the surprising surprises that get you

That... that's not a failure, that's exactly the point of this requirement

As you say, it reduced the rate of drinking. It had a few other side effects.

Lots of social programs are not working to improve lives of adults that are already messed up.

Aiming at kids is the way to go.

That’s how the YouTube influencers got big, they basically groomed generations of followers.

Whereas many of the interventions mentioned in the article were seemingly logical but indirect: If we do A, it will cause people to behave B, resulting in C.

In places where you can take the metro to the bar, or walk to the bar, it's a lot different.

  * Measurement requires substantial logistical overhead; both up-front in creating a plan for measurement, and during intervention due to the extra record keeping involved. This is a (probably frequently large) drain on program efficacy. Just listen to any schoolteacher's take on standardized testing, evaluation rubrics, etc.
  * Some contexts, and some types of intervention are more amenable to measurement than others. Insisting that nothing which is not measured can be valuable biases our decision-making towards the most measurable actions, which are likely not the best. Strong correlation with simplistic actions, short timelines, and lack of responsiveness to individual context.
  * Goodhart's law.

https://www.experimental-history.com/p/the-one-science-refor...

mainly, interventions are experiments, and therefore we need more of them, you never know. but they are not experiments, otherwise we wouldn't repeat failed ones over and over, or let existing failing ones run on and on.

maybe the answer really is more simple, that most programs are indeed politically motivated endeavors, indeed largely signals of a worldview, less programs to bring about material improvement.

fwiw - i dont know the answer, but having your cake and eat it, being perceptive of the absurdity, yet, not daring to call it out, is an author's disposition, less a truth. so maybe, if we do the one hand other hand game, accept the bleak truth glimpsed in the facts, but also accept that there's nothing to be done about it within existing frameworks, and not point at it as something desirable.

The fact, it costs taxpayers' money is bad enough. It often reduces freedom of choice, introduces bad second order consequences, etc.

Generally, it is not a government's job to do social engineering.

The author is assuming good faith on behalf of the Arkansas state government that the purpose of the requirement was to increase employment, rather than merely cutting down on the number of citizens successfully receiving Medicaid.

If I ignore intent, then I would expect the outcome of the additional requirement to be fewer people on health insurance and an increase in employment. (Obviously, both of those metrics are heavily influenced by other factors, so it may just not be possible to measure accurately.)

It's unrealistic to run pilots, data collection, and followup in 4 years for many political decisions.

The best policy recommendation here is simply to enforce data collection for every decision. This would also incentivize politicans to only propose policies they truely believe will work.

Money will still be burned, but at the very least over long term our understanding will grow, and bad policies will gradually be disincentivized.

The paper's abstract says "First, we find no impact of debt relief on credit access, utilization, and financial distress on average. [...] Third, we find no effect of medical debt relief on mental health on average, with detrimental effects for some groups in pre-registered heterogeneity analysis." which is exactly the kind of meaningless result you'd expect from doing ~20 statistical tests over random numbers. (The paper lists hundreds of p-values.)

Medical debt with a face value of $169 million was wiped out for 83,401 people, yet the study failed to find any benefit. I suspect this is representative of the quality of the reasoning in the rest of this article.

On page 20 they describe using the Westfall and Young method to adjust for multiple hypothesis testing and report adjusted and unadjusted p-values.

They also pre registered their analysis with depression as a primary outcome and breaking down by medical debt quartile as pre registered subgroup analysis.

In the highest quartile of medical debt group, 45.9% percent in the no debt forgiveness group had at least moderate depression vs 58.3% in the debt forgiveness group, p=0.002

Dismissing this as a meaningless result seems unfair given that they preregistered their analysis and adjusted for multiple comparisons.

I’m not a stats expert and haven’t spent much time with the paper, but I’d suggest checking the methods section before making claims.

in other words, government is time-constrained, so when a problem is sufficiently complex, there's just not enough time to really flesh out a good solution.

also, humans can be selfish and will occasionally try to game the system, which changes your problem into an active adversary. and when you have active adversaries, you're playing a completely different game (indeed, when designing the policy in the first place you may not have even considered it a game with opponents!).

That being said, I wonder if there is a case to be made that for there to be more social & policy interventions that actually are evidence-based and more rigorous, do we need to have technocrats that are not beholden to the short-termism of the democratic political system? But wouldn't they just accumulate too much power without enough accountability? Who will watch the watchers?

The "lying flat" movement seems to persist, the willingness of the young to accept 996 jobs is close to zero, and all attempts to raise the abysmal Chinese fertility rate have so far failed. The (long abandoned) One Child Policy on the other hand worked way too well and might have cost them 200 million young people.

Social engineering is hard.

There's good evidence that we can improve accuracy on the predicted outcomes of policies, e.g. delphi method or prediction markets. See the "replication markets".

I'm no expert, but after the destruction of the Second Temple on the Temple Mount in Jerusalem in AD 70[1], animal sacrifices ended in Judaism as they're restricted to one chosen site[2], and that site has ever since then had no temple on it. If you are game enough to sneak in during the night and build a new Jewish temple on the Temple Mount, animal sacrifices may have to resume.

Although I think a red heifer is needed?

[1] https://en.wikipedia.org/wiki/Siege_of_Jerusalem_(70_CE)#Des...

[2] https://www.biblegateway.com/passage/?search=Deuteronomy%201...

Most interventions are embarked upon by working backwards from unproven (often political) theories and an extremely limited range of possible tools (“give money”, “take money”, “physically punish”). That leads to the mother of all searching-only-under-the-streetlight fallacies.

But that doesn’t necessarily indict the social science academy/frontiers of understanding (which, again, are plenty flawed—just perhaps not quite as stone-aged as the author suggests). Usually, it indicts the state/designers of the intervention in question.

I’m reminded of the conservation efforts for the spotted owl. Famous academic naturalists were asked to design a conservation plan that worked with logging. They did. The state replied “fine, but we need to clear-cut 90% of the forest you asked for, so make do with 10% of the habitat”. The program did poorly, but not because the academic understanding of how to address the problem was as flawed.

This could have been much shorter.

"Generally, you should have skin-in-the-game. If you're going to propose something that has the potential for harmful waste, be sure that this is taken into account before doing it."

https://spot.colorado.edu/~huemer/papers/passivity.htm

I went and looked at the first study on the School Improvement Grants,

    > In 2009, Congress spent $3 billion on “School Improvement Grants” for the worst-performing schools in the country. As far as we can tell, those grants did nothing.

At one end it says, "Although schools implementing SIG-funded models reported using more SIG promoted practices than other schools, we found no evidence that SIG caused those schools to implement more practices."

What are these practises? Well... it seems that as a pre-requisite to getting funding states were made to divide schools into four categories / intervention modes,

    > ED required that each SIG-awarded school under Tier I or Tier II implement one of four school intervention models. These models required specific practices:
    > 
    > • Transformation. This model required schools to replace the principal, adopt a teacher and principal evaluation system that accounted for student achievement growth as a significant factor, adopt a new governance structure, institute comprehensive instructional reforms, increase learning time, create community-oriented schools, and have operational flexibility.
    > • Turnaround. This model required schools to replace the principal, replace at least 50 percent of the school staff, institute comprehensive instructional reforms, increase learning time, create community-oriented schools, and have operational flexibility.
    > • Restart. This model required schools to convert to a charter school or close and reopen under the management of a charter management organization or education management organization.
    > • School closure. This model required districts to close schools and enroll their students in higher-achieving schools within the district.
    > 
    > These required practices can be grouped into the four main topic areas promoted by SIG. Table ES.1 lists these four broad areas and the objectives promoted by SIG within each. For example, replacing the principal falls under the “identifying and rewarding effective teachers and principals and removing ineffective ones” objective within the “developing and increasing teacher and principal effectiveness” topic area. 
    >
    > The objectives listed in the table cover all practices promoted by SIG, which different models designated as either required or permissible.

But then they start the summary by saying that there's "no evidence that SIG caused those schools to implement more practices" – the ones which were heavily promoted by the program and were a part of it?

Surely that can't be right.

What am I missing here?

Also some of these graphs look wrong. The error bars on Figure ES.3. go from ~ -0.2 to 0.25...

No, the mere act of picking imbues their ticket with a bit of guarantee that you haven't maliciously selected numbers that don't win to cheat them. And I'd expect people's psychologies to pick up on that even if they can't explain why.

Even if that actually was the study that was conducted, it still wouldn't support belief in juju as an explaination.

Such spendings exist mostly not to achieve what is told, but to make contractors more reach. Any government spending should be treated like this in the first place.

Without mentioning Effective Altruism, it explains why it's important. Yes, there are a lot of expensive, failed interventions out there.

If you want to fund something that works, the simplest way is to copy GiveWell's answers.