It is pretty offensive and misguided at minimum to argue that having more black people in your neighbourhood will depress housing prices. 

I think that's the wrong lens to look at this through. I'm happy to concede your statement about it being offensive is true (although I think from a purely statistical perspective, correlations with poverty, etc probably make the assumption correct. Before 2015 or so when we all lost it, it would only be racist to say there was a causal relationship between race and price, not a correlation). Anyway, that's all an aside.

It's the purging of a dataset, a toy dataset in this context, for a reason of political correctness, that I don't support. If you look hard enough at anything, you can probably find a way to call it racist or some similar slur. If we start applying this lens to tools like scikit learn, we go down a path I don't agree with, that's completely performative in terms of actually addressing any wrongs, and is a continuing distraction from what could be useful work. Debating if and how racist this is is immaterial imo to whether or not we should erase everything doesn't align with modern hypersensitivity about political correctness

> If you look hard enough at anything, you can probably find a way to call it racist or some similar slur.

Which is such an insane premise that it's hard to take the rest of your point seriously.

You're missing the fact that there are a lot of reasons the data itself is actually bad... another commenter shared a reasonable analysis: https://medium.com/@docintangible/racist-data-destruction-11...

So even if you want to ignore the fact that the data was outright used to discriminate in the past, the data itself is actually flawed in several ways...

I’m sure we can find flaws in any number of the included data sets, but we’re not excluding those, right?

Why are people singling out this one dataset?

because this one is flawed and comes with some racist presumptions made by the people that originally correlated the data (the presumption being that black people moving into a neighborhood decreases property values... a particularly white-biased correlation without causation)

> offensive and misguided

The data is the data. The data isn't suggesting that "having more black people in your neighbourhood will depress housing prices." That's your take on what a racist causal interpretation would look like.

The correlation is very real and turning a blind eye to it is worse: https://www.brookings.edu/testimonies/how-racial-disparities...

That's actually a racist causal interpretation made about the data when it was originally analyzed:

> At low to moderate levels of B, an increase in B should have a negative influence on housing value if Blacks are regarded as undesirable neighbors by Whites.

Isn't that saying that lower house prices are caused by racism?

In an incredible whitewashed way that's very familiar to anyone that lived during the Jim Crow era, yes. Note that it doesn't say that white people leave because they're intolerant of black neighbors, but paints black neighbors as undesirable.

Either way, it's a dramatic oversimplification. Black people are also significantly poorer than white people, so it's also likely that black people can only afford to move into a neighborhood as property values decrease... yet the people who collated this data chose to make a different assumption.

That's actually a racist causal interpretation made about the data when it was originally compiled and analyzed... which raises a lot of alarm bells:

> At low to moderate levels of B, an increase in B should have a negative influence on housing value if Blacks are regarded as undesirable neighbors by Whites.

> At low to moderate levels of B, an increase in B should have a negative influence on housing value if Blacks are regarded as undesirable neighbors by Whites.

Isn't that true? It's almost a tautology.

(Bad things can be true.)

It's significantly more complicated. This presumption (made by white people) assumes that black people are responsible for depressing housing values and ignores a myriad of other factors. It's such a narrow view of the problem that it makes me question the motivations of collating such data to begin with.

For example ignores the fact that on the whole black people are significantly poorer.

In Boston the average white family has a net worth of $200k+ while black families in the same city have a net worth of <$10 (that's not a typo, it's less than ten dollars). Poorer people by nature can not afford houses in more expensive neighborhoods, so naturally you have a concentration of black people in poorer neighborhoods... it's not because white people find black neighbors undesirable, it's that black people are disproportionately poorer.

This kind of oversimplification tends to perpetuate a lot of negative stereotypes about black people while hand-waiving away the chronic issues black people are faced with that creates this kind of disparity.

Offensive to who? It sounds like a great way to compare racism in different areas.

The more it correlates with a lower price, the more race issues.

How did they arrive at .63?

It's seems like a weird choice. If there are 62% or 64% then the feature will yield the same value.

I think it would make more sense just to include percentage of households where at least one member of the household has a certain ethnicity.

I don't think it's a offensive to analyze demographic information in the aggregate. In fact this happens all the time.

I wonder if that was the overall proportion for the entire surrounding area at the time. If so, then B=0 would mean "this looks like what you'd expect if where people lived had nothing to do with race", and increasing values of B would be increasing evidence to the contrary.

It is kind of strange, I don't understand why they would use that. Here's a plot of the function:

https://www.wolframalpha.com/input?i=plot+%5B1000*%28x-0.63%...

Isn't that a measure of how integrated a neighborhood is? And even accepting for the sake of argument that such a variable is evil, why not just exclude it instead of ditching the whole dataset?

a) No. If 50% of the community is African-Americans and live on one half and everyone else on the other then it would be exactly integrated. Except of course it isn't.

b) It is not scikit-learn's responsibility to alter third party datasets.

> a) No. If 50% of the community is African-Americans and live on one half and everyone else on the other then it would be exactly integrated. Except of course it isn't.

IIUC, you're arguing that "whole-town" level aggregation is misleading. So if we get more granular, we could do it my neighborhood, street, building, apartment/car/shelter, bedroom, bed, bed @ time of day, etc.

Any one of those aggregation levels could hide interesting distinctions that could be made if only the data were reported with even more granularity.

So are you arguing against aggregation in general? Or just whole-town aggregation specifically?

data is just that, not an argument in itself. how you use it to disprove something is up to you.

Except that this is not just data! The feature is just overall bad.

More bad than evil?

Even if it wasn't evil, it's still bad is my point.

Data can show a correlation. What the underlying causation is is not what the dataset aims to answer.

But it’s a fact that—due to the very racism you’re pointing out—house prices tend to be lower the more minorities there are. That’s in part because of the racist policies of banks and the real estate industry. Ignoring it doesn’t do anyone a service. Now, whether it’s used/included responsibly in this dataset is another matter entirely.