News for hackers. Stuff that matters.
Re:Boston Housing Price Dataset was Removed from scikit-lear... by josephcsible on Friday December 30, 2022 @04:53PM (#34190555)
The "B" variable measures how integrated a neighborhood is, and that snippet seems to be saying that its existence is the "ethical problem" that led them to purge the dataset. How is any of that different than what I said?
[dead]
Re:Boston Housing Price Dataset was Removed from scikit-lear... by trinsic2 on Friday December 30, 2022 @05:01PM (#34190615)
Stop what?
Re:Boston Housing Price Dataset was Removed from scikit-lear... by anigbrowl on Friday December 30, 2022 @07:52PM (#34192020)
Expressing incredulity over one piece of context without acknowledging the existence of other parts which modify it. There was a full detailed explanation at the medium blog, and all that was necessary to do was follow the citation given and read it. Repeatedly proclaiming confusion when already in possession of the explanatory material, rather than challenging that explanation, is not a credible posture.
Re:Boston Housing Price Dataset was Removed from scikit-lear... by trinsic2 on Saturday December 31, 2022 @11:34AM (#34198013)
If you say so...
Re:Boston Housing Price Dataset was Removed from scikit-lear... by threeseed on Friday December 30, 2022 @05:03PM (#34190627)
Please provide a source that says that "number of blacks in my neighbourhood" is a measure of "neighbourhood integration".

It's a ridiculous and offensive premise from my perspective.

Re:Boston Housing Price Dataset was Removed from scikit-lear... by josephcsible on Friday December 30, 2022 @05:33PM (#34190902)
> Please provide a source that says that "number of blacks in my neighbourhood" is a measure of "neighbourhood integration".

It isn't. Bk is "number of blacks in my neighbourhood" as you put it, and the whole point of using B instead of it was so that an all-black neighborhood wouldn't count as more integrated than one with a mix of races.

Re:Boston Housing Price Dataset was Removed from scikit-lear... by CoastalCoder on Friday December 30, 2022 @05:44PM (#34191006)
> Please provide a source that says that "number of blacks in my neighbourhood" is a measure of "neighbourhood integration".

I wonder if the dataset design makes more sense in the context of Boston in particular: [0].

[0] https://en.wikipedia.org/wiki/Boston_desegregation_busing_cr...

Re:Boston Housing Price Dataset was Removed from scikit-lear... by CoastalCoder on Friday December 30, 2022 @05:45PM (#34191027)
> It's a ridiculous and offensive premise from my perspective.

Can you elaborate on the problem you have with this?

(I'm just trying to not guess at your meaning.)

Re:Boston Housing Price Dataset was Removed from scikit-lear... by stevenbedrick on Friday December 30, 2022 @06:17PM (#34191300)
So, the issue is more subtle than that - if it was just a matter of including demographic data, there wouldn't be an issue. The problem is that the "B" column is _not_ measuring how integrated the neighborhood is- it is a transformed value whose calculation begins with data about integration levels, but the details of how and why that transformation is performed are super important. The calculation the original authors did to produce that column rests on a model about the relationship between a neighborhood's level of integration and its property values, and that model's assumptions are frankly racist and also factually incorrect (and were known to be incorrect at the time of its original publication back in the '70s). As a result, if one were to _use_ the "B" column in a model, one would be getting results that at best were wrong and useless, and at worst would make a model that literally encodes broken 70s-era ideas about how real estate and race interact in the US.

And the transformation itself is non-invertible, so it's not possible to recover the original values for about 7-8% of the rows in the dataset. The commit diff links to a thorough investigation of the data[1] in which the author takes a crack at linking up the ambiguous rows with the original 1970 Census data that supposedly went in to generating this dataset, and long story short, it looks like the original dataset's authors may have made some errors in their calculations on top of everything else.

1: https://medium.com/@docintangible/racist-data-destruction-11...