savagedata

Was PayPal trying to verify your identity or that you know your recipient's identity (who's coincidentally your mother)?

I'm surprised if PayPal expected you to know your recipient's birthday, but "What's your mother's birthday?" would be a common question to verify your identity. They should have moved on to another question if you had a moral objection.

On the other hand, scammers will often ship goods to a nearby address and pick them up off the porch, so verifying that you know your recipient might actually be a fraud countermeasure.

I've found them at HEB and Randall's too!

One of my favorite data science factoids is how "regression" (to return to a former state) came to mean "prediction of a continuous variable".

In the late 1800s, Sir Francis Galton noticed that extremely tall or short parents usually had children that were not as tall or short as themselves, i.e. the children's heights were regressing (returning) to the mean. He collected hundreds of data points, graphed them, and estimated a coefficient describing this relationship, thereby inventing "linear regression."

We call them "regression" models simply because the first linear regression model was created to demonstrate the concept of regression to the mean.

https://en.wikipedia.org/wiki/Regression_toward_the_mean#His...

Santa Clara County (South Bay Area) shares a dashboard with cases by age group and deaths by age group: https://www.sccgov.org/sites/phd/DiseaseInformation/novel-co...

They approximated income, education, and ethnicity at the neighborhood-level (400-600 people) based on census data. That doesn't seem granular enough to me if the poorest families in each neighborhood live closest to the major roads.

The problem is with multiple-day recovery periods after the surgery. The later in the week your surgery is scheduled, the more likely you'll overlap with the weekend when care is hypothesized to be lower quality.

> Choosing Monday for a surgery increases your chance of success 2 times

The title seems poorly worded. The researchers studied mortality rate, not success of procedures. Since these were elective surgeries, I assume the success rate was high. If your surgery has a baseline success rate of 90%, what would it mean to increase that 2 times?

The conclusion of the study was instead that the odds of death were 44% higher on Friday compared to Monday, and 82% higher on the weekend compared to Monday. Basically, there was a gradual increase in the odds of death from Monday to Thursday, but a big jump up on Friday and again on the weekend.

You're looking to test the difference in proportions. I entered 0.93 / 1000 / 0.91 / 650 into this online calculator [1] and got a p-value of 0.14 which means that the difference is not statistically significant. Technically you would want to do some correction for multiple comparisons since your actual question is "are Americans more happy than workers in every other country?" not "are Americans more happy than Netherlanders specifically?" but it's a moot point since the difference already isn't significant.

A more important question is probably how the sampling was conducted. How did they guarantee a representative sample in each country? How did they account for the gig economy, small businesses, part-time employees, and employees paid under the table? So many ways a survey like this could be wrong.

[1] https://www.socscistatistics.com/tests/ztest/default2.aspx

When Irma was making landfall two years ago, I did an animation demonstrating this for r/DataIsBeautiful. [1] The real hurricane path is in red with the forecast at each timestamp in black. If you look at the forecasts through time, you can see that they're frequently off by a distance the width of Florida.

(This final graphic is an amalgamation of all the useful comments I got on the original thread. It was a really cool experience getting so much feedback so fast, like a giant collaboration with all of Reddit. You can see the original not-so-beautiful here. [2])

[1] https://github.com/savagedata/hurricane-animation

[2] https://www.reddit.com/r/dataisbeautiful/comments/6z0w20/tim...

There are always potential issues when a machine learning algorithm is applied over time.

Example #1: Let's say that cancer rates are increasing over time and cameras are improving over time. You might end up with a weird artifact in your model that higher resolution images are more likely to indicate cancer.

Example #2: Let's say that cancer-detecting algorithms are widely successful and so someone makes an app that lets you upload images of skin and the app tells you the probability of you having cancer. Suddenly a model that was trained on suspicious lesions is being used on normal freckles that people uploaded for fun. You end up with a lot of false positives. Maybe you try to combat that by including images uploaded to the app (that you somehow obtain labels for). But now you have a model that predicts that photos taken in brightly lit medical offices are likely to be cancer and blurry images taken in bathroom mirrors are not cancer.

You could argue that Example #2 is more about the difference between training data and data to be scored, but the fact remains that outside of tightly controlled scenarios, the way data is collected nearly always changes in time and ends up affecting model performance in unexpected ways.

Thank you for the useful feedback! I'll have to look up GUIDE trees.

> This is interesting. I hear it was sqroot(total number of predictors).

I was probably looking at the randomForest R package documentation [1], which says:

> mtry Number of variables randomly sampled as candidates at each split. Note that the default values are different for classification (sqrt(p) where p is number of variables in x) and regression (p/3)

I checked the H2O implementation of random forest [2] and they use the same defaults.

I'll add a note about the one third default being specific to regression since that seems like an important distinction.

[1] https://www.rdocumentation.org/packages/randomForest/version...

[2] http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/d...

I wrote this regression tree tutorial a few years back that might be a good complement to the tutorial above since it covers regression instead of classification and goes on to talk about bagging vs random forest, out-of-bag samples, and tuning parameters: https://github.com/savagedata/regression-tree-tutorial I wrote it at the start of my career and haven't shared it beyond my study group, so I'm happy to hear feedback.

I wonder if this study takes into account that, because some professions are skewed toward women and others are skewed toward men, on average women will interact more with women than with men on a daily basis. Instances of incivility from women and from men should be normalized by number of female coworkers and male coworkers.

I wish I could access the PDF to see if the researchers accounted for this.

I couldn't find a more recent source (this is from 2013), but it claims that 12% of SF (San Francisco-San Mateo-Redwood City, CA) jobs are in "high tech" and 29% of San Jose-Sunnyvale-Santa Clara, CA jobs. My guess is the difference between the two sources is that yours is looking at the whole population, which would include children and retired folks.

Close to a third of jobs in one region being in tech is an insanely high number, and it's no wonder that people would feel that the Bay Area is lacking diversity.

https://web.archive.org/web/20130308072135/http://www.bayare...

This is my favorite statistics factoid! Regression/reversion to the mean is the idea that if you observe an extreme value and remeasure it, it will tend toward the average value on the second observation. For example, students who have a very high or very low score on a test will have a more average score if you retest them.

Sir Francis Galton demonstrated the idea of regression to the mean by inventing linear "regression." He plotted the heights of children vs the heights of their parents and showed that very tall parents have children who are not as tall.

The fact that we call linear regression "regression" has nothing to do with reverting to the mean other than, coincidentally, linear regression was first used to illustrate the concept.