Re:The Rise and Fall of 'Social Bot' Research (2021)
by mike_hearn on Monday January 24, 2022 @04:11AM
(#30055106)
"Section 3.2: the 10%+ of already-deleted accounts were removed."
You've been implying that this is a methodological flaw of the paper we're talking about but Section 3.2 is an attempt to replicate someone else's study, one which claims Botometer isn't flawed. As they clearly point out the study wasn't replicable because the original authors didn't record/supply the data used to do the original classification, just numeric user IDs. Non-replicable research is exactly the kind of problem they're calling out here!
Gallwitz and Kreil have pointed out by this paragraph that the way the accounts were pre-filtered will already avoid Botometer's biggest weaknesses. Yet, despite this fact, "A surprisingly high number of the 121 alleged bots in our sample are in reality individuals with academic or professional credentials, many of them directly related to the topic of vaccines" and "A number of the 121 alleged bots are, in reality, the official Twitter accounts of health-related organisations", and "some of the alleged vaccine bots are, in reality, Twitter accounts related to agricultural topics and pets".
Section 3.2 points out that the study isn't replicable, which is by itself sufficient to discard it. Yet even when this problem is ignored and a replication is attempted anyway, a convincing takedown is easy to mount. It shows clearly that the authors never bothered to look at their "true positives" and is thus a great example of the problem in action.
If you think it's easy to do a much better rebuttal to all these bad papers than these two German authors managed, great, let's see it? It sounds like you would be pointing out flaws in a widely used competitor, after all. I agree completely that Botometer is dumb and data starved, but where we seem to disagree is that you think it's possible to give social 'scientists' better tools. I don't think it is. Users consent to sharing their data with services when they tweet, post, etc but they don't consent to sharing it with random academics. The whole field of academic social bot research is DOA because of this problem, yet, academic institutions like universities, journals and granting bodies systematically cannot accept this. Anything they produce will always be a data starved PRNG.
You've been implying that this is a methodological flaw of the paper we're talking about but Section 3.2 is an attempt to replicate someone else's study, one which claims Botometer isn't flawed. As they clearly point out the study wasn't replicable because the original authors didn't record/supply the data used to do the original classification, just numeric user IDs. Non-replicable research is exactly the kind of problem they're calling out here!
Gallwitz and Kreil have pointed out by this paragraph that the way the accounts were pre-filtered will already avoid Botometer's biggest weaknesses. Yet, despite this fact, "A surprisingly high number of the 121 alleged bots in our sample are in reality individuals with academic or professional credentials, many of them directly related to the topic of vaccines" and "A number of the 121 alleged bots are, in reality, the official Twitter accounts of health-related organisations", and "some of the alleged vaccine bots are, in reality, Twitter accounts related to agricultural topics and pets".
Section 3.2 points out that the study isn't replicable, which is by itself sufficient to discard it. Yet even when this problem is ignored and a replication is attempted anyway, a convincing takedown is easy to mount. It shows clearly that the authors never bothered to look at their "true positives" and is thus a great example of the problem in action.
If you think it's easy to do a much better rebuttal to all these bad papers than these two German authors managed, great, let's see it? It sounds like you would be pointing out flaws in a widely used competitor, after all. I agree completely that Botometer is dumb and data starved, but where we seem to disagree is that you think it's possible to give social 'scientists' better tools. I don't think it is. Users consent to sharing their data with services when they tweet, post, etc but they don't consent to sharing it with random academics. The whole field of academic social bot research is DOA because of this problem, yet, academic institutions like universities, journals and granting bodies systematically cannot accept this. Anything they produce will always be a data starved PRNG.