Re:Almost half of local Twitter trending topics in Turkey ar...
by telchar on Wednesday June 02, 2021 @09:48AM
(#27368620)
I suspect this rather obvious flaw (from the article: not taking deletion into account when calculating trends) is due to it being easier to create a data pipeline that is purely additive at the scale they're operating at. Not having to consider deletions and updates simplifies development. This is too much of a vulnerability not to fix, though. It amazes me Twitter somehow manages to remain relevant with how broken it is in various ways.
[ Parent
| on HN ]
Re:Almost half of local Twitter trending topics in Turkey ar...
by zozbot234 on Wednesday June 02, 2021 @09:57AM
(#27368745)
Deletion is easy if you're using ML based on feature vectors to extract "trends", because you can just add one of the opposite vector and the system is still purely "additive" at scale.
Re:Almost half of local Twitter trending topics in Turkey ar...
by a1sabau on Wednesday June 02, 2021 @10:40AM
(#27369213)
Not if you're using probabilistic data structures like Count-Min[0]. You don't care about the exact count. You just want an estimate with a certain probability. Such structures only support addition, not removal.
[0] - https://en.wikipedia.org/wiki/Count%E2%80%93min_sketch