I got a bot which follows everything that gets posted, or at least it is supposed to. I built it originally to gather the data to build a model which can predict if an article will get upvoted based on the headline, later I wanted to use it to prevent my YOShInOn RSS reader and agent from posting duplicates. I learned that the two things contradict because if you scan everything quickly everything has zero votes and zero comments!

It used to use arangodb but the license for that sucks so I just tore it down to use postgres, I developed an adaptation layer that makes postgres look like arangodb, at least enough to port my applications. I added the data structures so it can recrawl posts after two weeks so it can be fresh and have accurate counts. I am talking with a friend about reranking comments so I am planning on using the bot to suck down comments for some big discussions and rerank them.