Re:Ask HN: What are the best tools for web scraping in 2022?
by olvy0 on Wednesday August 10, 2022 @09:33AM
(#32411624)
Recently I wrote a small firefox extension to scrape only the Hacker News submissions which I upvoted, and also my own submissions, and create browser bookmarks from them.
It also scrapes all my comments I upvoted and if those have links inside them it creates bookmarks from them too.
That's because I often find myself searching for some submission I upvoted but can't find it, especially if there were many similar ones, whereas Firefox bookmarks manager has a nifty search feature...
I had to scrape since the HN API doesn't expose ability to get information about upvoted submissions/comments. The extension assumes you are logged in, it doesn't ask for your username or password.
It's mostly for my own use and not very polished, but it works. I uploaded it to the Firefox extensions gallery, and you can probably find it there, but I don't think it's ready for public consumption yet...
It also scrapes all my comments I upvoted and if those have links inside them it creates bookmarks from them too.
That's because I often find myself searching for some submission I upvoted but can't find it, especially if there were many similar ones, whereas Firefox bookmarks manager has a nifty search feature...
I had to scrape since the HN API doesn't expose ability to get information about upvoted submissions/comments. The extension assumes you are logged in, it doesn't ask for your username or password.
It's mostly for my own use and not very polished, but it works. I uploaded it to the Firefox extensions gallery, and you can probably find it there, but I don't think it's ready for public consumption yet...