Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Ask HN: Need some tips for web crawling.
A simple way would be a headless browser [1]
But there are also hosted tools that work like a website builder.
The best way is: keep it simple and keep back (check once an hour or day and not every minute).
Many shops use Schema.org markup. So if they support it, you don’t have to write it for every site.
You could also use a library that works with raw html and css. Then you could just use css selectors for extraction.
[1] https://www.atlantbh.com/building-a-dynamic-crawler-with-pup...
There are no comments on this comment yet.
A simple way would be a headless browser [1]
But there are also hosted tools that work like a website builder.
The best way is: keep it simple and keep back (check once an hour or day and not every minute).
Many shops use Schema.org markup. So if they support it, you don’t have to write it for every site.
You could also use a library that works with raw html and css. Then you could just use css selectors for extraction.
[1] https://www.atlantbh.com/building-a-dynamic-crawler-with-pup...