Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on A Web Crawler with Asyncio Coroutines. Parent comment
1) I check whether or not the page we just scrapped has any of the tags we are looking for.
2) We then extract any information within those tags (images, etc.)
3) We follow trough every link and if it's not in the seen/scrapped list, we add them to the queue.
Not sure if this helps to narrow it down.
Thanks!
There are no comments on this comment yet.
1) I check whether or not the page we just scrapped has any of the tags we are looking for.
2) We then extract any information within those tags (images, etc.)
3) We follow trough every link and if it's not in the seen/scrapped list, we add them to the queue.
Not sure if this helps to narrow it down.
Thanks!