News for hackers. Stuff that matters.

A comment on Show HN: Spider Pro – easy and cheap way to scrape the internet. Parent comment

Re:Show HN: Spider Pro – easy and cheap way to scrape the in... by Cyberdog on Thursday October 10, 2019 @01:07PM (#21216885)
Sure, everyone's time is free too, after all.

(Alternate smart-ass reply: Surely you coded the browser, operating system, and network stack involved in posting this response by yourself, right?)

[ Parent | on HN ]
Re:Show HN: Spider Pro – easy and cheap way to scrape the in... by nirvdrum on Thursday October 10, 2019 @01:20PM (#21217026)
I'm not the OP, but it looks like it was in reply to the "cheapest way" part of the title. Incidentally, that line doesn't appear on the web site.
[ Parent | on HN ]
Re:Show HN: Spider Pro – easy and cheap way to scrape the in... by JustFinishedBSG on Thursday October 10, 2019 @01:44PM (#21217284)
> Surely you coded the browser, operating system, and network stack involved in posting this response by yourself, right?)

No because they were all free.

And I suck too.

But it was free.

[ Parent | on HN ]
Re:Show HN: Spider Pro – easy and cheap way to scrape the in... by sillydinosaur on Thursday October 10, 2019 @01:18PM (#21216997)
https://www.crummy.com/software/BeautifulSoup/bs4/doc/

I mean...

soup.find(id="link3") # <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>

Is this hard?

[ Parent | on HN ]
Re:Show HN: Spider Pro – easy and cheap way to scrape the in... by Cyberdog on Thursday October 10, 2019 @01:22PM (#21217045)
For HN's audience, no, that's not hard. But this product is clearly not only for us, and you've also cherry-picked a very simple example.

There's also the aspect of downloading the page in the first place and dealing with things like authentication and bot detection which a product like this helps solve.

I personally don't have a use for this product right now, but I won't be so bold as to say I'll never find a case where using it wouldn't be easier or more cost-effective than hacking up my own solution.

[ Parent | on HN ]
Re:Show HN: Spider Pro – easy and cheap way to scrape the in... by slig on Thursday October 10, 2019 @03:46PM (#21218469)
Good, now do that on a page that uses JS and authentication. Trivial on this extension, not so much using BS.
[ Parent | on HN ]
Re:Show HN: Spider Pro – easy and cheap way to scrape the in... by penagwin on Thursday October 10, 2019 @02:18PM (#21217582)
If you show that to non-programmers they'll take one look at that and understand that about as well as egyptian hieroglyphics.

"Soup? Python? What's an 'id'? class? href? I just want to select the title!"

[ Parent | on HN ]