Re:Ask HN: What is your most impressive hackathon project?
by rprameshwor on Thursday May 25, 2017 @12:20PM
(#14418249)
I'm interested to know more about the tools/techniques you used to scrap the sites and clean the results. I am working on scraping contents from a couple of sites and it seems to be a pain, probably because i don't have lot of experience around it.
[ Parent
| on HN ]
Re:Ask HN: What is your most impressive hackathon project?
by wasi0013 on Thursday May 25, 2017 @02:38PM
(#14419172)
If you are familiar with python then try scrapy[1]. You can also scrape websites using beautifulsoup4[2]
[1] https://scrapy.org/ [2] https://pypi.python.org/pypi/beautifulsoup4
Re:Ask HN: What is your most impressive hackathon project?
by codefined on Friday May 26, 2017 @03:16AM
(#14422475)
We actually used Node.JS' request module, combined with some NLP (using natural) in order to pick out the main content. This worked pretty well, but for our purposes we didn't need it to be perfect because anything like headers would be removed when we processed the content (not being full sentences).