itsokaywelldoit

Re:Ask HN: What can we do with 20B links from sites on Monday September 04, 2023 @10:00AM (#37380543)

Sorry, missed the point of the question. Specifically finding links - because of that client. As we were working on his project we slowly began to realise that cross-referencing sites, links to competitors and stuff like that might be useful for someone. Also, with links u can go deeper into the site, and scrape links from all pages.

And maaybe we got too in love with the concept so now we're firmly stuck in it, finding reasons for our behaviour that are not "woohoo it's so fun to play with it and we have justification to spend time here - it's gonna work for us!"

Re:Ask HN: What can we do with 20B links from sites on Monday September 04, 2023 @09:55AM (#37380503)

We had a client that needed similar project, but for small amounts of domains, and we had the list of domains 'just lying around', so we thought 'wait, if our client knows where to use it then there's a market', and, well - wrote some more and combined the two.

Ask HN: What can we do with 20B links from sites

Posted by itsokaywelldoit on Monday September 04, 2023 @09:39AM
from the ask-hn dept.

Hello, geeks of the world. It's my first time writing here. I and my friend developed a project that receives list of domains like www.wikipedia.org(example, example!) and saves all of the links in the html of it's starting page. Sorts them into images, internal site links etc. We had list of like 200 mil domains and we parsed them all, resulting in 20 billion links from sites paired with...

( Read More... | 4 comments | on HN )