News for hackers. Stuff that matters.

A comment on Abusing AWS Lambda to make an Aussie search engine.

Re:Abusing AWS Lambda to make an Aussie search engine by peterburkimsher on Monday September 27, 2021 @12:37AM (#28667129)
The writeup is very detailed and entertaining, and it's refreshing to see some competition to Google!

I tried 2 queries. Vegemite worked great:

https://bonzamate.com.au/?q=vegemite

The (unofficial) national anthem, less so.

https://bonzamate.com.au/?q=%22men+at+work%22+%22down+under%...

Have you seen the CommonCrawl dataset? It's a great source if you're looking for a full-web index to do some Big Data analysis on, and scale up your search engine to something closer to DuckDuckGo. The CommonCrawl data is 320 TB in size though, so I hope you've got plenty of free disk space!

https://commoncrawl.org/

You can also add "Show HN" to your titles in future, if it's a project you worked on. Then it'll show up in the "show" subset (linked from the orange top bar) and usually attract more encouragement because Hacker News readers usually really like original authors coming to hang out :)

[ Parent | on HN ]
Re:Abusing AWS Lambda to make an Aussie search engine by boyter on Monday September 27, 2021 @01:52AM (#28667496)
Yes, I still need to fix the highlighting for it, which I think is causing the issues for the men at work issue. Something I am aware off and when I get some time will resolve.

I have seen common crawl. It's one of those things I should look into more when I get the time. Honestly thats the limiting factor for everything I do these days.

[ Parent | on HN ]