A few years ago, before JumpTap pivoted to be a mobile ad network, they built a search engine using off-the-shelf parts on a hosting service. Some of the founding team members were from Lycos, and they had a pretty good idea of how to do this.

It was not run at general-purpose production scale, but they did show they could produce credible search results at a relatively small scale.

There are several open source spiders out there. As Web search portals increasingly aim at the broad middle, I sometimes wonder if doing your own domain-specific spidering might find interesting things.