Re:Meanpath - Search the page source, server headers and vis...
by mwotton on Monday July 22, 2013 @08:54PM
(#6087509)
This is true :) been a great tool to write a spider in, fast native code, a strict type system and very cheap threads has made it much faster than it otherwise would have been. ZeroMQ has also been a great help in modularising it. I'll try to get a piece out on the technical architecture later. (I'm the CTO, btw.)
Re:Meanpath - Search the page source, server headers and vis...
by etherael on Monday July 22, 2013 @09:39PM
(#6087669)
My favourite bits of the project;
Elasticsearch
ZMQ
Golang & Haskell
Unbound DNS
And a bit of old Python glue to hold everything together.
It's funny how long things can last when you think you would have outgrown them pretty quickly. A tiny round robin domain server that maxes out at around 60k domains per second on just a single core to distributed spider bots though is just about easy with Python + ZMQ. Plenty to keep a swarm of far more complex and well engineered Haskell crawlers humming away and throwing the results to Golang indexers to push into the Elasticsearch core for ...
Well, I'll leave the rest of it to mwotton's blog post, suffice to say it's been one of those life changingly awesome projects of doom. I'm sold on the benefits of embracing polyglot.
Considering I'm doing my own Haskell based tech in my business (and I'm somewhat actively in the core community), It pleases me greatly to see Haskell/ghc used in a rich array of interesting businesses!