Re:Crawl Order and Disorder
by muscomposter on Thursday March 27, 2025 @11:13AM
(#43494519)
based on what you said, and my assumption that Elixir (thanks to Erlang’s VM) is pretty good at concurrency, then yes Elixir would be good for crawling
The hard problems in crawling is mostly dealing with a very large and potentially highly mutable state. It's a concurrent problem, but a fairly easy one and a good use case for a traditional thread pool since there is latency is mostly a non-issue.