Re:New Elixir 1.12 – The developer’s point of view
by prophesi on Wednesday June 23, 2021 @01:53PM
(#27607647)
> I think a few companies use Elixir to power their web crawling/scraping tools.
What do they use for headless browser scraping? I tried Hound a few months ago, but it seems too geared towards testing to be used more generically. We ended up just using Oclif and Puppeteer for scraping via NodeJS.
Re:New Elixir 1.12 – The developer’s point of view
by vereis on Wednesday June 23, 2021 @03:43PM
(#27609118)
Might fall into the same category as Hound but Wallaby exists and works.
Otherwise have you heard of Crawley?
Re:New Elixir 1.12 – The developer’s point of view
by prophesi on Friday June 25, 2021 @10:18AM
(#27631058)
I heard of Crawley and thought it only did HTTP-based crawling, but now I see it has a browser rendering option via Splash[0] that looks like it'd fit the bill. Thanks!
(Also had issues configuring Wallaby to be used outside of testing)
Re:New Elixir 1.12 – The developer’s point of view
by michaelcampbell on Wednesday June 23, 2021 @03:05PM
(#27608647)
> if you won't need the full bird.
Never heard this phrase before, but I like it.
I think a few companies use Elixir to power their web crawling/scraping tools. This makes intuitive sense as a good candidate for the process supervisor and parallel work architecture OTP encourages.
Nerves (embeddable Elixir) has come a long way. I switched to Nerves for some Raspberry Pi projects and the amount of time I waste dealing with hardware/config has gone to nearly zero. I am a hardware novice and was able to setup flashing firmware over-the-air updates to the Pi with very little effort. I'm sure the companies that use Nerves in production have more to say about it.
I'm not very tuned into the updates to Scenic, a project for display/UI on embedded screens, but it looks like they've hit some big release/stability milestones.
Phoenix is the way to go for web interfaces, and is an excellent toolset, so alternatives haven't been demanded. For more lightweight http people usually reach for Plug, a key building block of Phoenix, if you won't need the full bird.