News for hackers. Stuff that matters.
Re:Lessons learned scraping 100B product pages by pdimitar on Tuesday July 10, 2018 @01:14PM (#17500123)
Thank you man (I know you are the author of Meeseks), I didn't know that. Always knew that the current info was the Meeseks was faster than Floki but it seems that OTP 21 largely eliminated that as you said.

Valuable info, thanks!

Re:Lessons learned scraping 100B product pages by mischov on Tuesday July 10, 2018 @01:47PM (#17500468)
It was pretty interesting to see Floki get a lot faster and Meeseeks actually get a little slower with OTP 21. I'll enjoy figuring out why. I hope to get a chance to work on the OTP 21 performance of Meeseeks before too long.

On the plus side there were some nice memory improvements for Meeseeks in OTP 21.

Re:Lessons learned scraping 100B product pages by pdimitar on Tuesday July 10, 2018 @02:14PM (#17500715)
(off-topic alert)

Don't let this sound patronizing because it's not -- but have you looked at how many times is the boundary between the BEAM and the Rust code crossed? I haven't inspected Meeseks' code so can't talk, just wildly guessing.

My ancient experience with Java <-> C++ bridges has taught me that if your higher-level language calls the lower-level language very often then the gains of using the lower-level language almost disappear due to the high overhead of constantly serializing data back and forth.

Anyhow, we should probably take this discussion to ElixirForum and not here. :)

(I am @dimitarvp there and almost everywhere else on the net, HN is one of the very few exceptions of inconsistent username for me).