i'll be the first to ack that popularity doesnt matter if you personally feel productive in it; but it does matter when talking about jobs and leveraging an ecosystem (which i dont know elixir's ecosystem so i cant comment on, i had not heard of Broadway / Genstage before this comment). but career-wise, i do observe that developers going after a theoretically better language routinely sacrifice a degree of industry relevance they never get back since they voluntarily enter an extremely niche subculture that for whatever reason never catches on. seen this time and time and time again and i think HN disregards this for the love of the new shiny without a balanced analysis of the career risk/reward
It takes time for technologies to catch on. Elixir and Rust both are relative drops in the bucket in respects to industry adoption, making it difficult for practitioners with the languages to continue using them beyond their current employment. Web development often acts as the foot-in-the-door, and Elixir's Phoenix is doing superbly well in this domain. However, Elixir's got an even more impressive position for data engineering related work.
Were all those devs picking Ruby instead of Java in 2010 ruining their careers? I doubt it. Were the devs picking Java instead of Ruby for their startups? Very possibly, many of them.
Once your business comes into contact with the market, tool productivity matters more than popularity. Popularity can affect productivity, but at the end of the day, it’s the productivity that matters.
See: https://twitter.com/logicmason/status/1371255029412233218
but is working for yc startups some good thing? like desired?
I was speaking more from a founder-perspective, but being an early employee with equity in a startup that becomes very valuable is pretty good, too.
I know about the book Concurrent Data Processing in Elixir when I am trying to pick up Broadway and GenStage. Do you have any other recommendations?
https://www.youtube.com/watch?v=Y2Nr4dNu6hI
"Chris Grainger (Amplified) delivers the Friday Keynote at ElixirConf US 2022.
We switched our machine learning research, production inference, and ETL pipelines to Elixir. This is the story of how we made that decision and how it's going...."
There aren't many data engineering related jobs in either Elixir or Rust, yet, but as someone with depth in both I can use either and lead a team to use them successfully in short time.
Actually, to even the playing field, a bald guy in a blue shirt just looked at your LinkedIn profile. ;)
I manage Elixir systems for data engineering related work, so I can attest to its use in this domain within a production environment. I also have used Rust for comparable data-engineering related systems.
Elixir is a phenomenal language for data engineering. I've written message processing pipelines in Rust and didn't get anywhere near the level of design considerations that Broadway / Genstage have. Some day, there may be robust open source offerings in Rust as there are in Elixir, but the ecosystem hasn't reached this state, yet. Rust asyncio is also in a minimum-viable-product condition, lacking the sweet structured concurrency that Erlang-OTP solved long ago and Elixir benefits by. Data pipeline processing in Elixir, utilizing all available cores exposed to the runtime, is straightforward and manageable. Telemetry patterns have been standardized across Elixir ecosystem. There are background worker processing libraries like Oban that help with audit trail/transparency. Smart, helpful developer communities.
Elixir is not going to beat Rust on performance. CPU-bound work is going to take orders of magnitude longer to complete with Elixir than Rust. You could extend Elixir with Rust in CPU-intensive situations using what is known as NIFs but you'll need to become familiar with the tradeoffs associated with using Rust NIFs.
Writing in Rust involves waiting for compilation. When a developer waits for compilation, they switch to something else and lose focus. You can use partial compilation for local development, and that speed things up. You also need to have a very modern workstation for development, preferably an M1 laptop or 16-core Ryzen, with at least 32GB ram and SSD. Elixir, however, has quick compile times as it doesn't do anywhere near the level of work that Rust compiler does. There is a tradeoff for that, though. Elixir is a dynamic language and consequently has all the problems that dynamic languages do that are automatically solved by a strongly-typed compiled language such as Rust. You also discover problems at runtime in Elixir that often would be caught by the Rust compiler.