Erlang / Elixir are languages that are easy to begin learning, but difficult to master. In our case, the team was able to get the basics up and running quickly, but the service became increasingly difficult to manage as our product requirements expanded beyond the strengths of the Erlang ecosystem's core competencies.
In our case, the original team of developers was excited to write a greenfield service in Erlang, but much less excited to maintain and scale it long-term. Some of them used their experience to speak at Erlang conferences, which led to them being recruited away by companies who were desperate for Erlang developers to maintain their legacy code after the core team left. Ironic.
At first we tried to train existing engineers on Erlang. Doesn't work. Not because Erlang is bad, but because people don't want their companies changing the direction of their career development just to support someone else's legacy project that requires an on-call team to support.
So your two options are: Hire at least 2, preferably 3 Erlang engineers for the reasons discussed above. Harder than it sounds, because AFAICT there are a lot of companies with 2010-era Erlang backends that made sense at the time but now require dedicated maintenance engineers.
Or you can rewrite it in a more common language, which is the route we chose. Rewrites are a difficult decision for well-known reasons, but ultimately it was the correct choice. Being able to drop 10+ engineers into a project as needed is a huge accelerator compared to funneling 100% of your work through the 2 Erlang gurus in the corner who maintain a codebase that no one else wants to touch.
Hiring is nearly impossible, come support our legacy system, and what ever you do, don't introduce Erlang into any other part of the company. Why? We feel burnt on this one project, and god forbid we get any deeper than we are now. What this isn't appealing?
If you have $200k "python engineers" on the payroll who wouldn't jump at the opportunity to do some additional Erlang, maybe it's time to reconsider your hiring practices and that is the real cautionary tale.
Back in the day people would balk at hiring Python programmers saying, "there are so many more Java programmers", and I used to say, "Why would you hire a Java programmer who was unwilling or unable to learn Python?"
Same logic applies here: Why would you hire a Python programmer who was unwilling or unable to learn Erlang? (Especially if you're going to pay them to do it!)
If you can't switch languages you're a kind of technician not a programmer.
FWIW, having done it professionally for ~15 years I'll never write another large project in Python again. Erlang's runtime is so much better than Python's it's just ridiculous. I feel stupid for not realizing this sooner.
The moment we can get rid of it, you bet we will. But it's much easier tearing it down when you understand it enough to discern designs driven by the limitations of the tech vs requirements for the product. Black box ground up replacements are hard and expensive compared to ports. Also, thankfully it's Python and nothing something more difficult/complex.
At the end of the day, it's a job, and there's an expectation of professionalism to do what's necessary. A lot of people on HN like treating tech jobs like a paid hobby which I think is detrimental to career development.
I have met lots of engineers who are smack dab in the middle of the flock when it comes to tech stacks, or even _behind_, but they were more than capable. Hire for what you need. But what you think you need and what you actually need is probably not know to you.
Your example of Java to Python doesn't represent the gravity of switching from Java to say Erlang or Haskell. You assume someone says no because they are unwilling or unable. We are unwilling, sometimes, because of the opportunity cost we have to pay for your fuck up.
Why would learning a new tool narrow rather than widen your opportunities?
It's not like you forgot all you know about Java.
Languages and specially the tool chain and the way of doing things around them evolve. If you spend like 5 years doing something else and come back to it, you'll find things have changed a lot. You may argue that it's easy to relearn this. But from a job perspective, the guy who has been using the exact same tech stack in the recent years will often be a better bet than a guy who wrote Java in version 1. This is very true for C#. Syntax, design principles, tools, method of deployments all have changed rapidly. Professionally, you want to be as up to date as possible in the area you worked in or you get rusty. You can learn new things on your time for their own utility. Flushing down 40-60 hours a week on a useless piece of knowledge is bad all round to me.
That is only if it's actually useless. I haven't programmed in haskell for many years, yet I am very glad I did. It was hard at first, but it made me think about code, data and transformation from a different perspective, expanding the repertoire of paradigms, regardless of which language I code in.
YMMV, but personally, I'd much rather hire or work with someone with 3y java + 1y erlang experience, than just 5y java experience. In general a polyglot rather than monoglot, especially if it's a different coding paradigm. Nevertheless, I may not actually want them coding random bits of work in erlang or whatever random language.
I get being interested in Erlang if you're doing Python. But interested in doing Python as a Java dev? Probably not.
And no fat-binaries. Nor self executing binaries. It's just as old and having em basically just boils down to enforcing conventions. But nope, nothing. There have been attempts by some such as shiv, but really... None are even remotely as usable as maven is.
And no, it's not the same. Not even remotely. The only thing that helps with are quick and easy deployments.
The thing which makes maven special is the ease of producing artefacts for deployments, not the final step of starting it up on the server.
I've been a happy Python programmer for over fifteen years, so this comes from a place of love and respect.
Beyond roughly 150k LoC you start to run into difficulties keeping everything straight over time, especially if you use a lot of indirection or other magic. The aspects of the language that help you go fast when you're prototyping (is "RAD" still a thing? Rapid Application Development) start to trip you up when the codebase matures. There are a lot of fancy footguns in Python that catch you if you stop dancing.
Tooling is getting a little better (MyPy, et. al.) but at the same time the overhead of adding explicit type information erodes the advantages Python has over, say, Java or Haskell.
There was at least 1 really large error that slipped into production because we were mocking an untyped service incorrectly (a mistyped property name), and the test passed, but prod exploded. It was embarrassing, and I certainly would have caught it with more mypy discipline, but overall the pros outweighed the cons.
At big companies that use mypy, however, this is problematic. If teams only define types for the straightforward parts of the system, then the types become less useful, especially as the company scales. From what I’ve seen, individual teams use an all-or-nothing approach with MyPy, so it’s unusual for an individual dev to sparsely type anything.
They can put only "Python" on their résumés, right?
For example, I wouldn't want PHP or Fortran on my resume, and, consistently with that, I avoid them. I don't have an opportunity to actively avoid them, but that defense mechanism would leap into action, if called upon.
However, I know a thing or two about these; I'm not simply applying "meme-based reasoning".
Would you want to pass on Lisp because of the ramblings of a rookie web developer with no Lisp experience? That's what you might do if you believe the "Lisp Curse".
I've almost exclusively worked on the usual procedural /OO languages professionally (java /c#)..I like to experiment and I've picked up a bunch of languages along the way (perl, python, coffeescript, f#, nim etc).. And its usually because I want to see what's different and spend enough time to do something useful (but not large) with it. Also, it helps you become a better programmer IMO. otoh, if I'm hired for my skills but then have to work on js( don't actually like the js eco system) or haskell (I gave up after trying and I doubt I'd become very good at it anytime soon if I were forced into it) I'd be not happy either (and IMO, with valid cause that doesn't reflect on me or my abilities)
Ah, sure they might be willing, but as a lead I'm not willing to invest in it.
1. I'm not sure I want someone who's got 4 weeks of experience in a language working on system critical infrastructure. Pick your time, does it take 6mo?
2. They have existing work, learning and working on an entire new language _AND_ new project is a huge task. Especially for non-trivial projects.
We could, but in our case it was a better engineering decision was to retire the old system and move on.
I have made mistakes like that (introducing monads/future and making the team learn guava) and I still regret it. Its not to say futures are useless - there were a lot of async logic in the codebase and futures solve a real problem, but its not clear if the benefits outweigh the cost.
I'm just beginning to get interested in the Erlang ecosystem. Could you please add more color to this and give some examples where Erlang shines? IIRC, Whatsapp backend used to be on Erlang.
You can hot swap all languages.
"All languages" provide you the ability to run arbitrary code, and thus theoretically support hot-swapping, but if you try to put it into practice you'll soon run into real obstacles you hadn't considered that either bite you or at the least make your job a lot harder. Erlang provides those practical solutions that make hot-swapping easy (in other words, practically possible).
TL;DR: On the question of buy vs. build when it comes to hot-swapping abilities, Erlang falls on the left while "all languages" start at the right.
Hot swapping Erlang in production is rarely done, and if it is done, it mostly done during debugging, not serving production traffic.
I invite you to do a straw-poll on the erlang mailing list.
I'm going to have to call your bluff and ask you to substantiate your claim. Please show where Python provides, as a part of the language and/or the standard library, the equivalents of these Erlang built-ins:
1. Module versioning (via vsn)
2. The ability to run multiple versions of the same module at the same time, transparently and for as long as needed.
3. The versioning (via Module vsn), tracking (via handle_call/3 and derivatives), and migrating (via code_change/3) of State.
4. The dependency tracking and ordering between code updates of multiple modules (via release_handler)
These are the native features that make hot-swapping in production not only feasible, but also powerful and easy. Of course, these are built on top of foundational primitives that are possible in any language, sure, but not all languages (including Python) have these natively available.
----
> Hot swapping Erlang in production is rarely done, and if it is done, it mostly done during debugging, not serving production traffic.
> I invite you to do a straw-poll on the erlang mailing list.
I'm sorry, but this point you're introducing now is unrelated to the discussion we're having, so I'll refuse to respond to this in any way except to remind you of the point we _are_ discussing:
1. adrr's claim: " Erlang was developed for telecommunication systems where you need performance and the ability to hot swap code." https://news.ycombinator.com/item?id=23286646
2. To which you responded: "Erlang's hotswap and Python's hotswap are of comparable power, ie, both are crappy. Being able to do something isn't the same as doing it well.
You can hot swap all languages." https://news.ycombinator.com/item?id=23287624
Used to be? I know WhatsApp switched from FreeBSD to Linux and swapped hundreds bare metal servers to thousands of VMs. I dont think WhatsApp ever swapped / rewrote their system in another language under Facebook.
Key phrase right there. Because it doesn’t matter what language a system is written in: if management doesn’t bother to hire replacements until after everyone who understands that system and why it works the way it does has already left, of course it’s going to crater shortly thereafter.
That’s not a language problem, that’s a business continuity problem. And it’s lamentably common as dirt.
Any monkey can learn to churn code. Learning the business domain, what its problems are, and how to solve them; that’s the bit that actually matters.
PEBKAC, at every level.
If you're an up-and-coming Golang developer, how happy are you going to be when your company takes you off of a Golang project and makes you the new Erlang person?
Now how happy are you going to be when you're informed that you're learning Erlang to maintain an old, flakey codebase written by 2 guys who left the company and are now unreachable?
Oh, and you're going to need to be on call for this because no one else knows how to fix it.
In my mind good developers are not <language> developer, they are engineers that solve problems with the right tools.
Given infinite time and no deadlines, sure.
But in the real, you can't expect your engineers to learn entire programming languages and associated ecosystems on the fly as they debug issues in production.
If your Erlang-based cloud backend starts going down, you can't afford to wait around while engineers teach themselves Erlang so they can begin to even debug the problem.
The point is that if you write critical infrastructure in a specific language/framework/ecosystem, you need to have people proficient in that ecosystem who are ready to go. When dealing with production systems at scale, it would be downright negligent to assume you could simply learn the language and framework on the fly if any problems come up.
Maybe the emphasis should be more on the disappeal of your role being changed to maintain an old, flakey codebase than on the language.
I'm not a <language> programmer. I'm a programmer period.
If I'm a python developer, I'm now spending less time on my core competency, giving other python developers an edge. Seems like you're actually narrowing my career, unless I wish to apply for jobs seeking mediocre Erlang programmers...
The OP said "I'm a python programmer", and discounted the value of learning something new like elixir because it would undermine this due to the opportunity cost, which implies they may be falling into the trap I mentioned. All other things being equal, someone who is reasonably competent at both Python and Elixir is going to probably solve many problems in a better way than someone who didn't have a broader perspective. (In part because learning Elixir stretches well beyond learning a programming language, it is an introduction to an entire distinct continent of distributed systems knowledge and technology - OTP, BEAM, etc.)
There are lots of senior roles that require oversight of multiple projects that are working with different tech stacks. Some companies specialise in a single language. But lots of companies have say 2 or 3 backend languages (say C++, Go, and Python), plus a website in JavaScript plus mobile apps in Java/Kotlin and Swift/Objective-C. Being able to dive into all of those makes you incredibly valuable to this kind of company.
So that everyone knows you're full of shit? It takes years in a technology to become an expert at it. There's no replacement for the experience of taking multiple projects from inception to production to scale. That's how you learn about the inscrutable errors, fiddly configuration issues, and pitfalls that make you an expert at something.
For Java it might be enterprise Java, it might be Android, in which case you need familiarity with those frameworks to be productive. For Swift it'll be iOS.
No one's "diving into" the codebase for an iOS app starting from "I mostly do backend web app development in Golang".
Are you an "expert" in all those things? I doubt it. In fact the worst possible thing you can generally write anywhere is that you're an expert in C++ which at this point is oh-so-many subdialects.
I mean, I've certainly been expected to at previous jobs (my background being backend and frontend web). It was a steep learning curve, but in retrospect it was also an excellent learning experience.
> Are you an "expert" in all those things? I doubt it.
No, but give me time. I'd consider myself an expert at frontend JavaScript/TypeScript and backend JavaScript/PHP. I have a decent amount of experience (enough to get a job an X developer, not enough to call myself an expert) in a number of other technologies (e.g. Python, Rust). And at my current day job, I'm responsible for a React-Native project through which I'm getting exposed to a lot of the quirks of the Android and iOS platforms.
Go I've not used. But Go's a relatively small ecosystem to learn. I don't put Go on my CV, but I'd be confident applying to a Go job. C++ I've done a little of, but I'm absolutely not an expert. Although Rust experience means that I understand most of the low-level concepts and best-practises that aren't C++-specific. Still, I absolutely don't advertise that on my CV either. One day though maybe. I'd like to do some embedded stuff at some point.
> In fact the worst possible thing you can generally write anywhere is that you're an expert in C++ which at this point is oh-so-many subdialects.
C++ is a bit of a special case there though. Because
1. C++ is huge: it takes a lot longer to master C++ than it does other technologies.
2. C++ is wildly unsafe. Therefore the consequences of not knowing what you are doing with C++ are much greater. You'll likely end up with a crashy program riddled with security holes.
In contrast, I wrote a production Rust project with no prior experience, and all the things that would have been crashes or security issues in C++ were compile errors. So the only consequence of my lack of experience was that it took longer to complete the project.
so this argues against the theory that learning other languages and other programming paradigms makes you a better programmer in the languages you already know.
This might be true based on your specific situation (relatively new to the language, but already know some other varied languages); I don't know.
Other comments say an engineer is supposed to solve issues with the tools given to them. I can solve issues in other languages, I just don't want to. How does that make me a bad engineer?
Second, because you are also likely to favor recruitment of other devs based on this cultural choice not on their merit, eventually creating a clique.
If I run a Java shop I want a clique of Java devs, not Golang or Rust devs.
A "multi cultural", Jack-of-all-trades kind of team, that have a broader knowledge of many tools and techniques but less in depth familiarity with any of them, is more likely to use less features from the languages and tools that they have picked for a few of their strong points only, and less likely to indulge themselves with a level of sophistication that benefits their ego (and job security) more than the project.
> If I run a Jave shop
So I know a Python shop must be a business that sells reptiles, but what does a Java shop sell?
>So I know a Python shop must be a business that sells reptiles, but what does a Java shop sell?
Coffee obviously.
The workers are settled and and don't want change. And there's nothing wrong with clocking in and out and not giving a shit about your "career". But there is something wrong with the managers falling for this.
If your product-critical cloud backend depends on people being knowledgeable in a specific programming language, you don't make it a 1/3 time side project for multiple engineers.
Also, if you start giving people "1/3 time" responsibilities, you're one step away from "everything is top priority" territory.
The point is: If you're going to write critical pieces of your infrastructure in a specific language, it's a requirement to have at least one person full-time dedicated to that piece of infrastructure.
When things go wrong at scale, you can't depend on a group of people for whom this is a part-time side project.
You'll have to trust me that we didn't just throw our hands up in the air when dealing with the Erlang situation. We tried a lot of the suggestions here and more.
Erlang is one of those languages that, for whatever reason, lulls engineers into a false expectation that they can pick it up over a few weekends and start knocking out IoT-scale solutions serving 6- and 7-figure connection counts. Speaking from direct experience, that's not only untrue, but it's a dangerous misconception that leads to a lot of pitfalls.
Don't get me wrong, Erlang is great if your problem set matches up neatly to core OTP functionality and well-defined patterns, but you don't have to stray far until you start finding difficult pitfalls that aren't easily addressed by engineers learning Erlang on the fly.
> Also, if you start giving people "1/3 time" responsibilities, you're one step away from "everything is top priority" territory.
I like to maintain and improve legacy systems and usually happy to do it. When I got a separate side-project I had constantly fight with my team why I should work on project X instead of contributing to our core project. If it was my whole team responsibility they were happy that somebody does it. Also, they wanted to invest enough time so it won't break.
You don't need someone "learning" Erlang. You need someone who works with it.
* Healthcare startup that basically outsourced its entire infra/security team from a consulting company. The consultants rewrote everything and took over engineering leadership, basically acting as gatekeepers for anything SRE-related.
* Worked at an older DNS company that hired a bunch of consultants to help engineers migrate their web framework from Play to Spring. Said engineers used Play because it was shiny but didn't have the Scala expertise to keep maintaining it, so they moved to pure Java.
For example good static typing is not just about adding type declarations but about taking advantage of the type system to the right degree. (Like don't ever overdo it, it's killing productivity most times).
But what degree falls into the productivity boosting spot and from which point on it will hinder productivity depends a lot on the team.
The problem often starts when the team changes and e.g. a pure scala/rust/C++/Ocml team get people cross entering from languages without any usefull static typing.
What had been reasonable and useful might now be a hindrance to the team!
I ran into a situation where this caused massive delay. (Well through the actual root cause was miscommunications, people not complaining about problems and instead trying to find ways to fix them behind the back of other people force pushing nice looking simple solutions which just happen to fail many edge cases, killed the write, refactor, test cycle and increase maintenance cost longtime :( )
So well 1st never overuse type systems. 2nd communicate problems, especially if you have problems understanding anything even if it just takes longer then you think it should.
What I mean is code which not only uses types and takes advantage of the type system but overdoes it. In many languages with a powerful type system you can encode much more then just basic types and this is useful to reduce possible error cases. But if you overdo it the type system part becomes too complex, type error messages too unreadable and it hinders productivity and on-boarding.
I'm a big fan of static typing and believe that more encoding more constraints in the type system is _generally_ good. But if the resulting types become to complex it often not worth it.
With simpler, more gradual designs like what are in Typescript or Haxe or even static, native-code languages like D, you have more escape hatches available, so you can iterate on the specification a few times and get code running quickly but also aim to make it tighter over time.
This is a great question to ask. It’s worth extending beyond language to frameworks and libraries as well.
It’s important to optimize for the development experience across all the apps in a team. Especially once the excitement of using a new language is gone and there are bugs to fix.
I‘ve rewritten a few projects in a common language for the org. Each time the biggest benefit was how straightforward it became to work on. Problems you’ve found somewhere else become easy to spot (and search for). Writing another test is the same as the last project you had open. Anyone in the org can contribute with a minimal learning curve. This can be the difference in getting bugs fixed in a day or a month.
Then consider how hard it would be to get more people that have used such language/ecosystem in anger.
Side note: Pagerduty comes with many hilarious sounds, including a very realistic meow. Thus we practice meow driven development: I don't want my phone to meow desperately.
Not Elixir, but a cautionary tale from our Erlang project. ~8 years ago a our IoT backend was written in Erlang, this was the early days of IoT, so sure it made sense as a technology, could scale well, handle all the symmetric sessions, etc. It was a good tool for the job and in theory could scale well.
But, there's always a but. We're a Python shop on the backend and C on embedded devices. Engineers move on, there some lean times, and after a few years there was no one left who could support an Erlang system. So we hired for the position, and it's key infrastructure, but not really a full time project, so we hired a Python+Erland person.
But! Now they're on call 24/7 since the regular on call roster are python people and when the Erlang service goes wrong, it's call the one engineer. So, do you hire 2 or 3 people to support the service? No, you design it out. At the time is was 2018, and IoT services were a plenty, so we could use an existing service and python.
Another way of looking at it, let's say it's a critical service and you need at least 3 people to support it/be on call. If each engineer costs $200k/year, that's $600k/year. Does this new language save you that much over using a more generic and widly known language in the org?