Hot code reloading with Erlang

Re:Hot code reloading with Erlang by toast0 on Sunday December 06, 2015 @03:20AM (#10684722)

Not the original asker, but it would be nice to be able to upgrade between OTP releases, without having to restart my application (but I have no expectation of that being possible ever when there are VM changes, and unlikely for the effort to be spent to do it for the user-space erlang bits either). I have to use DNS for load balancing [1] and big mnesia tables, so I have to wait a long time for traffic to drain, and then another long time for the application to start back up.

Working for 4 years in an Erlang environment where hotloading is the norm, makes me wish for it everywhere! Why do I have to reboot to fix kernel bugs in tcp? :(

[1] the load balancers I have access to where we host had more downtime than our hosts, so not actually helpful

[ Parent | on HN ]

Re:Hot code reloading with Erlang by simoncion on Sunday December 06, 2015 @06:39AM (#10685005)

ISTR that, you can reload core Erlang modules, but that there's some sticky_directory stuff that prevents it from happening by default.

I'm pretty sure that I can reload the inet module: [0]

    Eshell V7.0  (abort with ^G)
    1>  l(inet).
    {error,sticky_directory}
    
    =ERROR REPORT==== 6-Dec-2015::03:37:00 ===
    Can't load module 'inet' that resides in sticky dir
    2> code:which(inet).
    "/usr/lib/erlang/lib/kernel-4.0/ebin/inet.beam"
    3> code:unstick_dir("/usr/lib/erlang/lib/kernel-4.0/ebin/"). 
    ok
    4> l(inet).
    {module,inet}
    5>

Not that this is a good idea, mind, but I'm fairly certain that it's doable. :)

(Also note that you can reload the mnesia module without hassle. Its ebin directory is not marked as sticky. :) )

> I have to use DNS for load balancing [because the load balancers fall over often].

Oh lord. That's a terrible situation to be in.

[0] Which is part of the kernel application, which is one of the applications that hot upgrades require that you restart the emulator to upgrade. [1]

[1] http://www.erlang.org/doc/system_principles/upgrade.html

[ Parent | on HN ]

Re:Hot code reloading with Erlang by toast0 on Sunday December 06, 2015 @04:47PM (#10686793)

Yes, you can reload all the beam files (even if you have to unsticky them -- you can do that), but that doesn't mean r16 -> r17 didn't make changes you can't safely load. Some parts of OTP have state upgraders, and some don't; some may depend on bif/nif changes as well.

>> I have to use DNS for load balancing [because the load balancers fall over often].

>Oh lord. That's a terrible situation to be in.

It's not terrible, it's just not great. Our hosting environment is generally very reliable, so if I don't screw things up, my systems won't fall over. It's just their loadbalancers are crap, the suggested upgrade path was run a load balancer in a VM appliance; which seems like maybe I should just run CARP myself on the hosts (or something), instead, and skip a layer, but I'll probably never get around to that, because it doesn't come up that often :)

[ Parent | on HN ]

Re:Hot code reloading with Erlang by simoncion on Sunday December 06, 2015 @08:55PM (#10687653)

> ...but that doesn't mean r16 -> r17 didn't make changes you can't safely load.

Oh yeah. I know. I would expect for some-to-many minor things (like the TCP bug you mentioned), this wouldn't be an insane way to do a hot upgrade. Guess one would need to make a close study of the diffs.

Regardless, I mentioned the fact that this isn't a good idea for any novice who might stumble across this comment months or years hence and get the wrong idea. :)

> ...maybe I should just run CARP myself on the hosts...

Oh, CARP is so cool. It's a bit of a pity that ucarp doesn't support IPv6. But, how does CARP replace a load balancer? Isn't it used for single host availability, or is my understanding too narrow?

[ Parent | on HN ]

Re:Hot code reloading with Erlang by toast0 on Sunday December 06, 2015 @09:26PM (#10687729)

Oh, I meant a os kernel bug; no way to hotload those.

I have two servers, if I get two carpable ips, and may each server primary on one, and put them both in DNS, I have load balancing and failover. If I just use one IP, at least I have failover, anyway I need each server to be able to handle full load, so I could have hot-warm instead of hot-hot.

(with more than two servers, need to figure something else out, probably two boxes running a simple load balancing in front of the rest of the cluster.

[ Parent | on HN ]