A behavior is simply a list of functions you've declared that your module will export -- and a convention on what they might do. gen_server.erl is going to make lots of callbacks into your code, and rather than pass a huge list of funs, instead we pass the module name, and gen_server calls the exported functions from that module (this style means all the callbacks will hit your new code if you hot load, without you doing anything special; processing type changes is up to you, of course)
It seems odd to me that they would include a unique feature like live-updating if it shouldn't be used.
I grant that live-updates and gen_servers may be anti-patterns, but my assumption was to consider the effects of static types on OTP and these are part of it.
If you identify some subset of erlang+OTP that is easier in some ways, great, I'm all for it.
I am just pointing out some complexities without making assumptions about what should be included or discarded. ( I do not know what erlang shops do in the small or in the large).
Perhaps what we want then is static types for "OTP-Lite"
It could be done without a big lock by splitting into three steps:
1) Push an upgrade that changes the types and adds the conversion functions. The valid type is the union of the old type and the new type. Wait until all nodes complete the upgrade.
2) Push an upgrade that instructs the nodes to convert their data and start using the new types by default. Wait until all nodes complete the upgrade.
3) Push an upgrade that removes the old types and conversion functions.
The problem is whether the function can or cannot handle the content of the new message and that's completely orthogonal
You could still use Erlang's crashing and supervisor technique, but you would have the additional benefit of using static typing across a distributed system (where each node may or may not have received the upgrade yet).
No, that's not how you do it. You marshal things directly into the desired types. Check out either aeson for Haskell or how Go does things via either the json modules or the generic Text/Binary Marshaler/Unmarshaler.
"but also the myriad deployment & upgrade & versioning scenarios."
The answer to all of those things is mostly that even a lot of Erlang shops don't use live upgrading. You really have to have a very particular use case for that to be the best solution vs. a rolling upgrade and server restarts. Even if the language is capable of it, it still requires you to write services that can handle being upgraded, and it's much easier to write services that can handle being restarted, especially since you 100% have to write that anyhow because services get restarted anyhow. Most people don't have that use case. Web services certainly don't have that use case.
Once you drop that, it's a lot simpler.
"Even thinking about something like a gen_server"
gen_server is partially as complicated as it is as a side-effect of other decisions in the language. While the concept of a gen_server is a strength in Erlang, the specific implementation of gen_server as this "behavior" thing is mind-blowingly complicated for what you actually get. (It reminds me of Python's "metaclasses". I spent many hours wrapping my head around what that was, but in the end, all that it amounts to is what is now called a class decorator, which is way more sensible. A metaclass isn't a class decorator in theory, but in practice, class decorators are way easier to understand and cover 99.9% of the use cases, if not 100%.) When I implemented supervisor trees in Go, my solution for gen_server/gen_fsm/gen_* was just to... not. Behaviors are just a very, very weird half-object-ish system with a lot of limitations. They are easily replaced by simply having some sort of "interface" system, be it via conventional classes or interfaces. It's why you don't see "behaviors" as Erlang defines them anywhere else. Erlang has a lot to learn from and copy from, but that part isn't it.