News for hackers. Stuff that matters.
Re:Migrating Millions of Concurrent WebSockets to Envoy by endisneigh on Wednesday March 17, 2021 @03:04PM (#26495490)
I wonder if Slack has considered using webrtc to do peer to peer chats on the client side and then gathering up the chat metadata and having each client periodically send their version of the history and reconciling it server side.

This would also have the effect of allowing slack to peer more or less normally even if Slack was down (of course bots, search, etc wouldn’t work).

Re:Migrating Millions of Concurrent WebSockets to Envoy by lovedswain on Wednesday March 17, 2021 @03:47PM (#26495979)
It's possible to implement all of this without inheriting the additional infrastructure and networking complexity WebRTC brings along with it, not forgetting WebRTC still relies on centralized components to coordinate. Don't use WebRTC unless you really need the features it offers, routers in many scenarios hate it and even where they allow it, the combinatorial explosion in possible configurations to support and diagnose between peers is a problem nobody should willingly invite unless they can't achieve a solution any other way

With WebRTC you give up the nice ultra-low-common-denominator "outbound port 443/TCP needs to work" requirement and replace it with "UDP networking generally healthy, possible to establish port mappings, possible to maintain stable port mappings over time, possible to not have mappings go away due to lack of traffic" etc etc

Re:Migrating Millions of Concurrent WebSockets to Envoy by meheleventyone on Wednesday March 17, 2021 @03:55PM (#26496061)
Hah, this is so true. Am building a little hobby project to try out WebRTC for game development. On my ISP provided router a Mac and Windows computer can’t see each other over WiFi due to some mDNS issue likely the router support for multicast. Using Chrome flags to turn off mDNS and they can connect fine but obviously expose internal IPs. Wire one of the machines and mDNS works. TURN is essentially a necessity but then why not use a server (particularly for a chat app).
Re:Migrating Millions of Concurrent WebSockets to Envoy by mypalmike on Thursday March 18, 2021 @06:26AM (#26501224)
If your topology is essentially client server, you don't need a TURN or even a STUN server for webrtc connectivity. The client can send dummy candidate IPs in the offer, but receives valid server IPs in the answer. The client starts initiating transport to the server and reaches it. Peer reflexive candidate processing by the server allows it to communicate back to the client even if transport is UDP.

It's definitely more complex than websockets, but removing STUN servers from the equation does simplify connection initiation a bit.

Re:Migrating Millions of Concurrent WebSockets to Envoy by meheleventyone on Thursday March 18, 2021 @06:35AM (#26501285)
I'm aiming for peer to peer. :)
Re:Migrating Millions of Concurrent WebSockets to Envoy by mypalmike on Thursday March 18, 2021 @07:16AM (#26501499)
Interesting! There aren't a lot of peer to peer multiplayer game architectures nowadays. Good luck!
Re:Migrating Millions of Concurrent WebSockets to Envoy by meheleventyone on Thursday March 18, 2021 @07:27AM (#26501566)
Thanks, it's actually more common than you'd think particularly in certain genres. For big examples with a large player bases you have GTA Online and the Destiny games. Fighting games also tend to run peer to peer both because there is (usually) only two people in a fight and because they are quite latency sensitive.

My own interest comes from many years of building multiplayer games professionally and wanting to make smaller scale multiplayer games for fun without needing to invest too much in infrastructure.

Re:Migrating Millions of Concurrent WebSockets to Envoy by littlestymaar on Thursday March 18, 2021 @06:12AM (#26501157)
If you're using the DataChannel only (which is common when using WebRTC for games) you don't need a TURN (that adds complexity and you don't need it useless you're dealing with media streams) instead all you need is to re-use your signaling server as a fallback to get your messages from two peer who cannot establish a connection to one another.
Re:Migrating Millions of Concurrent WebSockets to Envoy by meheleventyone on Thursday March 18, 2021 @06:21AM (#26501195)
That's merely equivalent though. Ultimately TURN is just a common specification for a relay. You also run head first into latency issues where you can run one signaling server globally but need regional relays to keep latency down.

Could you expand a bit on why MediaChannels need TURN specifically? Is there some complication that prevents just dumb relaying of the packets ala what you're suggesting?

Re:Migrating Millions of Concurrent WebSockets to Envoy by SahAssar on Wednesday March 17, 2021 @05:11PM (#26496829)
Sounds like you mean STUN, not TURN.
Re:Migrating Millions of Concurrent WebSockets to Envoy by meheleventyone on Wednesday March 17, 2021 @05:38PM (#26497085)
No, I’m using a STUN server. This issue is unrelated and due to the local IPs being masked by mDNS addresses so that local network topology isn’t leaked to the world at large and my routers handling of mDNS. Which is why everything works over the local network if I disable mDNS use in Chrome. TURN is the ultimate fallback to being unable to NAT punch.

Ironically getting machines connected across the internet with WebRTC has so far been relatively smooth sailing.

Re:Migrating Millions of Concurrent WebSockets to Envoy by mypalmike on Thursday March 18, 2021 @06:30AM (#26501247)
TURN should never be necessary if one of your endpoints has a public IP address.
Re:Migrating Millions of Concurrent WebSockets to Envoy by littlestymaar on Wednesday March 17, 2021 @04:58PM (#26496702)
There's some truth in what you said, but also a few exaggerations.

First of all, while WebRTC has its share of complexity when using it for videoconferencing, here we are talking about using the DataChannel, which is really straightforward to use and doesn't need additional infrastructure.

> not forgetting WebRTC still relies on centralized components to coordinate

It needs a centralized component to setup the connection (signaling), if it fails later, your communication channel is still up. And the good thing if you have a websocket-based chat service, is that you can directly use it for the signaling purpose with zero modifications on the back-end side.

> routers in many scenarios hate it and even where they allow it, the combinatorial explosion in possible configurations to support and diagnose between peers is a problem nobody should willingly invite unless they can't achieve a solution any other way

When using the Datachannel, your failure mode is can't establish a connection, not some hard to understand Heisenbug. All you need is to provide a centralized fallback for clients who cannot establish a connection. This fallback will depend on the centralized service being up, but in case of failure you'll keep most of your users without disturbance (at least in the first world, the network is not as WebRTC friendly in other places of the world). And because the DataChannel's API is close to the WebSocket's one, implementing the fallback is straightforward.

Though, in Slack's situation there is a good reason not to use WebRTC: they can have several thousands of people in the same channel (IIRC IBM uses Slack and have most of their employees in a shared channel for official announcements). You won't be able to do that with WebRTC[1] if a user needs to establish a connection with every other users in the channel (there's just not enough ports available). And even worse, back in 2016, Chrome's implementation of the DataChannel was so poor, you could not establish more than a handful of PeerConnection before feeling the browser's becoming sluggish (this wasn't the case in Firefox so maybe Google fixed that since then).

Also, Slack's users are likely to be in some enterprise network, which makes WebRTC more likely to fail than when you customers are home, which reduces the opportunity.

Main takeaway: WebRTC-based chat is probably not a great fit for Slack, but don't be afraid of using it: it's not hard, it combines well with your already existing centralized infrastructure, and can massively reduces the load on it.

[1] unless you want to build some fancy sparse mesh network, but this is likely overengineering.

Re:Migrating Millions of Concurrent WebSockets to Envoy by SahAssar on Wednesday March 17, 2021 @05:09PM (#26496813)
It sounds like the only thing you did was signaling, not STUN and TURN.

If you do both STUN and TURN it works on most networks. I've worked at really restricted work sites, and while STUN fails at those if you have a TURN server then it almost always works.

These sort of comments are why people think webRTC is unstable while the same people use slack calls which literally use webRTC.

I might be wrong, but please don't talk about network reliability in webRTC without specifying if you have a working STUN and/or TURN setup.

Re:Migrating Millions of Concurrent WebSockets to Envoy by lovedswain on Wednesday March 17, 2021 @07:51PM (#26498234)
What has WebRTC got to do with me personally? Please try to speak to the topic, it makes for better reading, and threads much less likely to lead in the wrong direction.

> If you do both STUN and TURN it works on most networks

TURN requires clean outbound UDP/TCP connectivity which is far from ubiquitous, there are numerous corporate firewalls where "CONNECT <x>:443" is the best that could be hoped for, and even in some of those, where if the resulting connection did not include an obvious SSL handshake the connection would be immediately reset.

Re:Migrating Millions of Concurrent WebSockets to Envoy by ryanianian on Wednesday March 17, 2021 @03:28PM (#26495781)
I suspect there may be regulatory restrictions about allowing text-based communications that aren't available during an audit.
Re:Migrating Millions of Concurrent WebSockets to Envoy by ssss11 on Wednesday March 17, 2021 @03:37PM (#26495885)
What regulation do you think would apply? And how/why would this regulation differ for e2e encrypted chat products like Signal, Telegram, WhatsApp etc that can’t access text based chat messages?
Re:Migrating Millions of Concurrent WebSockets to Envoy by zonotope on Wednesday March 17, 2021 @03:49PM (#26496000)
IANAL, but the enterprise companies that make up Slack's customer base are often under regulations to preserve their employees' official communications in case they are needed for future investigations. Those same regulations prevent them from using the products you listed as official communication channels.
Re:Migrating Millions of Concurrent WebSockets to Envoy by toomuchtodo on Wednesday March 17, 2021 @03:58PM (#26496087)
FINRA recordkeeping and retention requirements, as well as SEC statute around records and reporting requirements (finance industry specific).
Re:Migrating Millions of Concurrent WebSockets to Envoy by detaro on Wednesday March 17, 2021 @03:47PM (#26495980)
Companies that have such audit requirements do not use Signal et al either for internal comms.