jbryu

Whoa did not expect the CEO of Cloudflare to comment here! Thanks for the response. The extended periods of high latency was concerning, but I did some more digging and saw that your team is aware of this and working on it: https://www.answeroverflow.com/m/1409539854747963523 Hoping things work out!

Thanks for the response. After doing some more digging it looks like this is a known issue at Cloudflare and they're actively working on it: https://www.answeroverflow.com/m/1409539854747963523

Side-quest productivity is a great way to put it... It does feel like AI effectively enables the opposite of "death by a thousand cuts" (life by a thousand bandaids?)

For forking and changing a few things here and there, I could see how there might be less of a need for LLMs, especially if you know what you're doing. But in my case I didn't actually fork `ts-rest`, I built a much smaller custom abstraction from the ground-up and I don't consider myself to be a top-tier dev. In this case it felt like LLMs provided a lot more value, not necessarily because the problem was overly difficult but moreso because of the time saved. Had LLMs not existed, I probably would have never considered doing this as the opportunity cost would have felt too high (i.e. DX work vs critical user-facing work). I estimate it would have taken me ~2 weeks or more to finish the task without LLMs, whereas with LLMs it only took a few days.

I do feel we're heading in a direction where building in-house will become more common than defaulting to 3rd party dependencies—strictly because the opportunity costs have decreased so much. I also wonder how code sharing and open source libraries will change in the future. I can see a world where instead of uploading packages for others to plug into their projects, maintainers will instead upload detailed guides on how to build and customize the library yourself. This approach feels very LLM friendly to me. I think a great example of this is with `lucia-auth`[0] where the maintainer deprecated their library in favour of creating a guide. Their decision didn't have anything to do with LLMs, but I would personally much rather use a guide like this alongside AI (and I have!) rather than relying on a 3rd party dependency whose future is uncertain.

[0] https://lucia-auth.com/

nvm I'm dumb lol, `ts-rest` does support express v5: https://github.com/ts-rest/ts-rest/pull/786. Don't listen to my misinformation above!!

I would say this oversight was a blessing in disguise though, I really do appreciate minimizing dependencies. If I could go back in time knowing what I know now, I still would've gone down the same path.

Just last week I was about to integrate `ts-rest` into a project for the same reasons you mentioned above... before I realized they don't have express v5 support yet: https://github.com/ts-rest/ts-rest/issues/715

I think `ts-rest` is a great library, but the lack of maintenance didn't make me feel confident to invest, even if I wasn't using express. Have you ever considered building your own in-house solution? I wouldn't necessarily recommend this if you already have `ts-rest` setup and are happy with it, but rebuilding custom versions of 3rd party dependencies actually feels more feasible nowadays thanks to LLMs. I ended up building a stripped down version of `ts-rest` and am quite happy with it. Having full control/understanding of the internals feels very good and it surprisingly only took a few days. Claude helped immensely and filled a looot of knowledge gaps, namely with complicated Typescript types. I would also watch out for treeshaking and accidental client zod imports if you decide to go down this route.

I'm still a bit in shock that I was even able to do this, but yeah building something in-house is definitely a viable option in 2025.

unintuitively, less cores ended up being the fix... I did a small writeup here: https://news.ycombinator.com/item?id=44436679

Nut has been cracked! https://news.ycombinator.com/item?id=44436679

And yeah, I've been using prometheus' `collectDefaultMetrics()` function so far to see event loop metrics, but it looks like node:perf_hooks might provide a more detailed output... thanks for sharing

Big thanks to everyone who commented so far, I wasn't able to reply to everyone (busy trying to fix the issue!) but grateful for everyone's insights.

I ended up figuring out a fix but it's a little embarrassing... Optimizing certain parts of socket.io helped a little (eg installing bufferutil: https://www.npmjs.com/package/bufferutil), but the biggest performance gain I found was actually going from 2 node.js containers on a single server to just 1! To be exact I was able to go from ~500 concurrent players on a single server to ~3000+. I feel silly because had I been load-testing with 1 container from the start, I would've clearly seen the performance loss when scaling up to 2 containers. Instead I went on a wild goose chase trying to fix things that had nothing to do with the real issue[0].

In the end it seems like the bottleneck was indeed happening at the NIC/OS layer rather than the application layer. Apparently the NIC/OS prefers to deal with a single process screaming `n` packets at it rather than `x` processes screaming `n/x` packets. In fact it seems like the bigger `x` is, the worse performance degrades. Perhaps something to do with context switching, but I'm not 100% sure. Unfortunately given my lacking infra/networking knowledge this wasn't intuitive to me at all - it didn't occur to me that scaling down could actually improve performance!

Overall a frustrating but educational experience. Again, thanks to everyone who helped along the way!

TLDR: premature optimization is the root of all evil

[0] Admittedly AI let me down pretty bad here. So far I've found AI to be an incredible learning and scaffolding tool, but most of my LLM experiences have been in domains I feel comfortable in. This time around though, it was pretty sobering to realize that I had been effectively punked by AI multiple times over. The hallucination trap is very real when working in domains outside your comfort zone, and I think I would've been able to debug more effectively had I relied more on hard metrics.

I'm somewhat buffering right now - Everytime the current turn player types I buffer their input on the backend, and I have a job setup that broadcasts typing events every ~200ms using this buffer.

I could increase this interval, but I'd like to keep it as short as I can afford to to keep that realtime feel (i.e. other players can see what the current turn player is typing).

Ahhhh I see what you mean now. You just gave me some good ideas. Alas because of the nature of my game, it will always have first person shooter esque networking problems despite it being turn-based. But it's good to know that I'm dealing with a non-trivial level of throughput.

Yeah changing the sysctl options was a shot in the dark... I really hope it's my app code. But the fact that the same bottleneck occurs even when I add more containers which decreases the load per container confuses me. I mentioned this in another comment but I wonder if socket.io broadcast calls share the same I/O resource or something. Maybe a lock?

Good point. I actually don't know what performance looks like with 500 real users. The way I'm mocking right now is by running a script on my local machine that generates 500+ bots that listens to events to auto join + play games. I tried to implement the bots to behave as closely to humans as possible. I'm not sure if this is what you mean by keeping traffic internal to my box's OS, but right now this approach creates lag. I didn't consider whether spinning up hundreds of websocket connections from a single source (my local machine) would have any implications when load testing hm

So when running `top` WCHAN shows `ep_poll` most of the time and sometimes `-`. Even when the game starts lagging this pattern stays pretty consistent.

There is no cross-room communication. I could spawn a process per room but I was trying to address this issue with my current Docker setup where I have multiple `game` containers that run a single node.js process and each process can host multiple rooms.

Not having to use Docker sounds simpler but it's that's where I'm at atm haha.

I agree that the network load feels very small. Maybe it's a socket.io related issue where when many broadcasts are being fired at once, then a shared I/O step gets bottlenecked?

Here's my actual typing broadcast code, I was originally broadcasting from the socket event callback itself but I found performance improved slightly by batching broadcasts per player in a setInterval loop (also note that only 1 player in a given room can be typing at once, so batching broadcasts per room shouldn't address the bottleneck).

  /**
   * Used to handle very frequent typing events more gracefully to avoid overloading CPU
   */
  const TypingUsersMap = new Map<
    ConnectionId,
    {
      socketId: string | null; // doesn't exist for bots
      roomId: PublicRoomId;
      userId: UserId;
      currentInput: string;
    }
  >();

  type ConnectionId = `${UserId}:${PublicRoomId}`;

  // ! this should be same as client throttle interval
  const TYPING_BROADCAST_INTERVAL = 200;

  export let typingBroadcastInterval: NodeJS.Timeout | undefined = undefined;
  export const startTypingBroadcastJob = () => {
    typingBroadcastInterval = setInterval(() => {
      const freshTypingUsersMap = new Map(TypingUsersMap);
      TypingUsersMap.clear();

      if (freshTypingUsersMap.size === 0) return; // Nothing to do

      // Go through each user that has a pending update
      for (const [_connectionId, data] of freshTypingUsersMap.entries()) {
        const socket = data.socketId
          ? io.sockets.sockets.get(data.socketId)
          : undefined;

        // Use the data we stored to perform the broadcast
        if (socket) {
          // emit to other players
          socket
            .to(data.roomId)
            .volatile.emit(
              SOCKET_EVENT_NAMES.USER_TYPING_RES,
              data.userId,
              data.currentInput
            );
        } else {
          // bots emit to everyone
          io.to(data.roomId).volatile.emit(
            SOCKET_EVENT_NAMES.USER_TYPING_RES,
            data.userId,
            data.currentInput
          );
        }
      }
    }, TYPING_BROADCAST_INTERVAL);
  };

  export const stopTypingBroadcastJob = () => {
    if (typingBroadcastInterval) {
      clearInterval(typingBroadcastInterval);
      typingBroadcastInterval = undefined;
    }
  };

  // this is called from the USER_TYPING socket event callback. so effectively every throttled keystroke by the user gets queued.
  export const queueTypingEvent = ({
    socketId,
    roomId,
    userId,
    currentInput,
  }: {
    socketId: string | null;
    roomId: PublicRoomId;
    userId: UserId;
    currentInput: string;
  }) => {
    const connectionId: ConnectionId = `${userId}:${roomId}`;
    TypingUsersMap.set(connectionId, {
      socketId,
      roomId,
      userId,
      currentInput,
    });
  };

Yep just triple checked. If distributing the load on a single server by adding more backend containers doesn't decrease ping then maybe this is just the natural upper bound for my particular game... The only shared bottleneck between all backend containers I can think of right now is at the OS or network interface layer, but things still lag even when I tried increasing OS networking limits:

  net.core.wmem_max = 16777216
  net.core.rmem_max = 16777216
  net.ipv4.tcp_wmem = 4096 65536 16777216
  net.ipv4.tcp_rmem = 4096 87380 16777216

Perhaps the reality for low latency multiplayer games is to embrace horizontal scaling and not vertically scaling? Not sure.

I thought this might've been the issue too, but because the game is turn-based there should only ever be 1 person typing at once (in a given room).

Interesting, what was bad about your Redis experience on Coolify? I was just about to do the same for a personal project.

Looks like they recently updated their ToS as well: https://www.diffchecker.com/w4dbxWwt/

That's fair. Given that you can't do this programatically with their api (it disallows scraping prompts), it feels less prone to abuse. And even if a bad actor tried to leverage their web api instead of their official api to get around prompt limitations they could easily just ban you.

It's cool that scarcity can breed innovation as well. An example I like is how the Swiss went from manufacturing general watches to very successful luxury ones because they were scarce in labour resources and decided to make a pivot given their constraints at the time.