How exactly would an open source anti-cheat do its job effectively?

Being opensource doesn't imply the game server will allow modified builds to work. Being able to see the code doesn't mean you will find a suitable way to circumvent the anticheat.

That being said, I agree it would be harder to maintain an open source anticheat effectively.

> Being opensource doesn't imply the game server will allow modified builds to work

The client can lie about being modified. E.g. it can send false hashes.

The same way open source crypto libraries do.

Please explain in more detail? I am curious to hear your theory on this.

By using hardware attestation? Same mechanism used by secure boot and DRM could be used for anticheat.

How would hardware attest to the game server that the player cannot see through walls, and that their aim is not nudged (subtly or overtly) in the direction of enemy faces? Keeping in mind that the cheater has full control over the software running on their computer, so they can decide what to send to the server. Also the cheater doesn't need to alter the game itself, they could access the game memory and implement their aimbot by having a clever mouse-driver.

It will always be a game of cat and mouse anyway

The same way that other open source security-critical software does it.

I think it's a choice with lots of positives and negatives tradeoffs on both sides (in-kernel anti-cheat vs userland anti-cheat), where any choice is not gonna make everyone happy.

How much data a in-kernel or user-land anti-cheat can easily be detected by observing the traffic that flows out from your network, so it really doesn't matter if it's open source or not.

The biggest roadblock to a open source in-kernel anti-cheat is not "exposing the amount of data they extract from you", but rather it exposes how the anti-cheat is working, which is working against the efficiency of the anti-cheat. If you know how they detect you're cheating, it's much easier to overcome that hurdle.

In most cases, security-by-obscurity is obviously flawed, but when it comes to cheat/fraud detection, exposing how it happens makes it's core value less efficient.