Depending on the complexity of the geometry this is within the realm of reason. Fairness might not be a huge issue as long as you make sure visibility is a two way street. People will probably complain about high latency players popping into existence after they're already most of the way around a corner though.
The server already has to do a full 3D simulation for each client. The server just doesn't send updates for players that are not visible to another player. So yes the server has to do a bit more work, on the other hand it reduces server bandwidth.
darkplaces quake does this. see sv_cullentities_trace.
Wouldn't that require the server to do some amount of 3D logic every netcode tick for every player? And then you'd get into real latency problems, since either you do rollbacks to account for when people are allowed to see each other or you let the fastest connection win.