Assuming you a male, along with your brother, looking at IQ graphs there would be equivalent males on the dumb side, who arguably would be in prison right now if they did what you did.

Also without you at a normal school some students might be worse off. Not your problem of course, but it is at a societal level.