It widely varies with modern games, but any experienced modder facing an isolated system will use some degree of reverse engineering from analyzing file formats to scanning functions, hooking, and building trampolines, etc.

This is also how cheats for games are developed, and some communities use more malicious forms of these modding techniques for technical challenge and the fun of it, and sometimes possibility of making a little side cash.