I assume you make a new mach-o file that hasn't been signed by apple and try to exec into it. If it runs, definitely jailbroken. Obviously a jailbroken phone could load a kernel module that detects this and stops it from running in this specific case, leading to a cat and mouse game between jailbreak developers and these apps. iBooks tried this once.