Mac layouts are arbitrary trees (or probably DAGs, but I've never checked) so it wouldn't be difficult to port an X Compose file.

(XKB manages to be both over-complicated and under-capable.)