Yeah unfortunately there's a steep learning curve, but worth it for anything math-related.

Here's a sample chapter laid out with LaTeX: http://docs.google.com/viewer?url=http://betterexplained.com...

The #1 rule is to not use the default styles unless you want your book to look like a master's thesis.