LaTeX with PGF/TikZ gave me higher quality typesetting of text, mathematics, and technical diagrams than other alternatives I considered. Learning PGF/TikZ made me suffer, but I wouldn't choose anything else now, especially for complicated or exact diagrams. For a build tool I used latexmk. The memoir document class easily handled a full length book. Wrapping text around images was reasonably effective with the wrapfig package, albeit slightly buggy. I used the Gimp to remix public domain clipart just for fun illustrations, and Blender for some crazy looking 3D cover art. If I were better at Blender I would have used it more extensively. My workflow was limited to a PDF target because I couldn't imagine any ebook or web format doing justice to the math, but it did a nice job of embedding hyperlinks into the document for things like equation numbers and section numbers that are usable in PDF readers such as evince. As others have noted, this advice might not be practical if you're working with a publisher and have to use whatever boring enterprise software the publisher demands.