Mathematics: Its Content, Methods and Meaning
A. D. Aleksandrov, A. N. Kolmogorov, M. A. Lavrent'ev
www.amazon.com/Mathematics-Content-Methods-Meaning-Dover/dp/0486409163/
The book gives historical contexts for the problem, prompts you to (try to) solve yourself first, and then gives the complete mathematics.
Most of the ones below seem somewhat too basic for me.
In university libraries you can sort of pick out narrative/casual books in the math section by working off of the form factor. If you pull out books with a novel-like size and shape and read the backs you might find ones that are the right mix for you.
One last clue is that publishing houses like Springer will put out series of more narrative or casual works from time to time. Best of luck!
More seriously: From glancing at the recommended list (bottom of the linked page), a few books I heartily recommend: "Uncle Petros and Goldbach's Conjecture", "Logicomix", and of course: "Godel Escher Bach".
I would love a list of books that are actual Math books, but for a popular audience. I'm not talking about something like "Fermat's Last Theorem", as it does not actually teach real math.
My best example of such a book is "Journey Through Genius", which is an amazing book - a look at 10 great mathematical theorems, from explaining the basic theory behind them, on through their history, and ending with a full proof of the theory itself. This is how textbooks should be written, if our goal is for people to enjoy mathematics - explaining why mathematicians came up with the theorems they did, including the always-interesting human angle of the history. But still keeping full mathematical rigour.