For starters, you need to define what you consider success and failure. If it's to make a ton of money, you should probably reconsider your options right there. If it's to spend a bunch of time learning and thinking about a topic that may be something else. If it's just a fun side project (just wrapping up another one of those), that's fine too.

Totally agree. If you focus on the money you'll just be frustrated. I could have made more money spending those hours driving for Lyft instead. I deliberately haven't tracked my hours so I won't be discouraged at my hourly rate. Even spending that time on leetcode could get you a better job and more income.

But, there's nothing quite as satisfying as setting out to do something really difficult and seeing it come together. And unlike most other things in your career where you're part of a team and sharing your success, a book basically rests solely on your shoulders. So it's easy to point to it and truthfully say "I did that".

Plus, it turned me into a really good writer.

I used to publish and have some anecdata you may find encouraging. In my experience, if you’re thoughtful enough to wonder if you have something interesting to say, you likely do.

Aside from that bit of data, there is one definite advantage to writing. The process of writing a book is a great way to learn more and get better at explaining what you know. This is a double edged sword because if you spend six months on a first draft, you will likely see a major difference in quality between the first section you wrote and the last.

You may make some money, but you’ll almost certainly learn something.

To some extent, this is true of many things where you do work in advance and then try to sell it. That’s just what a lot of business is. In this case I assume the author got a lot of feedback over the years in the form of ‘I’m a programmer and I liked your article and I always wanted to learn mathematics and never learned how’. And, like, that sort of thing doesn’t mean writing a book on the topic would be successful, but it does seem like a bit of a hint.

I think in general the answers might be something like ‘look for clues’ or ‘try to get feedback on whether the idea is good early’ but I also think a lot of the time it’s just people writing books they want to write or that they guess people will be interested in. But I’m definitely guessing and I don’t really know.

It's hard for me to pin it down, but after blogging for 10+ years and working on my second book now, I kind of just have a good feeling for what will work. The topic and my approach to it feels true and speaks to my experience.