You can play with an online synthesizer to create timbres whose octaves are dissonant:

https://raw.githack.com/CindyJS/ScaleLab/master/index.html

There's the book by Dave Benson [0] (available online), mostly about the mathematical modeling of instruments, that has a nice ethnomusicological compendium of instruments with weird timbres.

And then there's the infamous book by Sethares [1] that is all about the dependency of harmony on timbre.

[0] https://homepages.abdn.ac.uk/d.j.benson/pages/html/maths-mus...

[1] https://sethares.engr.wisc.edu/ttss.html