You could grab one or two from a thrift store and see how you like it, then upgrade if it seems like something you want to keep doing. Nice and cheap way to try it out. Difficulty: a nice light linen jacket's probably gonna be a little hard to find at a thrift store, so it may be best to start this attempt in the Fall.
I wear a short sleeve polyester t-shirt in black. I own a dozen of them, all identical, and wear them most days. My wife hates it.
The black is not good on very hot days. It’s surprising how much difference a light color makes. But, I can’t seem to keep light stuff clean.
I don’t know if I could stand my arms in a sport coat, but I guess it’s no different from a regular jacket in the winter.