As an aside, I have said for a long time that I wish it was socially uncontroversial for a guy to wear a loose skirt or dress in public, and that if I found one I liked I would wear it all the time.
Yeah it’s becoming a bit less uncommon but not by much.
Maybe you could make up a Scottish heritage and wear a kilt? Other options would be overalls (there’s overall high-fashion) you might have the same shape issue as pants but they won’t fall down thanks to built-in suspenders (which you can play with e.g. cross, halves, …), or onesies (kigurumi) if you don’t mind being seen as a weeb.
If the pants fit, I really don't think anyone is going to notice.
Avoid the pants with pink flowers on them.
That being said, you might want to go to Kohl's and try on Lee women's pants. Those aren't bad, although I hate the way Kohl's marks stuff up and then puts a normal price on it and calls that a sale.
Have you considered looking at the women’s? Might be no better in the end, but this reads like what’s conventionally a feminine anatomy, and garment designers will generally go with conventional anatomy, if not outright prescribe completely imaginary ones you have to conform to.