I think it's an language thing. Shoes I would wear for indoor activities like the gym I'd generally refer to as "gym shoes". Any other form of indoor shoes I'd just refer to as shoes. Indoor shoes to me immediately brings up inside the house, especially because that's how OP used "Indoor shoes" in context