I guess it depends on your sensibilities. Tokyo in the summer matches where I live in the summer, so it's no real change. Also I'd say weather is all part of the experience. Places like Sapporo or Sendai are great to visit in the winter, because you get a real winter experience. Something we don't get in my country.

Most countries have more than 4 season in practical terms though, and going during the "seasons inbetween" is my recommendation. In Australia, the indigenous have up to 6 different seasons they identified depending on the location/tribe, and they have more nuanced events expected for each season. Most countries are like that in real terms, so intimate knowledge can get you a great budget off-season holiday experience.

https://en.wikipedia.org/wiki/Indigenous_Australian_seasons