The youtube captions toggle is guilty of this too. It toggles between black on white and white on black. I just have to click it and wait until I see someone's lips moving with no captions then try the other icon mode.
It's like the move to touchscreen erased everyone's common sense for stateful positional input feedback.
For example, in this specific app, the flag/no flag is easier to discern its use compared to the cloud/sun icon. It doesn't follow the same logic as the flag/no flag. It could be cloudy/not cloudy cloud or sunny/not sunny, and then it follows the flag. Consistency between buttons is important
While I've seen similar problems the captions toggle hasn't been one of these for me. On desktop it uses a red underline when activated which is very clear that it's currently activated. On other platforms, I've seen the button's background toggle between translucent button with outlined icon to white button with filled icon, which I find intuitive but I guess others might not.
There is a horizontal row of elements for the Seasons of a show, and then below that is a horizontal row of thumbnails for each episode in that season.
The only way you know whether you are currently focused on the Seasons or the Episodes row is that the element you are focusing is perhaps 2.5% larger than the other elements.
There is no border, no shaders over the element - nothing to tell you where you are without arbitrarily moving the focus around just to gather some context.
And you certainly aren't going crazy either, I shipped a last-minute bug-fix where I got the cloud icon the wrong way around compared to everything else
I'm also having the same issue on a current project. There's a set of +/- buttons that control the size of a time range window. Clicking the +/- buttons increase/decrease the size of the window, but for the start time + makes the time seconds value smaller, and - makes the seconds value larger. For the end value, it is exactly the opposite. But for the window size, both sets of buttons do what one expects. So it's a matter of perspective that needs to be clear to the user.