It can be confusing when other languages get added to the mix, because they don't necessarily follow that convention. E.g. in Russian, you traditionally translate the main meal of the day as "dinner" in English - but it's actually eaten early in the afternoon (usually around 1pm), so it's more like lunch in that regard, and honestly should probably just be rendered as such. I often found that whenever I would refer to "dinner" in my conversation, my (native English speaking) friends would automatically assume the end-of-the-afternoon meal; and the idea of having a dinner with colleagues implies some sort of highly formal event.

On the other hand, the after-work evening meal is usually less heavy, and is rendered as "supper" in English. And, again, I find that Americans usually assume that it takes place much later than it normally does, and is also much lighter than it normally is (basically a snack rather than a proper meal), if I use that word.

So clearly there is some implicit time-of-day mapping there.