It's a term from the UI design world, and it goes way back. A mode is a state where the app accepts certain inputs from you, the user. A new mode means previous buttons and shortcuts are inaccessible, and the same input may produce different results. Either coincidentally or directly related, Vim's modes illustrate the idea well—in contrast to editors where input, motion and shortcuts for extra functions are available together.

https://en.wikipedia.org/wiki/Mode_(user_interface) and https://en.wikipedia.org/wiki/Modal_window

The trouble with elaborate modes is that the user has to mentally keep track of which mode they're in (which is what the term describes); and that other functions are inaccessible even though they might be useful to resolve the situation. However, modal dialogs are a streamlined and visually distinct extreme of modes, with an established history, so some don't consider them problematic. (Modes that are difficult to notice are still a no-no.) OTOH dialogs are frequently used as cheap bail-out by programmers, especially in desktop interfaces—since they're very easy to produce, completely synchronous, and shift all problems onto the user.