That's a problem for many common actions -- for instance, I normally want to resize the view box to enclose the content tightly before saving an edited file, so why is that common action buried multiple layers deep in the UI? File->Document Properties->Page->Custom size->Resize page to content->Resize page to drawing or selection->Seriously? It's almost easier to do this in a text editor.
Finally, people who write apps that don't remember their previous desktop window size and position, but that instead auto-size themselves to cover the entire screen every time they're launched, should die in a grease fire. Unnecessarily-aggressive hyperbole aside, this kind of obvious forehead-slapping behavior is never good news because it means that the maintainers don't use their own app.
All that aside, InkScape is really a very powerful application that could be a lot harder to use. A great deal of hard work has clearly gone into it, and at the end of the day it is free software, so it's hard to complain without seeming ungracious. If it weren't such a great app, it wouldn't be worth criticizing at all.
> the authors seem to think I most likely want to drag the entire image around.
I don't think that is the default, unless the whole image happens to have been grouped into a single element, which might of course be the case.
Anything particular you have in mind?