Public/state workers and university students were a small elite until the 20th century, as were schoolchildren until a fairly late date. Also, the demographics you speak of were males, meaning that you are leaving out a half of the population.

Certainly German words in Croatian testify to contacts at some level of the population. Romanian, too, abounds in German words in some domains, for example, but this doesn’t mean the average Romanian would have been able to speak any German. But in Croatia knowledge of German varied greatly from the country’s northwest to its south. Again, the sociolinguistics of the Austro-Hungarian Empire are well-described.