Some of it has been open violence, of course, with lynching and race massacres being the most obvious. There was also plenty of more quiet violence, the unmarked graves and the vicious but survivable beatings.
But that's relatively rare because it is backed up by a host of more subtle things. Things that might lead to violence, especially if an uppity person persisted in acting like an equal. Threats, of course, but also menacing looks, harsh words, bad attitudes, etc.
This is summed up in ADL's pyramid of hate. The top layer is built on the layers beneath: https://www.adl.org/sites/default/files/documents/pyramid-of...
So we talk about microaggressions because the societal system of white supremacy uses both macroaggressions and migroaggresions as a continuum of actions that maintain the racist status quo, continuously informing both black and white people of their assigned place.
A misunderstood social signal (e.g. a raised eyebrow) can be called a micro-aggression.
I don't think you actually know what "microaggression" means. You should educate yourself on its definition.
The main source of microaggressions are in fact words, and words that while rooted in ignorance (and people, like you, should educate themselves), are not in fact in any shape or form "aggressive" or even have any form of negativity associated with them.
Take the "you're the whitest black person I know", from the "I, Too, Am Harvard". Well, that's sure a stupid thing to say. But is it "aggression"? Obviously not.
If you think that's not the case, go out and try doing that to the first 10 men you see on the street. Heck, try it with a couple of cops.
So yes, calling more modest unwanted touching a microaggression is perfectly appropriate.