"The little guy" means something similar, but with the added connotation of being an underdog (which sounds like what you describe "Der kleine Mann" as). Tiny idiomatic differences!