Exactly, port scans on my public IP address are not an attack, but crossing the boundary to my localhost and private networks is malicious behavior.