I'd argue that the modern definition has nothing to do with immigrant status, and is entirely about whether or not you can pass as a native English speaker. It just so happens that most people who have accents are immigrants, and you can reasonably assume that anyone who didn't grow up speaking English as a native language is going to have an accent.

You are not a fob if you moved here from China at 9 years old, didn't read or speak your first word of English until then, and have no discernible accent as an adult.

No one would call a Hong Konger with a HK English accent a fob either.