Hmm, I'm no expert. But AFAIK open memory mapped files are counted towards a process's virtual memory (as displayed by utils like 'top' and 'ps). Dunno if FF uses memory mapped files extensively.
There's a utility called 'pmap' that you can use to display the memory mapped files and subtract it from the total etc.
Finding out how much memory a program is using is surprisingly tricky to compute. Good luck.
I'm running Firefox 15 / Iceweasel 10 on Ubuntu and Debian respectively, watching VIRT (total virtual memory size). Currently sitting at 2.3 GB, FML.