I don't think there is a spec dictating that the default location for saving files needs to depend on the file tile - which program is doing this? I think Firefox remembers the last used path per file type, even across restarts.

The Desktop, Downloads, Pictures, etc. directories themselves are not from the XDG Basedir spec and not set by XDG_ environment variables but instead in the $XDG_CONFIG_HOME/user-dirs.dirs file. I also find these less useful than the more generel data/config/cache dirs and have most theme set to my home directory - perhaps that is affecting how Firefox works?