As a woman who also don't like to track her fitness by pounds, dress size is not a particularly useful metric, because it's so highly variable among manufacturers. I have pants that range from size 8 to size 16. If I tried to measure my fitness by what pant size I wore today, I'd go crazy in record time.