Well fitting clothes are comfortable too. You don't need to get brand name items either. There's nothing wrong with not caring about fashion - indeed, people shouldn't be judging others negatively just because of fashion. However, why not also look good? It can benefit the social aspect of your life a lot, by giving you some added confidence. So if you're looking to improve that part of your life, dressing well is a good place to start.