You can start by imagining how traumatic is it to have a huge team of armed men break into your home and violently restrain you or a loved one. If you've been SWATted you don't know why these people are there. Heck even if you recognize they're SWAT, you're not going to have any clue if they really are or of they're people posing as SWAT to kidnap you for ransom or worse.

Also that SWAT team isn't just going to come in, realize you're not armed, and laugh at the situation before letting you go. For all they know, you are as dangerous as the person who called in said and were just putting on an act when you saw the SWAT team roll up. So now you're at least going to deal with questioning which is going to be extremely difficult to deal with in a state of panic and confusion.

And no, it's very unlikely that the SWAT team would be held responsible for anything. Police have too much protection against that. To spin it for them, they were operating in good faith on the assumption that they needed to raid the house in the manner they did otherwise people would die.