You've provided insufficient data. What's your employment history and compensation? How "small" is the company and the team? Your depression and anxiety "worsened", so they existed before this situation? What are "ambitious" and "unreasonable"?

Don't lose sleep over the manager's opinion. He's not unbiased.