"You don't need a framework" assumes your developers will write code that operates with everyone else's code nicely, without accidently writing something that will be painful to remove in a year's time.

Frameworks generally enforce a pattern where things are quite well encapsulated, don't pollute global spaces, don't mess with prototypes, and manage callbacks well. Code written by one person, or a group of well-organised experts, also does that, but as soon as you have a less talented team, or a couple of juniors who aren't being overseen because the seniors are too busy, things creep in to the code that have the potential to blow up later. Frameworks help your team avoid those footguns (and introduce some different ones, but at least they're usually easier to manage).