Most dark matter is expected to reside in a halo, much larger than the visible part of the galaxy:

https://en.wikipedia.org/wiki/Dark_matter_halo

So it's more weakly bound than most of the stars. Technically, it's outside the tidal radius:

https://academic.oup.com/mnras/article/366/2/429/1213793

What it seems like is most likely is that dark matter is going to turn out to not be some mysterious unobtanium new physics substance, but rather likely just things which aren’t very bright — dust and rocks and rogue planets and gas and such... stuff that is hard to see and therefore hard to account for in our galaxy formation models.