What I've also seen is that a team with plenty of churn or no time for maintenance will have exactly the problems you describe. As knowledgeable people leave the team, the old code becomes "legacy", new people come and partially refactor it, maybe adding some new gem and planning to migrate the entire codebase later on to it (which is very hard to make it happen). Stakeholders' pressure will force the team to focus on delivering features and only tackling debt when it gets to an unattainable state, and then is when consultants are called. I think this is why many freelancers suffer the worst of Rails (or any other framework) if they don't usually work with greenfield projects.
But this is easier and more rewarding in a garden that is well layed out and allows flexibility to move things around.
An old, crumbling shed, in the middle of that garden limits what you can do with it. It is always in the way, always visible and often defines the entire garden, when you really don't find it that important.
A gem, like devise, defines what your users are (it limits your freedom to define the Domain model), how your onboarding flow works (which is probably your most important money-maker) and what features you can implement and what not.
When you started with the garden, it was handy to have a quick and ready place to store your tools. So the first cheap, off-the-shelve shed, is the best solution. But once your garden is defined, you probably want to move it aside, nicely integrated in your garden and well maintained and exactly large enough for your tools, so it leaves as much space for your flowers and trees -the stuff that matters-.
I'm on the other end of the spectrum.
I mean, I see the benefit of such "roll X quick by adding a gem" for a proof of concept, a demo or even an MVC.
But even for the latter, it breaks down rapidly. Such gems bring their own libraries. They force you to "do it their way". A gem like devise comes with its own addons (omniauth) that paint you even quicker into a corner.
I've consulted and developed at many Rails codebase where most of the problems were caused by years of development on top of such gems. A weak, wobbly base, and the entire castle built on top of that. Where 90% of the devise features were unused or worked around. Where more code was present to customise and work around devise-isms, AASM-weirdness, CanCan-troubles, than a complete "roll your own from scratch" would have ever caused.
So, my "devise advise" always is: sure, use it to get your product to market rapidly. But be sure to pull it out and replace it with a focused, fitting architecture, sooner rather than later.
Such gems fit very well in the problematic development-workflow where one "searches for an existing gem which offers some version of Foo" rather than "figuring out how we want to do Foo, then look for a gem that does it our way".