But this is easier and more rewarding in a garden that is well layed out and allows flexibility to move things around.
An old, crumbling shed, in the middle of that garden limits what you can do with it. It is always in the way, always visible and often defines the entire garden, when you really don't find it that important.
A gem, like devise, defines what your users are (it limits your freedom to define the Domain model), how your onboarding flow works (which is probably your most important money-maker) and what features you can implement and what not.
When you started with the garden, it was handy to have a quick and ready place to store your tools. So the first cheap, off-the-shelve shed, is the best solution. But once your garden is defined, you probably want to move it aside, nicely integrated in your garden and well maintained and exactly large enough for your tools, so it leaves as much space for your flowers and trees -the stuff that matters-.
What I've also seen is that a team with plenty of churn or no time for maintenance will have exactly the problems you describe. As knowledgeable people leave the team, the old code becomes "legacy", new people come and partially refactor it, maybe adding some new gem and planning to migrate the entire codebase later on to it (which is very hard to make it happen). Stakeholders' pressure will force the team to focus on delivering features and only tackling debt when it gets to an unattainable state, and then is when consultants are called. I think this is why many freelancers suffer the worst of Rails (or any other framework) if they don't usually work with greenfield projects.