We built the engine when we realized that these scrapers were a competitive advantage for the company and we'd be writing a lot of them (when I left, we had around 250 of them with a backlog of near 1000 to be written). It became worth the investment to write the framework because there were features we wanted that would have been hell to integrate any other way - for instance, killing long-running scrapers that may have been hung up. The engine bought us lots of other niceties too - for instance, centralized error reporting. If one particular scraper returned lots of errors, odds are that the site changed and we knew to give the code some attention.
When you describe it as a "big enough fence", that's only partially true. It's more about a series of independent tasks that don't depend on each other. When you have that and those tasks have only minimal impact to the system as a whole that you have a scenario which suggests drone programming.
When I left, I mentioned in the exit interview that, for my salary and benefits, the company could hire 4 interns and put them to work. Even if they were twice as slow as I was, which was unlikely given all the work that went into the framework, the company would have a net productivity win, not to mention the benefits that come with having interns around to recruit from. Last I heard, they took me up on that suggestion and picked up 10 interns who are all writing and maintaining scrapers and freeing up full-timers to work on less routine work.
It can work, but you need a steady supply of relatively routine work that's set up in such a way that nobody can break the system with a simple bug or logic error.