It has been demonstrated possible for a large range of projects that are similar in complexity. For example: web browsers (Firefox), OS kernels (Linux), 3d-modelling applications (Blender), compilers (GCC), databases (PostgreSQL).

Won't be easy for sure, neither on technical nor marketing level. Many of the successful examples took 10+ years before they got really viable. But to say it is impossible to do this for a game engine, just because no-one has done it yet, is defeatist.