The accuracy isn't in the process of grinding; it's in the process of measurement. All the grinding process needs to do is to remove only a small amount of material, which is itself a nontrivial problem; a single large grain of grit can ruin weeks or months of work, so amateur astronomers may get very angry at you for things that seem unreasonably minor, like opening a door without washing your hands. But that's not a problem that scales superlinearly with mirror size.

Really large mirrors are segmented anyway.