> It occured to me recently, why don't they talk about the holes in the multiplication table?

Because there aren't any. The prime numbers would be gaps if you pivoted the table into a division table but, alas, that's not how it's presented (even in the context of teaching division.)