The universe's expansion, and a black hole's increase in mass over time, are unrelated phenomena. We could have one without the other. In fact, because of Hawking radiation, in the far future we might see a larger universe accompanied by smaller black holes.