Isn't it true that if you were falling into a black hole, you would see the end of the universe before you crossed the event horizon? Therefore from our frame of reference nothing has ever actually fallen into a black hole? Or maybe I'm thinking of reaching the singularity at the center of the black hole.

That's not true. If you are falling into a black hole, to you time will flow normally until your sphaghettified demise. To an observer at infinity, they'll never observe you going through the event horizon because no light can reach the observer from within the event horizon and because gravitational time dilation.

But doesn't the rest of the universe speed up to such an extent that billions of years pass (our frame of reference) before you cross the event horizon? It would just pass extremely quickly (from your frame of reference).