Given that the colleges are different, this can mean it's important to choose the right one for you. For example, at Cambridge, Trinity College has an illustrious history in mathematics, and attracts excellent staff, so if you're accepted there then you might end up being taught one-on-two by a world renowned mathematician (Fields Medal winning Timothy Gowers for example). Other colleges with less of a reputation might have more trouble attracting the same type of staff. (this isn't to say that being an expert in your field makes you an excellent teacher of course...)
You can also make an "open application" which is similar to pooling where you say you don't mind which college you get into.
Generally applicants tend to put too much emphasis on the "right" college, the academic differences aren't that great, and a lot of it is just candidates wanting to get into the more prestigious colleges for prestige's sake.
As students become more specialised, or if their college doesn't have many professors in their particular subject, they may have tutorials with academics in other colleges.