Make people fill out proof-of-right-to-work-in-the-country paperwork when they get employed and proof-of-right-to-live-in-the-country paperwork when they rent or buy housing. That's essentially what most countries do. Border security is definitely helpful--especially if you want to deter outright criminals from trafficking drugs and human beings--but it's not sufficient, because if you enter the country on a time-limited visa and your visa expires, you still need a mechanism for that.

Literally not enforcing the Irish border and relying on everyone having their paperwork in order when they deliver truckloads of goods or apply for jobs or whatever isn't a completely insane idea. There's a moderate risk of UK tariffs and trade barriers being circumvented by people smuggling Czech stereo systems across the Irish border in an unmarked van and selling them on the streets of Liverpool, but maybe that's okay.