Detection and logistics, mainly. Weapons are a special case, since if we're still talking mainly about the US/Mexico border, I would expect that weapons do not move in the same direction as drugs, and expertise in smuggling one way across a border may not translate to smuggling the other way.
Nuclear material will require at least some shielding to avoid detection, and it would seem to me that a well-implemented detection system would notice abnormally quiescent cargo as well. That would mean that you have to carefully tune shielding to present the right level of radioactivity, rather than just piling on more to make it as low as possible.
Smuggling people of any kind requires life support; there are a lot of tactics you wouldn't be able to use without losing them. For some types of human cargo, the smugglers might find some losses acceptable, but angering terrorists by losing a container of their buddies might not be healthy. In any case, it seems that there's already a lot of people walking across the border into the US, and I'd expect that if there were an obviously safer or more reliable way, they'd use that. Lots of assumptions, there, of course.
Consider the cargo container. A large metal box. It'll hold several tons of drugs, or just as easily 50lbs of radioactive material and 1 ton of shielding. Or 5 people and several jugs of water.
It's not clear to me what the challenges are with smuggling radioactive material, or how good detection protocols are. That doesn't mean I can't think of any, or solutions to same, but it means that I expect there are at least some consequences I'm not thinking of, and it's certainly not as easy as some sealed, inert cargo.
The largest cocaine bust was on the order of 48,000 pounds. Drug smugglers, on the high end of things, have access to submarines and secret tunnels that cross nation boundaries.
Child trafficking happens, and it the overlap between the major drug movers and the child traffickers is significant -- because they are both serving black market commodity on the level of distributors.
There are, however, a whole lot less people out there interested in buying children compared to the number interested in buying pick-a-substance. The reduced amount of profit available to high-level black-market distributors when you take away the market for drugs would, in my opinion, make it harder to find people willing to smuggle children.
In a lot of places (US Southwest especially), the penalties for drug smuggling are far higher than for escorting groups of people -- if you get caught, you might go free if it's just a group of workers. The humans can transport themselves on foot. The humans can be held hostage afterward and you can try to extort more money from their family; if a $20k kilo of coke is held for ransom, no one is going to pay more than the discount to an alternative kilo for it.
Taken literally, this seems extremely unlikely.