I am not sure that the time it takes to have an effect is a good metric for whether or not something has potential for abuse. Sure SSRIs take a few weeks to have a noticeably effect but they also takes weeks to tail off. The area under the curve is not necessarily different.
https://www.quora.com/Why-are-SSRIs-typically-prescribed-bef...
"This last item makes SSRI's abuse-safe: contrary to benzodiazepines or opiate-based painkillers, there's no risk of abusing them for pleasure, as they're, once again not "happy pills" and there's no high. They even worsen symptoms for most people at first. That has to do with their mechanism and is their major flaw but serendipitously it strongly protects against recreational use."
Also, I do believe personality comes into play. When I was taking the max dose of Tramadol for over a year, I stopped dead three weeks post-op. The only noticeable side-effect was a runny-nose for a few days, hardly a big deal.