That can definitely be a problem. Using handles makes it easier to swap out your allocation mechanism if you ever have to optimize your cache efficiency.

Hell, projects that need to squeeze out that level of performance will typically have multiple different memory allocation mechanisms.