Yeah I don't even really have a system for this; I basically have the mostly statically managed memory which is living for the length of the scene/program, and for per-frame stuff I just wait for fences on a cleanup thread.