It seems you have a producer-consumer problem, where feeding in work to do faster than it can be processed only leads to lengthy queues, and running more tasks in parallel than you have cores only increases memory usage and task switching overhead.

Such processes rarely are memory-bound, and from what you write (it would use CoreFilters that support GPU acceleration "when available"), yours aren’t either. So why would you want to limit the number of consumers by looking at memory usage?

ioquatix’ comment to measure latency is better, but more work, so my first go would be to manage queue length, as that is simpler to do and often effective enough.

Determine how many consumer processes your system can really handle in parallel and limit the number of queued GCD tasks to, say, twice that number. That factor of two depends on how bursty the producer produces work items and on how CPU-bound the consumers are.

If memory of I/O aren’t your bottleneck, you want to minimize the number of items ‘in flight’ needed to keep your CPU or GPU at 100% usage.