Excellent write up! Running a single long running Spark Context that accepts and runs jars compiled from user input is a curse I thought we were suffering alone at my company. We too have fought the ContextCleaner, though for us it was G1GC and `-XX:+ExplicitGCInvokesConcurrent` on the driver that saved us.