Are you using custom weights? I'm assuming you are but there is a major difference between using the default RunwayML 1.5 weights and using a model finetuned for a specific purpose.

Generally the trade off is that any of the impressive finetuned models are far less generalizable then the default weights, but in practice this is not a big deal and the results can be a substantial improvement.