Doesn't really matter if the relationship isn't linear like that. I suspect it's more likely conversions start to drop off after certain thresholds are met e.g. 3 seconds causes conversions to drop by 5%, 20 seconds causes them to drop by 30%.

Just averaging out those numbers could result in engineering time being wasted chasing rapidly diminishing returns, if the site is already below one of those thresholds.