You could probably improve speed forever and always see a measurable improvement, but if that's costing more than you're seeing in added conversions, or you're doing it instead of improving features that would net a greater return, then you're doing the wrong thing.
There's indeed a point where optimizations stop yielding meaningful results, but it is certainly not connected to the performance of competitor websites. As soon as an user lands on your website, only its own performance and accessibility matters and positive ROI can still be achieved well beyond the point when your start doing better than competition.
Where did you get this idea about competing websites from?
However if your site is at 1800ms range it's worthwhile to increase to 1200ms range