I guess it is all about which questions you are trying to answer. Generally speaking at the OpenAI and Deepmind end of the scale the question is "(How) can we escape the diminishing returns of applying ever more data and compute to improve upon the SOTA?" rather than "(How) can we improve upon the SOTA without applying more data and compute?".