If you are in a team that is looking at problems that justify massive hardware (in the sense that solving them will pay back the capital and environmental cost) then you will have access to said hardware.
Most (almost all) AI and Data Science teams are not working on that kind of problem though, and it's often the case that we are working on cloud infrastructures where GPU's and TPU's can be accessed on demand and $100's can be used to train pretty good models. Obviously models need to be trained many times so the crossover point from a few $100 to a few $1000 can be painful - but actually many/most engagements really only need models that cost <$100 to train.
Also many of the interesting problems that are out there can utilize transfer learning over shared large pretrained models such as Resnet or GPT-2 (I know that in the dizzying paced modern world these no longer count as large or modern but they are examples...) So for images and natural language problems we can get round the intractable demand for staggeringly expensive compute.
Imagine that you had got a degree in Aeronautical Engineering, you are watching the Apollo program and wondering how you will get a job at NASA or something similar... but there are lots of jobs at Boeing and Lockheed and Cessna and so on.
I know how you feel, but I also think stories like this may be a wake-up call for some groups to invest more in hardware. The Baker and Tsinghua University groups are not small. They can afford more than 4 GPUs.
Probably it's more about setting up a good pipeline. Once you get about 4 GPUs you need more than one machine to run them. Hopefully in the next years we'll see more open source tools to make it easy to "build your own GPU cloud".
Anyway these days I see a lot of industrial investment in middle size computing datacenters full of GPUs. Sure Google scale is not within reach but I'm sure there's room for scrapier algorithms and training methods to demonstrate feasibility and then paying a lump sum to AWS (or some ml-expert-for-cloud-training SME) for the final step.
Anyway, I thought the expensive part was data acquisition and labeling? I like the 'surrogate network' approach of learning a very-expensive-to-compute simulation or model, that doesn't need data, but the output of a costly simulation.
From what I understand, deep learning, although opaque and relying on tons of data, is a bit magical : although one would say "it's just probabilities", it actually does probabilities at a level where it actually figures some things.
Plus, and that's very much a problem to me, Google does it at 100x the scale of a regular researcher. Since I just invested a year in studying data sciences, that worries me a lot : where am I suppose to work if, to produce meaningful results, you need way-too-expensive hardware...