1 week is fine for a production training run. But that’s after you’ve locked in your hyperparams on small prototypes. Small prototypes should take no more than two days to give results. If your prototype absolutely must be a large model, you should have many (perhaps even a dozen) attempts training simultaneously.

It’s hard to imagine someone spending an entire week waiting to see whether their latest tweak worked, then tweaking again and waiting another week…

I had models like that, but they were exploratory side projects, not my primary research focus. It’s stuff you try for fun, not for work. That way a week is “whatever” if it doesn’t work out.

But when you’re hunting for a result, no way. Prototype results are measured in “N completed runs per day”. Which means you should have multiple runs going at all times.

It gets tricky to manage, but… I don’t know. Either you’re right and I got incredibly lucky, or determined programmers can do the same thing I did. I’m not that smart.

Try training Imagenet. For some reason this is called a "medium" sized data set.

I do want to point something out though that's a big problem with a lot of your comments. We're seeing the results of what happens when you "just implement a DL pipeline." Lots of people are using these powerful tools without understanding. It's like handing a monkey a machine gun. We're seeing that models don't generalize like people think they do. We're seeing people use models pre-trained on ImageNet or CelebA and think that there is no bias in them. When people are saying you need to learn theory, this is really what they are concerned with.