The 168MB is for storing all of the weights (parameters of the model) and the activations from passing one sample through the network. When you are training, you want to pass a whole minibatch (32 samples in this article) through the network, so the activations (not the weights) are repeated 32 times - that's where the 2GB figure comes from. You need to keep these activations around for backprop to work, so you either store them all, or as the article later suggests, you keep the ones that are expensive to compute and re-calculate the cheap ones. Also note that when training you need to store the gradient of every weight too, so this adds 104mb in this example - almost negligible next to 2GB of activations.

For running a trained model, it depends how it's being used. If you're computing one sample at a time, then the 168MB figure is an upper bound on that (one copy of weights + one set of activations). You could optimize this by only keeping the activations for the current layer because you don't need to save the intermediate activations. If however you're batch processing samples at inference time, that's where you want to make most efficient use of the compute resources, so chances are minibatches are going to be the best route; again the same kind of optimization with not keeping intermediate activations applies here.

As for whether GPUs are useful at inference time, that heavily depends on your model and available deployment hardware. TensorFlow has some support for 8-bit quantized models for targeting mobile/embedded/CPU, so there are ways to optimize for this use case. Even at inference time though, most of your compute work is going to be in matrix multiplies or convolutions so there's plenty of cases where GPUs can help.

Apologies, I did not mean to say that the information was not in the article. It mostly was. (I think the only thing missing is that you are saying that of that 168 megs, 104 are the weights and 64 are the activations. This gives 64 * 32 for the 2G of memory. Right?)

The difficulty for me is merely in visualizing.

And I should have said that I was not looking for a single answer on whether GPUs make sense in predicting. A chart with rough guidance on when you will target specific execution environments is all I'm hunting for. My bias is that lower power targets are more common. Phones and the like. But, I would not be willing to bet heavily on that.