News for hackers. Stuff that matters.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by atylerrice on Tuesday February 14, 2023 @09:59PM (#34799479)
Well the batch size also helps with making training more stable. So a larger batch size computes the gradient across more examples so it ends up being more stable than only using a batch size of day one. A batch size of one is only fixing the error on one example rather than many.

a larger Batch size can help keep the gpu fed but usually that’s not a problem with these larger models.

There are no comments on this comment yet.