It might be related to git and not Gogs/Gitea itself.

I've found Gitea to handle larger repos fine after running the following:

  git config --global core.packedGitWindowSize 16m
  git config --global core.packedGitLimit 64m
  git config --global pack.windowMemory 64m
  git config --global pack.packSizeLimit 64m
  git config --global pack.thread 1
  git config --global pack.deltaCacheSize 1m

This reduces the memory used by git during certain operations.