It might be related to git and not Gogs/Gitea itself.
I've found Gitea to handle larger repos fine after running the following:
git config --global core.packedGitWindowSize 16m
git config --global core.packedGitLimit 64m
git config --global pack.windowMemory 64m
git config --global pack.packSizeLimit 64m
git config --global pack.thread 1
git config --global pack.deltaCacheSize 1m
This reduces the memory used by git during certain operations.
The only (minor) issue I've had with it was when I tried to push the OpenCV repository to it, just for the hell of it, on a heavily constrained VM (Debian with 256 MiB of RAM). The poor thing just couldn't handle it without crashing until I upped the memory to a couple gigabytes.