Usually layers itself do not make a PSD big. A big PSD file is usually because the DPI is huge and >500mb PSD files are very common in the print industry where designers have typically design between 300 to 600 dpi. A B2 poster at 600dpi will easily be >500mb.

*A PSD file has a max height and width of 30,000 pixels, and a length limit of 3 Gigabytes. Hence, Photoshop created PSB filetype which supports max height and width of 300,000 pixels and the length limit to around 4 Exabytes. https://en.wikipedia.org/wiki/Adobe_Photoshop#File_format