Note that according to the first google result for "2006 computer spec"[1], 1GB RAM was common at the time.
Oh, Firefox 1.0, we miss you so!
Disk storage grew along with it. Hundreds of KB with the earliest machines I used (on floppies and cartridges), hundreds of MB in the 90s (and like 1.1GB when Dad added a second hard drive to that machine!!), 6.4GB in my own first one, and so on.
It's amusing to think of my son saying things like "I've used a computer with only 4GB of RAM!" when he's older.
For the most part, the cpu in the 328 doesn't have general read-write access to the 32k flash memory (used for program storage), outside of some basic functionality mainly limited to bootloader usage (at a machine code level, and IIRC only for the first block of memory), and read-only capability otherwise (so you can store data prior to loading it into flash to be read later by the code).
The 2k static RAM (SRAM) is read-write always and anywhere, anytime - and so it used mainly for variable storage. There's also a small amount of general purpose flash memory available, which is also read-write capable (but has a limited lifespan just like any flash memory - though from testing, in general you won't reach the lifespan unless you do something really stupid) that is typically used to store configuration settings and whatnot between reboots.
Whereas on the VIC-20, the CPU had complete access to all of the RAM and ROM, and full control over the address and data lines of the bus.
Each architecture has its pluses and minuses, but in general the Harvard scheme is used when you don't want the code to be able to alter itself (either intentionally or accidentally due to a buffer overrun or something), which might cause a problem that could have consequences; for an embedded microcontroller, this makes sense, as you just want the thing to run continuously and reliably on every reboot or restart, without worry that the code will corrupt itself and lead to problems.