Run a Windows VM with Visual Studio. That will need at least 1.5 Gigs. Then run some heavy stuff on OSX too and it will become slow. If you have an SSD, it won't become unbearably slow, but clearly straining.
Which is to say, 4 Gigs are fine for a VM every now and then, but not for everyday use. At least not for me.
Couldn't resist pointing out that I have a 4GB MBA, and I have yet to encounter any memory issues whatsoever. I have run some hefty software on there - e.g. photoshop + iWork + Pages all at once. Granted, it's not the same as VMs, but it seems that the ultra-fast SSD makes swapping a non-issue. Maybe someone who does run VMs on an MBA can chime in?