"Well, we are limited by the physical size of our brains"

I don't think that's something that's clear at all. Men and whales have larger brains than women, on average. Are they proportionally smarter? Without knowing how brains work, it's impossible to say what constraints exist due to size.