English is my second language and I got ~13,500 words. I was surprised that this was only half the average. I read a lot of scientific papers in my field and almost never come across a word I don't know. Writing technical papers well also requires not much vocabulary (it's more about clarity of expression). I believe to get beyond that number you simply have to read lots of books. I notice lots of words I've never heard of when reading, say, Bill Bryson but I very rarely bother to look them up because they usually make enough sense from the context.
Beyond this, it's pretty much necessary to live abroad for an extended period of time, or be exceptionally good (driven) at languages and watch TONS of TV, use tons of online chat, etc.