It feels like there's a certain number (maybe Dunbar's like you said) at which people stop seeing a community as a group of disparate individuals and start seeing it as some sort of a collective hivemind.
You see this regularly with comments like "HN/subreddit/etc is so hypocritical. One day they're in favor of X, the next day they're against it", in which the poster seems to fail to realize that in a community of tens of thousands of people, chances are it's entirely disjoint subsets of the group taking those positions.
I don't see that kind of thing as often in smaller message boards. When you recognize everybody's name (or at least their avatar), it's a lot clearer that it's just Bob and Alice who support X and Eve and Mallory who don't.
It seems natural that such a failure to see the individuals that make up a group would lead to a decline in civility.
Smaller communities tend to be more familiar - people recognize each others names, and people have each other's backs, when someone is trolling or something. When things grow beyond a certain size - maybe related to Dunbar's number - the group dynamics change.
In some cases this can lead to the community to loosing coherence or even becoming toxic - but I don't think that's inevitable. I believe it IS possible to grow a community to a larger size and have it stay nice. I think though, that in order to keep a community nice when growing beyond a certain point, some effort is required - effort which wasn't necessary while the group was smaller and kinda self-organized.
Also, growing slowly rather than quickly might help - as newcomers have some time to acclimatize and internalize those unspoken social rules that govern a group's dynamic - and will be able to pass on the knowledge to those coming after. People who join a group will eventually adapt to the group - but that process needs a bit of time. But when growth is too fast, a newcomer might have more interactions with other newcomers rather than long-standing community members, and thus will adapt to something quite different.