Something wierd about a vanilla minecraft server is that sometimes less RAM is better. If you are using a good enough cpu going from 4 gb allocated to 2 gb allocated massively increases performance for low numbers of users because garbage collection runs more efficiently. You can tune the garbage collection manually, but super counterintuitively this fixed most of my problems.