It's probably from creating a backup copy. Compare performance when invoking 'view' instead of 'vim'.

It could be from your plugins if they're doing something goofy.