Yeah "on accident" jumps out as wrong to me (a Canadian) but I can appreciate this it is symmetric with "on purpose". I've never heard anyone say "by purpose".