In my opinion you shouldn't punish 96% of your users because of those 4%. As others have mentioned you could use something like modernizer to take advantage of progressive enhancement. That way the users with modern browsers could benefit from newer technologies and the legacy users could still view your site properly. It's more work but I think it is worth the effort.