It never occurred to me that back links from other websites would hurt ranking, I thought it's OK as long as a website don't point to other spamming sites... So this is really scary, it means that other people can attach your ranking...