The biggest danger that's specific to leaded solder is accidental ingestion. Both the common methods of cleaning your iron (damp sponge and brass wool) produce many tiny little balls of solder. They're difficult to see, and because they're round and dense they easily roll and bounce to unexpected places. They can get caught in your clothes and potentially end up falling in food.

The fumes are flux fumes, not lead fumes. They're still bad to breathe but not specific to leaded solder.