Activation of the reactor walls. The neutrons don't disappear; they're absorbed, and some fraction of the atoms that absorb them become radioactive themselves. I've seen lifetime estimates ranging from five to ten years for the walls, after which they'll be high-level waste.

Isn't the breeding blanket suppose to prevent that?

The breeding blanket does slow the neutrons, but there needs to be a first wall material that does not ablate into the plasma. You need very little of non-hydrogen material in the vacuum to cause a density collapse. If you made the wall out of liquid lithium then there would be a lot of lithium in the plasma.

Tungsten is a good choice for a first wall material because of its uniquely high melting point, low rate of embrittlement in high neutron flux, and short-lived radioisotopes.