This seems like a perfect example of a pattern applied in the wrong way. The builder pattern is supposed to be used to construct new instances (in other languages named parameters make it pretty redundant). I could perhaps understand allowing people to batch up changes but when you're forced to switch to a builder and back again to make a change to one field you've definitely got a poor design. The example seems particularly egregious because you're also forced to walk the domain where really you should just be able to do:

    user.addFavourite(album);