I've run Debian for two decades, so I've definitely ended up going down the path more than a few times. Yes, the actual building process wasn't so hard. It was the maintenance burden afterwards. Maybe I just didn't commit hard enough with my own apt repo, pinning, staying ahead of what was coming through backports, etc. But point is that it effectively never stuck long term, the end result was always going back to vanilla upstream. Whereas in Nixos such patches are some lines in the larger config file, which feels a lot more persistently manageable.