As someone who’s done UX for 20 years, I agree with you entirely. Some of the value of a UX process is that decisions are reversible. When you user test it can be difficult to integrate the results, especially if many things are wrong. With a thought-out design and a good paper trail of decisions, design decisions can be evaluated atomically and the team can react faster to user feedback.
Sadly, some of the value of a UX process is that many teams will not test with users at all. A process can give people a better shot at building something useful to begin with.
But your point stands well for most software: sketch something Good Enough, build it quickly, and get feedback.
As both a long-time developer and, more recently, a UI designer, I'd assert that having a UI designer for a 6 person user-base doesn't really make sense, unless maybe it's 6 highly paid full-time users that are doing almost all of their work through it— in which case it would probably be critical.
In my 20 year tech career, I've seen far fewer cases of "why was the UI designer so involved, here" than cases of "Oh, the developers will know what makes sense to users, so we'll just have a designer come in at the end and make it look nice." The latter is a much bigger mistake than the former.
1. The "sketch" step yields about 80% of what you'll benefit from the whole process. If you can't do anything else, at least do that.
2. Very often, getting something, anything, built and working, and letting your users test and critique it will reveal far more than any amount of thoughtful design process will.