Is it really that difficult? I’m imagine there are things I am missing, but isn’t it pretty much: build foundation, build floor, build frame, build out roof, build walls, etc.?

As long as I research, do proper leveling, cutting, etc. and follow along with other builds’ process, I am thinking I will be fine. Would be great to hear where I might be wrong though so I don’t sink money into a failed build.

Yes, unless you have prior experiences, it is that hard. I agree with the others here who recommend you build an on-site tool/storage shed as practice. And no, it's not just about building foundations, floors, walls etc. You will need to consider how to rough-in for HVAC, plumbing, electrical, etc. as part of the build. Your wall-framing techniques have to take into consideration things like openings for doors, windows, and even how you will hang sheetrock.

Your biggest hurdle is probably going to be the permitting/inspection processes, assuming you want a legally habitual dwelling.

Agreed there, I know not to mess with plumbing, electric, and/or heating myself. Was debating going with spray on insulation done professionally, although I understand costs will rise the more I hire contractors.