There are some hobbies where starting without any prior knowledge and learning through a process of trial and error is fun and rewarding.
IMHO aviation is not one of these hobbies. It tends to be rather unforgiving of the "error" part of the equation, especially if the error is something like "wings not correctly attached". :)
...but not know how to fly it.
The barrier to entry for aviation is not steep at all if you approach it with the attitude that flying with an instructor is still enjoyable flying. People often think that you need to drop a few thousand bucks on training before you can fly. But not so! You'll be flying right from the start. You'll be doing it with an instructor, but if you pick an instructor whose company you enjoy, that can be a bonus, not a drawback.
Using their plans as a guideline for building an ultralight instead of a light sport aircraft would reduce the cost dramatically. Ultralights have much lower limits but don't require any formal training/licenses.
In the end you could probably build an ultralight for the cost of a pilots license.