This is how psychometric testing works. It's inherently difficult because in order to estimate the point of subjective "loss", which we call the "just noticeable difference" (or JND), one has to sample more in the area of the variable (amplitude, frequency, etc) that is more difficult for you to distinguish. Consequently, one will always walk away from such an experiment with an impression of having "guessed" and being really not sure if you gave the right answers. But that's because they're trying to estimate exactly that: they're trying to find the point at which you really aren't sure whether you hear something or not.
Basically this: if you guessed perfectly every time that you heard something, that would be a 100% recognition rate. If you always said with 100% certainty that you didn't hear anything, that would be a 0% recognition rate. So logically, the point of hearing loss occurs somewhere between those two extremes.
In order to determine more precisely where, the procedure has to "zoom in" on the point at which you answer correctly 50% of the time, a bit like a binary search. (Or sometimes they want 75% or the time, etc.) In any case, to do so, they need to sample the probability of you answering correctly or incorrectly in that region. This sketches out a probability curve, and then they can fit that curve and figure out the 50% or 75% point on the curve.
They'll sample using either a constant spacing method, random sampling, or a staircase method that adaptively moves towards the 50% point. The latter is more efficient, in the sense that it requires fewer answers from you, so that is what is often used in practice. However, by its nature it is also much more frustrating for the patient, because it will be sampling much more frequently in the region where you are "not sure" of the answer.
I'm really sorry they didn't explain this stuff to you, and allowed you to walk away thinking it was a badly done experiment!
If you are reluctant about getting a hearing aid, I would totally totally recommend it. It turned my social and professional life around - I was beginning to avoid conversation with certain softly-spoken people and couldn't follow arguments in conference rooms that had any sort of noisy aircon. As a Brit, I have access to free NHS hearing aids and consumables (batteries and the tiny tubes that go into the ear), which helps. I had been planning to buy a smaller in-ear device, but in fact the external device is acceptable in terms of size and appearance (silvery grey).
The device I use has an external control button (a tiny stud) which controls volume but which could alternatively be programmed to trigger different modes (e.g. noisy room vs quiet room). The fact that I almost never have to change the settings also suggests that the initial test (which is programmed into the hearing aid) was somewhat accurate.
Its been great for me. I've had mine for 6 months. They are not perfect but they are a big improvement. See my other comment above.
I know what you mean though ... especially if you suffer a bit from tinnitus, it's sometimes really difficult to distinguish if you are hearing a high freq test tone, or just the tinnitus 'noise'.
I'll re-iterate the point other commenters have made, though: even if you're not entirely sure you heard a sound, press the button. It's all useful data: if you're pushing the button during a high-frequency part of the test when there's actually no sound, that's a sign things are not entirely right. And if you only push the button when there is a sound, but don't push it half the time the sound is played, that's also useful data.