I think you mean "I have had flu-related symptoms twice". Given that influenza is a virus with a huge number of variations, and almost nobody is ever tested for infection with any of them, there's every chance that you've had many more influenza infections, but no symptoms.

Nope these were confirmed cases.

Yes, I understand that the times you had symptoms, the infection was confirmed to be flu.

What wasn't tested (or detected) were the times when you were asymptommatic but still infected with one the many influenza variants.

English is a little problematic when it comes to describing infectious disease. In all cases, there is the disease agent (e.g. an influenza virus) and there are the disease symptoms (e.g. "flu"). But we tend to say "I had the flu" without it being clear if we mean "the disease agent was present" or "i had the symptoms".

Following your logic I should have had residual immunity to keep most of the variants in check. The first time I had the flu in college it was brutal and I was out of commission. The second time diagosed I got tamiflu after the flu test and it was mild. I also knew I was sick. Anything in between was unnoticed and I have had a flu shot since they first came on the market.

Not necessarily. The immunity fades with time, so it is possible that you could be infected again. The residual immunity might be strong enough to prevent symptoms however. Each such exposure can boost your immunity. Nevertheless, influenza virus has many mutations and sometimes can cause breakthrough disease even in vaccinated.

Many influenza infections are asymptomatic. Unless you've been getting tested every week, it's entirely possible that you have had the flu more than twice.