Baseline cholesterol levels are genetically determined, it's true, but diet is far from irrelevant. A cholesterol-rich diet can easily push your overall cholesterol level above that baseline into the danger zone. This has been verified in numerous gold standard direct intervention studies. Anecdotally I have several friends that have brought high cholesterol levels down through diet far enough that they didn't have to take statins.

Most of the studies that have claimed to show that diet doesn't raise cholesterol have involved populations of people that already have abnormally high cholesterol from a bad diet (i.e. the typical American).

Important points summarized here:

https://www.youtube.com/watch?v=vBtfzd43t8o