Linear algebra is quite a beautiful, approachable subject; and a certain amount of it is necessary to make the leap from single variable to multi-variable calculus. Without a good grip on calculus, you can’t really what’s going on under the covers with linear algebra. What you need to do is precalculus (Simmons) -> single variable calculus -> very introductory / elementary linear algebra -> multi variable calculus (Apostol) -> less introductory linear algebra but still fairly basic (Gilbert Strang Intro to Linear Algebra) -> mathematical analysis (Apostol) -> linear algebra done right (Axler). You have to apply a spiral method where you return to subjects as you gain the tools you need to understand them better. You’ll never be done understanding geometry, algebra, or analysis.

Also, math is a problem solving art, and you can’t solve problems by reading, you solve them by thinking. Seek out problems that challenge and consolidate your understanding. You should be able to prove everything in Simmons and it should seem totally natural and intuitive. Then you’re ready to struggle with calculus, which is a subject humanity struggled with for centuries before getting a rigorous handle on. You probably want to get a handle on the mechanics and intuition, first, and for that I’ve heard that “Calculus Made Easy” by Silvanus Thompson is good.

Don’t try to eat too much all at once, you’ll make yourself sick. Don’t try to cheat yourself of the patient struggle to understand, confusion is completely natural when striving to really know something.