With the Last Jedi, I honestly don't think the central problem is that the creators were trying to appeal to everybody at once. I think that better describes the Force Awakens, which miraculously succeeds at doing so at the cost of being kind of forgettable (see my point about Star Trek 2009 above). I think the issue with the Last Jedi is that it makes a very specific set of choices that don't please everybody.
The thing is, in some ways I'm the kind of old fan you're talking about, though I may be younger than you. I pored over books of EU spaceship schematics, read the Star Wars Legacy comics, played the video games, etc. Maybe my experience was different than yours because I watched the prequels at a similar age to the originals, because they were coming out in theaters when I was a kid.
I am not a casual fan, but I'm also not a superfan. I think it's easy to dismiss people who liked The Last Jedi as casual, but Star Wars is such a dominant cultural phenomenon that there is a massive spectrum of how much people care about it and what it means to them.
This has all been kind of a scattered argument. I'm not sure what my conclusion is. Maybe that the real issue with The Last Jedi for many people isn't that it ignores the lore. The prequel trilogy already directly contradicts loads of aspects of the original trilogy that EU creators and fans had to no-prize away. I think it's more to do with how The Last Jedi conceives of Star Wars as a story, and what it considers important and what it does not.
I agree with your hesitation to start a debate about The Last Jedi, but I think your point about what rules are ironclad is why reactions to that movie were so mixed. For me (and many others), the mechanics of the Star Wars world are paramount. I have trouble experiencing strong in-universe emotions when those mechanics are broken, because when that happens, I'm pulled out of the story and have trouble immersing myself in what's happening to the characters. But to others, the emotional impact of what happens to the characters, and how they act, is the most important thing, as you suggest, with all other concerns secondary.
The writers of the Star Wars sequel trilogy had a very tall order on their hands: the "original" Star Wars fans are older (some might say much older) now, and they needed to tell a story that could appeal to those people (such as myself), but also appeal to children and young adults. It's very hard (some might even say impossible) to create a narrative that speaks to superfans who have been steeped in the mythology and mechanics of a world for decades, but also attracts a more casual viewer who just wants to watch a cool story with lightsabers and spaceship battles. My opinion is that they sacrificed the first bunch in order to attract a fresher audience, but it doesn't really matter if my opinion is "correct" or not; the fact that the movie (or entire trilogy, even) was so controversial, with so many opposing views, is all we need to know: they tried to appeal to too many different types of people, didn't do a great job of it (if a great job of it is even possible) and in the end a lot of people were unhappy.