This is vestigial thinking from the MacOS perspective. The concept of "saving" is gone. The concept of "memory state" versus "disk" is gone, from the user perspective. Those are all implementation details now. For the user it's simply, your windows stay open until you close them. Quitting does not close windows, so it's not unusual to see them when you return to an app.
By the way, Safari in iOS does exactly the same thing. And given that iOS kills apps on its own, there would be no question that randomly killing private sessions would be a bug.
I mean maybe you could distinguish between a force quit and an OS quit, but why? Just close the windows if you want to clear them. The private window guarantee is there will be no trace after you close the window. It's just vestigial thinking that quitting apps also closes windows.
Given that, it is logical that it reopens private windows, just as, for example, TextEdit or Numbers reopen windows with content you never explicitly saved (even across reboots)
(You can go back to the old behavior via a system setting: "Close windows when quitting an app")