Is it possible that our understanding of entropy is flawed? Yes. But unless you have some novel physics experiment or mathematical model to share with us, I'll stick with the second law of thermodynamics.
>But this does not by itself imply entropy of the system increased. The reason is the system got to a non-equilibrium state which has no macroscopic thermodynamic description and no unique way to assign entropy and detect its changes.
Entropy is the tendency to equilibrium. Your earlier statement, that entropy is only a measure of systems that are in equilibrium, is false. If you were right, entropy would be a totally meaningless notion because there are no closed systems and true equilibrium (as opposed to the near-equilibriums that we work with in practical terms) can only exist in one.
doesn't matter what way it got there, entropy of the gravitational system oil-water-Earth is lower and the gravitational potential is higher when oil bubbles are inside the water when compared with the same system in the state when all oil is on top of the water. Thus system naturally moves from the former state to the latter, from lower entropy to the higher, from higher gravitational potential to the lower.
in short - entropy of the system with lower gravitational potential is higher (https://en.wikipedia.org/wiki/Entropic_gravity). When oil is mixed into water, the thermodynamic entropy "St,mix" of oil/water system is higher than "St,unmixed" while the entropy "Sg,mix" of the gravitational system of oil/water/Earth is lower when mixed than the "Sg,unmixed" of the unmixed case. In case of bad solubility, like oil in water, we have ["St,mix" - "St,unmixed" < "Sg,unmixed" - "Sg,mix"] and thus system on its own will evolve toward unmixed state.
What about you?
>there is no possibility of deriving equations of motion of mechanics from the laws of thermodynamics
i didn't say that. It actually works the other way - thermodynamics is derivable as an aggregation from the laws of motion. And thermodynamical entropy is just a facet of the entropy (which in general can be defined using relative volume in the space of states of possible configurations of the system for a given values of generic coordinates)