Moving an object to higher gravitational potential decreases the entropy of the system (and thus requires work, unsurprisingly).
Mixing oil into water means you displace some volume of water which were closer to Earth with oil bubbles (which are lighter than equivalent volume of water) while you force the displaced water volume higher/further from Earth. Thus by mixing lighter oil with heavier water you decrease the entropy of the gravitational system consisting of the oil, water and Earth. Left to its own devices, the system evolves along the entropy increasing gradient - heavy water goes down displacing lighter oil up, thus decreasing the gravitational potential of the system (the friction during that "unmixing" is the work of the gravitational force that ultimately results in the heat that is lost away into the environment).
Is it possible that our understanding of entropy is flawed? Yes. But unless you have some novel physics experiment or mathematical model to share with us, I'll stick with the second law of thermodynamics.
>But this does not by itself imply entropy of the system increased. The reason is the system got to a non-equilibrium state which has no macroscopic thermodynamic description and no unique way to assign entropy and detect its changes.
Entropy is the tendency to equilibrium. Your earlier statement, that entropy is only a measure of systems that are in equilibrium, is false. If you were right, entropy would be a totally meaningless notion because there are no closed systems and true equilibrium (as opposed to the near-equilibriums that we work with in practical terms) can only exist in one.
doesn't matter what way it got there, entropy of the gravitational system oil-water-Earth is lower and the gravitational potential is higher when oil bubbles are inside the water when compared with the same system in the state when all oil is on top of the water. Thus system naturally moves from the former state to the latter, from lower entropy to the higher, from higher gravitational potential to the lower.
in short - entropy of the system with lower gravitational potential is higher (https://en.wikipedia.org/wiki/Entropic_gravity). When oil is mixed into water, the thermodynamic entropy "St,mix" of oil/water system is higher than "St,unmixed" while the entropy "Sg,mix" of the gravitational system of oil/water/Earth is lower when mixed than the "Sg,unmixed" of the unmixed case. In case of bad solubility, like oil in water, we have ["St,mix" - "St,unmixed" < "Sg,unmixed" - "Sg,mix"] and thus system on its own will evolve toward unmixed state.
What about you?
>there is no possibility of deriving equations of motion of mechanics from the laws of thermodynamics
i didn't say that. It actually works the other way - thermodynamics is derivable as an aggregation from the laws of motion. And thermodynamical entropy is just a facet of the entropy (which in general can be defined using relative volume in the space of states of possible configurations of the system for a given values of generic coordinates)
Entropy does not always increase in time, because entropy is rarely a function of time. The most common meaning of the word is a thermodynamic quantity that is not, in general, defined in non-equilibrium states. It is a function of state variables rather than time.
What people are talking about when they say "entropy increases in time" is actually some variant of minus Boltzmann H-function. This kind of resembles thermodynamic entropy, but is a distinct and less general concept. There are processes where entropy gets higher but "entropy" (minus Boltzmann H-function) get lower.
If you mix oil and water, you will see the result does unmix itself again. Does this mean entropy decreased and time direction got reversed? Nah. It is not even clear how to define entropy of the system in between, since it is in highly non-equilibrium state.