The only really good way of understanding the variational principal in my experience as a physicist who has chewed on it informally since getting out of grad school is to recognize that energy, potential or kinetic, comes after the variational principal, not before it. In school we are taught to write down the kinetic and potential energies and then to form a lagrangian and turn the crank, but it is in fact the lagrangian formalism which determines which expression we write down for both terms. All the physics before, including the characterization of kinetic and potential energy as concepts, is fumbling towards that idea.

Really, if you look at Hamiltonian Mechanics this is more clear, since most of the ideas in Hamiltonian mechanics flow from the basic idea that p generates q AND either that paths in state space don't cross and/or that time evolution is unitary (depending on whether you want classical or quantum mechanics to shake out). These are more or less unavoidable assumptions which we impose or our efforts to understand reality and from these basic ideas we simply fiddle around with expressions until we find the mechanics that match our observations and from which we can sometimes identify objects which behave like the potential and kinetic energies we learn about in grade school.

Another way to appreciate this is the difficulty of coming up with a good totally fundamental definition of energy independent of a physical system. Energy is a handy way of thinking about how things will move. That is all.