Not even the founding fathers of RL (Bertsekas et al) have made such claims.
RL almost always solves an approximation of the dynamic program. Aka no guarantees of consistent optimality.
RL does not guarantee constraint satisfaction.
RL needs tons of data to learn the state space and still is not enough. LPs solve nowadays Traveling salesman problems to optimality with millions of nodes. RL close to optimality for less than 1000 nodes.
They're two different methodologies that are most of the time applied to vastly different problems.
I don't follow. Linear programming (with the simplex method) finds the optimal solution set for any linear program eventually. How can reinforcement learning be "better" apart from efficiency?
actually i guess they're shilling - not that there's that much of a differnce.
There are many problems that are hard to represent such that LP can compute them. That process alone is long, and hard to adjust even though clients need to make those adjustments.
These problems include multi-echelon inventory optimizations, and load-balancing across multiple assembly lines in factories when using hundreds of delivery vehicles, among others. These are properly multi-agent problems that RL solves well if given a valid simulation of the environment. LP often cannot solve it at all.
In cases where we went head to head with well known LP software vendors, we were able to move the Pareto frontier by double digit percentages.
We open sourced our software here after the acquihire. Depends on Gym among other repositories. Does something similar to Microsoft’s Bonsai. All are welcome to use.
The main challenge is to optimize these decisions. With optimality guarantees. Consistently. Day in, day out, with various uncertainties.
That is what math optimization has been doing for decades.
And no, RL is nowhere to be seen in production, except from some consulting projects that are abandoned.
No, this is mathematically impossible. You cannot out-optimize a mathematical global optimization solver. Whether that is LP, MILP, or NLP. Exactly because they provide a certificate of optimality.
"But there are lots of situations that LP can't optimize while RL can"
I will be satisfied if you just show us 1.
Stay in your lane, and state that you built a fast heuristic that works in the problems you tried. Nobody can dispute this. But don't make claims you cannot support.
Also, LP often offers perfectly acceptable optimization and latency. The only case where this isn't true for offline problems is where LP times out or where relaxations of the problem are required to prevent timeout.
If I had to choose one method to learn, I guess I would learn RL. But it's a false choice. Given that I know both well, I'd reach for LP unless there was a good reason to use RL instead. And from a user perspective LP is much easier to learn than RL (see cons), so I think this is good general advice. Definitely worth having both tools on your belt if you plan on working on problems where either is a good fit.
RL makes sense when you don’t know the dynamics of the environment and have interactions with it.
Linear programming happens when you can encode a problem as linear constraints and therefore know the dynamics. You can then find the optimal set of points of the convex polyhedron defined by the constraints via interior point methods.
That’s it. You don’t need exploration and you can get an optimal solution to the constraints because the feasible space is convex.
If the problem is too complex for LP then you're probably not going to get a global optimal from RL either.
However, unless there is some aspect of the problem which is not known (e.g., you don’t exactly know the objective or constraints), so you model it as a distribution over LPs, I really don’t know how RL will help you. Gradient-based methods can give you improvements if your problem is very large scale and doesn’t have, e.g., sparse structure, but the above claim is bold.
From that background, given a problem in LP, there are three cases, the problem is (1) infeasible, (2) feasible and bounded, and (3) feasible and unbounded.
In case (2), at least one of the extreme points (intuitively, a sharp point) of the set of feasible points is optimal. For an algorithm, the famous simplex algorithm will find an optimal extreme point in finitely many iterations in finite time. For the computing, about all that can go wrong is from a problem being too big or causing poor numerical accuracy.
Thus, given an LP, the claim
"objectively better optimization",
that there was any "optimization" at all means that the problem is in case (2), feasible and bounded. Then claim
"objectively better optimization"
is essentially not possible.
This is giving a ton of charity to the opening post, but fun for me to consider. As you say, given a model, if it is bounded than LP is an obvious route to get optimal. However, models being what they are, there are often times when the prescribed solution is unrealizable for reasons. Reinforcement learning being what it is, is an interesting way to add new features to a model that can "cost in" the penalties that come from non-realized optimal solutions.
That make sense? I'm assuming this has been studied, and I simply don't know the names to look into.
That is, if you know what your total exposure to risk is, then LP is still ultimately the answer. If you don't, you need some way to monte carlo different scenarios to get an idea. And at that point, I feel you have backed yourself into RL?
The main appeal of RL is for cases where an explicit model cannot be derived. Think of very complex chemical reactions or other system dynamics. There indeed, you can go with RL and cross your fingers.
If your process is dangerous, we must also have a supervisor system, to ensure that the RL actions will not lead to catastrophic situations (from violating constraints). Like Teslas targeting firetrucks.
Mainly trying to find a way to use charity for that opening post. At face value, I don't know what they were getting at.
Changes in the costs c, the right hand side b, or the non-basic columns of matrix A would let the new solution be found usually quickly.
1 by n c
n by 1 x
m by n A
m by 1 b
1 by 1 z,
maximize z = cx
subject to Ax = b, x >= 0
That is, an LP is a fully precisely stated problem in math. There is no randomness, ambiguity, flexibility, uncertainty, etc.
Now when move to the real world and have a problem, e.g., in mixing animal feed, cracking crude oil, we have some real world considerations that are not really LP. But in some cases, maybe it appears, after some examination, that we can take the real problem and formulate it as an LP. The formulation may have some of these real considerations, but once we have our LP, all we can hope to get from the LP is the usual LP results -- an optimal solution, maybe infinitely many alternate optima (other solutions also optimal), some sensitivity analysis on how much we could change some of the input data and have the solution we found remain optimal, and some more, but none of this goes very far on resolving the real world issues of randomness, etc.
So, sure, maybe we (1) several times adjust the input data and solve the LP again, (2) move to non-linear programming (quadratic programming, the Kuhn-Tucker conditions, etc.) or integer linear programming, or (3) have some models of the randomness and move to optimization under uncertainty, e.g., stochastic dynamic programming.
Main point: There is (A) the math and (B) the real world, and the two are not the same. Hopefully the real world problem can be solved nicely with some math, but we should know when we jump from the (a) real problem to the (b) math problem and have good understandings of the relevant issues of both (a) and (b).
For a (crude) analogy, the math, e.g., LP is just a tool. E.g., recently somehow my plastic mail box got whacked with some cracked plastic and a hole in the top. Fixes used some tools, tape measure, hand saw, bridge saw, circle saw, hand planer, wire brush, electric drill, epoxy glue, spray paint, ... -- but none of these were directly "mailbox fixes" but just tools useful in fixing the mailbox. In this analogy, the math is not a real fix but just some tools that might yield a real fix.
Specifically, use LP to craft a plan in an iteration of scenarios that is varying a cost function through iterations. I suspect I'm fumbling in the multi period planning realm more so than the RL one, here.
I further suspect, though, that it is more "if you can formalize your problem such that LP solves it, use LP." That is, the opening post was largely just nonsensical?
LP (linear programming), given an optimal solution, has some sensitivity analysis that nicely addresses robustness.
In the LP find x to solve
maximize z = cx
subject to Ax = b, x >= 0
small changes in c, A, and B can result in large changes in x. But, still, due to some continuity in the linear algebra, the changes in z should be (I am avoiding doing the derivations) will be small and continuing to use the optimal x from before the changes should (again, some derivations are needed) be not far off (omitting details).
The feasible region is much like a diamond with flat sides and sharp points (the extreme points). And the feasible region is convex, i.e., for x_1 and x_2 in the feasible region and t in [0,1],
What you are pointing to, seems to be degrees of freedom on an input? The more options you have in an input, the better. Robust systems also have to deal with the unknown adversaries of the system that will work against it, though? Not just, "we were sensitive to the costs of feed changing, but there is also now an avian flu going around and we have to kill our flock." A simulation is easy to imagine that does this and forces you to see what is next. Is that common in LP formulations?
If you want to consider the effects of war, inflation, something like the 2008 housing crash, economic depression, COVID, hurricanes, train derailments, etc., sure, if those hypothetical changes still yield an LP (linear program), can just solve the resulting LPs and display the resulting data in a way can visualize and get some intuitive insight.
But there is another way: Let insurance companies consider history and set insurance premiums for disasters, all disasters, ever seen before or not, and they will have quite good data, likely, sorry, better than get from RL (reinforcement learning) or, really, anything that can be obtained from anything like current analysis of data from Web crawling. A common assumption, understanding in business is that it is possible to get insurance on nearly anything. E.g., the whole thing about the book and movie The Big Short was that it was possible to get insurance on the BBB tranches of housing mortgage bonds, make a bet that the bonds would fail. For this would have to pay monthly premiums but when the bonds failed make huge returns on the premiums paid.
With their good data and history, an insurance company will be able to set insurance premiums that give you some good information about the seriousness of the disasters, on your list of disasters or not, you are concerned about. If seeing the premiums you are still highly concerned, the pay the premiums and relax.
Sure, if I were to invest, what for me are big bucks, in raising a flock of chickens, maybe I'd get some insurance against any disease, e.g., bird flu, attacks by rabid foxes, wolves, raccoons, fire, flood, etc. destroying the flock.
Sure, once I did some Monte Carlo work essentially to evaluate the odds of some survivors in a really complex, goofy situation -- with those odds, could have found premiums for writing n insurance policies. The issue was, if there were global nuclear war limited to sea, how long would the US SSBN fleet survive? For input, I used some encounter rate probability derivations Koopman did as part of ocean search in WWII. Then I used some detection radii of Red-Blue pairs and battle result probabilities (one dies, the other dies, both die, neither dies). My work passed technical review by a world-class probability professor. My summary results were a graph over time of the expected number of SSBNs surviving. Also the Monte Carlo result could have also reported the probabilities of other aspects of the scenario, e.g., the probability that at least 10 survived for 30 days, enough to write an insurance policy!
Lesson: Sometimes it's possible to do Monte Carlo under some seemingly wildly complicated situations. In particular, it's possible to get insurance to protect nearly anything from nearly anything.
Net, I would advise pricing some insurance instead of some DIY (do it yourself) with either LP or RL.
More broadly and by analogy, a hammer and a screwdriver are both good tools, but no sense in asking which is better. Maybe the same for LP and RL. And for one more tool, insurance policies.