Your response makes me wonder if the idea is not that RL can give more optimal solutions, but perhaps more robust ones?

This is giving a ton of charity to the opening post, but fun for me to consider. As you say, given a model, if it is bounded than LP is an obvious route to get optimal. However, models being what they are, there are often times when the prescribed solution is unrealizable for reasons. Reinforcement learning being what it is, is an interesting way to add new features to a model that can "cost in" the penalties that come from non-realized optimal solutions.

That make sense? I'm assuming this has been studied, and I simply don't know the names to look into.

If you need robustness you can explicitly model an uncertainty set for your problem coefficients, and solve to optimality the robust optimization problem. That way you know exactly your degree of “insurance” to uncertainty. You also know that this is the absolute best you can do in view of this uncertainty set.

Right, I am asking if RL may be a somewhat easy way to get that in there? Since a lot of robustness is best known after a series of losses.

That is, if you know what your total exposure to risk is, then LP is still ultimately the answer. If you don't, you need some way to monte carlo different scenarios to get an idea. And at that point, I feel you have backed yourself into RL?

Fair point, but there are no shortcuts. Either you will use domain knowledge and you will build a model-based approach (LP, IP, MINLP, CP/SAT, DP), or you will have to find terabytes of good quality samples to feed the RL beast, hoping it will approximate a good enough policy / value function.

The main appeal of RL is for cases where an explicit model cannot be derived. Think of very complex chemical reactions or other system dynamics. There indeed, you can go with RL and cross your fingers.

If your process is dangerous, we must also have a supervisor system, to ensure that the RL actions will not lead to catastrophic situations (from violating constraints). Like Teslas targeting firetrucks.

Absolutely! I said in another post that I assumed you'd use LP if your decision space was modeled in such a way that that can work, but RL if you instead have more of a simulator and a cost function. Even that, I grant, is over simplified.

Mainly trying to find a way to use charity for that opening post. At face value, I don't know what they were getting at.

Small point: If you want to do Monte Carlo on an LP (linear program), then using the simplex algorithm, for the first Monte Carlo trial, go ahead and find the optimal solution. Then for the rest of the trials, start with the optimal solution, the basic variables, and the matrix inverse of the basis matrix and from those find the next optimal solution -- that solution will nearly always come quite quickly. So, you will be able to have a good collection of Monte Carlo trials for surprisingly little more computing than for the first optimal solution.

I knew you can carry forward solutions in most solvers. I'd assume you would be changing constraints some, though I imagine most constraints would not be changed such that a lot of the discovered solution would still be good?

The change that would make finding the next solution maybe not quickly would be to the basis matrix and its inverse from the last solution. Still, finding a new basis inverse would be much faster than starting from just the whole problem. Yes, changes in the basis matrix could make that basis infeasible, but could attack that the usual way with artificial variables, and there may be still better ways.

Changes in the costs c, the right hand side b, or the non-basic columns of matrix A would let the new solution be found usually quickly.

A linear programming (LP) problem is essentially just, for positive integers m and n and real or rational matrices

1 by n c

n by 1 x

m by n A

m by 1 b

1 by 1 z,

maximize z = cx

subject to Ax = b, x >= 0

That is, an LP is a fully precisely stated problem in math. There is no randomness, ambiguity, flexibility, uncertainty, etc.

Now when move to the real world and have a problem, e.g., in mixing animal feed, cracking crude oil, we have some real world considerations that are not really LP. But in some cases, maybe it appears, after some examination, that we can take the real problem and formulate it as an LP. The formulation may have some of these real considerations, but once we have our LP, all we can hope to get from the LP is the usual LP results -- an optimal solution, maybe infinitely many alternate optima (other solutions also optimal), some sensitivity analysis on how much we could change some of the input data and have the solution we found remain optimal, and some more, but none of this goes very far on resolving the real world issues of randomness, etc.

So, sure, maybe we (1) several times adjust the input data and solve the LP again, (2) move to non-linear programming (quadratic programming, the Kuhn-Tucker conditions, etc.) or integer linear programming, or (3) have some models of the randomness and move to optimization under uncertainty, e.g., stochastic dynamic programming.

Main point: There is (A) the math and (B) the real world, and the two are not the same. Hopefully the real world problem can be solved nicely with some math, but we should know when we jump from the (a) real problem to the (b) math problem and have good understandings of the relevant issues of both (a) and (b).

For a (crude) analogy, the math, e.g., LP is just a tool. E.g., recently somehow my plastic mail box got whacked with some cracked plastic and a hole in the top. Fixes used some tools, tape measure, hand saw, bridge saw, circle saw, hand planer, wire brush, electric drill, epoxy glue, spray paint, ... -- but none of these were directly "mailbox fixes" but just tools useful in fixing the mailbox. In this analogy, the math is not a real fix but just some tools that might yield a real fix.

Right, I get what an LP is. As I said in the other thread, I'm mainly trying to use charity to the opening post to find a way it can be meaningful. With an idea that mayhap you could use RL to try and explore a solution space that you don't know the penalty for non-compliance to a solution.

Specifically, use LP to craft a plan in an iteration of scenarios that is varying a cost function through iterations. I suspect I'm fumbling in the multi period planning realm more so than the RL one, here.

I further suspect, though, that it is more "if you can formalize your problem such that LP solves it, use LP." That is, the opening post was largely just nonsensical?

> Your response makes me wonder if the idea is not that RL can give more optimal solutions, but perhaps more robust ones?

LP (linear programming), given an optimal solution, has some sensitivity analysis that nicely addresses robustness.

In the LP find x to solve

maximize z = cx

subject to Ax = b, x >= 0

small changes in c, A, and B can result in large changes in x. But, still, due to some continuity in the linear algebra, the changes in z should be (I am avoiding doing the derivations) will be small and continuing to use the optimal x from before the changes should (again, some derivations are needed) be not far off (omitting details).

The feasible region is much like a diamond with flat sides and sharp points (the extreme points). And the feasible region is convex, i.e., for x_1 and x_2 in the feasible region and t in [0,1],

Ish? Sensitivity analysis measures, by definition, sensitivity to inputs. I suppose you can model it so that some extrinsic factors are themselves written as inputs, but I haven't seen this explored with the idea of robustness, much. (Granted I may just be ignorant in that area.)

What you are pointing to, seems to be degrees of freedom on an input? The more options you have in an input, the better. Robust systems also have to deal with the unknown adversaries of the system that will work against it, though? Not just, "we were sensitive to the costs of feed changing, but there is also now an avian flu going around and we have to kill our flock." A simulation is easy to imagine that does this and forces you to see what is next. Is that common in LP formulations?

> common in LP formulations?

If you want to consider the effects of war, inflation, something like the 2008 housing crash, economic depression, COVID, hurricanes, train derailments, etc., sure, if those hypothetical changes still yield an LP (linear program), can just solve the resulting LPs and display the resulting data in a way can visualize and get some intuitive insight.

But there is another way: Let insurance companies consider history and set insurance premiums for disasters, all disasters, ever seen before or not, and they will have quite good data, likely, sorry, better than get from RL (reinforcement learning) or, really, anything that can be obtained from anything like current analysis of data from Web crawling. A common assumption, understanding in business is that it is possible to get insurance on nearly anything. E.g., the whole thing about the book and movie The Big Short was that it was possible to get insurance on the BBB tranches of housing mortgage bonds, make a bet that the bonds would fail. For this would have to pay monthly premiums but when the bonds failed make huge returns on the premiums paid.

With their good data and history, an insurance company will be able to set insurance premiums that give you some good information about the seriousness of the disasters, on your list of disasters or not, you are concerned about. If seeing the premiums you are still highly concerned, the pay the premiums and relax.

Sure, if I were to invest, what for me are big bucks, in raising a flock of chickens, maybe I'd get some insurance against any disease, e.g., bird flu, attacks by rabid foxes, wolves, raccoons, fire, flood, etc. destroying the flock.

Sure, once I did some Monte Carlo work essentially to evaluate the odds of some survivors in a really complex, goofy situation -- with those odds, could have found premiums for writing n insurance policies. The issue was, if there were global nuclear war limited to sea, how long would the US SSBN fleet survive? For input, I used some encounter rate probability derivations Koopman did as part of ocean search in WWII. Then I used some detection radii of Red-Blue pairs and battle result probabilities (one dies, the other dies, both die, neither dies). My work passed technical review by a world-class probability professor. My summary results were a graph over time of the expected number of SSBNs surviving. Also the Monte Carlo result could have also reported the probabilities of other aspects of the scenario, e.g., the probability that at least 10 survived for 30 days, enough to write an insurance policy!

Lesson: Sometimes it's possible to do Monte Carlo under some seemingly wildly complicated situations. In particular, it's possible to get insurance to protect nearly anything from nearly anything.

Net, I would advise pricing some insurance instead of some DIY (do it yourself) with either LP or RL.

More broadly and by analogy, a hammer and a screwdriver are both good tools, but no sense in asking which is better. Maybe the same for LP and RL. And for one more tool, insurance policies.