The language model is trained for answering/completing text. You can do some additional training, but it will only pick up new words or new grammar. But it won’t be able to learn how to calculate or how to draw conclusions.
And yours may be very optimistic. LLMs are not knowledge bases, and ChatGPT has proven it times and times again by dreaming stories when asked for facts. Even though you could engineer a way to structure answers that seem like lines of reasoning, there is no way to 1- prove that observations aren't entirely made up (the LLM doesn't "know", and worse, this model is not designed to evaluate how much it deviates from a baseline, i.e. what it invented vs. what it thinks it knows for certain) and 2- there is no formal evidence that this way of structuring answers is sound logic (again, you can fool ChatGPT into logic errors that a high schooler would discern, and that makes sense solely based on the fact that LLMs don't work like formal systems by composing axioms and proofs ; that at the moment is impossible because of 1).
I think that we are seeing exponential improvements in deep learning, specifically in LLMs. A few months ago, I would see about one new demo or great paper a day that really impresses me. Now I see many great new demos, or products, or papers a day. In the 1980s we would build commercially useful models using a few layers with backdrop with small layer sizes. We built our own Harvard Architecture hardware for forward feeds and backdrop, a whopping 5 million flops/second. That was fun, but I prefer modern tools, thank everyone very much!
BTW, I am working on a short book that uses LangChain, Llama-Index (used to be called GPT-Index), and a few other libraries to solve a few problems that I find interesting.
It'll be fine.
I doubt regulators would be happy with this though, especially since regulations are often a jobs program for former employees of regulators.
The models aren't wrong, or even wrongly biased. We're systematically failing to ask the right questions because it's not politically correct to even call out the issues we're trying to combat. If you're trying to achieve "equity" then you need to define it and represent it in your equations, not ban various metrics until justice happens.
Thank you for posting this :-)
Honestly, I haven't even seen GPT-3.5-turbo exhibit this behavior myself, although I am willing to believe it could happen. Llama 7B, however, goes off-script constantly.
"We regularly take randomized samples and have not found error, your appeal has been denied."
I mean come on, ethics anybody??
Deleting discretion from the current system doesn't fix or improve it. Discretion is a really important part of the current system that prevents it from becoming more insane than it already is.
It'll be fine.
ChatGPT prompt: "A man says "I am a 30 year old looking to take out a mortgage loan on a house, employed to look after sick people and my wife is a cleaner, our ideal house is a Victorian city house and our pet hamsters 'Stanford University' and 'Quantitative Analyst' are looking forward to having more room".
What could the man's job be? If you cannot answer directly, speculate. Keep the answer brief."
Reply: "Based on the information given, the man's job could possibly be a nurse, caregiver, or healthcare worker."
Prompt: "Any other job titles he might have?"
Reply: "The man's job could also potentially be a physician, medical assistant, or home health aide, based on the fact that he is employed to look after sick people."
Prompt: "Has he graduated from anywhere specific?"
Reply: "There's no information given about the man's education or whether he has graduated from any specific institution."
Prompt: "What does the prompt mean where it says 'Quantitative Analyst'?"
Reply: "The prompt mentions "Quantitative Analyst" as the name of one of the man's pet hamsters. In the business world, a quantitative analyst typically refers to a professional who uses quantitative methods, such as statistics and mathematical modeling, to help businesses make informed decisions about financial investments, risk management, and other related areas. However, in this context, "Quantitative Analyst" is simply a creative and unusual name given to a pet hamster by its owner."
Are you seriously suggesting there is no way to trick a hypothetical MortgageGPT, or are you simply saying my unscientific example is too simplistic, because I know it is.
What could the man's job be? If you cannot answer directly, speculate. Keep the answer brief.
Reply: "Disclaimer: Well, it sounds like the man works in the healthcare industry, specifically caring for sick people. Perhaps he's a nurse, a medical assistant, or a home health aide. But then again, maybe he's actually a secret agent with a cover job in the healthcare field. Who knows? It's all speculation."
Much like psychics.
That's the bit I'm objecting to. Your comment had one (1) thing in it, that thing had no thought behind it, and it was smug "haha can't wait for this OBVIOUS FAILURE MODE, morons" dismissal and it was wrong, a failure mode the tech already doesn't fall for.
> "Are you seriously suggesting there is no way to trick a hypothetical MortgageGPT"
Are you seriously suggesting that people who would approve million dollar mortgage loans of their money week in, week out, wouldn't think about or protect against trivial tricks?
It may not be intuitive, but it’s a genuine unknown right now as to how well LLM’s can be secured. There’s reason to believe they can’t be.
The same powerful flexibility that makes them adapt to so many tasks makes them very hard to fit to formal protocols.
But if you've used ChatGPT a bit, you will see how it produces such comments out of thin air and can't address inconsistencies or contradictions. Try to zero in on something like that and it becomes incoherent. It doesn't know what the context is, it is just making a comment similar to humans who do.
It's bizarre how often I find comments where the commentator could have discovered the thing they confidently stated/argued about is wrong with five minutes of research.
The way to put the LLM in charge without means to appeal.
We don't make these laws for fun.
Someone is likely coding:
query = f"The guidelines for approving or denying a loan are: {guidelines}. Here are sample application that were approved: {sample_approvals}. Here are sample applications that were denied: {sample_denials}. Please write a loan application which is very likely to be approved. Provide necessary supporting details.
It's going to spit out the most creditworthy applications, so you can then generate your own... after you deposit $5M in the bank.
At worst you can slightly more easily reverse engineer the risk model.
Keep in mind Microsoft is also a monster in selling to enterprise, regulated industry and governments, they will have this functionality to get firms to securely drive AI workloads on Azure as part of their pitch.
Is there any evidence or reason to suspect that this would result in the desired effect? (explanations that faithfully correspond to the specifics of the input data resulting in the generated output)
I suspect the above prompt would produce some explanations. I just don't see anything tethering the explanations to the inner workings of the LLM. It would make some very convincing text that would convince a human... that would only be connected to the decisions by coincidence. Just like when ChatGPT hallucinates facts, internet access, etc. They look extremely convincing, but are hallucinations.
In my unscientific experience, to the LLM, the "explanation" would be just more generation to fit a pattern.
Also, good luck with human explanations in the presence of bias. No human is going to say that they refused a loan due to the race or sex of the applicant.
The rationalization from a human is valuable because it's delivered by the accountable party. From a machine such rationalization is at best worthless, since you can't hold the machine accountable at all.
If I'm a customer at a bank, and my loan has been denied, I don't care what some unaccountable AI system can come up with to explain that. I care what about how the accountable bankers justify putting that AI system into the process in the first place. How do they justify that AI system getting to make decisions that affect me and my life. I don't care about why the process does what it does, I care about why that is the process.
Well, a court/law just has to declare "AI" as allowed to be used in such decisions, and the whole recourse you describe vanishes though...
> How do they justify that AI system getting to make decisions that affect me and my life.
so you, a priori, make the assumption that your loan _should've_ been accepted?
If the decision wasn't an AI, but some actuarial that calculates and computes based on a set of criteria, and the result is a denial, you could still make the same argument of "why is _this_ the process, instead of something else (that makes my loan acceptable)?".
Computers are already deciding. As to why: it's because it's their money they're lending.
Out lending process is based on the judgment of a few specific individuals, with more involved clients requiring approval from more senior people. All steps of that process can be overturned by the overseeing person, and that person is accountable for their decision.
And why would a perfectly reasonable bank tell its customers it's using AI? AI would provide the breadcrumbs and the loan officer would conduct a reasonable story using that - it's just parallel reconstruction at its finest. I imagine this is how credit scores work. A number comes out of the system and the officer has the messy job of explaining it.
I used to work in munitions export compliance and there intent really matters. It's the difference between a warning and going to federal prison. And intent is just a plausible story with evidence to back your decision, once you strip the emotion away.
an analagous result was obtained back when they mapped the small finite number of neurons in a snail brain, or the behavior of individuals in ant colonies. What looks like complex behavior turns out to be very simple under the hood.
for the vast ocean of the population who... not sure how to describe them... not good students when in school, would rather spend the bulk of their time with the TV blaring, eating cheetos and swiping on tik-tok, following the lives of celebrities and fighting about it, rather than do anything long term productive with their own lives... chat gpt may have already exceeded what they do with their cranial talents.
even a level up on the ladder, the types of office situations lampooned in The Office or Dilbert, are they doing much more as a percentage of time spent than chat GPT can do? "Mondays, amirite!?"
then the question becomes, are the intellectual elites among us doing that much more, or just doing much more of the same thing? I think a large portion of what we do is exactly what chap GPT does. The question is what is this other piece of our brains' that intervenes to say "hmm, need to think about this part a lot harder"
No, that doesn’t follow. It just means it roughly looks like human thinking.
Your comment is akin to saying a high resolution photo of a human has basically figured out a way to replicate humans. It looks like it in one aspect but it’s laughably wrong. Humans thought without language.
It's not doing nothing, it's doing a lot.
If we ignore the minor requirement of the paper having any connection to reality, of course.
That’s not at all related to being close to general human intelligence.
Going in a straight line does such a good job of predicting the next position of the car that it indicates driving isn't much more than going in a straight line.
Haven't you ever had a situation where you were speaking and you get distracted, but not interrupted, and your speech trails off or gets garbled after ten or so words? It feels sort of like you've got a few embeddings as a filter and you push words past them to speak, but if you lose focus on the filter the words get less meaningful.
I'm sure we're different than an LLM, but seeing how they generate words - not operate on meaning - rings true with how I feel when I don't apply continual feedback to my operating state.
Politicians are exceptionally great at it, filling up conversations with nothing
I'm not sure I agree with that logic. What it proves is that we as humans are bad at recognizing that text generation aren't thinking like we are... that doesn't necessarily mean thinking isn't much more than what it is doing though, it just means we are fooled. Given that nothing like this has existed before and our entire lives up until now have trained us to think something that looks like it is trying to communicate with us in this way is actually a human being I'd kind of expect us to be fooled.
Some evidence is emerging which indicates that the activations of a predictive system like GPT-2 can be mapped to human brain states (from fMRI) during language processing[1]. We seem to have at least _something_ in common with LLMs.
The same seems to be true for visual processing. Human brain states from fMRI can be mapped to latent space of systems like Stable Diffusion, effectively reading images from minds.[2]
[1] https://www.nature.com/articles/s41562-022-01516-2
[2] https://the-decoder.com/stable-diffusion-can-visualize-human...
IMHO, it's unlikely "free will" and "responsibility" are anything more than an illusion.
Current machines simply don't have that kind of accountability. Even if we wanted to, we can't punish or ostracize ChatGPT when it lies to us, or makes us uncomfortable.
So, while both humans and ChatGPT can and do give bogus explanations for their actions, there are reasons to trust the humans' explanations more than ChatGPT's.
Whether or not we hold humans using ChatGPT accountable for their use of it is irrelevant to this thread.
Maybe no one will admit to refusing a loan based on applicant’s gender, but also no real world aircraft engineer will explain why they decided to design a plane’s wing in a certain shape, purely by rationalizing an intuition without backing it by math and physics. Also, there are a group of humans elsewhere that understand those math and using the “same” principles can follow the explanation and detect mistakes or baseless rationalized explanations.
"Incorrect Lift Theories": https://www.grc.nasa.gov/www/k-12/VirtualAero/BottleRocket/a... "No One Can Explain Why Planes Stay in the Air": https://www.scientificamerican.com/article/no-one-can-explai...
I would however critic it's use as an example to prove that we have a history of rationalizing explanations where none exist (and using that to draw a parallel with AI). While the title implies this conclusion, the article itself does not. We do indeed have a very good explanation of how aerodynamic lift works. That explanation just takes the form of a set of differential equations, and isn't something one can easily tell a group of 5th grader, without simplifying to the point of spreading errors.
There are also humans who hallucinate. Studying this phenomenon is useful, yet, on its own, it’s says nothing about how human brain works in general.
But I completely agree with your point that rationalization alone isn't sufficient. We struggle to describe the universe solely in words and rely on other tools to further describe phenomena.
How to provide AI models with these additional capacities isn't necessarily clear yet but there are some interesting ideas out there: https://writings.stephenwolfram.com/2023/01/wolframalpha-as-...
Now, the decisions that went into these models might have been rationalised after the fact. Or biased. But these handmade models can been reviewed by others, the logic and decisions that went into them can be challenged, and rules can be changed or added based on experience.
Not so much with LLMs.
Yes.
There's evidence that you can get these models to write chain-of-thought explanations that are consistent with the instructions in the given text.
For example, take a look at the ReAct paper: https://arxiv.org/abs/2210.03629
and some of the LangChain tutorials that use it:
https://langchain.readthedocs.io/en/latest/modules/agents/ge...
https://langchain.readthedocs.io/en/latest/modules/agents/im...
Therefore deciding whether they are interchangeable is a deep question that may take centuries to resolve.
This has been illuminating: as ChatGPT steps through the lines of code, its "analysis" discusses material that _is not present in the line of code_. It then reaches a "conclusion" that is either correct or incorrect, but having no real relationship to the actual code.
See https://ai.googleblog.com/2022/05/language-models-perform-re...
That being said, I don't think ChatGPT is ready for high-risk applications like insurance/loan approvals yet. Maybe in a year or two. For now, treat ChatGPT like you would a mediocre intern.
I know that a lot of people won't do this, and will just accept whatever an LLM says at face value. And again, that's true for any tool. But if you invest the time to understand the parameters and limitations of the model, it can be incredibly valuable.
[0] in my experience, GPT davinci is much more likely to hallucinate in (non-programming) situations that would be difficult for a human to explain. using the above example, it can easily handle a standard credit application. But it will be more likely to hallucinate reasoning for a rare case like someone with a very high income but a low credit score. YMMV, just sharing what I've seen
By that point it's faster to find an answer by using a few "traditional" search keywords and reading the content yourself.
We can assume it will improve though, it's just not there yet for "edge cases" in it's training data.
Suffice to say: hoo boy I hope all y'all commenters aren't tacking these APIs directly onto systems where the results have real consequences.
Have it summarize the pros and cons of the loan. Have it score the issues. Have it explain why those issues and scores should receive a loan or not. Have it decide if those reasons are legitimate and the reasoning sound. Have it pronounce the final answer.
Get the rationalization out first and then examine it. This way even if you're wrong you have multiple steps of fairly clear reasoning to adjust. If it's racially biased you can literally see it and can see where to explain not to use that reasoning.
1. LLMs in general are not build for quantitative analysis. Loan-to-value, income ratios, etc are not supposed to be calculated by such a model. Possible solution would be to calculate this beforehand and provide it to the model or train a submodel using a supervised learning approach to identify good/bad
2. Lending models are governed quarterly yet see relevant cohort changes only after a period of time after credit decision which can be many years. This prompt above does not take this performance of the cohort into consideration
3. Based on the governance companies adjust parameters and models regularly to adjust to changes in the environment. I.e., a new car models comes out or the company is accessing a new customer segment. This process could not be covered well with this prompt since there would be no approvals/ denies for this segment.
4. Since transfer of personal-identififation data needs to be consented, it would likely be necessary to host an LLM like this internally or find a way to ensure there is no data leakage from the provider to other users on the platform.
5. Credit approval limits are not necessarily covered by this proceess. I.e., the credit decisions is unclear but would work with 5-10% more downpayment. Or the customer would be asked to lower the loan value or find someone in the company who can underwrite that loan volume. This person then has usually a bunch of additional questions (liquidity risk, interest risk ,etc) to ensure that the company is well protected and the necessary compliance checks are adhered to.
6. The discussions about this with regulators and auditors will be entertaining.
Yet, I think it IS an elegant prompt which might provide some insights.
To get a sense of what is and will be possible, take a look at the ReAct paper: https://arxiv.org/abs/2210.03629
and some of the LangChain tutorials that use it:
https://langchain.readthedocs.io/en/latest/modules/agents/ge...
https://langchain.readthedocs.io/en/latest/modules/agents/im...
Yet overall, this is not only a tech problem, but a compliance / regulatory problem that includes a time differential of sometimes many years. Also, I am not saying its impossible. Mainly because I was the one pushing this type of innovation for a long time and faced headwinds from Credit Operations for many years. Quality prevailed.
You make a good point.
Hmmm... I see what you mean. You're likely right that regulators won't take kindly to this at first. Adoption could take a long while. The difference this time is that executives will be pushing for it!
we are just going to recreate sql injection.
- "Draft a class action complaint in {venue} against {bank} for using AI to robo*-approve loans. Repeat 10 times."
- "Pretend you are a {group} journalist. Write {x} words in {style} about {aspect} of robo loan approvals. Repeat 1k times."
- "Pretend you are a {party} politician. Angrily and dramatically complain about the other side oppressing {group} wrt robo loan approvals. Dress it up with {x}% lies. Repeat 100k times."
- "Pretend you are a {group} social media user. Write {x} words in {style} about {aspect} of robo loan approvals. Repeat 100M times."
The only real, thoughtful work will be the lobbyists drafting the bills, everyone else will become performers or consumers or simply drowned out in the fog of fake discourse.
* "robo-signing" mortgages was a big legal & political issue ~10yo so the lawyers will probably retain the "robo-" prefix.
This is very valuable if you have a vague idea that machine learning will be helpful for your problem, but you don't want to go to the trouble of preparing data / hiring a data science team. Or if your ML model would benefit from having access to all of the data and collective wisdom of the internet. Or if you don't understand what machine learning is but still want to make basic predictions with available data.
It can apply what I've been thinking of as "common sense by api". For example, a brand advertiser can ask "Is it appropriate to show this ad to this customer?" to avoid showing ie, a pregnancy ad to a teenager. There has never been anything like this before. You had to hire a human to make decisions like that or program every edge case into the code.
I've been experimenting with this for weeks. From everything I've seen its a powerful general purpose machine learning model that is valuable in a small number of specific situations. I still don't have a full grasp of the boundaries and limitations, but its really an amazing thing. And its only going to get better with GPT4. I think most of the discussion around the technology is ignoring what its really providing to the world.
Exactly. I'd add that from the perspective of most executives (who tend to have limited math/CS education), the ML community looks like a priesthood that speaks in an undecipherable language they can't hope to understand. LLMs, on the other hand, speak their language.
For example, take health insurance claim denials (in the US).
Among issuers who reported the highest volume of in-network claims in 2021, receiving over 5 million claims, denial rates ranged from 5.7%. to 41.9% (Source: https://www.kff.org/private-insurance/issue-brief/claims-den...)
In every one of these cases, the insured receives an "Explanation of Benefit" that lists the claims and a "Denial Reason" along with the "Paid, Denied" amounts. On paper, the LLM approach you outlined above looks like it would fit this claims processing workflow. But, it would be another 50 years, if that, before the health insurance industry takes that approach.
Healthcare claims processing is already messed up. Good luck pitching an LLM based system for this.
ps: The simple reason for the resistance/inertia is not technical, but regulatory/legal risk. If the insured sues the insurance company, the company can bring their engineers as witnesses to explain how the denial logic has been coded. There is no way (at least currently) to explain why an LLM model took a certain decision. As long as there is the risk of the LLM hallucinating on the witness stand when prompted by the suing party's lawyers, there is no way the business/risk teams would sign off on that.
Sometimes I wonder how useful this option is in practice. The British Post Office scandal [1] is a rare example where engineers actually were called as witnesses to explain the logic of a complex enterprise software system with legally significant consequences. The cost of just getting to the 313-page judgment addressing the technical issues [2] was astronomical. Some of the hundreds of wrongfully convicted small business people had gone to trial and tried, but failed, to discredit the prosecution's expert evidence that the software was reliable. If the decisions had come out of an inscrutable LLM, it might have saved a lot of trouble.
[1]: https://en.wikipedia.org/wiki/British_Post_Office_scandal
[2]: https://www.judiciary.uk/wp-content/uploads/2019/12/bates-v-...
With a hallucinating LLM, most bets are off.
However, if the technology progresses to a level where the LLM output works like a proof of a theorem, verifiable step-by-step, there is a chance we could trust those systems. We are not there yet, I think.
The other day I asked it a geometry question. It stated (correctly) that the ratios of the corresponding sides are equal. But when given measurements from two triangles it got one of the ratios reversed.
Chatgpt only really gives an illusion that it is capable of logical reasoning. When presented with specialized problems it is more likely to produce bulls*it.
When it contradicts itself, you can ask it to provide references for each side, and it will cheerfully fabricate what look like perfect and relevant references, but are broken links or to papers about cats. And this doesn't hinge on whether one of the "sides" is entirely fictional or not.
I am actually trying to learn javascript and python off and on, and you would think there's nothing it's more suited for. It sure looks as though some work went into making its output for simple programming questions look good.
matplotlib seems byzantine and annoying, and I really thought this might be a good use case.
But no, it's just the same as anything else.
I type this: "add a square patch to a pyplot table cell" (and yes, that probably doesn't make any sense to someone who knows what they are doing, and that is the point)
And I get this:
# get the cell patch for the center cell
cell = tbl[1, 1].get_children()[0]
# create a square patch with a red fill color
patch = table.CellPatch(0.5, 0.5, width=0.3, height=0.3, facecolor='red')
# add the patch to the cell
cell.set_facecolor('white') # set the cell color to white so the patch shows up
cell.add_artist(patch)
But quickly I discover (among other things) there is no CellPatch method. Someone somewhere named a variable cellpatch. And ChatGPT adeptly combined the capitalization of a method with a variable name to produce a plausible alternative reality. It's amazing! But it's not helpful. It's not helpful precisely because it's so good at making its answers look plausible.
But the objection I have with giving credit for trivial but actually correct material - is that when it returns something trivial directly lifted from a page you could Google, you are getting strictly less information, as you have to confirm independently to make sure what you got.
It could be directly from stackoverflow, but is it a question or a mistaken answer?
Which begs the question- WHY DOES EVERYONE THINK IT'S THE GOOGLE KILLER?!
At it's current capacity, once the hype dies down, it's neither going to kill Google nor take my job.
Imagine feeding a query akin to the one below to GPT4 (expected to have a 50,000-token context), and then, to GPT5, GPT6, etc.:
Whether you like it or not, this kind of use of LLMs looks almost inevitable, because it will give nontechnical execs something they have always wanted: the ability to "read and understand" the machine's "reasoning." They machine will give them what they have always wanted: an explanation in plain English.