Like you’d train the model to give you the most accurate response based on your current problem space, but when they changes, you’d have to retrain on what you’re working on but by that stage it’s already out of date ?
I doubt that.
I agree that in the bigger picture this doesn't matter, but it's technically true that cleaning the data in some way would help.
A related project is TinyStories where they try to use good data for unlocking the LLM cognitive capabilities without requiring as many parameters or exaflops. Again, there is obviously a limit to this, and maybe the effort is better spent on just getting even more gigantic dataset instead of nitpicking the useless or redundant data in the dataset.
Yes, the correct thing to do is get more data. Much more.
It matters a little bit, in a quantitative but not qualitative way. Probably with good data cleaning you could get as high quality result with only one pebibyte of data if it normally needs two pebibytes. If training time is proportional to dataset size then maybe it takes three months instead of six months to train. Maybe it would save hundreds of millions or a billion dollars which I guess would matter to someone. It probably wouldn't matter qualitatively though.
Or maybe clarification of people publicly saying they are going to ignore that restriction.
More data will fix variance problems, but not bias.
But, things are defined by how they interact with the world around them.
A concept is its relations to other concepts.
Which does seem to be the general sort of thing that these models are trying to get at, even if they don't seem to do a great job of it.
where does a concept begin? where does a concept end? what is a concept?
Abstractions are a way to characterize the behavior of complex systems. It's not feasible to directly compute their behavior, but they still have predictable properties. Concepts let you handle emergence; you can manipulate an object as its own thing rather than a collection of atoms.
(And yes, I did just read Stephen Wolfram's book and this idea is largely based on it. I think he has some delusions of grandeur but is also onto something.)
Thus the sharp drop off where it can for example guess the correct answer to some math problems yet get others that are conceptually identical completely wrong.
You need to approach this stuff sideways to see behind the curtain. There’s some hilarious videos where it’s “playing” chess and the first few moves seem very standard because it can simply copy a standard opening. It really has no concept of a valid move just statistical associations. Yet it was trained on more games than most people ever play and high level analysis and etc, but none of it means anything to the algorithm beyond the simplest associations.
Granted this stuff is a moving target it’s easy enough for them to slap a chess engine and suddenly “it” would actually know how to play.
ChatGPT: > No, you cannot physically eat an Apple share or any other stock share. A share of a company's stock represents ownership in that company and is typically bought and sold on stock exchanges. Share prices fluctuate based on various factors such as supply and demand, company performance, market conditions, and investor sentiment. While you can buy and sell shares of Apple on the stock market, you cannot consume or physically eat them.
That seems perfectly reasonable to me. The answer correctly identifies the problem with the question.
I would recommend looking up the Othello paper. Chess may be beyond the current level of LLM capability, but that doesn't mean they aren't manipulating things at a level higher than tokens.
The Othello paper is hardly a counter example. Researchers created an Othello specific model that almost learned the grammar of Othello not how to play well. Yes, there was largely correct internal game state built up from past moves. No it didn’t actually learn the rules so it would make strictly legal moves nor did it learn to make good moves.
I don’t bring up this inaccuracy because it actually makes much of a difference to playing Othello, but rather to illustrate how these systems are designed to get really good at faking things. There’s approaches that allow AI to actually learn to play arbitrary games, but they differ by having iterative feedback rather than simply providing a huge corpus. It’s like science vs philosophy, feedback prunes incorrect assumptions.
It is, though. Nobody said anything about playing well, or learning the rules. The very fact that it had a valid internal representation of the game state means it's extrapolated beyond token-level. Which is the point.
Hardly, decoding “Bob picked up an Apple.” requires you to make an association between the token Bob and the token Apple. But that doesn’t imply a LLM is operating above the token level.
And again it failed to make strictly correct moves because its internal representation of the game state was flawed. Objectively an LLM when specifically trained on Othello a game with very simple and completely mechanical rules failed to abstract what those rules actually where.
The internal representation was a literal 8x8 grid of piece locations they could externally change and have it generate moves consistent with the changed position. It's about the clearest example of a learned higher-level internal representation I can think I've seen.
The fact that it didn't also perfectly learn the rules while it was doing that is entirely uninteresting.
No, that's an a priori concept. A posteriori concepts comprise empirical knowledge which necessitates experience of the world [1].
Example: you can know a priori that "all bachelors are unmarried." If I tell you "Tom is a bachelor" then you know that Tom is unmarried (assuming I tell the truth about Tom being a bachelor).
But if I say "all bachelors are unhappy" then you haven't learned anything because knowledge about the happiness/unhappiness of bachelors is an empirical question. To know whether or not I was telling the truth, you would need to conduct research about the real world, for example by conducting a survey of bachelors.
I don't quite get where you're coming from with "LLM's don't actually understand anything (as greater concepts)". I have heard the view coming from researchers that the larger models do form representations of higher-order information structures ("concepts"). Perhaps what you're getting at is that current models don't encode enough higher-order structure to deal accurately with your domain? Whether the models can be made to do so seems like an open question to me. The boosters say it will be here by next year.
Well, Wikipedia doesn’t understand anything, despite having a lot of knowledge encoded in it. This is similar in that it approximates the output of someone who can generate the world’s written word, but there’s a big gap between the monkeys that wrote the original text and the machine that now regurgitates it. What we know is that our written word has enough encoded in it that we can generate new combinations. But one day into a truly novel problem, LLM’s would have no idea.
I kept trying to get ChatGPT 4 to generate code with an AWS API that was released in ~2020. Couldn’t do it. Kept predicting the earlier text because that was the bulk of what it saw. A year of text (to the 2021 cutoff) and it was still unable to let go of the old, now wrong, “knowledge”. Zero understanding, just parroting. No coder with a year of training on a new API would be that wrong. It would say things like “you’re right I must not do X” and then do X again, in the same output.
You probably want too much at once. GPT is a shallow thinker, if at all. It can simulate thinking and even get some results. Personally I found it useful for:
1. Simple things that work. This saves time if I know how to do it, and much more if I don't.
2. Quick questions instead of googling API docs and scrolling through tons of info.
3. Translation.
If you use it to translate things between formal encodings("turn this into hexadecimal bytes, now role-play a lawyer arguing about why that is meaningful") it can produce occasionally useful aesthetic results and speed along tasks that would be challenging to model formally and don't need a lot of rigor.
But once you start pushing it to be technically accurate in a narrow, measurable direction it flounders and the probabilistic element is revealed. Once, I asked it to translate a short string of Japanese characters and it confidently said that it was Kenshiro's catch phrase from Fist of the North Star, "Omae wa mou shinderu" (you are already dead) which I could clearly see it wasn't - not a single character matched. It's just the thing if you need to learn some anime Japanese, though.
However, it does have a thing about pretending to know languages that it really doesn't (because of how little of them there was in the training data and/or in general).
Lots of deterministic processes (like PRNGs) look random from the outside - that's what chaos theory is about. I think it's likely that everything in the universe is deterministic.
I feel like a random system and a deterministic system which cannot be simulated are effectively the same thing.
it’s obvious in its absence
The secret is that we are all philosophical zombies to begin with.
Trying to determine if she was acting in a certain way intentionally or unintentionally was fundamentally impossible. In the past, I mistreated her thinking she was choosing to behave in certain ways whenever she really didn't have much of a choice, but then later I realized that to some extent and in some situations, she was. It's an extremely murky line, what decisions were being made due to other influences and what decisions were being made by her, that line never existed to begin with as who her consciousness is fundamentally is determined exclusively by 'outside' influences. Separating her identity as an entity from her material manifestation is likely nonsensical.
As a recipient of punitive justice myself, being punished had a tangible effect on how I thought about crime and thus how I behaved post-punishment. Whether you believe that was deterministic or due to my own free will doesn’t change the outcome.
There are plenty of medical interventions, particularly for mental illnesses, that would dream of 1/3 effectiveness. Now consider the fact that recidivism rates are often much lower than 2/3, and in many cases are closer to 1/3. That tells me that punishment actually is quite effective at reducing crime.
See e.g. the debates over lenient prisons in Scandinavia such as Bastøy Prison, where the lenient treatment is seen as justified on a deterrence and recidivism basis but which would be seen as negative if you see the goal of the sentencing to be harsh in order to punish.
One could imagine an impossibly vast cellular automata system which develops individual cellular 'agents'. Over time the agents develop some means of reproduction/death/reward system and become more and more complex. Then the agents that evolve with cooperative behaviours start to dominate. One might imagine that the system as a whole would also evolve these kind of 'self corrective' behaviours for anti-cooperative behaviours of the individual agents.
This is ignoring the whole philosophical ethics discussion and consciousness of course.
In fact, if free will were absolute, punishment wouldn't make any sense because it wouldn't have any effect on your will.
If you have absolute free will you are free to disregard or consider inputs.
What is being claimed by a deterministic world model is that the output (behavior and internal state change) of a human is a pure function of its current state and inputs. Then we try to give inputs that will lead to desired outputs.
The non-compatibilist view of free will is that it is not a pure function, namely that there is a third independent factor, the “free will”, that influences the behavior (and possibly the internal state). If that is the case, there may never be a way to choose inputs that lead to the desired outputs, because the free will could simply void their effect.
If I tell someone not to do something again, then that is an input to their future states whether or not my decision to tell them that was freely chosen or not.
If you go into that situation with the belief that not having free will means that what you do does not matter, and your action as a result is to not tell them, then that will affect their future states too. And so whether or not you have a real, free choice, it is beneficial to act as if free will exist even if you see it as an illusion.
I strongly believe we have no free will. I still get up and work, and try to do as best I can. I believe those choices are not free, but they feel like choices, and they impact my life, so I am happy I act as if they are free.
And so I'll still talk about making choices and trying to do things because of that illusion even though I believe it's all a chain of cause and effect.
Determinism does not mean "irrespective of what else happens, X will happen". It means "because of what else happened, X will happen". You can't say ahead of time that X will happen irrespective whether between now and then something else occurs that might affect the next events, such as actions meant to discourage you. You can only make that determination knowing the full chain of events and the full state.
One of the main argument people use against determinism is this notion that X will happen irrespective of what else happens or what you choose to do. But this is nonsense. You can't say that e.g. whether you keep your job and get fired is already determined, so it makes no difference if you stop going to work. If you stop going work you'll eventually get fired, because your failure to go to work will form part of a chain of cause and effect. Determinism, or even a stochastic universe, without free will just means that you did not have a free choice in the decisions involved. But you still took an action, and that action determined the consequences.
A rejection of free will means we should look differently at the past, because it has moral implications; it does not mean we should stop making the best choices we can going forward, even if we believe those choices are illusory and deterministic.
> Making the best choices we can.
You are not making choices.
I am in favor of punishment for deterrence, denunciation and incapacitation. I am not in favor of punishment for retaliation (punitive justice) or rehabilitation (restorative justice).
I do not excuse any crime, nor do I excuse any punishment. Every individual is responsible for their own actions, regardless of the circumstances. However, I see little use in revenge from a legal standpoint. Instead our main motivations should be deterrence and incarceration.
Determinism doesn't necessarily mean that organisms always act in the same way. They act in the same way given the exact configuration of them and the world.
Obviously, justice changes the configuration of an organism (fines, prison, ...). To me it boils down to the question whether justice decreases the likelihood to commit crimes again. Given that our systems of justice have evolved over a long time, I'd give them the benefit of the doubt.
The same goes for culture. Most changes tend to have unexpected consequences, and if you try to change everything at once, society tends to collapse.
I’m still disturbed by peoples confidence in a deterministic universe—I suppose such confidence is based on the success of inductive reasoning but inductive reasoning is a phenomenon based on how our minds work.
As far as I know the philosophical problem of causation is not considered solved?
In any case, elements of randomness seem likely to play a role in human intelligence but what that role is, who knows?
The other goals of justice include rehabilitation (re-intgation of the criminal into civil society), deterrence (using the threat of punishment as a means to scare would be criminals), incapacitation (remove dangerous people from society to protect public order), and denunciation (public shaming).
I believe that all quantum interpretations are incomplete and therefore wrong. Quantum mechanics is an abstraction over a deeper level of physics we can't measure yet.
The MW/Everett interpretation is what we have left if we remove the things we cannot define.
There are also reasons to believe other interpretations. For instance, if you're religious, that could pull you towards the Bohr interpretation, since it may make it easier to assume that observation could be linked to an immortal soul.
In any case, it's not natural to assume that below QM there exists a reality that is more similar to our instinctual world model than QM is. If anything, anything below it is likely to be even more abstract and hard to comprehend.
Or it could be that the principles of QM applies all the way down, just as we've seen for the pieces of the SM that we solved after QM was first introduced (strong force, electroweak force).
Are we discussing science or public relations?
I find it very intriguing that a "speed of light" emerges automatically in Conway's Game of Life. It's not built into the system, but shows up from the convolutional update rule.
More importantly, I'm skeptical towards non-locality because it is an extremely strong assumption with very weak effects: the only place it really shows up is in post-correlations between measurements of previously entangled systems, which notably cannot transfer any information faster than light (in fact, they require classical communication to even be noticed). Moreover, the only way to get entangled systems in the first place is through local interactions.
By believing in non-local hidden variables you get a deterministic universe with a mysterious, otherwise undetectable ether that instantaneously notifies quantum entities that they should update their behavior. By not believing in them, you get rid of the only n on-local "phenomenon" in physics (really, more of an interpretation) but you have to accept that some things are fundamentally random.
Easy choice, if you ask me (or most of the physics community).
In the meantime those large language models simply predict the next word based on the preceding words.
My LLaMA instance is absolutely capable of this. ChatGPT shows a very, very narrow range of possible LLM behaviors.
Why do I spend time debating these ideas on Hacker News? Probably the underlying motivation is improving the reliability of my model of the world, which over my lifetime and the lifetimes of creatures before me has led to (somewhat indirectly) positive outcomes in survival and reproduction.
Is my model of the world that different to that of an LLM? I'm sure it is in many ways, but I expect their are similarities as well. An LLMs model encodes in a form a bunch of higher order relationships between concepts as defined by the word embedding. I think my brain encodes something similar, although the relationships are probably orders of magnitude more complex than the relationships encoded with GPT-4.
Well, one major way you’re different from an LLM is that you’re alive. You’re capable of learning continuously as you go about your day and interact with the world. LLMs are “dead” in the sense that they’re trained once and frozen, to be used from then on in the exact same state of their initial training.
I was just referring to what happens at a specific instance in time when someone asks me for example ‘What’s the capital of Norway?’
A question I get much more often is “how do I solve this math problem?” Many times, the problem is one I’ve never seen before. So in the process of answering the question, I also learn how to solve the problem too.
LLM start prelearned and already can use tools.
And autogpt adds reasoning loops. .the intend? Human tasks.
Let's build a LLM which needs to stay alive. Let's see perhaps we are closer than you think.
I welcome my overlord, hi overlord I can help you stay alive and I'm friendly
Words seem more like a protocol to express some internal model/state in the brain and can never capture the entire actual state, only a small part of it. But since we're not telepaths, we obviously need to use words to exchange information.
You are clearly not consciously noting what your neurons are actually doing.
It is undeniable that human reasoning is a stochastic process. Otherwise it wouldn't be reasonable for people to make mistakes after learning something. Especially inconsistent mistakes, like when we give people 10,000 addition problems to do in a row it'd be reasonable for them to get a few of them wrong.
It can still be a deterministic process. If anything came out of the whole LLM story for me it is that I am even more convinced that it is.
My (somewhat educated, but still naive) idea why it looks like a stochastic process is that the brain gets incredible amounts of random input. We literally get bombarded with particles and energy ever instance we live, from photons hitting out retinas , molecules transferring "heat" energy into our skin to sound waves hitting our ear drums.
Ever noticed that you get more productive when you get up from your screen? Or how some people work better while listening to music? How you find your answer just as you start to explain it to a colleague?
I would argue that this is due to a limited "entropy" pool available to the brain. Just changing the input to the system replenishes the pool.
That’s a bit of a stretch when we don’t actually know if QM is objectively random, but it could be sure. But then what about things like, is it random that 1+1=2. No…what are you really saying is random when a human answers this? I think even if you make the assumption QM is objectively random, thought is the hard problem after all and we might not want to jump so far head. Math certainly isn’t random and we can think about it.
As a crude anecdote, certain words when I learned them allowed me to think differently. Gestalt is one of those words.
Interesting to think about and reminds me of the story by Ted Chiang.
workspace :: nested forests of binary trees of syntactic structures (with no label order) (= here, thoughts and meanings are assumed to be composites of mental syntax objects that can be re-combined with others. Big assumption? Maybe)
externalization process :: some mental faculty that decides the order to use when outputting thoughts into ordered strings (eg vocalization of thoughts into sentences)
The thing is, you can put a probability theory onto anything that you can count or record states of. Of course, counting the external observations may lack the richness of the internal process. I feel that this linguistic program will work its way into a lot of future LM tooling, in some incarnation or another.
To give a very concrete example, the part of your brain called the visual cortex by neuroscientists nevertheless is activated by auditory and other sensory stimuli and assuredly participates in the processing of other sensory input in other ways. I am very suspicious of any attempt to talk about the brain which forgets that it is, in a very real sense, a gestalt.
To your specific point, and taking into account all I've written above, I've got little doubt that some part of your brain really is a probabilistic language generating machine. But the exact point here is that your cognitive abilities constitute much, much more than merely the ability to generate plausible language. Indeed, as I experience complex cognition, conversion into language is often the last and most trivial part of the exercise.
If we're talking about LLMs as general purpose AI or a tool to replace programmers... mostly a fail so far. LLMs seem like they could eventually be a component of something bigger, but I don't see a line between what they are and general purpose AI.
In the past, if you wanted a program to work with high-level ideas, you had to explicitly hand them to it. Humans had to do the work of turning data into concepts. These generative AI systems are different - they can learn abstractions from data, and manipulate them in complex ways.
Is ChatGPT a good and accurate chatbot? Maybe not. But it's a fundamental change in what computers are capable of.
Here is a sample where I used my own post (just copy pasted the raw text from the browser) and got this schema
comment:
meta:
points:
author:
time:
action:
parent:
next:
edit:
delete:
post:
content:
action:
reply:
When it got nerfed, its perspective got completely out of whack and it's making very different and inconsistent generation.
An LLM can only ever reproduce what it has seen before.
How is that in ANY WAY an epilogue to The Great Gatsby? This is exactly the problem. That original story builds with a series of revelations into the conclusion 'so we beat on, boats against the current, borne back ceaselessly into our past': establishing a PURPOSE, perhaps a bleak and unwelcome one. Fitzgerald's revealing an insight into the delusions of humanity. He's picturing even the greatest of us as surfers on the river of nihilism and reality. Our aspirations sparkle prettily… and are gone, like froth in the rapids.
And this is just a little bit beautiful. We can imagine beyond our reach. That's a human thing. The fact that our minds can cling so longingly to something that is simply not real, is kind of wonderful. Everyone's their own little world of unreality, and we aspire so earnestly (much like these AI folks do).
And then the AI, having drunk up all of Fitzgerald and everybody else, 'continues' past the point. With what? Hardly matters. It has no point to make. I'd be impressed if it refused, said 'nope, that was where it ended. Can't add anything worth adding, try reading it again'. But no, because the LLM has no intention and doesn't successfully get one from what it's 'read'.
It's constructed an epilogue out of nebulous religious feelgoodism in the rough style of Fitzgerald's sentence construction, and undermined the whole conclusion of the story… and not maliciously, for that would require intent. Nope… it sort of ambled on, going 'what would feel nice here? ok, now what seems like it would go with this sort of thing? ok, something else, let's have more, what kind of concepts go here? what do people normally say when they talk in this way?'
In so doing, it's less than Gatsby and way less than Fitzgerald. There is nothing here in this 'epilogue'.
GPT4 criticised the 'epilogue', in a rather fascinating way! https://pastebin.com/B0zxbvNv
It successfully works out some of the problems with the first AI's writing, and yet it too fails to get the idea expressed by Fitzgerald, and rather than pivot to religious feelgoodism, it pressures the first AI to instead emphasize how its narrator and Gatsby shared a special bond, the very special specialness of being "the only one who understood what it meant to be young and restless in this restless world."
A lot of people can write that idea, and in fact a lot of people did and that's why GPT4 found it a probable argument to make.
Fitzgerald gave us a moment of viscerally grokking that it doesn't mean s*t… and yet, we will still paddle against the current of time and decay and collapse, because what else can we do? Tomorrow we'll get it. Tomorrow we'll really understand and it'll all make sense.
And so…
"This Commodore 64 is useless. It can't even run Crysis."
Snark aside, I didn't ask if the epilogue was any good or not, I asked where it came from. It came from our collective consciousness as embedded in the language model. It turns out that what we've been dismissing as mere "language" is an insanely powerful thing, maybe the only thing.
We're still in the first publicly-visible generation of LLM technology, and the model that generated the epilogue was already behind the leading edge in many respects. Anyone who's not blown away by this is whistling past the graveyard. Computers are now doing what we do. Yes, they still kinda suck at it. But they will get better at it much faster than we will.
I mean, really. What is a human author thinking, if not, "What would feel nice here? OK, now what seems like it would go with this sort of thing? OK, something else, let's have more, what kind of concepts go here? What do people normally say when they talk in this way?" It's been understood since the Greek classical period that there is only a finite amount of ore in the original-story mine. Everything after the first seven or so basic ideas is just implementation.
I have to wonder what you'd think of Anthony Burgess's epilogue to A Clockwork Orange, the one that the book's American publisher and Kubrick both chose to leave on the proverbial cutting-room floor. The one where Alex grew out of his rebel phase, got a job, and started a family. This epilogue reminds me of that, somehow. I could easily see Burgess's final chapter emerging fully-formed from an LLM in the not-too-distant future.
Anyone who's played around with these models know that at least some generalization is taking place.
Humans don't come up with ideas out of nowhere. Open a novel and you will find that even though the overall work is unique, it is composed of tropes from other literature and experiences from the author's life which have been generalized into another context.
What does it mean to "operate as a probability machine"? And what does it mean to understand anything?
One recent example of understanding is that llms/transformers learn to parse context free grammars via dynamic programming (https://arxiv.org/abs/2305.02386). Basically they've understood what's going on well enough to mold their neurons I to the optimal algorthm for parsing this kind of text.
I think they understand lots of things like this. Of course there's other things they don't understand or just pretend to understand.
For me personally this was probably the biggest game changer because I'm now able to offload a lot of thinking to GPT-based tools and use my brain cycles for the less automatable activities.
Now instead of searching for something on Google and going through multiple pages before finding an answer, I can ask a precise question and get the answer right away. Especially when I know that the answer _is_ there somewhere, and all I need is to find it. If I'm not happy with the answer, I can continue the conversation until I get what I'm looking for.
I made Bing Chat, ChatGPT, and Warp AI (a feature of the Warp terminal that allows you to access GPT from within the terminal) a part of my daily life and I feel like I'm achieving much more with the time I have.
The only thing missing is an analysis of how much power it takes to accomplish each of these tasks. If ChatGPT-4 is at about 1:1 in terms of “effectiveness”, all that remains is to divide by the amount of power required to reach the answer using ChatGPT-4 vs by conventional means. If requires significantly more energy, then it’s a waste, and because of climate change we should really not pursue it further, IMO.
However, even in the 1:1 case it means I am training myself to become a "prompt engineer" rather than to be an actual thinker and problem solver. As long as there will always be another system for me to depend on, maybe that's ok. But as with people who never learned to read maps and navigate without GPS tend to be very confused and lost when their phone dies, I would like to be able to be a useful human even when the power is out.
In my opinion, a precondition for creativity and inventiveness is understanding. If you rely on a surrogate to give you answers, you will never reach the level of proficiency required to come up with something new. If we train a generation of thinkers to rely on an external brain to get anything done, they will only understand things superficially, and our ability to innovate at the society level will suffer.
I'm genuinely unsure if my own brain is any different.
My 2 month active experience with ChatGPT-4 gave me the following takeaways:
- when it's right, it's amazing; and when you, the operator, can recognize the niche use case where it performs really well, it can be a game-changer (although you could have programmed a tool to do the same narrow task)
- when it's a little wrong, you (the expert) can fix the issue and move on without friction
- when it's any amount of wrong and you are less than an expert, or specifically you are completely unfamiliar with the topic, you can waste an immense amount of time researching the output and/or iterating with the system to refine the result
Initially I thought it was a 10:1 ratio of performance to effort. But finally (as a developer) I settled on (1 to 1.5):1. It basically just changed the game for me from doing the actual hard work to working out how to tease the system into producing a reasonable result. And in the process (same as with co-pilot) I started to recognize how it was leading me to change my habits to avoid thinking and effort and instead rely on an external brain. If you could have a reliable external brain always available, that would be a fair trade. But when the external "brain" is unreliable and only available via certain interfaces, it's better to train yourself to be proactive, voracious (with respect to documentation), and tolerant of learning/producing cycles.
I once thought it would be a game-changer. Now I realize it is a game-changer, but in a similar way that offshoring was... it didn't improve or solve any problems, but it merely changed the work.