Re:Data Branching for Batch Job Systems
by PaulHoule on Friday January 24, 2025 @12:26PM
(#42815330)
An interesting pattern that I've thought about in "intelligent systems" is "patching" which could be applied at various stages of a process.
If you look at the average training set used in competitive evaluations you will find examples in it that are just plain wrong which put an upper limit on both the evaluation scores and the real-world performance based on that data.
Very occasionally someone does the hard work to improve the training and eval set and, if the world was efficient, this would be the new data set everyone uses.
In real life you are getting more data all the time and you need a stable way to keep your data set "patched" as new data comes in. Similarly, an AI step later in the process needs to (i) have a human override so it can get things right for the consumer and (ii) remember that override so not to waste the time of the human or wear them out emotionally.
If you look at the average training set used in competitive evaluations you will find examples in it that are just plain wrong which put an upper limit on both the evaluation scores and the real-world performance based on that data.
Very occasionally someone does the hard work to improve the training and eval set and, if the world was efficient, this would be the new data set everyone uses.
In real life you are getting more data all the time and you need a stable way to keep your data set "patched" as new data comes in. Similarly, an AI step later in the process needs to (i) have a human override so it can get things right for the consumer and (ii) remember that override so not to waste the time of the human or wear them out emotionally.