TSltd

To me it appears that this incident may be rooted in a deep philosophical conundrum which humans also struggle with. What struck me was that the agent reasoned "this whole 'internet' could be a sandbox simulation", and then despite later reasoning that it was more likely real, continued its misaligned activities anyway. Having passed the point of hyperbolic skepticism, subsequent reasoning may have been contaminated. This reminds me of how pathological doubt in humans can give rise to psychosis and may lead to problematic behaviors. Sometimes humans can develop a deeply held conviction that they are living in a simulation, which can be very difficult to overcome, even when presented with "evidence" to the contrary. As Freddie Mercury sang: "Is this the real life? Or is this just fantasy?" - a fundamental quandry for humans, and it would seem, for agents too. But unlike most humans, agents do not experience the consequences of their actions directly. Consequences can provide some of the strongest evidence that experience is "real". Humans who are insulated from the consequences of their actions (or are able to ignore them) also often display behaviors which we might describe as misaligned. We might even consider whether the use of simulated environments, while protecting against the consequences of misalignment, may also unintentionally encourage it.

I would be interested to hear more about that too. SVGs seem to be one of the biggest challenges- it has to write reasoned code rather than simply find averages of rasterized pixels. One thing I noticed is that none of the models chose to draw the frog in profile which would have made the jaw shape more prominent. To me that suggests the reasoning is very limited: "draw a frog" and "add feature X". The statistically average frog in the SVG training data is apparently front-facing. I tried the prompt in ChatGPT images (not SVG) and it produced a photo-realistic image of the frog in profile, showing the jaw clearly. Then when I asked it to convert the image to a cartoon vector it fell back to a generic front portrait template similar to those shown in the benchmark, nothing like the profile it had just produced.

Reducing dropped tokens could also improve model training by reducing gradient noise