An idea I want to try out is pre-training a model on lots of synthetically generated data that is just "given this small synthetic program, predict the output". Code like:

    x = True
    y = False
    if y:
        x = y
    else:
        x = not x
    print(y)

Just very simple snippets that have basic branching, loops, mutation, etc. Easy to synthetically generate infinitely many of these programs.

The hypothesis is that pre-training on this task will force the model to learn a lot of circuits that deal purely with in-context, causal reasoning.