Why on earth would you tell the model the task in english? The tasks have extremely simple JSON representations that would make for a way, way more natural input. The colorful grids are just to make it easier (for humans) to visualize the symmetries needed. Eg: Here is an (abbreviated) train/test example for task 007bbfb7 from https://github.com/fchollet/ARC/blob/master/data/training/00...

    "train": [{
    "input": [[0, 7, 7], [7, 7, 7], [0, 7, 7]], 
    "output": [[0, 0, 0, 0, 7, 7, 0, 7, 7], [0, 0, 0, 7, 7, 7, 7, 7, 7], [0, 0, 0, 0, 7, 7...

    "test": [{
    "input": [[7, 0, 7], [7, 0, 7], [7, 7, 0]], 
    "output": [[7, 0, 7, 0, 0, 0, 7, 0, 7], [7, 0, 7, 0, 0, 0, 7, 0, 7], [7, 7, 0, 0, 0...

I imagine it would be pretty easy to fine tune a davinci model on these and then test, no? Has anyone done it?