The scratchspace could be in natural language, preserving some debugeability and letting us know about the model mental process.

This is doable but it introduce a sequential dependency which would make the training significantly slower.