The problem is that facts are not differentiable. If I have a statement “The Declaration of Independence was signed in 1776” and the model replaces 1776 with 1775 that’s incorrect but in some sense just as incorrect as 1876 or 1976. So the loss at all but the correct output should be 1.0, making it pretty hard to learn. It’s like finding a hole in an infinite flat plane with trillions of dimensions.