That is for books and must cover mentions of both Watt’s and Joule’s name and of the unit (and, likely, way more of the names. The units typically are abbreviated), so I don’t think that proves or disproves much about wide use of the units.

The difference before 1860 probably is there because James Prescott Joule was less than a year old when James Watt died in 1819 (at the age of 83)

Edit: https://books.google.com/ngrams/graph?content=kJ%2CkWh%2Ccal... seems to support the claim that “Joule” came up relatively late.