Something isn't right. Sounds like you're recalculating the cache digests each time..

Make sure you have your cache store set to dalli (production.rb)

      config.assets.cache_store = :dalli_store