You could do everything client side as well and just store encrypted indices on the server. Probably not a good idea, but not the end of the world.

I don't think users actually care though. Google's attempt to redefine "end-to-end encryption" to include decrypting something mid route bothered me. This... meh.