You can use the more barebones Worker Sites and retain file extensions. Pages has nicer UX (automatic PR preview sites!) but the old school Worker Sites setup is dead simple and does exactly what you want - it's just a template setup to load assets from KV storage.

https://developers.cloudflare.com/workers/platform/sites/