That jogs my memory; in fact, aha, I used get_user_pages in an early version, and switched to remap_pfn_range; then I nuked all that complicated non-linear access code. The ioctl for attaching the buffer then allocated the memory, and its interface changed to passing down a size and receiving a pointer.
Open question: would it have helped to have a CMA[1] region managed by a kernel module and mapping that memory/regions of that memory to user-space using remap_pfn_range[2] instead?
[1] CMA documentation file: https://lwn.net/Articles/396707/
[2] https://www.kernel.org/doc/htmldocs/kernel-api/API-remap-pfn...