That's what I meant by each thread having their own map (i.e. root), and within each map they have a shared branch, a branch being a pointer to a subtree. In this case, what I had in mind was the shared branch being at the root; basically slicing the user land address space in half (or 1/radix size...), though, that's probably getting too detailed.
Perhaps by "branch" you thought I meant a jump operation in code. That's certainly not how most MMUs work, which walk a data structure constructed by the OS. But some MMUs actually fault to OS provided code to resolve an address, in which case "branch" in the meaning of software code isn't necessarily wrong. (But to be clear, it's not what I had in mind.)
As to software-managed TLBs, very few processors these days support such functionality, instead opting for hardware mapping table walkers in their MMUs which is the context for my comment. I am literally the author of memory mapping code for multiple architectures in a commercial operating system and even I do not need to consider such hardware.
I'm not describing processes because I'm not describing anything that currently exists, at least not in Linux or any other popular OS.
I may not be a maintainer for any current VM subsystem, but I've been around long enough to know it's ridiculous to get into semantic arguments about processes, threads, light weight processes, or other similar labels. The meanings behind such terms continually evolve and are dependent on context--particular operating system, etc.
If it's not possible to implement the OP's notion, explain please why it wouldn't be possible with current hardware; why one process'/thread's/LWP's/whatever's mapping table couldn't have a subtree shared (not copied) with another mapping table which could be manipulated with the appropriate semantics. I don't know if it's possible or not; that was the gist of my question. I don't need a lesson in how "process" and "thread" are currently defined and modeled in typical systems. Obviously they don't model what the OP was suggesting.
As to your actual question, the underlying concept is fairly easy to do and fully supported by the hardware if done appropriately. There are no difficulties with TLB coherence because the problems are a subset of multithreaded TLB coherence, so any kernel supporting multithreading should already have those mechanisms readily available for repurposing.
The only difficulty is if the software abstractions of the memory manager you are working with disagree with the concept. I am unfamiliar with the Linux kernel, but if the abstractions are wrong for the concept then you probably need to go look and modify code related to processes rather than threads.
MMUs expect a tree. You can not give them “Tree A, except at Node N substitute a different sub-tree”. To have a different sub-tree, you must have a different root. However, you can share sub-trees amongst different trees.
To do what you are expecting, you would actually want multiple processes with their own mappings/tree and you would share a substantial common/shared mapping/sub-tree. The difficulty there is making sure all the processes are aware of the common sub-tree and properly share and synchronize with it.