In my opinion the right approach to downsizing is to train at full size and then either distill or prune the model to keep the parts that are relevant to your problem.
Unfortuntaly, distillation of smaller models can take a fair bit of time. However, there is a lot of recent work to make distillation more efficient, e.g. by not just training on the label distributions of of a teacher model, but by also learning to emulate the teacher's attention, hidden layer outputs, etc.
More parameters is inherently better, with the knowledge we have now.
If your concerns are about over fitting there are lots of regularization techniques used in practice like dropout, weight decay, and data augmentation.
There's nothing preventing you from sharing weights across layers, and would be interesting to see some research about that.
E.g. the ALBERT model does that:
https://arxiv.org/abs/1909.11942
I have done model distillation of XLM-RoBERTa into ALBERT-based models with multiple layer groups and for the tasks that I was working on (syntax) it works really well.
E.g. we have gone from a finetuned ~1000MiB XLM-R base model to a 74MiB ALBERT-based model with barely any loss in accuracy.
I'm surprised we haven't seen more research on modular models. Random forests and boosting both work by building a collection of weak models, and combining their results to produce final predictions. It seems like building an ensemble of some sort of unsupervised learners, then training a smaller supervised network on that ensemble could produce good results, while also being more reusable and easier to train.