The question, again, is why you'd use MongoDB even for document storage when there are so many alternatives that are safer and better-engineered. Even Postgres has a JSON data type that, by some reports, works better than Mongo and is much safer to use.
Another thread here[1] has gone into some detail about alternatives.
The most mainstream and interesting of these is Cosmos:
https://docs.microsoft.com/en-us/azure/cosmos-db/connect-mon...
(Cosmos is actually one of the most interesting DB products available for production right now, in my opinion.)
- FEATURES table (~400k rows), primary key: Feature_ID
- SAMPLES table (tens of thousands), primary key: Sample_ID
- OCCURRENCE table (what you observed), with the fields Feature_ID, Sample_ID, Occurrence_Count.
This is pretty much a standard solution well proven in data warehousing...
So our sample metadata has the generic metadata about each sample (where it's from, when was it collected, etc.). Our feature metadata is the particular feature metadata (DNA amplicon sequence variant, taxonomy, all seven taxa levels pre-split). These would work perfectly fine in a standard SQL database, but the problem is when we get into our sample-by-feature table/collection. Currently, we have over 400k unique features and tens of thousands of samples. If we were to try and map the frequency of each feature in our samples in a standard row/column relational database table, our table would be un-usably large (or just entirely not work as PostgreSQL, MySQL have a hard column limit). Thus, the benefit of MongoDB's document system fixes all the issues this causes.
Each of our documents in the sample by feature table can consist of ONLY the counts of the features that were in the sample, ignoring all the other N-thousand features that may be present in others. Building a pandas DataFrame from a query of this data will just fill the absent features with NaN, and we can just fill those NaN's with zeroes and carry on working with our target subset of the database.