Re:Add 10 GbE to your system with an M.2 2280 module
by rglullis on Wednesday January 19, 2022 @08:28AM
(#29993336)
Then again, if it is public (or not-sensitive) data, why make it part of your infra?
If you are working with ML and have public datasets, just use the original source. Make a public mirror if you want to collaborate. Make it accessible via BitTorrent. You can bet that the amount of savings you will get from AWS will be enough to give each dev a NAS with a few tens of TB, which then they can even then use to host/mirror the public data by themselves.
Re:Add 10 GbE to your system with an M.2 2280 module
by jiggawatts on Wednesday January 19, 2022 @09:25PM
(#30003609)
My specific example was a private data set hosted on-premises (satellite imagery).
The analysts were pulling down a ~0.1% sample to develop the ML algorithms locally, and then deploying the final thing in the cloud to run through the whole petabyte.
Local development has its advantages, such as high flexibility and very fast cycle times. It's fantastically expensive to outperform a modern PCI 4.0 NVMe drive in the cloud. Locally it's a $200 part.
I think there are different philosophies, just like 'no comments allowed in code', this is a matter of judgement.
I saw 5 different corporations setup centralised workflow, and usually you end up with a massive cloud bill, incorrectly configured admin rights, under-resourced VDI, and a ticketing system that takes a week to resolve the smallest issue.
If we assume for the moment that data in question is not highly sensitive (for example public satellite imagery), there is nothing wrong with getting every dev a beefy workstation with 50 TB of storage and a 20-core CPU, and it's often even cheaper.