News for hackers. Stuff that matters.
Re:Ask HN: Is it just me, or do most data scientists hate th... by PaulHoule on Wednesday August 25, 2021 @09:16AM (#28301276)
Actually "data rich products" are complicated because they involve evolving code and data. There are tools that are good at one but not tools that are good at both.

For instance Docker is a problem instead of a solution if part of your system is a several-gigabyte language model that changes frequently.

Re:Ask HN: Is it just me, or do most data scientists hate th... by fundamental on Wednesday August 25, 2021 @12:30PM (#28303659)
I tend to agree with tools either being good at code or data. I'm sure eventually there will be a solution, but for now it's a lot of bespoke tooling.

Docker can be a headache at times, but being able to maintain a consistent environment is very handy. For the large binary resources (trained models, dataset, etc) I've found it easy enough to just use docker volumes to mount read only resources. Other people have resorted to leaving read only assets on the local network which might be fast enough for your needs. As long as you're not unintentionally copying data, mounting a TB of data read only takes no time in my experience.