Good observation, we've made sure that datasets can evolve better as you go. Specifically for your use case, you can query subsets of data and materialize it on the fly to be streamed, and then go back to a specific dataset "view" (i.e. saved query), as needed.
See how this works at around 5:50 here - https://youtu.be/SxsofpSIw3k
As for your last question, when the jpeg is appended using its file path, the compressed bytes get stored in the dataset without decompression/recompression. When the data is accessed as a numpy array, then the jpeg bytes are decompressed.
For researchers in Academia, our Growth plan is free. Since you work at a startup, the trial for Growth plan is for two weeks. If you want access, hit us up in the Community slack (slack.activeloop.ai - or you can just test the querying on public activeloop datasets!)
https://docs.deeplake.ai/en/latest/deeplake.html?highlight=l... https://docs.deeplake.ai/en/latest/deeplake.html#deeplake.re...
Some questions/observations: - Python API is great and definitely more useful than CLI when it comes to regular use. But CLI would also be nice for some operations, such as import/export, see history metadata, etc. - Just a nit, but if I didn't already by know about activeloop I might be less interested due to the "deep lake" branding. There's already so much "data lake" stuff that I have no interest in since it's historically not useful for image data. - In academic contexts, it's typical to have a static dataset and always use that. In my use cases, typically I have an every-growing "raw" dataset and extract/preprocess subsets periodically for (re)training, annotation, etc. I typically consider these subsets ephemeral, as I can regenerate on demand. Last time I tried activeloop, it seemed like it was more geared towards the static dataset use case, but browsing the docs now it seems like there's more consideration of the latter case, so I'll have to look at that. - In the examples I've seen, it seems like if a JPEG image is added to the dataset with jpeg compression, it first gets decompressed, so it would have to be recompressed - a lossy operation, right?