News for hackers. Stuff that matters.

A comment on When your data doesn’t fit in memory: the basic techniques. Parent comment

Re:When your data doesn’t fit in memory: the basic techniques by nautilus12 on Monday November 11, 2019 @08:23PM (#21510661)
Both indicators (along with python) that this is an article written by a yet another data scientist that thinks engineering is unecessary. Data science is a field that is plagued by individuals having very specific narrow experiences that are often easy to engineer making them think they are experts in data engineering. Vertical scalability has a hard bound, if your algorithm changes complexity it could become literally impossible to move forward. Horizontal scalability gives you options, this is why we like it, not just because we like to overengineer systems, but because we don't like embracing hard limitations on the systems we engineer, especially when we often don't know yet what problem we are solving. Engineers often don't get the liberty of solving extremely specific instances of problems like data scientists do.
[ Parent | on HN ]
Re:When your data doesn’t fit in memory: the basic techniques by realusername on Monday November 11, 2019 @08:59PM (#21510857)
> Data science is a field that is plagued by individuals having very specific narrow experiences that are often easy to engineer making them think they are experts in data engineering.

That's on point, some of the worst and most inefficient code I had to maintain was written by data scientists. They are for sure leagues above any of my data science knowledge in their respective field but the average engineering knowledge and best practices seems pretty low.

[ Parent | on HN ]
Re:When your data doesn’t fit in memory: the basic techniques by kyllo on Tuesday November 12, 2019 @12:47AM (#21511822)
Most of the time it's not that data scientists think engineering is "unnecessary," it's that they're expected to deliver analyses and models without the help of engineers, because the engineers are on another team, behind a massive backlog. So they make do with the tools they have.
[ Parent | on HN ]
Re:When your data doesn’t fit in memory: the basic techniques by nautilus12 on Tuesday November 12, 2019 @09:05AM (#21514130)
This is totally fair, why we need to bring the two disciplines closer together rather than have one try to obsolete the other.
[ Parent | on HN ]
Re:When your data doesn’t fit in memory: the basic techniques by Animats on Tuesday November 12, 2019 @02:56AM (#21512334)
Um, yes. From the article:

"The simplest and most common way to implement indexing is by naming files in a directory:"

    mydata/
        2019-Jan.csv
        2019-Feb.csv
        2019-Mar.csv
        2019-Apr.csv
Um.
[ Parent | on HN ]
Re:When your data doesn’t fit in memory: the basic techniques by kyllo on Tuesday November 12, 2019 @11:41AM (#21515477)
This is time partitioning, not indexing, but it's commonly done in data lakes for raw telemetry/log data.

    mydata/
        2019/
            11/
               12/
                   2019111200.csv
                   2019111201.csv
Where each one of those CSV files could be 10-100 GB in size.

Usually you want to process it into a columnar format like Parquet from there, though

[ Parent | on HN ]