The statement was "fits on", not "matches the speed of".
Re:Big data is dead (2023)
by Dylan16807 on Monday May 27, 2024 @03:49PM
(#40494006)
Several gigabytes per second, plus RAM caching, is probably enough though. Latency can be very important, but there exist some very low latency enterprise flash drives.
I think the point is that if it fits on a single drive, you can still get away with a much simpler solution (like a traditional SQL database) than any kind of "big data" stack.
I always heard it as "if the database index fits in the RAM of a single machine, it's not big data". The reason being that this makes random access fast. You always know where a piece of data is.
Once the index is too big to have in one place, thing get more complicated.
Re:Big data is dead (2023)
by justsomehnguy on Tuesday May 28, 2024 @02:05AM
(#40497794)
980 is an M.2 drive, PCIe 3.0 x4, 3 years old, up to 3500MB/s sequential read.
You want something like PM1735: PCIe 4.0 x8, up to 8000 MB/s sequential read.
And while DDR5 is surely faster the question is what the data access patterns are there.
In almost all cases (ie mix of random access, occasional sequential reads) just reading from the NVMe drive would be faster than loading to RAM and reading from there. In some cases you would spend more time processing the data than reading it.
PS all these RAM bandwidth rates are good for the sequential access, as you go random access the bandwidth drops.
https://semiconductor.samsung.com/ssd/enterprise-ssd/pm1733-...
[0] https://www.tomshardware.com/reviews/samsung-980-m2-nvme-ssd...
[1] https://www.tomshardware.com/reviews/ram-speed-tests,1807-3....
[2] https://en.wikipedia.org/wiki/DDR5_SDRAM