News for hackers. Stuff that matters.

A comment on A Modern App Developer and an Old-Timer System Developer Walk into a Bar. Parent comment

Re:A Modern App Developer and an Old-Timer System Developer ... by jdmoreira on Sunday February 14, 2016 @05:21PM (#11100265)
"if the data set becomes too large to fit in memory on a single machine?"

You just use mmap. If it become bigger than the disk capacity of single machine, you can still mount a distributed filesystem and use mmap.

[ Parent | on HN ]
Re:A Modern App Developer and an Old-Timer System Developer ... by voltagex_ on Sunday February 14, 2016 @07:30PM (#11100758)
Slightly off topic: http://yourdatafitsinram.com/ and more seriously https://www.youtube.com/watch?v=SiSY1b0am5w which is a really good talk about the challenges of sysadmining for scientists. What do you do when 2TB of RAM isn't enough?
[ Parent | on HN ]
Re:A Modern App Developer and an Old-Timer System Developer ... by pdkl95 on Sunday February 14, 2016 @07:47PM (#11100829)
You add an index and smaller-block loader. The size and structure of a "smaller-block" depends on the project and how you are using it.

In more extreme situations, you to learn all about materialized views, cache invalidation, etc.

[ Parent | on HN ]