R17 on spinning disk faster than PostgreSQL on SSD

Re:R17 on spinning disk faster than PostgreSQL on SSD by skimbrel on Monday August 15, 2011 @11:23PM (#2889706)

At first glance it looks like doing a query involves streaming the entire dataset into memory while selecting and projecting on the fly. If that's true, what happens when you have truly massive rows (i.e., things containing MEDIUMTEXTs or worse)?

Okay, reading further down you only get very basic data types. Still, nothing in the spec appears to prohibit very long rows, and I'd imagine performance starts to fall off once you're throwing around tens of kilobytes per row. Any plans to support pushing the projection operation into the read phase so you can work with massive individual records?

And where's the source? I want to see exactly how much this differs from a modern SQL engine.

[ Parent | on HN ]

Re:R17 on spinning disk faster than PostgreSQL on SSD by matthewnourse on Tuesday August 16, 2011 @12:29AM (#2889900)

It streams the dataset into memory 256K (more for longer rows) at a time. It doesn't load the whole dataset into RAM unless it must eg for a join, sort or grouping.

I don't currently have plans to push projection into the read phase, but the phases are all pretty close together :) so maybe it wouldn't be required. How massive is "massive" for you? 10s of K? Megs?

R17 is not currently open source, but I haven't ruled it out.

[ Parent | on HN ]