It streams the dataset into memory 256K (more for longer rows) at a time. It doesn't load the whole dataset into RAM unless it must eg for a join, sort or grouping.

I don't currently have plans to push projection into the read phase, but the phases are all pretty close together :) so maybe it wouldn't be required. How massive is "massive" for you? 10s of K? Megs?

R17 is not currently open source, but I haven't ruled it out.