Re:How Python became the language of choice for data science
by ma2rten on Friday November 22, 2013 @06:37AM
(#6780271)
This is not a major problem for data science. We usually use numpy arrays (or scipy sparse arrays), which are memory efficient. There are also some other efficient data structures for python:
http://kmike.ru/python-data-structures/
For preprocessing you can usually work on a line-by-line basis (e.g. don't keep the entire dataset in memory).
This is a bigger problem for web apps, imo. But there you usually just throw money (e.g. more servers) at the problem.