Re:What you need may be “pipeline +Unix commands” only
by geofft on Sunday March 03, 2019 @11:15AM
(#19295148)
> In the real world, those 5M lines are growing by several hundred thousand every day, and after a few months, grows beyond what a single computer or awk can handle.
This is a frustrating viewpoint, and it feels to me like the same poisonous worldview as "If your company isn't growing 10% month-over-month, it's not worthwhile and we won't invest in it." There are plenty of meaningful and useful things you can do for the world at a static size, and doing continued interesting work with a dataset of the same size doesn't mean you're not part of the "real world."
This is a frustrating viewpoint, and it feels to me like the same poisonous worldview as "If your company isn't growing 10% month-over-month, it's not worthwhile and we won't invest in it." There are plenty of meaningful and useful things you can do for the world at a static size, and doing continued interesting work with a dataset of the same size doesn't mean you're not part of the "real world."